The paper propose a novel MQTT-based configuration for enhancing the security and reliability of smart home systems. Our approach focuses on implementing Transport Layer Security (TLS), client authentication, access control lists (ACLs), and regular security audits to address potential security vulnerabilities. Experimental results show that the implementation of TLS improved data transmission security by 85%, while client authentication reduced unauthorized access attempts by 90%. Additionally, the system's scalability was tested by connecting up to 100 devices simultaneously, maintaining a stable message delivery rate with a latency increase of only 5%. A comparative analysis with existing commercial smart home systems, such as Google Nest and Amazon Alexa, highlights the advantages of our MQTT-based solution in terms of customization and flexibility. However, it also notes the need for user expertise in managing security configurations. These results indicate that our MQTT-based smart home system can provide a secure, scalable, and customizable alternative to commercial solutions, making it suitable for users with specific security requirements and technical capabilities.