Experimental and data-driven evaluation of transport-layer and application-layer security for MQTT-based IoT networks

The rapid expansion of Internet of Things (IoT) deployments has intensified the need for secure and efficient communication mechanisms tailored to resource-constrained devices. Message Queuing Telemetry Transport (MQTT) is widely adopted due to its lightweight design; however, it lacks native security support, requiring external protection mechanisms that may significantly affect system performance. This paper presents a comprehensive experimental and data-driven evaluation of two security paradigms for MQTT-based IoT networks implemented on ESP32 microcontrollers: transport-layer security using TLS and application-layer encryption based on elliptic curve cryptography (ECC) for key exchange combined with AES symmetric encryption. The analysis jointly evaluates memory utilization, end-to-end latency, energy consumption, and resistance to passive traffic interception under identical experimental conditions. In addition to conventional metric-based comparisons, multivariate statistical analysis and unsupervised learning techniques are employed as exploratory tools to characterize the system-level behavior induced by each security scheme. Results show that Transport Layer Security (TLS) offers stronger confidentiality guarantees at the cost of higher memory overhead, while the ECC–(Advanced Encryption Standard) AES approach significantly reduces memory footprint with moderate latency penalties and comparable energy consumption. Multivariate analysis further reveals that each security mechanism induces a distinct performance regime, providing a compact joint characterization of the security–performance trade-offs across the evaluated configurations.