A Hybrid Deep Learning Framework for Anomaly Detection in High-Dimensional Streaming Data: Integrating Autoencoders and LSTM Networks

Abstract

The rapid growth of data generation, particularly in streaming environments, presents significant challenges for anomaly detection. High-dimensionality, temporal dependencies, and the sheer volume of data necessitate sophisticated approaches. This paper proposes a novel hybrid deep learning framework that integrates the strengths of autoencoders and Long Short-Term Memory (LSTM) networks for anomaly detection in high-dimensional streaming data. The autoencoder component reduces dimensionality and extracts salient features, while the LSTM network models temporal dependencies to identify deviations from normal patterns. The framework is evaluated on a real-world network traffic dataset and compared with state-of-the-art anomaly detection methods. The results demonstrate that the proposed hybrid approach achieves superior performance in terms of accuracy, precision, recall, and F1-score, offering a robust and efficient solution for anomaly detection in complex big data environments.