Adaptive hybrid ensemble framework for real-time anomaly detection in large-scale data streams

Karshiyev, Zaynidin, Sattarov, Mirzabek, Erkinov, Farkhodjon

Techscience.uz - техника фанлари долзарб масалалри · 2025-yil

Annotatsiya

This paper presents an adaptive ensemble framework for real-time anomaly detection in large-scale data streams, addressing the challenges of concept drift, high-velocity data processing, and computational efficiency in modern distributed systems. We propose a Hybrid Statistical-Machine Learning Anomaly Detection (HSML-AD) algorithm that combines sliding window-based statistical analysis with incremental machine learning techniques. The framework employs a three-tier architecture: (1) lightweight statistical pre-filtering using modified Z-score and interquartile range methods, (2) adaptive feature extraction through exponential moving averages, and (3) ensemble classification using online random forest with dynamic weight adjustment based on recent prediction accuracy. Experimental evaluation on five benchmark datasets (KDD Cup 99, NSL-KDD, CICIDS2017, Yahoo S5, and Numenta Anomaly Benchmark) demonstrates that HSML-AD achieves an average F1-score of 94.3%, precision of 93.8%, and recall of 94.7%, outperforming baseline methods including Isolation Forest (F1: 87.2%), LSTM-Autoencoder (F1: 89.6%), and SPOT (F1: 86.4%). The algorithm maintains processing throughput of 127,000 records per second with average latency of 7.8 milliseconds on commodity hardware. The novelty lies in the adaptive weight mechanism that dynamically adjusts ensemble components based on data stream characteristics and recent performance, coupled with a memory-efficient incremental learning strategy that limits model size to 45 MB while maintaining detection accuracy. The proposed framework is applicable to network intrusion detection, IoT sensor monitoring, financial fraud detection, and industrial system health monitoring, particularly in resource-constrained environments requiring real-time processing.

Maqola ma’lumotlari
MualliflarKarshiyev, Zaynidin, Sattarov, Mirzabek, Erkinov, Farkhodjon
JurnalTechscience.uz - техника фанлари долзарб масалалри
NashriyotЖиззах политехника институти
ISSN3030-3702, 3030-3702
Nashr sanasi2025-12-27
Jild3
Son12
Betlar74-93
TilIngliz
Ilmiy sohaTexnika
DOI10.47390/ts-v3i12y2025n09
Indeksga qo‘shilgan2026-08-26

DOI: 10.47390/ts-v3i12y2025n09 · Maqolaning asl sahifasi

Kalit so‘zlar

anomaly detection, data streams, ensemble learning, concept drift, real-time processing, adaptive algorithms, machine learning., anomaliyalarni aniqlash, ma’lumotlar oqimi, ansambl o‘qitish, kontseptual siljish, real vaqt rejimida qayta ishlash, moslashuvchan algoritmlar, mashinali o‘qitish.

Ilmiy soha

Texnika

Iqtibos keltirish

APAKarshiyev, Zaynidin, Sattarov, Mirzabek, & Erkinov, Farkhodjon (2025). ADAPTIVE HYBRID ENSEMBLE FRAMEWORK FOR REAL-TIME ANOMALY DETECTION IN LARGE-SCALE DATA STREAMS. Techscience.uz - техника фанлари долзарб масалалри, 3(12), 74-93. https://doi.org/10.47390/ts-v3i12y2025n09
GOST (OAK)Karshiyev, Zaynidin, Sattarov, Mirzabek, Erkinov, Farkhodjon ADAPTIVE HYBRID ENSEMBLE FRAMEWORK FOR REAL-TIME ANOMALY DETECTION IN LARGE-SCALE DATA STREAMS // Techscience.uz - техника фанлари долзарб масалалри. – 2025. – T. 3, № 12. – P. 74-93. – DOI: 10.47390/ts-v3i12y2025n09.
MLAKarshiyev, Zaynidin, Sattarov, Mirzabek, and Erkinov, Farkhodjon. “ADAPTIVE HYBRID ENSEMBLE FRAMEWORK FOR REAL-TIME ANOMALY DETECTION IN LARGE-SCALE DATA STREAMS.” Techscience.uz - техника фанлари долзарб масалалри, 3(12), 74-93, 2025. https://doi.org/10.47390/ts-v3i12y2025n09
ChicagoKarshiyev, Zaynidin, Sattarov, Mirzabek, and Erkinov, Farkhodjon. “ADAPTIVE HYBRID ENSEMBLE FRAMEWORK FOR REAL-TIME ANOMALY DETECTION IN LARGE-SCALE DATA STREAMS.” Techscience.uz - техника фанлари долзарб масалалри, 3(12), 74-93 (2025). https://doi.org/10.47390/ts-v3i12y2025n09
HarvardKarshiyev, Zaynidin, Sattarov, Mirzabek, and Erkinov, Farkhodjon (2025) ‘ADAPTIVE HYBRID ENSEMBLE FRAMEWORK FOR REAL-TIME ANOMALY DETECTION IN LARGE-SCALE DATA STREAMS’, Techscience.uz - техника фанлари долзарб масалалри, 3(12), 74-93. https://doi.org/10.47390/ts-v3i12y2025n09
BibTeX
@article{karshiyevzaynidin2025,
  title = {ADAPTIVE HYBRID ENSEMBLE FRAMEWORK FOR REAL-TIME ANOMALY DETECTION IN LARGE-SCALE DATA STREAMS},
  author = {Karshiyev, Zaynidin and Sattarov, Mirzabek and Erkinov, Farkhodjon},
  journal = {Techscience.uz - техника фанлари долзарб масалалри},
  year = {2025},
  volume = {3},
  number = {12},
  pages = {74-93},
  doi = {10.47390/ts-v3i12y2025n09},
  url = {https://doi.org/10.47390/ts-v3i12y2025n09}
}

Boshqa maqola uchun iqtibos tayyorlash

Techscience.uz - техника фанлари долзарб масалалри jurnalidan boshqa maqolalar

Techscience.uz - техника фанлари долзарб масалалри — barcha maqolalar · 2025-yil arxivi

Texnika sohasidagi boshqa maqolalar

Texnika sohasidagi OAK jurnallari