JoWUA

Abstract

Computer networks are exposed to attacks which have been increasingly more effective. To counter these emerging threats, researchers and security engineers work relentlessly to keep up with the arms race and offer improvements to intrusion detection systems as soon as possible. In the recent years, there has been an increase in the proliferation of systems employing deep learning and machine learning algorithms to detect suspicious patterns more effectively. To leverage AI effectively in a real-world scenario of intrusion detection, a scalable stream processing system to feed the detection algorithms with data samples in a timely and reliable manner has to be established. In this paper, two use cases of intrusion detection are presented. The first one shows a real-world example of data collected by one the largest telecom operators - ORANGE. The data was gathered for the SIMARGL project. The second use case presents the experiments and results of intrusion detection based on the Netflow scheme. The paper also proposes a scalable streaming architecture based on the Apache Spark and Apache Kafka technologies. The results of the evaluation of the effectiveness of detecting malicious behavior in network packets using several machine learning techniques in conjunction with the stream processing framework are presented.