Stream project: security in real-time with elasticity, analytic, and monitoring

Abstract

Most of current threats are detected long after they occurred, considerably increasing the risk of irreparable damages, and disabling any defense attempt. The late detection of those threats is a consequence of the high complexity of the attacks, more specialized every day, and the huge amount of data (Big Data) to be analyzed and monitored by security specialists. Although very hard analyze, attacks always leave traces, or trails, that can be detected with machine learning techniques through real-time stream processing. The efficiency of the defense mechanisms requires reducing the detection time of threats, from months to minutes or hours. Therefore, the STREAM project focuses on collecting, enriching data, and processing data in real-time to detect security threats. We propose to develop a platform to promptly detect security threat and start an immediate defense of the target. The project propose the development of a platform for real time threat detection, which is based in open source tools and released to the community. The services provided by the proposed platform ensure system security for both known and unknown attacks through various automated machine learning methods of attack classification and network anomaly detection. (AU)