The MetaFlows Security System

The MetaFlows Security System (MSS) detects and prevents cyber-threats using multiple, collaborative intelligence sources at once, rather than using a traditional, single-source, proprietary intelligence feed. Furthermore, instead of solely relying on a single functional component (like IDS or sandboxing), the MSS performs multifunctional traffic analysis looking for multiple red flags in the behavior of hosts on the network. This unique, multi-functional, multi-session, behavioral detection technology simultaneously lowers both false positive and false negative rates. The improved detection accuracy yields tremendous cost savings in terms of human capital and automation while improving security.

Architecture

The MSS is designed using open hardware and software standards. This approach provides an enormous amount of flexibility in meeting emerging network monitoring requirements while also yielding one of the best deep-packet-inspection cost-performance ratios in the industry. This is thanks to MetaFlows’ pioneering efforts in the commercialization of open source, course-based IDS parallelism (pf_ring).

Along side a standard browser, our system has two components: the sensor and the controller. The controller can manage from one to thousands of sensors and is today offered primarily as Software as a Service (SaaS). Most users do not need to deploy the controller but can use MetaFlows' SaaS offering at nsm.metaflows.com.

We do not offer high availability or backups since our services are for passive monitoring and do not impact availability of the production components. All our configurations are centralized in our SaaS cloud and any failures of the sensors are remediated by instantiating new sensors and reloading the configuration from the SaaS service. See https://docs.metaflows.com/Account_Management#Manage_Existing_Subscriptions

The SaaS user configurations and events are backed up daily and we have disaster recovery plans in place to restore the SaaS services within 24 hours.

Our service does not have support tiers. All customers have access to the maximum level of expertise at all times.

Sensors

The sensors run on standard Linux CentOS or Red Hat Enterprise Linux (RHEL) augmented with our multifunctional deep-packet-inspection software (which includes proprietary kernel drivers). The user has root access to the sensors and can easily customize the system with site-specific applications or site-specific configurations (if necessary). The Operating System (OS) update process is performed using standard package management operations while the MetaFlows’ software self-updates whenever a new feature or bug fix is published. We offer turn-key CentOS 7 appliances but also encourage users to also install software on their own hardware running CentOS or RHEL 6/7, VMware or EC2 instances.

The sensors receive promiscuous traffic from a mirror/span coming from a switch or a TAP (or a virtual mirror in case of Cloud-based_assets). We typically recommend to mirror both TX and RX packets of the ports connecting the switch to the firewall thus providing a copy of all packets entering or leaving the enterprise to the sensor. If multiple firewalls are used, it is possible to bond multiple mirrors on the same sensor. Depending on the hardware provided, the sensor can process anywhere from a few Mbps to 7 Gbps sustained (100 second average).

Controller

The controller (which also runs on CentOS or Red Hat) performs the following functions:

Provides a web Graphic User Interface (GUI) for the provisioning and configuration of all the sensors. This includes customizing the network configuration, the types of analysis performed, packet capture options, and Intrusion Detection System (IDS) configuration.
Receives and stores all metadata continuously exported by all the sensors.
Provides a web-based application used by analysts to browse and analyze the metadata or payload.
Automatically generates Reports and email alerts.
Automatically fetches and re-distributes security intelligence updates to the sensors (like IDS signatures or IP reputation lists).

The typical work flow is as follows:

A sensor detects suspicious behavior from analyzing the network traffic.
The metadata and the automated incident report are sent as real time events to the controller and/or a third party Security Information Event Management (SIEM).
The controller triggers an email alert sent to the analyst (user) according to a user-defined policy or the analyst sees interesting events on the Real-Time Event View.
The analyst analyzes the metadata/report through our Forensic Tools and/or the third party SIEM.
The analyst queries the sensor for payload data. This can be delivered through the Historical Flow and Payload Data web GUI or the Command Line Interface
1. in the form of text,
2. as a Packet Capture (Pcap) slice, or
3. as carved files.
The analyst files an incident report which includes relevant metadata (obtained from the controller) and payload data (obtained from the sensor).
The analyst instantiates remediation policies through Soft Intrusion Protection System (Soft IPS) or verifies that the current filtering policies adequately protected the enterprise.

Appliances

MetaFlows’ software can be deployed on the customer’s own hardware or can be provided through MetaFlows’ turn-key appliances. MetaFlows’ appliances are based on commodity hardware that is carefully selected and submitted through a rigorous Quality Assurance (QA) process. This translates to an availability that measures greater than five nines.

Appliance	Inspection Throughput (Mbps)	Operating System	Hardware	Application
MSS-UTM-1C	50 or Less	Linux CentOS/RHEL	1 CPU	Remote Office
MSS-Silver	300 or Less	Linux CentOS/RHEL, VMware	1 CPU (1 Core)	Small Enterprise, Cloud Security
MSS-Silver Plus	1,000 or Less	Linux CentOS/RHEL, VMware	1 CPU (4 Cores)	Medium Enterprise
MSS-APP-8C	1,000-2,000	Linux CentOS/RHEL	1 CPU (8 Cores)	Medium/Large Enterprise
MSS-APP-24C	2,000-5,000	Linux CentOS/RHEL	2 CPU (24 Cores)	Large Enterprise
MSS-App-64C	5,000-10,000	Linux CentOS/RHEL	4 CPU (64 Cores)	Large Enterprise

The table above illustrates the different appliance options. The primary differentiation is the maximum sustained (100 seconds average) inspection throughput. Additionally, the MSS-UTM-1C can replace the router of a small office by offering commercial firewall and wireless AP functionality.

Sensor Software

Each appliance is configured with the MSS’ sensor software. This software analyzes promiscuous network traffic obtained from a Switched Port Analyzer (SPAN)/mirror port or passive TAP. The appliance runs multiple applications, each performing a different analysis function on the traffic including IDS, file carving with network antivirus/sandboxing, flow analysis, and full packet capture (on the local sensor or a SAN storage). Each of these functions is parallelized across multiple CPUs to achieve multi-Gig processing if necessary. The full packet capture is always only stored on the sensor disk or SAN, while the metadata resulting from the analysis is exported. The metadata is exported to the controller and/or a third party SIEM in Common Event Format (CEF), to ArcSight for example, or syslog format, to Splunk for example. Most customers export the metadata to both the controller and a third party SIEM (if available).

In order to properly investigate and document security incidents sensor also store packet data on their hard disk for backtracking. The packet capture is limited by the amount of storage provided to the sensor. Older packets are deleted to make room for newer packets. The user can add as much storage as needed to achieve the desired packet retention time as a function of the bandwidth captured - anywhere from minutes to weeks.

Feature	Description	Required
Behavioral Malware Detection	Monitors your network traffic to quickly find malware infected systems or those that are part of a botnet. Unlike other IDS systems, it compares IDS alerts from multiple sessions and combines them to find typical infection patterns or abusive behavior.	Yes
Soft IPS	Blocks unwanted traffic in passive mode using techniques reversed engineered from the Great Firewall of China. Soft IPS blocks sessions by injecting spoofed TCP packets into the network to disrupt unwanted communications. This idea is coupled with a new algorithm that will safely predict which traffic to block based on observed communication patterns.	No
Antivirus/Sandboxing	Monitors the transmission of all notable files (.exe, .dll, .pdf, .zip, Microsoft Office formats, etc.) transmitted on your network. The digest of each file is passed to the network antivirus system, which consists of 50+ antivirus solutions. All files that test positive on three or more antivirus solutions generate high-priority reports for your analysis. Carved content can also be automatically sent to a cuckoo sandbox for execution.	No
Basic IDS/IPS	Events are generated by reconstructing a single session between two endpoints. Each session is examined for known security violation patterns. The component uses approximately 20,0000 IDS (ET-PRO and VRT) rules and is highly customizable. The update is every twelve hours.	Yes
SIEM/Third Party Log Management	Allows both importing and exporting Syslog, CEF, and OSSEC formats. This means that our system can export actionable network intelligence to any existing third party SIEM solutions as well as providing a unique correlation platform to view all your real time and historical security events feeds.	No
Flow Analysis	Generates flow records and our flow aggregation algorithm simplifies flow analysis by automatically choosing the most efficient invariant of a set of flows. This automatically highlights patterns that show scans or anomalous uses of network bandwidth.	No
Passive Service/Host Discovery	Discovers network services, user agents, DNS names, and DHCP leases by processing packets payloads. It provides the user with a comprehensive picture of which services or hosts are active on their network.	Yes
Vulnerability Scanning	Initiates external vulnerability scans (from Amazon EC2) against your own public facing systems, giving you an immediate overview of your exposure to attackers.	No
Historical Flow, Payload Data Storage and File Carving	TCP and UDP sessions are recorded on the local sensor disk for backtracking following an incident. Besides examining packet payloads and historical flow queries, this function also allows extracting files from the packet logs of the traffic being transmitted across your network. This is an important analysis feature which allows you to close the loop on suspected downloads, payloads from exploits, or policy violations, and to categorically identify malicious behavior or data exfiltration activities.	No
SSL Decryption/Interception	SSL traffic routed through the MetaFlows sensor is intercepted using a man-in-the-middle proxy. The users can load a locally generated CA certificate directly or the certificate can be offered as group policy. SSL interception is important for monitoring user access and identifying dangerous websites. It also plays a key role in Data Loss Prevention.	No

Multiple Session Analysis

The MSS coordinates and normalizes a mix of proprietary and open source packet processing functions aimed at providing multifunctional network intelligence correlation. One of the most important MSS applications is BotHunter, which performs real time multi-session analysis specifically designed to catch malware. Rather than relying on signatures, BotHunter looks for multiple red flags in the behavior of hosts on the network.

The red flags are consistent with the typical five phases of a bot infection and they include:

An inbound scan
Exploitation
An egg download
C&C communication
Propagation or data exfiltration activities

All of these phases need to be detected over multiple sessions with multiple hosts over an undetermined amount of time and therefore cannot be correlated by traditional IDS systems. MetaFlows uses BotHunter to perform high-speed real time correlation of these activities and therefore provides an anomaly detection function capable of detecting security threats that would otherwise go unnoticed. We recently extended BotHunter through our Correlation Engine Rules specification which we have used to provide additional behavioral correlation rules. This results in a deep packet inspection system with an unusual combination of low false positives and false negative rates.

For example, a typical campus of with 20,000 students would incur in 0.5-1 million events per day. These events are stored and made available for forensic examination but are only used as context (or backtracking). Our multi-functional correlation system processes these events in real time and offers the user only a very small fraction of the events in the form of correlation reports. A typical initial campus deployment would incur perhaps a dozen incidents per day (most of the time previously unknown) until proper IPS rules are implemented and use policies are enforced to remediate the issues.

Soft IPS

Placing a computer inline is not always desirable because of network reliability and latency concerns. MetaFlows' Soft IPS reliably blocks unwanted traffic in passive mode, removing the need for an in line configuration in most circumstances. MetaFlows does this by injecting several types of spoofed TCP packets into the network to disrupt unwanted communications. This idea (also employed by the Great Firewall of China) is not new but it has been refined to a point that makes a passive Soft IPS configuration as effective as an inline deployment, without the potential drawbacks of an inline device.

Summary of Soft IPS Benefits:

It runs entirely in software and can scale to 10 Gbps of network traffic on commodity hardware.
It runs in passive mode (not inline). This is can be a huge advantage because traditional, inline IPS configurations pose a higher risk to your network's availability
It uses powerful active response technology to block unwanted traffic (bots, spyware, P2P, etc.) and actively learns which hosts need to be isolated.

SIEM Import and Export

Our flexible Log Management solution allows both importing and exporting syslog, CEF, and Open Source SECurity (OSSEC) formats. This means that our system can export actionable network intelligence to any existing third party SIEM solutions as well as providing a unique correlation platform to view security events feeds from all your existing networked assets. The MSS can import and correlate the syslog events from a large number of devices including the following:

Type	Products
Firewalls	NetScreen, PIX, ASA, FWSM, Checkpoint, SonicWALL
IDS	Cisco, SourceFire (Snort), Dragon, Check Point Smart Defense
Antivirus	McAfee VirusScan Enterprise V8 and V8.5
Databases	MySQL, PostgreSQL
Mail	Imapd, Pop3d, Postifx, Sendmail, VPopMail, Microsoft Exchange, Courier Ipmapd, Pop3d, Pop3-ssl, Vm-pop3d, SMF-SAV, Procmail, Mailscanner
Web	Apache, IIS 5, IIS 6, Zeus, Horde Imp, ModSecurity
Cisco IOS Routers	All
Cisco VPN	All
Unix Based Servers	All
Microsoft Servers	All
OSSEC	All

File Transmission Logging and Network Antivirus

MetaFlows’ sensors monitor the transmission of all notable files (.exe, .dll, .pdf, .zip, Microsoft Office formats, etc.) transmitted on your network. The digest of each file is passed to the network antivirus system, which consists of 50+ antivirus solutions. All of the files that test positive on three or more antivirus solutions generate high-priority reports for your analysis. All of the potentially dangerous file transmissions (.exe, .dll, .pdf, .zip, Microsoft Office formats, etc.) are logged and correlated whether or not they are actually malicious. This allows you to see what your users are uploading or downloading (including BitTorrent). If a sample is unknown, these files are automatically submitted to MetaFlows' cloud-based sandboxing system (based on Cuckoo). The sandbox operation takes approximately 30-120 seconds per sample. Multiple virtual machines can be deployed to increase sandboxing throughput. We also recently released an Amazon EC2 sandbox capable of executing samples in an Amazon EC2 VPC.

Historical Flow and Payload Data Storage

The Historical Flow and Payload Data Storage function stores TCP and UDP flows and their associated payloads on the sensor disk. The sensor stores packets in four different databases.

Full Packet Payload (optional)
Tracker
IDS Events Packets
Sessions

Each of these four databases has a maximum size of 500GB by default (see below to change this). When each quota is reached the packet database is pruned by removing older packets automatically. If the disk on the sensor is less than 2TB, the system will start pruning packets to maintain 90% utilization. In this case, the pruning is performed in the order above (Full Packet Payload, Tracker, IDS Events Packets, and then Sessions) until 90% utilization is reached. A description of each of these packet payload databases is below:

Full Packet Payload Database

The Full Packet Payload Database stores all packets received by the sensor. This packet database is used to backtrack in time and reassemble complete TCP and UDP streams and optionally reconstruct files that were transferred. Full Packet Payloads are also important to file formal incident reports with law enforcement. To estimate the backtracking time for full packet capture, the following formula can be used:

storage=(24-hour-average-throughput/8*3600)*<number of hours of backtrack>).

So if, for example, the 24-hour-average throughput is 0.5Gbps, and we have a 500GB quota, the number of hours available for backtracking is approximately: 500,000,000,000/(500,000,000/8*3600)=2 hours. It is possible to increase the 500GB quotas by assigning the new desired quota the environment variable MAX_DISK. For example, to increase the quota to 1TB one would add the following statement to the /nsm/etc/mss.sh script:

export MAX_DISK=1000000000000

It is possible to disable full packet logging in favor of session storage in the sensor configuration page. Session storage of the first 512, 1024, 2048, or 4096 bytes of a flow instead of full packet capture will reduce storage requirements by approximately two orders of magnitude for the same backtracking time (or conversely increase backtracking time for the same amount of storage). Session storage might miss some information, but has a high probability of catching desirable payload which usually is in the first few hundred bytes of a session. In case session storage is not chosen through the sensor configuration page, an additional sessions packet store (see below) is enabled automatically to always provide at least one historical session flow data database.

Tracker Database

The Tracker Database is used to dynamically track specific IP addresses involved in high priority incidents. The correlation engine automatically starts logging full packet payloads of internal addresses as soon as a high-priority event is recognized for the next twenty-four hours. This database is useful because it dedicates a full packet payload store quota exclusively for the internal machines which are at most risk. This increases the likelihood that important packet payloads are recorded and available after an incident is recorded. The backtracking time for tracked IP address depends on the frequency of high priority events. Usually a 500GB quota allows backtracking for two to four months even on a fairly high speed network.

IDS Event Packets

Whenever an IDS rule fires, its associated packets are stored in this database. This store is especially useful when the analyst is trying to determine if an IDS rule is generating false positives. The backtracking time for this database is usually many months.

Session Packet Storage

Session Packet Storage is enabled whenever full packet storage is not in session storage mode or is disabled completely. This always guarantees that at least one data store is in session storage mode. If this database is enabled, the sensor will store the first 512 bytes of each TCP and UDP session. This allows users to perform historical NetFlow-like queries.


Previous Chapter	Next Chapter

Navigation

Search