1 (877) 664-7774 - support@metaflows.com
Sensor Setup
The MetaFlows system is easy to set up in a few minutes, easy to maintain, and yet provides a full-featured and sophisticated multifunctional network security monitoring system. Any of the configuration steps can be modified and later further refined through the browser.
Contents
- 1 Registering with MetaFlows
- 2 Adding a Sensor
- 2.1 Adding a Sensor (Short Form)
- 2.2 Adding a Sensor (Advanced)
- 2.3 Sensor Variables
- 2.4 Sensor Application Details
- 2.5 Automatic Blocking for Priority Rules
- 2.6 Manage Local Rule Source
Registering with MetaFlows
- Open https://nsm.metaflows.com/ with a Firefox or Google Chrome browser
- Select the "Create Your Account" link. You will be redirected to the Registration Form page shown in <xr id="fig:reg_form"/>.
- Enter your email address, which will be used for for all communication with the MetaFlows Security System (MSS), as your login ID
- Enter your password twice
- Answer the CAPTCHA challenge
- Fill in your contact information
- On the right side of the Registration Form are a few questions that will help us get to know your network security needs, please answer those before moving before you continue.
- Make sure to read the Licensing Terms and click on the checkbox to accept and to move forward.
- Look for a welcome message sent to the email address you provided. If it has not arrived in five minutes, check your spam folder to verify that the email was not blocked. Review your registration information in the email for accuracy, if there are any errors they can be corrected after logging in. Please login with your information to begin configuring and downloading your new sensor.
<figure id="fig:reg_form">
</figure>
Adding a Sensor
On your first login you will be shown this window:
If you choose to configure your sensor later you can always hit the "Add a Sensor" button on the homepage of the Dashboard.
Click the "Add a Sensor" button on the homepage of the dashboard or from the "Sensors" drop-down menu, select "Add" as shown in <xr id="fig:sensor_drop"/>.
<figure id="fig:sensor_drop">
</figure>
The sensor configuration and customization detail form is displayed next. This configuration is stored in your secure web space and can be modified at any time in the future.
Adding a Sensor (Short Form)
<figure id="fig:sensor_summary">
</figure>
The short form allows the addition of a sensor by just specifying essential parameters required for proper operation. This short form sets a large number of values to a default value that should work for most installations. We recommend using this for the short form initially and then modifying the sensor configuration later (if required) by clicking on sensor->view and modifying the sensor by clicking on its name and then clicking on Show more Sensor Options to reveal the advanced options.
The Sensor Name, Domain, Sensor Location are pre-filled in with some predefined values that can be modified to match your preference, if you do not want to use the system defaults. Simply edit these fields with some names that make more sense to you.
The Promiscuous Interface and the HOME_NET fields are very important and need to be modified to match your particular installation. We picked values that are very common but please verify their correctness before proceeding. If you do not enter correct values, your sensor will not run.
Adding a Sensor (Advanced)
Clicking on Show More Sensor Options reveals the advanced sensor settings for users who are familiar with networking and familiar with many concepts related to Intrusion Detection Systems.
<figure id="fig:sensor_details">
</figure>
The sensor details form is shown in <xr id="fig:sensor_details"/>. Note Based on the options you have chosen your form might look slightly different. This section explains all possible fields of the form.
- Sensor Name This is your text name for the sensor you are creating. This is usually the hostname where the sensor is installed.
- Domain This is the domain that groups all your sensors under one administrative domain. You should pick a domain such as mysensors or mycompany.com and use it for all sensors you add. Note that this field is not related to your actual company domain name or web address. This will not be used for DNS queries, so you can pick any name that makes sense to you.
- Sensor Location The location of your sensor, or another label you would like to use to identify the sensor.
- Promiscuous Interface This is the name of the interface the sensor will be monitoring packets from. Usually this field is eth1. Promiscuous Mode refers to a configuration of your network card(or virtual network interface) to allow packets to be sent to packet sniffing programs. You can learn more about Promiscuous Mode here
- Sensor as a Server In server mode the sensor accepts connection from the Browser directly. Enabling this option would permit you to monitor your sensor from outside your firewall. If you need to monitor your sensor from outside a firewall, you need to open/forward ports 843, 3001, 3002 from/to the sensor. The advantage of this mode is that the sensor real time data will go directly to your browser. Click on this button if you would prefer to assign a Static IP address and use this mode. When you click this button the browser adds a Real Interface Configuration section that you need to fill in. The flow of data for Sensor as a server is shown in <xr id="fig:sensor_server"/>.
<figure id="fig:sensor_server">
</figure>
- Sensor as a Client In client mode the sensor uses outgoing connections to MetaFlows' forwarder so that Browsers from outside your firewall can securely connect to it. This mode does not require any additional system configurations. The drawback of this mode is that the sensor real time data will always route through MetaFlows forwarder (located in Southern California) even though your Browser might be closer to the sensor. If you select this option you will be given an additional button DHCP as shown in <xr id="fig:sensor_details"/>. The flow of data for Sensor as a Client is shown in <xr id="fig:sensor_client"/>
<figure id="fig:sensor_client">
</figure>
- DHCP Selecting DHCP mode implies that you have a DHCP server on your network which assigns IP addresses automatically to clients requesting them.
<figure id="fig:real_interface">
</figure>
In this form, shown in <xr id="fig:real_interface"/> you will specify the Static IP addresses, Nameservers and other details as explained below. You will not see this form if you had chosen the DHCP option in the previous section
Sensor Details*Static IP IP address to reach a non-natted sensor. Ports 3001, 3002, and 843 need to be open or forwarded to this IP.
- Subnet This is the netmask or network mask of your real network interface. It determines which portion of your ip address is divided into subnetworks, and which portion is used to allocate actual machine addresses. It is usually of the form 255.255.255.0. Contact your IT administrator if you are not sure of this mask.
- Gateway This is the IP address of the gateway that your systems access the internet through.
- Nameserver 1 This is the Domain Name Server (DNS) address (DNS) for name resolution.
- Nameserver 2 The alternate or additional DNS server IP address can be specified here.
Log Management
Check this option if you wish to send syslog messages to the sensor. The sensor will normalize these messages and correlate them with the other security messages. The logs can can be sent from any device (including other Snort sensors) as long as they use standard syslog format. OSSEC syslog messages are particularly helpful and we highly recommend feeding them to the sensor.
Event Destinations
- MetaFlows DB This is the default event destination, all events will be sent to the MetaFlows cloud to be correlated and stored in our databases. Most users should leave this option checked.
- Local DB (Beta) The Local DB option logs events to a MySQL database that is running on the sensor, this allows events to be fetched from the local system instead of from our cloud. Some users with specific privacy concerns may wish to try this option. Note that some features are dependent on our correlation and cloud storage systems and so may work while running in local DB mode.
- Syslog Server The sensor can send events to an external syslog server. An example of the exported syslog is:
Feb 29 17:12:09 S-0.tcovel: {1,0} [1:2009375:3] ET CHAT General MSN Chat Activity {IP} 211.122.190.171:1730 <- 93.42.235.179:80 Feb 29 17:12:09 S-0.tcovel: {2,0} http:Server: Microsoft-IIS/6.0 {IP} 211.122.147.127:80 -> 63.251.63.121:3083
The format is: syslogdate> <sensor>.<domain> {<evtype>,<rank>} <msg> {<proto>} <srcip>:<srcport> <dir> <dstip>:<dstport>
where:
syslogdate=event time sensor=sensor name domain=sensor domain (usually one per customer location) evtype=1 or 2. 1=snort/BH. 2=service discovery. rank=priority. <0: false positive. 0: normal. >0 high threat msg=event message proto="IP" dir="->" or "<-" srcip,dstip,srcip,dstip=flow
Additional Details about the Local DB mode
In order to use the localdb you will have to accept an ssl certificate the first time that your browser connects to the sensor.
1. When you see this notification, click the link provided to go to the certificate page.
2. The certificate will appear as untrusted because you are using nsm.metaflows.com but the certificate is a self-signed certificate for your local sensor. If you are using Firefox Click "Add Exception" and then "Confirm Security Exception".
3. Once accepted the page should display a message showing that everything is setup correctly
4. Now, when you do a historical query you should see the following message.
Use Multiple Cores If Available
Check this if you want to run Snort is parallel to achieve performance greater than 100Mbps. If you check this box, the sensor will spawn multiple instances of IDS/IPS processes to achieve performance greater than 100 Mbps.
Use Inline Mode
Check this option if you would like to run the sensor in line. When running the sensor inline, packets are forwarded between the interfaces specified at Layer2. The sensor will not require additional IP addresses as the packets are bridged transparently. In inline mode any drop IDS rule will cause the packets not to be forwarded. After you check this option, you have to specify the interfaces which will be your input and output port.
Sensor Variables
<figure id="fig:sensor_vars">
</figure>
In this screen, shown in <xr id="fig:sensor_vars"/>, you will set the variables used to configure Snort and other packet monitoring devices on the sensors.
- HOME_NET This should identify the network address of the network you are monitoring and or protecting. Examples of HOME_NET variables are: 192.168.1.0/24 or [192.168.0.0/16,10.0.0.0/8]. Notice that you can list a number of network addresses if you have more than one subnet.This is the single most important parameter. If you do not know what this should be, please ask your IT or Network support team.
- Advanced Variables Beginning users are recommended to disable this option. The built in defaults for advanced variables will work for most installations. If you are an IDS (Intrusion Detection System) expert you can enable this button and you will see an expanded form as shown in <xr id="fig:adv_vars"/> below.
<figure id="fig:adv_vars">
</figure>
- EXTERNAL_NET This is usually the IP addresses external to your organization. This should be set to !$HOME_NET unless you really know what you are doing.
- DNS_SERVERS This is the list of IP addresses of Domain Name Servers being used in your network (such as [192.168.1.1,192.168.2.1]). If you do not know their addresses, leave this as 0.0.0.0. If you set DNS_SERVERS to $HOME_NET or any, there is a high probability that false positives will be generated (alerts that should not be generated).
- SMTP_SERVERS This is the list of IP addresses of Email Servers being used in your network (such as [192.168.1.1,192.168.2.1]). If you do not know their addresses, leave this as 0.0.0.0.
- HTTP_SERVERS This is the list of IP addresses of Web Servers being used by your network (such as [192.168.1.1,192.168.2.1]). This could also be a network address such as 192.168.1.128/23. If you do not know this, leave this as $HOME_NET.
- SQL_SERVERS This is the list of IP addresses of SQL Servers being used by your network (such as [192.168.1.1,192.168.2.1]). This could also be a network address such as 192.168.1.128/23. If you do not know this, leave this as $HOME_NET.
- TELNET_SERVERS This is the list of IP addresses of telnet servers being used by your network (such as [192.168.1.1,192.168.2.1]). This could also be a network address such as 192.168.1.128/23. If you do not know this, leave this as $HOME_NET.
- SNMP_SERVERS This is the list of IP addresses of SNMP Servers being used by your network (such as [192.168.1.1,192.168.2.1]). This could also be a network address such as 192.168.1.128/23. If you do not know this, leave this as $HOME_NET.
- HTTP_PORTS This is the list of ports that host HTTP services(Web server) in your network. 80 or [80,8080] are very typical values.
- SHELLCODE_PORTS This is the list of ports you want to monitor for SHELL Code exploits. !80 is a very typical value since shellcode does not work on HTTP servers while all other ports are at risk.
- ORACLE_PORTS This is the list of ports Oracle servers use. 1521 is a very typical value. If you have a non-standard configuration you can list the ports using the syntax [a,b,c].
- AIM_SERVERS: This variable lists the AIM servers and should not be modified unless you have a non-standard AIM configuration. The default value is [64.12.24.0/23,64.12.28.0/23,64.12.161.0/24,64.12.163.0/24,64.12.200.0/24,205.188.3.0/24,205.188.5.0/24,205.188.7.0/24,205.188.9.0/24,205.188.153.0/24,205.188.179.0/24,205.188.248.0/24].
- Filter This variable is a filtering expression to ignore packets not interesting to the sensor or that generate excessive data. The filter expression is a pcap expression (please see the official tcpdump documentation for more information about pcap expressions). An example filter expression is
not host 10.10.1.253 and not host 192.12.33.100 and not host 192.1.33.102
This example will ignore hosts 10.10.1.253, 192.12.33.100, and 192.1.33.102. Notice that you do not need quotes. As you are typing, the expression is validated in real-time and the text-box changes color to red if incorrect and to green if it is correct.
- CUSTOM_VAR If you are running your own Snort rules and have custom variables, you can define them as custom variables. Once you have added variables, you can click on the "Delete" button next to each variable to remove it. You may only add one instance of each variable. Multiple instances of a custom variable must have different names. For example:
MAX_DISK This custom variable can be used to set the maximum number of bytes of all MetaFlows logs on the sensor. The default is 10 gigabytes. For example, to increase the default to 100 gigabytes one would enter MAX_DISK=100000000000.
Sensor Application Details
<figure id="fig:sensor_app">
</figure>
As shown in <xr id="fig:sensor_app"/>, Several detection options are available. Typically all of them should be checked to get the most out of the system. In some cases, it is desirable to disabled some of them for performance reasons or for policy reasons.
Flow Analysis and Passive Service Discovery
The MSS embeds security event information within the context of real time flow data and analysis and allows the user to gain far greater visibility into their network. The MSS also allows users to record historical flow data with Ntop. The MSS flow aggregation algorithm simplifies flow analysis by automatically choosing the most efficient invariant of a set of flows. This automatically highlights patterns that show scans or anomalous uses of network bandwidth.
Network Analysis and File Carving
File carving extracts files from the traffic being transmitted across your network. This is an important analysis feature which allows the investigator to close the loop on suspected downloads, payloads from exploits, or policy violations, and to categorically identify malicious behavior or data exfiltration activities. Furthermore, with integration to VirusTotal, extracted files can be immediately scanned for known viruses and payloads. The file carving system can be launched from the real-time or historical records in the MetaFlows interface, or from the host flow data in the Ntop interface, and will precisely select packet logs which contain data about the host(s) and event(s) in question.
Malware Analysis (BotHunter)
BotHunter uses Multiple Session correlation to gather specific IDS alerts (also called dialog events) that form a typical behavior pattern for an infected host. Dialog events are fed directly into a separate correlation engine, where each host's individual dialog production pattern is mapped and scored against an abstract Malware Infection Life Cycle Model. When the dialog correlation algorithm shows that a host's dialog patterns map sufficiently close to the life cycle, the hose is declared infected, and an infection profile is generated to summarize all evidence about the infection.
Passive OS Fingerprinting
Passive OS fingerprinting attempts to discover what Operating System version the hosts are running by analyzing the way they communicate. This information can be helpful to understand how the security alerts affect a particular host based in its Operating System type and version. This option does not consume much CPU; so it is always safe to leave it on.
Store Packets On Sensor
This option makes the sensor store all packet logs on the disk. The storage is limited to the minimum of (1) 90% of the bytes available on the partition on which the logs are stored (/mnt/hgfs/logs) and (2) the value of MAX_DISK (in bytes); MAX_DISK is set to 500GB by default.
Block Communications in Passive Mode (Soft IPS)
MetaFlows Soft IPS technology blocks unwanted traffic in passive mode. Soft IPS does this by injecting spoofed TCP packets into the network to disrupt unwanted communications. This idea (also employed by the Great Firewall of China) is coupled with a new algorithm that will safely predict what traffic to block based on observed communication patterns. No rules are set to trigger IPS by default. So it is safe to enable this.
File Monitoring
The MetaFlows Security System (MSS) sensors reassemble interesting files being transmitted on your network (both inbound and outbound) on ports 25, 80, 110, and port 143. These are the ports through which most malware is propagated with browser-based exploits, Phishing, or Email Spam.
As shown in the diagram, all dangerous file transmissions (exe, dll, MS Office, pdf, zip, etc.) are logged and correlated whether or not they are malicious. This allows you to see what content your users are downloading or uploading.
The files that contain malware as reported by Virus Total are flagged as high-priority events for your analysis. Usually, any of these events need to be taken very seriously and appropriate remediation should be done quickly.
Please enter your free Virus Total key here if you have one. if you do not have one, we highly recommend registering with Virus Total at the link provided and getting a free key. If you do not want enter a Virus Total key, your malware detection will be limited.
If you do not want to log all file activity, uncheck 'Log All File Transfers'
Passive Modsecurity
Enabling File Carving also enables by default passive modsecurity. Tradidionally, modsecurity is deployed as a Web Application Firewall plugin directly on the web servers. It then uses a number of IDS modsecurity rules to inspect the inbound requests to the web server and blocks unwanted malicious requests. MetaFlows has developed the capability of deploying modsecurity inspection on the sensor. Whenever a web request is detected by the sensor, it is analyzed by the modsecurity rules and corresponding alerts are generated as system daemon syslog events detailing the exploit.
Modsecurity Alerts
These alerts are generated any time an external IP address (which is not in the HOME_NET specification) or an internal IP address attacks one or more web servers. An example outbound modsecurity event alert is shown below
modsec_out/CRITICAL/960010 Request content type is not allowed by policy - pan.baidu.com/rest/2.0/dss/online -- application/x-www-form-urlencoded
In addition to the flow information (not shown for clarity), the alert message specifies:
- whether it is an inbound (modsec_in) or outbound (modsec_out) attack (modsec_out in this case)
- the class of alert (CRITICAL in this case)
- the rule ID that triggered (960010 in this case)
- the description of the rule (Request content type is not allowed by policy in this case)
- the URL that caused the alert (pan.baidu.com/rest/2.0/dss/online in this case)
- the mime type returned by the request (application/x-www-form-urlencoded in this case)
Client or Server Mode
If Modsecurity is enabled, it can run in either client mode or server mode.
Client mode should be selected if the user is monitoring a network predominantly to detect client based malware. In client mode, the MSS adds a behavioral detection layer to filter out false positives due to the fact hat typically most client network trigger quite a few raw modsecurity alerts. This is due to the high heterogeneity of external web servers accessed by client-based networks. In this mode a modsecurity alert is only recorded if the same IP address triggers multiple modsecurity rules with multiple servers. Also, alerts are recorded only if the servers respond with 4xx or 5xx errors indicating the HTTP requests were malformed or prohibited. Client mode therefore only triggers if the same source of modsecurity exceptions is spread over time and multiple targets thus indicating a brute-force scanning behavior.
Server mode should be selected if the user is monitoring a network predominantly to detect attacks to their web servers. In server mode, the MSS generates mdosec_in or modsec_out alerts for every modsecuroty rule that triggers. It is recommended that in this mode the user carefully monitor the modsec alerts to eliminate false positives by disabling specific modsecurity alerts that are not considered harmful (for example allowing POST methods or allowing request without the host parameter in the HTTP request). The tuning of modsecuroty alerts can be performed using the Modsecurity Rule editor.
Modsecurity Rule Editor
The modsecuroty rule editor allows to edit, or disable modsecurity rules or configuration options. it is recommended that the initial configuration options not be modified. We recommend that the editor be used primarily to disable or enable particular rules. Note that any time a rule is disabled or edited it will be remembered and any following rule updates from Spiderlabs will not overwrite the user changes. If you want to revert the rules to a default configuration where they are updated by Spiderlabs distribution contact support@metaflows.com.
The modsecurity rule editor can be accessed through the Rules menu.
This will bring a sensor selection menu. Select a particular sensor to manage as shown in the picture.
There is a search capability to quickly find specific rule IDs of interest or any other specific text in the rules. Typing the search string will bring up the matching rules.
Once the appropriate rule or directive is found, hovering over it will show a disable/enable or edit menu. Disabling a rule or editing it will save a temporary copy of the rules.
Clicking on Validate will run a test on the modified rules to make sure they will run on the sensor without causing any faults. Once validated the user is prompted to restart the sensor in order to apply the change to the running system. This restart operation can be delayed or can also be executed by entering the following command on the sensor itself at any time. Until the sensor restarts the old rules will still be in effect.
/nem/etc/mss.sh restart
If the validation fails due to a syntax error, the system will provide some debugging information to the user and prompt the user to fix the issue before continuing.
Automatic Blocking for Priority Rules
You can enable a default block level for rules that are listed on the MetaFlows sid Priority map. These rules are collected and ranked each day using data from every domain currently being monitored by a MetaFlows sensor.
You can view the raw data for the rules at https://nsm.metaflows.com/sid_priority.map
Once a new default block level is chosen and the sensor is saved, you will be prompted to reload and save the rules in order to apply the changes.
<figure id="fig:sensor_autoblock">
</figure>
Manage Local Rule Source
The final Manage Local Rule Source is shown in <xr id="fig:sensor_view"/>.
Users who do not wish to manage rule sets locally will leave this field blank. Users choosing to manage rule sets locally can do so by entering the directory for the local rule set files here. If you want to manage your Snort rules completely independently from our system, check this box and store all of your rules in /nsm/localrules/ on the sensor. You will be responsible for updating and maintaining your ruleset in that directory. Note that if you manage your rules locally, the rule management interface will be disabled and you will not get any rule updates from us.
You are now ready to click Add Sensor. <figure id="fig:sensor_view">
</figure>
Clicking the "Add Sensor" button will redirect you to the sensor list page similar to Sensor View image displayed in <xr id="fig:sensor_view_1"/>. Your View Sensors page might look different based on the name(s) and number of sensors you have added. <figure id="fig:sensor_view_1">
</figure>
| Previous Chapter | Next Chapter |
