Log Management

From The MetaFlows Security System Documentation
Revision as of 01:16, 3 May 2018 by Wikiadmin (talk | contribs) (Log Management)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Sending Logs to Sensors

Syslog is a standard for logging program messages <ref name="Syslog">Syslog on Wikipedia http://en.wikipedia.org/wiki/Syslog </ref>. It allows the separation of the software that generates messages from the system that stores them and the software that reports and analyzes them. It also provides devices, which would otherwise be unable to communicate, a means to notify administrators of problems or performance.

Your sensor can accept syslog messages from a wide variety of devices. By enabling the log management feature and directing systems to send their log messages to the sensor, log events will appear in the Real-Time and Historical interfaces and will be correlated to the closest matching traffic flows. The sensor can also sniff syslog messages from the promiscuous interface if the logs are already being sent to a different device.

The sensor can parse messages in the following format:

  • Syslog
  • CEF
  • OSSEC syslog

Unix systems or most network security devices can be configured to send their existing syslog messages to the MetaFlows' sensor management interface IP address. Windows hosts or servers can be configured to send their event logs by installing the MetaFlows' Windows Agent (option 2 or 3).

The following sections provide more information on configuration for log management.

Configuring Log Management for Unix/Linux

Log in as root and open the file /etc/syslog.conf in your favorite editor. Add a line similar to the one shown below after replacing the IP address with that of your sensor.

# Change the IP address below to that of your sensor. Please make sure you have a space after the '*.*' 
*.* @

Our syslog plugin tries to guess your network domain configuration with the commands:

myip=`dig +short myip.opendns.com @resolver1.opendns.com`
mydomain=`host $myip | grep -Po '[^.]+\.[^.]+\.$' | cut -d . -f 1,2`

This is not a perfect solution and you might need to verify that this automated configuration works as intended.

For example, if your DNS domain was domain.com it would be appended to /etc/resolv.conf as

search domain.com

if a search parameter is not present already in /etc/resolv.conf.

Please confirm that you have the correct search parameter in /etc/resolv.conf. If not, please add it manually as:

search domain.com

Then, for a given hostname <host>, make sure that the commands:

dig <host>


dig <host>.domain.com>

both work correctly and return the same address.

Configuring Log Management for Windows

Please see the MetaFlows' Windows Agent

Exporting from the Sensor

The MSS sensor can export events to an external SIEM or syslog server by specifying the destination IP address in the sensor configuration page.

Syslog Format

By default, the sensor will export in syslog format on UDP port 514 using a standard syslog format. The following two lines exemplify the output syslog format:

        Feb 29 17:12:09 S-0.tcovel: {1,0} [1:2009375:3] ET CHAT General MSN Chat Activity {IP} <-
        Feb 29 17:12:09 S-0.tcovel: {2,0} http:Server: Microsoft-IIS/6.0  {IP} ->

The syslog format in general is:

<syslogdate> <sensor>.<domain> {<evtype>,<rank>} <msg> {<proto>} <srcip>:<srcport> <dir> <dstip>:<dstport>


event time
sensor name
sensor domain (usually one per customer location)
1 or 2. 1=snort/BH. 2=service discovery.
priority. <0: false positive. 0: normal. >0 high threat
event message
"->" or "<-"
flow information.

Note that in syslog format CEF types 5,6,7,8,9,10,11,12,13 will all be lumped as type 5. if you want to see these different types in syslog format, we recommend using the Splunk App . As in the case of the CEF format, it is possible to see what syslog messages are sent out by adding the line export CEFDEBUG=1 to write all the Syslog messages to the file /tmp/cefdebug (even though we are sending syslog). '"Remember"' to unset the CEFDEBUG variable in production otherwise, your disk will fill.

CEF Format

The MSS sensor can also export using the CEF 1.0 format. This can be enabled by adding the line export CEF=1 in the file mss.sh and restarting the sensor. By adding the variable export CEFDEBUG=1 to write all the CEF messages to the file /tmp/cefdebug, it will log all your events to a file so be mindful of its size and available disk space. '"Remember"' to unset the CEFDEBUG variable in production otherwise your disk will fill.

We generate the following types of CEF events (1,2,3,5,6,7,8,9,10,11,12,13) described below. All CEF event types have the following main fields and extension fields (constants are in bold):

  • version= 0
  • Device Vendor= MetaFlows
  • Device Product= MSS
  • Device Version= 1.4.3
  • Signature ID= <Snort ID>, service:<port>, ostype:0.00, 3:5 for events 1,2,3,5-11 respectively
  • Severity= 0|1
  • cat= event type [1|2|3|5]6|7|8|9|10|11
  • src,dst,spt,dpt = flow information
  • dvhost= sensor name
  • cs1= <- or ->
  • cs1Label= direction
  • Start= time (in milliseconds) the event actually happened NOT the CEF event was generated

These are the meanings of the extension fields by event type:

Type 1 (IDS Event) additional extension fields:

  • None

Type 2 (Service Discovery) additional extension fields:

  • app=name of application running on src (http, BitTorrent, etc..)

Type 3 (Host Discovery) additional extension fields:

  • dhost= DNS name1[,DNS name2,...] (these are the DNS names associated with dst address)
  • requestClientApplication= Agent1[,Agent2...] (these are the http agents associated with dst address)
  • cs2= os1[,os2...os3] (These are the OS detected for dst address.)
  • cs2Label= os
  • suser= user1[,user2,....] These are the user names or email associated with dst address. They are extracted from (emails, DHCP responses, or Syslog messages)

Type 5 (Syslog Message Generated by an External Device to the MSS) additional extension fields:

  • None

Type 6 (File Transmission Analysis) additional extension fields:

  • filetype= mime type of file being transmitted
  • shost= originating mail server
  • suser= originating user's email
  • duser= destination user's email
  • info= subject of the email
  • requestURL= url used to download file
  • report= optional report URL detailing why something is malicious as reported by VirusTotal

Type 7 (IPS Notification) additional extension fields:

  • act=block
  • cnt=number of blocked sessions
  • cs2=IPS rule number
  • cs2Label=IPS rule number
  • cs3=IPS rule name
  • cs3Label=IPS rule Name
  • cs4=0|1; indicates if the IPS system is actually sending spoofed TCP/ICMP messages (1) or not (0). When 0, it shows what the IPS would have blocked without blocking it.
  • cs4Label=Armed

Type 8 (BotHunter Analysis) additional extension fields:

  • report=report URL detailing why something is malicious as reported by BotHunter

Type 9 (Tracking Analysis) additional extension fields:

  • report=report URL detailing why something is being tracked

Type 10 (Ranking Increase/Decrease) additional extension fields:

  • trigger=the event (syslog format) that caused the ranking to be changed

Type 11 (Class Match) additional extension fields:

  • trigger=the event (syslog format) that caused the class to match

Type 12 (ModSecurity inbound) additional extension fields:

  • RequestURL= url used to download file
  • cs2=IPS rule number
  • cs2Label=IPS rule number

Type 13 (ModSecurity outbound) additional extension fields:

  • RequestURL= url used to download file
  • cs2=IPS rule number
  • cs2Label=IPS rule number
Previous Chapter Next Chapter

Splunk App

MetaFlows developed a splunk application to receive all sensor events on Splunk through an SSL encrypted channel called metaflows-syslog. The events are automatically categorized as follows:

Multisession Analysis
High Priority Events
IDS Events
Network Logs (3rd party logs sent to the sensors)
File Transmission Analysis
User Discovery
Service Discovery
Host Discovery
Mac Discovery
Suspicious URL Transmission Analysis
IPS Notifications
User Rankings

Events volumes are also broken down by addresses, ports and other important invariants like sensor names, domains types, applications, etc.. It is possible to click on all summary records fileds to either drilldown on Splunk or drill down on the MetaFlows' web application to obtain more detailed forensic information like packets payloads.

Splunk app Installation

The app is available from SplunkWeb<ref name="metaflowsapp">on SplunkWeb at https://splunkbase.splunk.com/app/3603/ </ref>. Please download and install the application using your Splunk application manager. In order for Splunk to receive MetaFlows's sensors event you need to add the following line to your /nsm/etc/mss.sh startup script

export SYSLOG2=<splunk_host_ip_address>:3015

where <splunk_host_ip_address> is the ip address of the host where Splunk is running. After this, restart your sensor(s) with the command

/nsm/etc/mss.sh restart

Make sure that tcp port 3015 is open and the sensor can communicate with your Splunk host.


<references />