Log Management

Sending Logs to Sensors

Syslog is a standard for logging program messages <ref name="Syslog">Syslog on Wikipedia http://en.wikipedia.org/wiki/Syslog </ref>. It allows the separation of the software that generates messages from the system that stores them and the software that reports and analyzes them. It also provides devices, which would otherwise be unable to communicate, a means to notify administrators of problems or performance.

Your sensor can accept syslog messages from a wide variety of devices. By enabling the log management feature and directing systems to send their log messages to the sensor, log events will appear in the Real-Time and Historical interfaces and will be correlated to the closest matching traffic flows. The sensor can also sniff syslog messages from the promiscuous interface if the logs are already being sent to a different device.

The sensor can parse messages in the following format:

Syslog
CEF
OSSEC syslog

Unix systems or most network security devices can be configured to send their existing syslog messages to the MetaFlows' sensor management interface IP address. Windows hosts or servers can be configured to send their event logs by installing the MetaFlows' Windows Agent (option 2 or 3).

The following sections provide more information on configuration for log management.

Configuring Log Management for Unix/Linux

Log in as root and open the file /etc/syslog.conf in your favorite editor. Add a line similar to the one shown below after replacing the IP address with that of your sensor.

# Change the IP address below to that of your sensor. Please make sure you have a space after the '*.*' 
*.* @192.168.1.100

Our syslog plugin tries to guess your network domain configuration with the commands:

myip=`dig +short myip.opendns.com @resolver1.opendns.com`
mydomain=`host $myip | grep -Po '[^.]+\.[^.]+\.$' | cut -d . -f 1,2`

This is not a perfect solution and you might need to verify that this automated configuration works as intended.

For example, if your DNS domain was domain.com it would be appended to /etc/resolv.conf as

search domain.com

if a search parameter is not present already in /etc/resolv.conf.

Please confirm that you have the correct search parameter in /etc/resolv.conf. If not, please add it manually as:

search domain.com

Then, for a given hostname <host>, make sure that the commands:

dig <host>

and

dig <host>.domain.com>

both work correctly and return the same address.

Configuring Log Management for Windows

Please see the MetaFlows' Windows Agent

Exporting from the Sensor

The MSS sensor can export events to an external SIEM or syslog server by specifying the destination IP address in the sensor configuration page.

Syslog Format

By default, the sensor will export in syslog format on UDP port 514 using a standard syslog format. The following two lines exemplify the output syslog format:

        Feb 29 17:12:09 S-0.tcovel: {1,0} [1:2009375:3] ET CHAT General MSN Chat Activity {IP} 211.122.190.171:1730 <- 93.42.235.179:80
        Feb 29 17:12:09 S-0.tcovel: {2,0} http:Server: Microsoft-IIS/6.0  {IP} 211.122.147.127:80 -> 63.251.63.121:3083

The syslog format in general is:

where:

syslogdate: event time
sensor: sensor name
domain: sensor domain (usually one per customer location)
evtype: 1 or 2. 1=snort/BH. 2=service discovery.
rank: priority. <0: false positive. 0: normal. >0 high threat
msg: event message
proto: "IP"
dir: "->" or "<-"
srcip,dstip,srcip,dstip: flow information.

Note that in syslog format CEF types 5,6,7,8,9,10,11,12,13 will all be lumped as type 5. if you want to see these different types in syslog format, we recommend using the Splunk App . As in the case of the CEF format, it is possible to see what syslog messages are sent out by adding the line export CEFDEBUG=1 to write all the Syslog messages to the file /tmp/cefdebug (even though we are sending syslog). '"Remember"' to unset the CEFDEBUG variable in production otherwise, your disk will fill.

CEF Format

The MSS sensor can also export using the CEF 1.0 format. This can be enabled by adding the line export CEF=1 in the file mss.sh and restarting the sensor. By adding the variable export CEFDEBUG=1 to write all the CEF messages to the file /tmp/cefdebug, it will log all your events to a file so be mindful of its size and available disk space. '"Remember"' to unset the CEFDEBUG variable in production otherwise your disk will fill.

We generate the following types of CEF events (1,2,3,5,6,7,8,9,10,11,12,13) described below. All CEF event types have the following main fields and extension fields (constants are in bold):

version= 0
Device Vendor= MetaFlows
Device Product= MSS
Device Version= 1.4.3
Signature ID= <Snort ID>, service:<port>, ostype:0.00, 3:5 for events 1,2,3,5-11 respectively
Severity= 0|1
cat= event type [1|2|3|5]6|7|8|9|10|11
src,dst,spt,dpt = flow information
dvhost= sensor name
cs1= <- or ->
cs1Label= direction
Start= time (in milliseconds) the event actually happened NOT the CEF event was generated

These are the meanings of the extension fields by event type:

Type 1 (IDS Event) additional extension fields:

None

Type 2 (Service Discovery) additional extension fields:

app=name of application running on src (http, BitTorrent, etc..)

Type 3 (Host Discovery) additional extension fields:

dhost= DNS name1[,DNS name2,...] (these are the DNS names associated with dst address)
requestClientApplication= Agent1[,Agent2...] (these are the http agents associated with dst address)
cs2= os1[,os2...os3] (These are the OS detected for dst address.)
cs2Label= os
suser= user1[,user2,....] These are the user names or email associated with dst address. They are extracted from (emails, DHCP responses, or Syslog messages)

Type 5 (Syslog Message Generated by an External Device to the MSS) additional extension fields:

None

Type 6 (File Transmission Analysis) additional extension fields:

filetype= mime type of file being transmitted
shost= originating mail server
suser= originating user's email
duser= destination user's email
info= subject of the email
requestURL= url used to download file
report= optional report URL detailing why something is malicious as reported by VirusTotal

Type 7 (IPS Notification) additional extension fields:

act=block
cnt=number of blocked sessions
cs2=IPS rule number
cs2Label=IPS rule number
cs3=IPS rule name
cs3Label=IPS rule Name
cs4=0|1; indicates if the IPS system is actually sending spoofed TCP/ICMP messages (1) or not (0). When 0, it shows what the IPS would have blocked without blocking it.
cs4Label=Armed

Type 8 (BotHunter Analysis) additional extension fields:

report=report URL detailing why something is malicious as reported by BotHunter

Type 9 (Tracking Analysis) additional extension fields:

report=report URL detailing why something is being tracked

Type 10 (Ranking Increase/Decrease) additional extension fields:

trigger=the event (syslog format) that caused the ranking to be changed

Type 11 (Class Match) additional extension fields:

trigger=the event (syslog format) that caused the class to match

Type 12 (ModSecurity inbound) additional extension fields:

RequestURL= url used to download file
cs2=IPS rule number
cs2Label=IPS rule number

Type 13 (ModSecurity outbound) additional extension fields:

RequestURL= url used to download file
cs2=IPS rule number
cs2Label=IPS rule number


Previous Chapter	Next Chapter

Splunk App

MetaFlows developed a splunk application to receive all sensor events on Splunk through an SSL encrypted channel called metaflows-syslog. The events are automatically categorized as follows:

Multisession Analysis
High Priority Events
IDS Events
Network Logs (3rd party logs sent to the sensors)
File Transmission Analysis
User Discovery
Service Discovery
Host Discovery
Mac Discovery
Suspicious URL Transmission Analysis
IPS Notifications
User Rankings
Modsecurity

Events volumes are also broken down by addresses, ports and other important invariants like sensor names, domains types, applications, etc.. It is possible to click on all summary records fileds to either drilldown on Splunk or drill down on the MetaFlows' web application to obtain more detailed forensic information like packets payloads.

Splunk app Installation

The app is available from SplunkWeb<ref name="metaflowsapp">on SplunkWeb at https://splunkbase.splunk.com/app/3603/ </ref>. Please download and install the application using your Splunk application manager. In order for Splunk to receive MetaFlows's sensors event you need to add the following line to your /nsm/etc/mss.sh startup script

export SYSLOG2=<splunk_host_ip_address>:3015

where <splunk_host_ip_address> is the ip address of the host where Splunk is running. After this, restart your sensor(s) with the command

/nsm/etc/mss.sh restart

Make sure that tcp port 3015 is open and the sensor can communicate with your Splunk host.

References

Navigation

Search