Correlation Engine Rules

From The MetaFlows Security System Documentation
Jump to: navigation, search

Introduction

Correlation Engine Rules (CER) process the events generated by the MSS components before they are stored in the database or displayed to the user in the real time interface. These rules allow to block flows, ignore events, prioritize events, track IP addresses and generate reports, or generate emails about specific combination of events.

The file /nsm/etc/classes on the sensor contains the user-defined CER rules and is maintained by the user. The file /nsm/etc/metaflows_classes is updated by the MetaFlows research team and is maintained automatically by the sensor. The user should not edit this file because it will be overwritten by the system.

After adding or editing CER rules, the rules are updated by the MSS with one of the following procedures:

  • Restart the MSS with the command line command /nsm/etc/mss.sh restart
  • Restart the MSS using the browser restart button
  • Execute the command killall -HUP mss from the command line to force the new CER rules to be reloaded

As the rules are reloaded errors will be displayed on STDERR and also added the mss logfile /tmp/msslog. Reloading through the killall -HUP command is the preferred method because it does not involve sensor downtime while it restarts.

The CER specification has the syntax:

{<ID>} {<name>} {<action>} {<cond>,<rpc>} <specification(1);specification(2);.. specification(n)> 

Where:

  • ID = unique numeric id for the CER (use values between 100-1000 to minimize potential ID conflicts)
  • Name = mnemonic name of the CER (we recommend limiting this to 10 characters or so)
  • Action = match, block, email, ignore, rank, trackint, trackext
  • <cond> <rpc> (see below)
  • Specification = one or more expressions separated by ';' (see below)

Actions

Block

Send spoofed Transmission Control Protocol (TCP) packets to prevent TCP communications. The TCP hijacking is performed by injecting spoofed packets in the packet stream through the management network interface. For this feature to work, the sensor needs to be able to route the IP packets to both source and destination IP addresses. The system also extracts invariants, so if particular IP addresses and ports combinations frequently trigger a block, they will be extracted as an invariant and blocked permanently until a Timeout. The Timeout is with respect to the last time the rule hit. If the rule keeps blocking connections, it will not expire.

If the rule does not block anything, it will be removed after a Timeout of 1 Year, 10 Minutes or 100 Seconds for:

  • A single IP block
  • 2 IP addresses or 1 IP address and one port
  • 2 IP addresses and one port

respectively.

Email

Send the matching events to the email address specified in the environment variable emailaddress. For this feature to work, it is necessary to execute export emailaddress=user@company.com before starting the MSS. It is best to add this command as the first line in file /nsm/etc/mss.sh so that this will be permanent. Emails will not be sent more than once per minute and will not send the same flow for at least ten minutes.

Ignore

Ignore the matching events. This action can be used to eliminate specific events that are to be simply ignored by the system.

Rank

Increase/decrease priority of matching events by the constant <rpc>. This action will also generate a system generated events to inform the user of the re-prioritization.

Trackint and Trackext

Start logging all packets for the internal (trackint) or external (trackext) IP addresses that match this rule. The packets will be logged for twenty-four hours from the last time the rule triggered. The MSS will generate a log message corresponding to this match and a report of why the tracking begun. It is also possible to extract files associated with the tracked IP through the report hyperlinks.

Match

This action is used to implement statefull multi-session rules. This action only sets a precondition and has no other side effects. Set a bitmask <rpc> for the internal or external IP address, the bitmask will then be used by other rules:

  • If <rpc> is negative (ex. -0x2) it will set the bitmask (ex. 0x2) for the external address of the event
  • If <rpc> is positive (ex. 0x4) it will set the bitmask (ex. 0x4) for the internal address of the event

<cond> and <rpc>

<cond> is used to match a precondition before the rest of the rule is evaluated. The precondition is used to implement stateful rules that depend from previous rules. The precondition can be 0 (always matches) or a positive hex number (ex. 0x20). A non-zero precondition must match a previously set bitmask either for the external address or the internal address bitmask or both before the rest of the rule (the <specification>) is evaluated. If the <specification> is evaluated successfully and the action is match, the <rpc> value is used to set the internal (if <rpc> is positive) or external (if <rpc> is negative) address bitmask. If the specification is evaluated successfully and the action is block,email,ignore,rank,trackint, or trackext, the particular action is taken.

<rpc> can be 0, a positive or a negative Hex number (ex. -0x20). <rpc> has completely different meaning if the action is match or one of block,email,ignore,rank,trackint, or trackext.

  • If action!=match, <rpc> increase/decreases the priority of the event if it matches <cond> AND the specifications.
  • If action==match,
    • A negative <rpc> will set the bitmask of the external address (not in the HOME_NET) if it matches the precondition AND the specifications.
    • A positive <rpc> will set the bitmask for the internal address (in the HOME_NET) if it matches the precondition AND the specifications.

Here is the pseudocode of the CER execution:

     if ((cond == 0 || 
          (time - addrt[extaddr] <1800 && addr[extaddr][cond] == true) || 
          (time - addrt[intaddr] <1800 && addr[intaddr][cond] == true)) 
          && specification == true) {
        if(action == match) {
           if(rpc<0) {
                 addr[extaddr][-rpc]=true
                 addrt[extaddr]=time
           } else {
                 addr[intaddr][rpc]=true
                 addrt[intaddr]=time
           }
        } else {
           priority=rpc
           perform action
        }
     }

Note: The address bitmask expires after 1800 seconds of inactivity.

Specifications

  • Each specification has the format <field> <op> <value>

Field

  • Flowbytes = the number of bytes exchanged by the flow (bidirectionally)) NOTE that flows with fewer than 20 packets will not be evaluated for performance reasons
  • Flowpackets = the number of packets exchanged by the flow (bidirectionally) NOTE that flows with fewer than 20 packets will not be evaluated for performance reasons
  • Flowstart = the age of the flow in seconds
  • Flowfrom = the name of the sensor sending the event
  • Flowranking = the priority of the event
  • Srca,dsta = Server/client IPv4 address.
  • Srcp,dstp = server/client port or port ranges like 80-88
  • Flowevents_service,flowevents_snort,flowevents_log = the content of the event message for service discovery, snort messages, or log messages

Op

Reg, Notnone, None, ==, !=, >=, <=, >,<

  • Reg matches a regular expression within the event messages (only valid for flowevents_service, flowevents_snort, flowevents_log)
  • Notnone, none matches if the event message exists/does not exists (only valid for flowevents_service, flowevents_snort, flowevents_log) and it does not require <value>
  • ==, !=, >=, <=, >,< correspond to their usual meaning

Value

This is the value to match against. A port range like 80-88 can only be used with the == operator. IPv4 addresses can be specified as 99.91.52.245/32 or a list (any of which may match) of the form 139.182.2.24/32,139.182.33.0/24. Built in Snort Variables like HOME_NET or user defined Snort variables can also be used.

Stateful Multi-session analysis

The statefull multi-session capabilities of the rules help in detecting behavioral anomalies that cannot be detected by one detection methodology alone but require the interaction of multiple applications. This API is used by the MetaFlows research team to publish new rules on a daily basis, or it can be used by the customer to write site-specific rules.

Here is a simple illustrative example that finds external hosts that scan or use covert channels and then exfiltrate data. The regular expressions SSH (?!(Scan OUTBOUND)) and scan (?!(OUTBOUND)) (rules 6 and 7) trigger whenever a Snort rule with those keywords in the description fires; unless the SSH and Scan keywords are followed by Scan OUTBOUND and OUTBOUND respectively. These conditions set the bit mask -0x20. The negative sign indicates the external address will be flagged associated with the Snort alert. The value 0x20 is the particular mask for this correlation rule. The last rule (rule 8) checks for the 0x20 bit mask and, if the same external address also exchanges more than 1000 packets (with any host), a tracking message is generated with priority of 0x100 (256).

      {6}{scan}{match}{0,-0x20}flowevents_snort reg SSH (?!(Scan OUTBOUND));
      {7}{scan}{match}{0,-0x20}flowevents_snort reg scan (?!(OUTBOUND));
      {8}{dataex}{trackext}{0x20,100}flowpackets > 1000;

This is just an illustration, the API is much more powerful and allows for much more complex and diverse actions.

Examples

Simple Matches: No Preconditions

These rules tell the system what to do when an event is generated without look back at any state. Notice that cond is always 0.

{110}{java rank}{rank}{0,0xa}flowevents_snort reg Vulnerable Java Version;
{130}{msn rank}{rank}{0,0xa}flowevents_snort reg General MSN Chat Activity;
{140}{tor email}{email}{0,0}flowevents_snort reg TOR 1;
{150}{ignore slammer}{ignore}{0,0}flowevents_snort reg Slammer Worm;
{160}{rank regex vuze}{rank}{0,0xa}flowevents_snort reg P2P\sVuze.*UDP.*\(\d\);
{170}{rank regex slash apache}{rank}{0,0xa}flowevents_service reg Apache/\d;
{180}{rank singleeq}{rank}{0,0x400}dstp == 80; dsta == $USER_DEFINED_IP_LIST;
{190}{filtersnort}{ignore}{0,0}flowevents_snort reg 1.2; srca==$WHITELIST;
{100}{filterservice}{ignore}{0,0}flowevents_service notnone;
{110}{Ranked}{email}{0,0}flowranking > 0;

One Precondition: Multiple Triggers and Multiple Actions

The first rule set the bitmask 2 flag on the external IP address if a snort event is generated with the string policy in the message and the client is in the 211.0.0.0/8 network range. Then if within 1800 seconds the same external IP address is involved in the generation of a snort rule with shellcode or trojan in the message an email alert is generated. Also, if the snort rules message contains trojan, the internal address is also tracked.

{150}{Policyd}{match}{0,-0x2}flowevents_snort reg policy; dsta == 211.0.0.0/8;
{130}{shellcode}{email}{0x2,0}flowevents_snort reg shellcode;
{160}{etprotrojan}{email}{0x2,0}flowevents_snort reg trojan;
{170}{tracktrojan}{trackint}{0x2,0}flowevents_snort reg trojan;

One Precondition and a Pure Flow Match

The first rule set the bitmask 0x20 flag on the external IP address if a snort event is generated with the string SSH. If the same external IP address then transfers more than 100 bytes in any flow, a ranking message is generated to alert the user of the event. The rank event will appear as an MSS system generated event.

{230}{scan}{match}{0,-0x20}flowevents_snort reg SSH;
{240}{sshflow}{rank}{0x20,0xa}flowbytes > 1000;


Previous Chapter Next Chapter