Deep Packet Inspection of Cloud-based assets

Overview

The advantage of the MSS in cloud security monitoring applications is that it can perform deep packet inspection in virtual cloud environments (like Amazon AWS or Microsoft Azure) without requiring access to the networking layer. Each MSS sensor can monitor up to 2 Gbps of traffic originating from cloud assets, inherits all the features of a physical deployment, and therefore, provides a complete IDS/IPS solution for a virtual cloud environment. This enables organizations to meet and exceed enterprise reporting and logging compliance requirements in a cost-effective and scalable way.

You can run the MSS software in any cloud environment by provisioning a dedicating Linux RHEL/CentOS virtual instance (MSS sensor) to monitoring up to 2 Gbps of traffic originating from other cloud instances.

• The MSS sensor monitors network traffic by (1) installing agents on the remote instances to forward promiscuous traffic through an encrypted tunnel or (2) receiving mirroring session provided by public cloud providers (see https://www.metaflows.com/assets/images/cloud_deployment3.png).
• The sensor replays the forwarded traffic to a dummy interface (dummy0), which it can monitor to generate security event data, packet logging and other advanced network security monitoring functions. Events generated by sensors are posted over TLS to the MetaFlows cloud.
• The agents installed on endpoints communicate with the sensor through a TLS tunnel established on TCP port 3005 and can optionally provide decrypted TLS mirroring of the local applications TLS sessions. The TLS tunnels over port 3005, by default, provide confidentiality but not authentication. The agent can be optionally configured to use preinstalled customer-generated certificates to prevent possible man-in-the-middle attacks.
• The MSS sensor can receive system logs from the remote instances on UDP ports 514,3004 and TCP port 3004. TCP port 3004 is used to send system logs encapsulated in a TLS tunnel accepting either a self-signed certificate or a configurable certificate signed by an external certificate authority.
• The MSS sensor can export any event to external syslog collectors in syslog or CEF format. The export can occur over UDP port 514 or a configurable TCP port through a TLS tunnel. The TLS tunnel can be configured to accept self-signed certificates or specific user-defined certificates.

It is recommended that users employ at least one sensor for each VPC to be monitored so that traffic monitoring may no increase intra-VPC WAN bandwidth charges.

To setup the MSS in cloud-based monitoring applications require some familiarity with Linux command line tools and root privilege on both the sensors and the endpoints where agents are installed.

The MSS does not require any services from public cloud providers other than Linux systems with root privilege and super user access to the instances being monitored.

Setup Process

Although not required, it is recommended that you dedicate a sensor for each virtual network to avoid incurring additional bandwidth charges for WAN traffic. As in the case of physical deployments, you can add multiple sensors to achieve this by repeating these steps for each virtual network you wish to monitor.

Setting up a sensor and configuring the appropriate security groups takes less than 1 hour. Agent installation on each endpoint takes about 10 minutes. Both operations require privileged user access to the instances.

(Note if if you are deploying on Amazon AWS you can use a MetaFlows pre-built AMI and skip steps 2 and 4.)

1. Configure the sensor as a collector
2. Instantiate a RHEL/CentOS 7 instance in the same virtual network or VPC where your cloud-based VMs are running.
3. Configure network access policies
4. Install and start sensor software
5. Add Linux or Windows agents

Configure the Sensor as a Collector

1. Log into your MetaFlows dashboard at nsm.metaflows.com.
2. Add a new sensor if necessary or edit your sensor configuration by clicking on the Sensors button at the top of the page, and then clicking on the sensor name.
3. Modify the sensor's HOME_NET value to reflect the networks or instances that you plan to monitor.
4. On your sensor configuration page, set the promiscuous interface option to: dummy0.
5. Save the configuration.

Configure Network Access Policies

When spawning virtual machines most cloud service providers default to a very restrictive network access configuration designed to limit exposure to Internet threats. The sensor however needs to be able to communicate to external services and to receive network data from your other instances. Therefore, most of the installation issues in the cloud deployments stem from not properly configuring the network port access policies. Please configure the security groups and ACLs according the requirements in the Firewall Setup.

Add Linux or Windows Agents

Linux Agent

Install tcpdump and ncat, on CentOS/RHEL - you can do this with the command:

yum -y install tcpdump nmap-ncat

If the above command does not work try:

wget https://nsm.metaflows.com/updates/ncat-5.51-1.x86_64.rpm -O ncat-5.51-1.x86_64.rpm
yum install ncat-5.51-1.x86_64.rpm  

Run the following to start the agent, add it to /etc/rc.local to run automatically when the system boots. Replace the IP address with the IP address of your sensor that the agent can reach:

while (true);do tcpdump -s0 -i eth0 -U -n -w - 'not port 3005' |ncat -vvn --ssl 10.10.254.13 3005;sleep 5;done

Note: The not port 3005 is very important; it makes sure that the agent does not monitors reflexively causing uncontrolled bandwidth usage and packet loss.

Note: The not port 3005 above can be changed to exclude additional traffic. For example not port 3005 and not port 443 would only send traffic that is not destined to port 443 and port 3005.

Windows Agents

This process installs Windows services nsm_agent and/or nsm_logc to forward, through an encrypted channel, all traffic and/or Windows Event logs respectively to the MSS sensor.

Prepare the sensor

ssh to the sensor and check what services are available by executing the command:

netstat -tpan

Make sure your sensor is listening on tcp port 3005 (for packet data) and/or tcp port 3004 (for log data). If you have a typical, physical sensor deployment you will most likely not have port 3005 enabled and therefore you will not be able to forward packet data but only syslog data (option 2 below). Cloud-based deployments will most likely be listening on both ports. If the sensor is not listening on tcp port 3004, you probably need to enable log management and restart the sensor with the command

/nsm/etc/mss.sh restart

Also make sure your network configuration allows communication to the sensor on these two ports from the Windows hosts you are trying to monitor.

Execute the command:

iptables -A METAFLOWS_CHAIN -p tcp --dport 3004 -j ACCEPT

On CentOS also execute:

iptables-save

On RHEL also execute:

/etc/init.d/iptables save

Install Agent on Windows

On the Windows machine you are trying to monitor:

• Download http://nsm.metaflows.com/updates/nsm_remote_windows.zip and extract it making a note to where the archive was extracted.
• Start a Command Prompt window with administrator privilege. To do this, you can right click on the command prompt icon and select run as administrator.
• Navigate to the directory containing the files extracted in step 1 and execute:

install

This will show a menu as shown below.

• Select option 1 or 2 or 3 (note that option 2 and 3 are not supported on Windows XP or earlier versions).
  1. will configure the agent to only forward packet data to the sensor.
  2. will configure the agent to only forward the Windows log messages and therefore will not provide deep packet inspection.
  3. (recommended) will configure the agent to forward both packet data and Windows logs to the sensor.
• Follow the prompts for the WinPcap installation.
• The installation script will list the network interfaces available on the system; enter the interface number that connects the host to the Internet (in most cases this is 1).
• Enter the IP address of the sensor collecting data for this host (ex. 10.2.3.4).

You now should see the promiscuous traffic and/or Windows Logs reaching the MSS sensor after a few seconds.

To verify the packet data is being forwarded execute the following commands on the MSS sensor:

tcpdump -i dummy0 -n host <Internal Windows IP address> -c 100

To verify that the Windows logs are being forwarded execute the following:

tcpdump -i [eno1|eth0] -n port 3004 and host <External Windows IP address> -c 100

Where: <Internal Windows IP address> is the IP address assigned to the network adapter (shown by doing ipconfig) <External Windows IP address> is the IP address used by the host to reach the sensor (it may or may not br the same as <Internal Windows IP address> due to Network Address Translation.

Both commands should show some packets.

Troubleshoot Agent Installation

To troubleshoot the installation please run the task manager to verify that the agent software is running. Depending on the option selected you should have running services nsm_agent and/or nsm_logc. You should also see the processes windump, ncat, and metalogc (if options 2 or 3 were selected).

If windump or ncat are not running (Options 1 and 3) or ncat or metalogc are not running (Options 2 or 3), try executing c:\nsm_remote\nsm_agent.bat and/or c:\nsm_remote\nsm_logc.bat respectively from the command line and record and try to fix any errors in the script.

It is also a good idea to restart Windows to make sure the agents start correctly after a reboot. If after a reboot you do not see ncat running, please stop and start the service manually. If ncat now appears, please increase the initial delay in c:\nsm_remote\nsm_agent.bat and c:\nsm_remote\nsm_logc.bat by editing these files using wordpad and increasing the number of delay pings from 20 to 40 or 60 to give the system more time to acquire an IP address before firing up the agent processes on startup.

In some cases it might be necessary to reinstall the agents. To reinstall the agents execute:

del c:\nsm_remote 

select A to remove all its contents and then repeat the installation process documented in this section.

Using the MetaFlows MSS AMI

A pre-built AMI image for Amazon EC2 is available in the marketplace and can be downloaded from https://aws.amazon.com/marketplace/pp/B008MAO9SE. This AMI is billed hourly by Amazon and allow to skip the MSS software installation step in the setup process in AWS.

Please make sure to configure the network access policies before accessing and configuring the AMI

Once running the MetaFlows AMI is accessible with using the ssh command using the default username 'metaflows':

ssh -i <your_private_key>.pem metaflows@<your instance address>

Once you have logged in as the MetaFlows user, you can start the sensor by running the command:

sudo /nsm/etc/mss.sh start

If this is the first time you have run the sensor, you will be presented with two options:

1. Register my account with MetaFlows and add my first instance - Use this option if you do not already have an account on nsm.metaflows.com
2. Add this ec2 instance to my existing account - Use this option if you have already registered an account on nsm.metaflows.com

Adding a Larger Disk for Log Storage

If your AMI has a small default disk, or you just want additional log storage capacity, follow these steps to add an EBS volume and re-map the log directories to it.

Adding the EBS Volume

1. From the AWS console add a new EBS volume to the region in which you are running the sensor. If you are not sure how big of a disk you need, 500GB is probably a good minimum.
2. Attach the EBS volume to the MetaFlows sensor instance.
3. Verify the name of the drive on your MetaFlows sensor by checking dmesg .

Preparing the Disk

1. Manually create one partition on the disk using fdisk, or run the command:

echo -e "o\nn\np\n1\n\n\nw" | fdisk /dev/xvdk

2. Add a file system to the disk with the command:

mkfs.ext4 /dev/xvdk1

Mounting the New Disk as Your Log Storage

1. Run the command:

/nsm/etc/mss.sh stop

2. Create a new directory for the logs:

mkdir /var/log/metaflows

3. Mount the new disk:

mount /dev/xvdk1 /var/log/metaflows

4. Move the old logs to the new disk:

mv /mnt/hgfs/logs /var/log/metaflows/; ln -s /var/log/metaflows/logs /mnt/hgfs/logs

5. Start the sensor:

/nsm/etc/mss.sh start

Making the New Disk Persistent on Boot

1. Add a new line to /etc/fstab like the one below to make sure the disk is re-mounted automatically:

/dev/xvdk1              /var/log/metaflows      ext4    defaults        0 0

MetaFlows Security Gateway on Amazon EC2

This deployment model requires considerable network administration expertise and it is not recommended in most cases.

Architecture

Setting up a MetaFlows EC2 instance as a VPC security gateway can monitor and protect your cloud assets. The MetaFlows EC2 security gateway routes IP traffic between the VPC and the internet, therefore it has complete visibility of the full-duplex traffic to and from your protected instances. The MetaFlows Security System (MSS) running on the MetaFlows EC2 instance will then allow you to identify and shut down threats through a standard web browser.

Setup Instructions

1. Launch a VPC

- Create a VPC and give it a network range (ex. 10.0.0.0/8).

2. Create Subnets

- Create a private subnet (ex. 10.1.1.0/24) and a public subnet (ex. 10.1.100.0/24).

3. Set Up the NAT Gateway

- Set up the NAT Gateway. - Launch an EC2 MetaFlows instance on the public subnet. - Configure the MetaFlows Security Software:

- Execute the command

/nsm/etc/mss.sh start. 

- Register the instance with an existing MetaFlows account or create a new account.

- Read and agree the license agreement.

- Specify the name of the gateway instance, and its domain.

- Assign an EIP address to the instance. This will be your externally routable address.

- Make sure to modify the network adapter to DISABLE src/dst checking. - Configure this instance as a normal NAT device for the private subnet:

Echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -s 10.1.1.0/24 -j MASQUERADE

4. Add Additional IP Addresses

Add additional IP addresses on the public subnet (if needed). - The EC2 will automatically assign a public subnet address to your instance once it is launched. Each instance can have additional IP addresses on the public subnet assigned to it in Step 2. - For each of these IP addresses, you can assign an EIP. Limits may apply depending on the type of instance you choose.

5. Set Up the Routing Tables

- The public subnet should have a default route for 0.0.0.0/0 to an Amazon IGW device. - The private subnet should have a default route for 0.0.0.0/0 to the instance id of the MetaFlows EC2 instance.

6. Launch the EC2 Instances

- Launch the EC2 instances to be monitored in the private subnet.

7. Add Port Forwarding Rules

Add fort forwarding rules to iptables for publicly accessible services.

- You can follow these instructions to do that at https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/4/html/Security_Guide/s1-firewall-ipt-fwd.html