Deep Packet Inspection of Cloud-based assets

From The MetaFlows Security System Documentation
Jump to: navigation, search


The advantage of the MSS in cloud security monitoring applications is that it can perform deep packet inspection in virtual cloud environments (like Amazon AWS or Microsoft Azure) without requiring access to the networking layer. Each MSS sensor can monitor up to 2 Gbps of traffic originating from cloud assets, inherits all the features of a physical deployment, and therefore, provides a complete IDS/IPS solution for a virtual cloud environment. This enables organizations to meet and exceed enterprise reporting and logging compliance requirements in a cost-effective and scalable way.

You can run the MSS software in any cloud environment by provisioning a dedicating Linux RHEL/CentOS virtual instance (MSS sensor) to monitoring up to 2 Gbps of traffic originating from other cloud instances.

  • The MSS sensor monitors network traffic by (1) installing agents on the remote instances to forward promiscuous traffic through an encrypted tunnel or (2) receiving mirroring session provided by public cloud providers (see
  • The sensor replays the forwarded traffic to a dummy interface (dummy0), which it can monitor to generate security event data, packet logging and other advanced network security monitoring functions. Events generated by sensors are posted over TLS to the MetaFlows cloud.
  • The agents installed on endpoints communicate with the sensor through a TLS tunnel established on TCP port 3005 and can optionally provide decrypted TLS mirroring of the local applications TLS sessions. The TLS tunnels over port 3005, by default, provide confidentiality but not authentication. The agent can be optionally configured to use preinstalled customer-generated certificates to prevent possible man-in-the-middle attacks.
  • The MSS sensor can receive system logs from the remote instances on UDP ports 514,3004 and TCP port 3004. TCP port 3004 is used to send system logs encapsulated in a TLS tunnel accepting either a self-signed certificate or a configurable certificate signed by an external certificate authority.
  • The MSS sensor can export any event to external syslog collectors in syslog or CEF format. The export can occur over UDP port 514 or a configurable TCP port through a TLS tunnel. The TLS tunnel can be configured to accept self-signed certificates or specific user-defined certificates.

It is recommended that users employ at least one sensor for each VPC to be monitored so that traffic monitoring may no increase intra-VPC WAN bandwidth charges.

To setup the MSS in cloud-based monitoring applications require some familiarity with Linux command line tools and root privilege on both the sensors and the endpoints where agents are installed.

The MSS does not require any services from public cloud providers other than Linux systems with root privilege and super user access to the instances being monitored.

Setup Process

Although not required, it is recommended that you dedicate a sensor for each virtual network to avoid incurring additional bandwidth charges for WAN traffic. As in the case of physical deployments, you can add multiple sensors to achieve this by repeating these steps for each virtual network you wish to monitor.

Setting up a sensor and configuring the appropriate security groups takes less than 1 hour. Agent installation on each endpoint takes about 10 minutes. Both operations require privileged user access to the instances.

(Note if if you are deploying on Amazon AWS you can use a MetaFlows pre-built AMI and skip steps 2 and 4.)

  1. Configure the sensor as a collector
  2. Instantiate a RHEL/CentOS 7 instance in the same virtual network or VPC where your cloud-based VMs are running.
  3. Configure network access policies
  4. Install and start sensor software
  5. Add Linux or Windows agents

Configure the Sensor as a Collector

  1. Log into your MetaFlows dashboard at
  2. Add a new sensor if necessary or edit your sensor configuration by clicking on the Sensors button at the top of the page, and then clicking on the sensor name.
  3. Modify the sensor's HOME_NET value to reflect the networks or instances that you plan to monitor.
  4. On your sensor configuration page, set the promiscuous interface option to: dummy0.
  5. Save the configuration.

Configure Network Access Policies

When spawning virtual machines most cloud service providers default to a very restrictive network access configuration designed to limit exposure to Internet threats. The sensor however needs to be able to communicate to external services and to receive network data from your other instances. Therefore, most of the installation issues in the cloud deployments stem from not properly configuring the network port access policies. Please configure the security groups and ACLs according the requirements in the Firewall Setup.

Add Linux or Windows Agents

Native Linux Agent

Install tcpdump and ncat, on CentOS/RHEL - you can do this with the command:

yum -y install tcpdump nmap-ncat

If the above command does not work try:

wget -O ncat-5.51-1.x86_64.rpm
yum install ncat-5.51-1.x86_64.rpm  

Run the following to start the agent, add it to /etc/rc.local to run automatically when the system boots. Replace the IP address with the IP address of your sensor that the agent can reach:

while (true);do tcpdump -s0 -i eth0 -U -n -w - 'not port 3005' |ncat -vvn --ssl 3005;sleep 5;done

Note: The not port 3005 is very important; it makes sure that the agent does not monitors reflexively causing uncontrolled bandwidth usage and packet loss.

Note: The not port 3005 above can be changed to exclude additional traffic. For example not port 3005 and not port 443 would only send traffic that is not destined to port 443 and port 3005.

Advanced Linux Agent

The advanced Linux agent provides important enhancement to a simple native Linux agent. Most notably it provides:

  • Mutual authentication
  • Passive TLS mirroring
  • Fail over redundancy
  • Traffic replay

Advanced Linux agents are available as pre-compiled binaries for any Intel/AMD 64-bit architecture. TLS mirroring is supported for Linux distributions Centos/RedHat 7|8, Amazon linux 2, and Ubuntu 18|20. Contact to get compatible Openssl libraries for other distributions. We also support nss and boring ssl upon request. Advanced Linux agents can be invoked as a traditional mirror to send packets to a remote packet inspection host (functionality similar to native Linux agent) or can be invoked to also send decrypted TLS traffic.

Permissions and Licenses

Invoking the advanced Linux agent for passive TLS mirroring requires a license tied to the specific endpoint where it is invoked. We grant 4 licenses for testing and personal use for each user who creates an account at on our SaaS system. Additional licenses or commercial licenses can be granted for a yearly fee. Invoking the advanced Linux agent as a traditional mirror (Normal Agent) or to receive mirrored packets (receiver) does not require a license.

The agent can be installed by connecting through ssh or putty to any supported Linux system. The installation and operation of the agent requires root privilege. Typically, on AWS EC2 instances one would obtain root privilege by executing:

sudo su
Detailed Installation

Once logged in and a the root prompt, the agent can be installed using the following steps:

tar xfz sslbinstall.tgz
cd sslbinstall

At this point the system prompts the user for what kind of installation or if to revert to a previous state:

  1. Normal Agent. Installs the agent in order to capture traffic from the host and sending it to the remote sensor over TLS
  2. Agent with TLS mirroring (license required; free for up to 4 servers). This option modifies the open ssl libraries so that the agent can also mirror decrypted TLS traffic to the remote sensor over TLS. Decrypted TLS packets will have the server port +4 (for example TLS 443 traffic will be re-mapped to 447 traffic in the clear). Also, decrypted TLS traffic src/dst MAC addresses will be remapped to 06:11:22:33:44:55 and 06:11:22:33:44:56. This option requires user registration at using a corporate email or a validation of the user identity by sending requests to The license is free for personal uses or testing for up to 4 individual server/agents. Additional or commercial licenses can be requested by sending inquiries to
  3. Remove TLS mirroring from host and restore original Openssl libraries. This option simply restores the original Openssl libraries thus disabling TLS mirroring
  4. Make Agent a persistent service This option assists in creating a service specification file so that the agent can be managed using systemd. Selecting this option will prompt for 3 main usage scenarios:
    1. Endpoint without SSL Interception This option will forward all packets to an IDS without TLS interception. Choosing this option will require entering the IP address of the IDS system receiving the captured packets.
    2. Endpoint with SSL Interception This option will forward all packets to an IDS with TLS interception. This option will require entering the IP address of the IDS system receiving the captured packets
    3. IDS system receiving packets This option will start the application as a receiver and will receive captured packets from the agents started (Option 1 and 2).

Once the selection is made the script will request confirmation and write the systemd specification file /usr/lib/systemd/system/sslcollector.service Note that there are a variety of options to further customize how the application is invoked. Please edit /usr/lib/systemd/system/sslcollector.service to customize the invocation and then restart the service with service sslcollector restart

If systemd is not available, add the following to /etc/rc.local to run automatically when the system boots.

NOWATERMARK=1; sslcollector [-i|-I] <monitored_interface> [-o <replay_interface>|-t <collector_ip>]


<monitored_interface> is the Ethernet interface that you want to monitor
<replay_interface> is the Ethernet interface to reply captured packets
<collector_ip> is the IP address of the system receiving captured packets
Local Test

To quickly locally test normal mirroring (option 1 above) execute: (replace eth0 with the name of your primary interface)

modprobe dummy
ifconfig dummy0 up
tcpdump -i dummy0 -w test.pcap -c 10 host &
/usr/local/bin/sslcollector -i eth0 -o dummy0 > /dev/null &
curl -o /dev/null
sleep 10
tcpdump -r test.pcap -n -nn -X

Ctrl-C and wait 10 seconds to kill the sslcollector

To quickly locally test TLS mirroring (option 2 above) execute: (replace eth0 with the name of your primary interface)

modprobe dummy
ifconfig dummy0 up
tcpdump -i dummy0 -w test.pcap -c 10 port 447 and host &
/usr/local/bin/sslcollector -I eth0 -o dummy0 > /dev/null &
curl -o /dev/null
sleep 10
tcpdump -r test.pcap -n -nn

Ctrl-c and wait for 10 seconds to kill the sslcollector

Advanced Agent Options

The agent has a number of other options that can be used to further customize its behavior:

Option Description
-i <iface> This option tells the agent to mirror all packets seen on interface <iface>
-I <iface> This option tells the agent to forge decrypted TLS packets corresponding to the traffic seen on <iface>. In order for this option to perform the decryption, it is necessary to install custom Openssl libraries choosing option 2 from the installation menu.
-p '<pcap expression>' Filter packets seen on <iface> according to <pcap_expression>
-o <iface> Instead of sending packets to a remote system using TCP/IP, replay the packets to <iface>. <iface> can then be physically connected to host or a switch as if it was a TAP in promiscuous mode.
-m <mtu> The agent automatically chooses the maximum MTU to send out or replay captured packets. This option forces the MTU to a user-defined value.
-t <ip_address> IP address of the remote system receiving the mirrored packets. The remote system must be running with the -c or -C option
-D When using the -I option, only mirror decrypted TLS packets and discard all packets on <iface>
-d <secret> Use <secret> to authenticate agents to remote systems. The remote system running with -c or -C must also specify -d <secret>.
-S To improve performance on high throughput systems, do not encapsulate packets in an SSL tunnel. The remote system running with -c or -C must also specify -S. Mirrored packets will be sent in the clear!
-a Authenticate the remote system through a known self signed SSL certificate. The remote system running with -c or -C must also specify -a.
-H This is the list of network addresses (ex. [,]) used to compute the effectiveness of TLS decryption.
-T <ip_address> Secondary remote system receiving the captured packets. If connection to the primary (-t) fails 3 times in a row, the secondary IP address is tried.
Receiver Agent Options

An agent receiver runs on the MetaFlows sensor or any other IDS or monitoring systems. It receives traffic mirrored by a native or advanced Linux agent and replays it to a local interface in promiscuous mode. The IDS or monitoring system than opens the local interface and processes the traffic as if it was originating from a mirror or a tap. It is possible to replay the traffic to an interface that is also receiving traffic from a traditional switch or a VPC mirror thus merging two mirrored sources onto one interface. To minimize contention, if possible, we recommend replaying traffic to a separate interface and either bonding multiple mirror interface or specifying multiple interfaces at the IDS level.

The agent receiver has a number of options that can be used to customize its behavior:

Option Description
-c This option puts the agent in receiver mode listening for incoming connections on port 3005. Port 3005 is setup to receive packet captured from native or advanced Linux agents.
-C This option puts the agent in receiver mode listening for incoming connections on ports 3005 and 3020. Port 3020 is setup to receive decrypted TLS sessions from Windows based agents.
-o <iface> By default the packets are replayed to the dummy0 interface. -o specifies an alternate replay <iface>. <iface> can be an existing input coming from a traditional mirror or span or can be an output connected to a switch or other host.
-m <mtu> The agent automatically chooses the maximum MTU to send out or replay captured packets. This option forces the MTU to a user-defined value.
-d <secret> Use <secret> to authenticate agents to remote systems. Remote agents must also specify the same -d <secret>.
-S To improve performance on high throughput systems, do not encapsulate packets in an SSL tunnel. Remote agents must also specify -S. Captured packets will be sent in the clear!
-a Authenticate the remote system through a known self signed SSL certificate. Remote agents must also specify -a.
-H This is the list of network addresses (ex. [,]) used to compute the effectiveness of TLS decryption.

Standard MetaFlows sensor run the receiver as:

sslcollector -C

Please contact if you want to modify this.

Patches and Upgrades

Registered users receive notifications when patches or upgrades are available for download. The patching and upgrading follows the same procedure as the installation. Patch notifications are expected to be issued within 36 hours of changes to the Openssl library offered by a particular distribution. Upgrades will be offered on an ongoing basis as needed.

Faults and Troubleshooting

Advanced Linux agents require the environment variable NOWATERMARK set to 1. This is added to the system's environment upon installation. if this variable is not set the agent will not start and display an error. To set this environment variable manually please execute


If the mirroring agent is not communicating with the receiver, it will output connection attempt messages every 30 seconds. Please make sure that:

  • The receiver is up and running and listening on port 3005
  • If the mirroring agent is invoked with the -a option, the receiver has also been invoked with the -a option (verify TLS certificate) and the file /nsm/etc/server-cert.pem is the same on both the mirroring agent and the receiver. Also, make sure that the Common Name in the certificate matches the IP address of the receiver.
  • If the mirroring agent is invoked with the -d option, make sure the receiver uses the same authentication string.
  • The endpoint and the receiver system clocks do not drift more than 10 minutes as the authentication requires a time stamp to avoid signature replication attacks.
  • The endpoint can reach TCP port 3005 on the receiver.

The mirroring agent -T option allows fault tolerance by specifying an alternate receiver. If the main receiver specified with the -t option does not allow connections, the mirroring agent will try the receiver specified with the -T option. After the main receiver comes back up, restart the agent to allow it to reconnect to the main receiver with the command:

service sslcollector restart.

In some very high bandwidth applications, the mirroring agent may disconnect from the receiver. If this happens, please upgrade ncat to version 7.91 with the commands below:

wget --no-check-certificate -O nmap-7.91.tar.bz2 2>>/tmp/updatelog
tar xvfj nmap-7.91.tar.bz2 2>>/tmp/updatelog
cd nmap-7.91
./configure --with-openssl=/usr/local/ssl/ --without-liblua 2>>/tmp/updatelog
make build-ncat 2>>/tmp/updatelog
cp ncat/ncat /usr/bin/ncat

It is very unlikely that the agent will affect the endpoint performance or stability. The only scenario we have been able to reproduce is when a high-performance endpoint saturates the network with several Gbps of sustained traffic over a prolonged period of time. in this case, the mirroring agent competes with the application in mirroring the traffic thus decreasing the endpoint application performance. In order to disable passive TLS mirroring to investigate performance or stability issues execute:

tar xfz sslbinstall.tgz
cd sslbinstall

And choosing option 3. This sill restore the original openssl libraries. Also execute the following command to stop the agent:

service sslcollector stop

the file /usr/lib/systemd/system/sslcollector.service can be removed to prevent the agent from starting in future reboots.

Software Recovery

The software can be recovered by repeating the installation steps outlined above. The TLS mirroring licenses are tied to the specific endpoints on which they are installed and cannot be recovered. Please contact to delete old licenses that have been lost due to a failure or reconfiguration.

Windows Agents

This process installs Windows services nsm_agent and/or nsm_logc to forward, through an encrypted channel, all traffic and/or Windows Event logs respectively to the MSS sensor.

Prepare the sensor

ssh to the sensor and check what services are available by executing the command:

netstat -tpan

Make sure your sensor is listening on tcp port 3005 (for packet data) and/or tcp port 3004 (for log data). If you have a typical, physical sensor deployment you will most likely not have port 3005 enabled and therefore you will not be able to forward packet data but only syslog data (option 2 below). Cloud-based deployments will most likely be listening on both ports. If the sensor is not listening on tcp port 3004, you probably need to enable log management and restart the sensor with the command

/nsm/etc/ restart

Also make sure your network configuration allows communication to the sensor on these two ports from the Windows hosts you are trying to monitor.

Execute the command:

iptables -A METAFLOWS_CHAIN -p tcp --dport 3004 -j ACCEPT

On CentOS also execute:


On RHEL also execute:

/etc/init.d/iptables save

Install Agent on Windows

On the Windows machine you are trying to monitor:

  • Download and extract it making a note to where the archive was extracted.
  • Start a Command Prompt window with administrator privilege. To do this, you can right click on the command prompt icon and select run as administrator.
  • Navigate to the directory containing the files extracted in step 1 and execute:

This will show a menu as shown below. Agentmenu.png

  • Select option 1 or 2 or 3 (note that option 2 and 3 are not supported on Windows XP or earlier versions).
    1. will configure the agent to only forward packet data to the sensor.
    2. will configure the agent to only forward the Windows log messages and therefore will not provide deep packet inspection.
    3. (recommended) will configure the agent to forward both packet data and Windows logs to the sensor.
  • Follow the prompts for the WinPcap installation.
  • The installation script will list the network interfaces available on the system; enter the interface number that connects the host to the Internet (in most cases this is 1).
  • Enter the IP address of the sensor collecting data for this host (ex.

You now should see the promiscuous traffic and/or Windows Logs reaching the MSS sensor after a few seconds.

To verify the packet data is being forwarded execute the following commands on the MSS sensor:

tcpdump -i dummy0 -n host <Internal Windows IP address> -c 100

To verify that the Windows logs are being forwarded execute the following:

tcpdump -i [eno1|eth0] -n port 3004 and host <External Windows IP address> -c 100

Where: <Internal Windows IP address> is the IP address assigned to the network adapter (shown by doing ipconfig) <External Windows IP address> is the IP address used by the host to reach the sensor (it may or may not br the same as <Internal Windows IP address> due to Network Address Translation.

Both commands should show some packets.

Troubleshoot Agent Installation

To troubleshoot the installation please run the task manager to verify that the agent software is running. Depending on the option selected you should have running services nsm_agent and/or nsm_logc. You should also see the processes windump, ncat, and metalogc (if options 2 or 3 were selected).

If windump or ncat are not running (Options 1 and 3) or ncat or metalogc are not running (Options 2 or 3), try executing c:\nsm_remote\nsm_agent.bat and/or c:\nsm_remote\nsm_logc.bat respectively from the command line and record and try to fix any errors in the script.

It is also a good idea to restart Windows to make sure the agents start correctly after a reboot. If after a reboot you do not see ncat running, please stop and start the service manually. If ncat now appears, please increase the initial delay in c:\nsm_remote\nsm_agent.bat and c:\nsm_remote\nsm_logc.bat by editing these files using wordpad and increasing the number of delay pings from 20 to 40 or 60 to give the system more time to acquire an IP address before firing up the agent processes on startup.

In some cases it might be necessary to reinstall the agents. To reinstall the agents execute:

del c:\nsm_remote 

select A to remove all its contents and then repeat the installation process documented in this section.

Amazon Web Services (AWS) Setup

Using the MetaFlows MSS AMI

A pre-built AMI image for Amazon EC2 is available in the marketplace and can be downloaded from This AMI is billed hourly by Amazon and allow to skip the MSS software installation step in the setup process in AWS.

Please make sure to configure the network access policies before accessing and configuring the AMI

Once running the MetaFlows AMI is accessible with using the ssh command using the default username 'metaflows':

ssh -i <your_private_key>.pem metaflows@<your instance address>

Once you have logged in as the MetaFlows user, you can start the sensor by running the command:

sudo /nsm/etc/ start

If this is the first time you have run the sensor, you will be presented with two options:

  1. Register my account with MetaFlows and add my first instance - Use this option if you do not already have an account on
  2. Add this ec2 instance to my existing account - Use this option if you have already registered an account on

Adding a Larger Disk for Log Storage

If your AMI has a small default disk, or you just want additional log storage capacity, follow these steps to add an EBS volume and re-map the log directories to it.

Adding the EBS Volume

  1. From the AWS console add a new EBS volume to the region in which you are running the sensor. If you are not sure how big of a disk you need, 500GB is probably a good minimum.
  2. Attach the EBS volume to the MetaFlows sensor instance.
  3. Verify the name of the drive on your MetaFlows sensor by checking dmesg .

Dmesg disk.png

Preparing the Disk

1. Manually create one partition on the disk using fdisk, or run the command:

echo -e "o\nn\np\n1\n\n\nw" | fdisk /dev/xvdk

2. Add a file system to the disk with the command:

mkfs.ext4 /dev/xvdk1

Mounting the New Disk as Your Log Storage

1. Run the command:

/nsm/etc/ stop

2. Create a new directory for the logs:

mkdir /var/log/metaflows

3. Mount the new disk:

mount /dev/xvdk1 /var/log/metaflows

4. Move the old logs to the new disk:

mv /mnt/hgfs/logs /var/log/metaflows/; ln -s /var/log/metaflows/logs /mnt/hgfs/logs

5. Start the sensor:

/nsm/etc/ start

Making the New Disk Persistent on Boot

  1. Add a new line to /etc/fstab like the one below to make sure the disk is re-mounted automatically:
/dev/xvdk1              /var/log/metaflows      ext4    defaults        0 0

MetaFlows Security Gateway on Amazon EC2

This deployment model requires considerable network administration expertise and it is not recommended in most cases.


Setting up a MetaFlows EC2 instance as a VPC security gateway can monitor and protect your cloud assets. The MetaFlows EC2 security gateway routes IP traffic between the VPC and the internet, therefore it has complete visibility of the full-duplex traffic to and from your protected instances. The MetaFlows Security System (MSS) running on the MetaFlows EC2 instance will then allow you to identify and shut down threats through a standard web browser.

Setup Instructions

1. Launch a VPC

- Create a VPC and give it a network range (ex.

2. Create Subnets

- Create a private subnet (ex. and a public subnet (ex.

3. Set Up the NAT Gateway

- Set up the NAT Gateway. - Launch an EC2 MetaFlows instance on the public subnet. - Configure the MetaFlows Security Software:

- Execute the command
/nsm/etc/ start. 
- Register the instance with an existing MetaFlows account or create a new account.
- Read and agree the license agreement.
- Specify the name of the gateway instance, and its domain.
- Assign an EIP address to the instance. This will be your externally routable address.

- Make sure to modify the network adapter to DISABLE src/dst checking. - Configure this instance as a normal NAT device for the private subnet:

Echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -s -j MASQUERADE

4. Add Additional IP Addresses

Add additional IP addresses on the public subnet (if needed). - The EC2 will automatically assign a public subnet address to your instance once it is launched. Each instance can have additional IP addresses on the public subnet assigned to it in Step 2. - For each of these IP addresses, you can assign an EIP. Limits may apply depending on the type of instance you choose.

5. Set Up the Routing Tables

- The public subnet should have a default route for to an Amazon IGW device. - The private subnet should have a default route for to the instance id of the MetaFlows EC2 instance.

6. Launch the EC2 Instances

- Launch the EC2 instances to be monitored in the private subnet.

7. Add Port Forwarding Rules

Add fort forwarding rules to iptables for publicly accessible services.

- You can follow these instructions to do that at