Command Line Interface

The MSS provides a Command Line Interface (CLI) tool to:

Retrieve historical flow and payload data directly from the sensor and
Retrieve historical event data from the MetaFlows database.

The CLI tool is a Perl script located on the sensor called /nsm/bin/getflows.pl. It provides a number of query options for both retrieving specific information (such as flow attributes or event types). It also allows displaying the data in different formats amenable to further processing with other CLI tools like grep, awk, sort, etc.

Initialization

Before the tool can be used, it is necessary to initialize the script so that proper credentials can be established for authenticating the queries. All queries performed by getflows.pl use SSL and, therefore, once a proper credential is established, such credential is only transmitted during an SSL session thus ensuring confidentiality. The initialization is performed by issuing the command:

/nsm/bin/getflows.pl -u user@mydomain.com -i api*:<key>

The command will prompt the user for the web login password used to access the MSS at nsm.metaflows.com. Once successfully authenticated, the new API key api*:<key> will be registered with the system. Every time a new registration is performed, it is necessary to restart the sensor so that the new key is assimilated. To restart a sensor, execute the command:

/nsm/etc/mss.sh restart

Once the sensor restarts, the user can perform CLI queries using the newly registered key as:

/nsm/bin/getflows.pl -u api*:<key> <myquery>

without having to enter the password. It is required that all API keys start with the letters api to distinguish API keys from normal logins using a password prompt. For example, the command below would register the API key api0:testdummykey:

/nsm/bin/getflows.pl -u user@mydomain.com -i api0:testdummykey

Multiple API keys can be registered by issuing multiple registration commands. Any registration or revocation of API keys can only be performed by executing getflows.pl directly on the sensor itself. To see a list of registered keys, execute the command:

grep api /nsm/etc/ntopkeys

Remote Execution

Once the registration is performed, the getflows.pl script can be copied off the sensor to any other system with perl and curl to perform queries remotely. Once the getflows.pl script is copied off the sensor, it can only be used for querying the particular sensor from which it was coped. It is recommended that the script name be changed to identify which sensor it was copied from so that users with multiple sensors can easily identify the script corresponding to each sensor. For example, the following command would install the CLI tool for sensors S0 and S1 on a remote workstation:

on Sensor S0:

scp /nsm/bin/getflows.pl user@<remote_workstation>:getflows_s0.pl

on Sensor S1:

scp /nsm/bin/getflows.pl user@<remote_workstation>:getflows_s1.pl

Then executing:

getflows_s0.pl -u api*:<key> <my_query>

would retrieve events or flows related to sensor S0, while

getflows_s1.pl -u api*:<key> <my_query>

would retrieve flows or events related to sensor S1.

Re-Initialization

Once initialized, the script is bound to the particular sensor on which it is initialized. The bounding is performed by rewriting certain variables inside the script to sensor-specific values. If the sensor name, packet logging mode (full packet logging or session logging), or mode of operation changes from client mode to server mode or vice versa, it is necessary to re-initialize the script. To re-initialize the script, edit the script and set the following variables to numeric 0s:

$PORT=0;
$IP=0;
$FORWARDER=0;
$DOMAIN=0;
$SID=0;
$SENSOR=0;
$FULLC=-1;

Then, repeat the initialization procedure.

API Key Revocation

To revoke a specific API key, re-initialize it with a different password. For example, if one wanted to revoke the API key api0:testkey on sensor s0, one would ssh to sensor s0 and execute:

/nsm/bin/getflows.pl -u user@mydomain.com -i api0:somedifferentkey

This will essentially overwrite the key to be revoked with a different credential, thus invalidating the key to be revoked.

Using Windows for remote queries

Windows does not provide perl and curl natively. In order to use getflows.pl from a windows machine please use the following links for installing these free packages.

http://strawberryperl.com/ http://www.confusedbycode.com/curl/

You can test perl by executing

perl -V

you can test curl by executing

curl -L https://www.google.com

Historical Flow and Payload Data Queries

The Flow and Payload Data Queries require that Network Analysis and File Carving plugin be enabled as the queries are served by the Ntop Web server on the sensor.

The getflows.pl CLI tool queries each of the packet databases in order to achieve the desired query type. Each of the queries also returns a pcap file containing the packets matched by the query.

Query Syntax

The general command syntax is below:

getflows.pl -u api*:key [ -s <IP Address> ] [ -d <IP Address> ] [ -p <port> ] [ -q <port> ] [ -t <UTC time> ] [ -w  ] [ -f ] [ -X ] [ -c ] [ -N ] [ -D ] [ -B ] [ -R ] [- J ]

-u is the only required option.

Default values are below: -s 0.0.0.0 (any address); -d 0.0.0.0 (any address); -p 0 (any port); -q 0 (any port); -t <current time>; -w 3600 (seconds); -l 100000 packets

-s and -d options: These are 32-bit IPv4 addresses for which it is desired to get flows. Only specifying one address, will return all flows for that address to any other address.
-p and -q options: These are integers between 1-65535 which identify the port numbers. Both TCP and UPD flows are returned for a given port. Only specifying one port will return all flows for that port to any other port.
-t option: This specifies the time in UTC seconds around which the query is to be executed.
-w option: This specifies the +/- time window around the UTC time for which the flows are requested.
-l option: This specifies the maximum number of packets.
-f switch: This specifies that the full-duplex flows should be displayed otherwise simple flows are reported.
-R switch: This resolves IP addresses to host names if possible.
-X switch: This indicates that the first 1024 bytes of payload should be displayed for each flow.
-c switch: This indicates that the query should be optimized for file carving and the SHA1 digests and the http request should be displayed for each file that is reconstructed using full packet payloads. This option also shows if VirusTotal has provided a verdict on the extracted files.
-N switch: This indicates that the query should be optimized for NetFlow-like data where file carving is not necessary. This yields faster query times because the query only looks in the session packet database.
-D switch: This instructs the application to automatically download all carved files to the current working directory. The file names are the SHA1 of the content with an extension indicating the type of content (ex. .exe, .dll, etc.).
-B switch: This restricts queries for carved files that were flagged as malicious by VirusTotal or the MetaFlows sandbox. Executing a query with -D -B will download all malicious samples to the local hard drive.
-J switch: This produces JSON output rather than text output.

Examples

Some example queries and sample return values are below.

getflows.pl -u api1:xxxx -s 139.182.52.108 -w 3600 -N

This query returns flows to and from address 139.182.52.108 for a one hour window. The output format is simplex and some of the flows have 0 payload data because they are generated by simple control packets like SYN packets. None of the reassembled content has been submitted to VirusTotal !ch. The query is made against the session database and therefore only the first part of the flows is returned; the query time low.

#Query parameters:
#Source address=139.182.52.108
#Destination Address=0.0.0.0
#Source port=0
#Destination port=0
#Time: Thu Sep 3 12:54:16 2015 +/- 3600 seconds
#Return a maximum of 100000 packets
#-------------------------------------------------------------------------------------
#Query optimized for raw flow data
#-------------------------------------------------------------------------------------
#ts         srca            srcp  dsta            dstp  Bytes   VT Type                          
1441306362 139.182.52.108  56376 132.245.44.18   00443       0 !Ch text/443                       
1441306428 139.182.52.108  54724 74.125.224.130  00443     470 !Ch text/443                       
1441306428 139.182.52.108  56380 54.176.113.249  00080       0 !Ch text/80                        
1441306428 74.125.224.130  00443 139.182.52.108  54724     470 !Ch text/443                       
1441306449 139.182.52.108  56409 153.2.224.50    00080       0 !Ch text/80                        
1441306449 153.2.224.50    00080 139.182.52.108  56409       0 !Ch text/80                        
1441306493 74.125.224.139  00443 139.182.52.108  56460     328 !Ch text/443                       
1441306493 139.182.52.108  56460 74.125.224.139  00443     222 !Ch text/443                       
1441306666 139.182.52.108  56383 74.125.224.102  00443       0 !Ch text/443                       
1441306666 74.125.224.102  00443 139.182.52.108  56383      63 !Ch text/443                       
1441306775 139.182.52.108  56503 50.31.164.173   00443     328 !Ch text/443                       
1441306845 54.176.113.249  00080 139.182.52.108  56499      29 !Ch jsonp/data                     
1441306845 54.176.113.249  00080 139.182.52.108  56499      36 !Ch jsonp/data                     
1441306869 50.31.164.173   00443 139.182.52.108  56503     165 !Ch text/443                       
1441306979 139.182.52.108  56499 54.176.113.249  00080     508 !Ch text/80                        
1441307098 104.215.146.200 00443 139.182.52.108  56537     384 !Ch text/443                       
1441307098 139.182.52.108  56537 104.215.146.200 00443     206 !Ch text/443                       
1441307285 54.193.159.143  00080 139.182.52.108  56549      30 !Ch jsonp/data                     
1441307376 139.182.52.108  56549 54.193.159.143  00080     266 !Ch text/80 
...            
#Returned 97 rows
#Return code=1
#pcap=/mnt/hgfs/logs/carver2//assets/0_139.182.52.108_0_0_0_1441310056.pcap (58749 Bytes)
#Fetching pcap file 0_139.182.52.108_0_0_0_1441310056.pcap

getflows.pl -u api1:xxxx -s 139.182.52.108 -w 3600 -N -f

This query shows the same data as the previous query but in a full-duplex view with bytes transferred in either direction.

#Query parameters:
#Source address=139.182.52.108
#Destination Address=0.0.0.0
#Source port=0
#Destination port=0
#Time: Thu Sep  3 12:50:14 2015 +/- 3600 seconds
#Return a maximum of 100000 packets
#-------------------------------------------------------------------------------------
#Query optimized for raw flow data
#-------------------------------------------------------------------------------------
#ts                    srca            srcp  dsta            dstp  Bytes         Type        
1441306845-1441306979 54.176.113.249  00080 139.182.52.108  56499 00508<->00065  jsonp/data text/80 
1441307642-1441307642 173.194.120.151 00443 139.182.52.108  61335 00898<->00705  text/443    
1441311088-1441311088 216.58.216.13   00443 139.182.52.108  56963 00216<->00391  text/443    
1441306292-1441306362 132.245.44.18   00443 139.182.52.108  56376 00206<->00324  text/443    
1441308072-1441308175 132.245.73.194  00443 139.182.52.108  56650 00206<->00324  text/443    
1441306326-1441306428 54.176.113.249  00080 139.182.52.108  56380 00246<->00058  jsonp/data text/80
...
#Returned 48 rows
#Return code=1
#pcap=/mnt/hgfs/logs/carver2//assets/0_139.182.52.108_0_0_0_1441309814.pcap (59046 Bytes)
#Fetching pcap file 0_139.182.52.108_0_0_0_1441309814.pcap

getflows.pl -u api1:xxxx -s 139.182.52.108 -p 80 -w 3600 -c -f -X

This query is performed against the full packet payload database (the option -N is not present); and the output includes the payload preview (only one flow is shown for brevity).

#Query parameters:
#Source address=139.182.52.108
#Destination Address=0.0.0.0
#Source port=80
#Destination port=0
#Time: Thu Sep  3 13:07:16 2015 +/- 3600 seconds
#Return a maximum of 100000 packets
#-------------------------------------------------------------------------------------
#Query optimized for carved content. Use -X to see a preview or -D to download the carved files.
#-------------------------------------------------------------------------------------
1441311004-1441311329 23.35.195.89    00080 139.182.52.108  56991 02873<->46736  text/80     
#--------content----------
HTTP/1.1 301 Moved Permanently
Server: nginx/1.4.6 (Ubuntu)
Content-Type: text/html
Location: http://www.iheart.com/live/kost-1035-los-angeles-193/<a%20href=/
Content-Length: 193
Date: Thu, 03 Sep 2015 20:10:04 GMT
Connection: keep-alive

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
HTTP/1.1 404 Not Found
Server: nginx/1.4.6 (Ubuntu)
Content-Type: text/html; charset=utf-8
X-Powered-By: Express
Content-Length: 46093
Date: Thu, 03 Sep 2015 20:10:04 GMT
Connection: keep-alive

<!DOCTYPE html>
<!--
                |))    |))
  .             |  )) /   ))            .oooooooo.    .ooooooooo.
  \\   ^ ^      |    /      ))        -sddddddddddy:::hdddddmmmmmy-
   \\(((  )))   |   /        ))      -ddddddhhhhhh     ddddddmmmmmd-
    / G    )))  |  /        ))       sddddhhhhhhhhhhhddddddddmmmmmms
   |o  _)   ))) | /       )))        odddhhhhhhhhhhhhdddddddddmmmmmo
    - ' |     ))`/      )))          -hdddhhhhhhhh     dddddddmmmmh.
     ___|              )))            -ydddhhhhhhh     ddddddmmmmy-
    / __\             ))))`()))        `+hdddddddd     ddddddmmh/`
   /\@   /             `(())))           ./ydddddd     ddddmdy:`
   \/   /  /`_______/\   \  ))))           `:sdddd     ddmdo-`
        | |          \ \  |  )))              .+hm     mh+.
        | |           | | |   )))               `+     /`
       /_/           /_/

  Come work with us in TriBeCa, NYC!
  We're hiring JS ninjas!
  Drop a line and resume to d2Viam9ic0BpaGVhcnRyYWRpby5jb20=
-->
<!--[if IE 8]>     <html class="ie ie8" xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://ogp.me/ns/fb#"> <![endif]-->
<!--[if IE 9]>     <html class="ie ie9" xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://ogp.me/ns/fb#"> <![endif]-->
<!--[if !IE]><!--> <html xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://ogp.me/ns/fb#"> <!--<![endif]-->
<head>
#------------------------- 
#--------content----------
GET /live/kost-1035-los-angeles-193/%3Ca%20href= HTTP/1.1
Host: www.iheart.com
Connection: keep-alive
Accept: image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
Referer: http://www.iheart.com/live/kost-1035-los-angeles-193/?autoplay=true&pname=3903&campid=play_bar&cid=main.html
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: adswizz_oaid=1433172695048_1730; __gads=ID=2e5c2bb7fde40ca7:T=1420479095:S=ALNI_MavZ5WXOvpOJdnbKNkc6pX_qNe6MQ; __qca=P0-1567117972-1426018580442; optimizelyEndUserId=oeu1434123445419r0.3702555079944432; SC_LINK=%5B%5BB%5D%5D; auuid=1440625665957; aid=gZEEBG8PuuZdx6Sryt1rfQ==; pid=133760285; country=US; s_cc=true; __utma=210027213.388875422.1420479095.1441203698.1441290120.81; __utmc=210027213; __utmz=210027213.1441290120.81.81.utmcsr=kost1035.com|utmccn=(referral)|utmcmd=referral|utmcct=/main.html; optimizelySegments=%7B%222788091586%22%3A%22false%22%2C%222800131998%22%3A%22referral%22%2C%222801751427%22%3A%22gc%22%2C%222810320950%22%3A%22none%22%7D; optimizelyPPID=133760285; optimizelyBuckets=%7B%7D; jwplayer.volume=60; s_sq=%5B%5BB%5D%5D; s_fid=77D9EE7C19140EAB-1BA937D6B4632F03; GED_PLAYLIST_ACTIVITY=[{"u":"8E/A","t":1441311001,"ed":{"i":{"w":{"tt":20864,"pd":20864,"bs":10}},"a":[{"kv":{}},{"kv":{}},{"kv":{}},{"kv":{}},{"kv":{}},{"kv":{}}]},"nv":0,"pl":20864}]

GET /live/kost-1035-los-angeles-193/%3Ca%20href=/ HTTP/1.1
Host: www.iheart.com
Connection: keep-alive
Accept: image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
Referer: http://www.iheart.com/live/kost-1035-los-angeles-193/?autoplay=true&pname=3903&campid=play_bar&cid=main.html
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: adswizz_oaid=1433172695048_1730; __gads=ID=2e5c2bb7fde40ca7:T=1420479095:S=ALNI_MavZ5WXOvpOJdnbKNkc6pX_qNe6
#-------------------------
...
#Returned 73 rows
#Return code=1
#pcap=/mnt/hgfs/logs/carver2//assets/0_139.182.52.108_80_0_0_1441310836.pcap (385070 Bytes)
#Fetching pcap file 0_139.182.52.108_80_0_0_1441310836.pcap
#Query took 218 seconds

getflows.pl -u api1:xxxx -s 139.182.52.108 -p 80 -w 3600 -c -f

This query shows the carver output format -c, which includes the requested URL. When combined with the -D option, this query would have created a number of carved files on the local disk.

 
#Query parameters:
#Source address=139.182.52.108
#Destination Address=0.0.0.0
#Source port=80
#Destination port=0
#Time: Thu Sep  3 14:27:16 2015 +/- 3600 seconds
#Return a maximum of 100000 packets
#-------------------------------------------------------------------------------------
#Query optimized for carved content. Use -X to see a preview or -D to download the carved files.
#-------------------------------------------------------------------------------------
1441315748-1441316073 173.223.3.173   00080 139.182.52.108  59005 00828<->23987  image/jpeg image/png text/80 
1441316117-1441316117 50.22.46.104    00080 139.182.52.108  59313 01235<->02793  application/javascript text/80 
1441315748-1441316073 173.223.3.173   00080 139.182.52.108  59014 01251<->41117  image/png image/gif text/80 
1441315685-1441315685 69.2.99.57      00080 139.182.52.108  58610 00419<->00507  text/80     
1441316977-1441316977 138.108.6.206   00080 139.182.52.108  59810 00945<->00000  text/80     
1441316357-1441316357 74.125.224.102  00080 139.182.52.108  59237 02621<->00086  image/gif text/80 
1441315762-1441315762 54.241.244.90   00080 139.182.52.108  59059 02497<->12480  text/javascript text/80 
1441315559-1441315559 69.2.99.57      00080 139.182.52.108  58448 00766<->00043  image/gif text/80 
1441316238-1441316238 67.195.33.91    00080 139.182.52.108  59250 03388<->00590  application/x-javascript image/gif text/80 
1441315589-1441315913 173.223.3.173   00080 139.182.52.108  58511 00415<->05143  image/jpeg text/80 
1441315544-1441315544 69.89.76.20     00080 139.182.52.108  58400 00000<->00000  text/80     
1441315748-1441315748 50.97.130.117   00080 139.182.52.108  59044 01238<->02788  text/80 application/javascript 
1441315379-1441315379 198.189.255.201 00080 139.182.52.108  57820 00930<->01481  text/xml text/80 
1441315729-1441316053 204.1.137.146   00080 139.182.52.108  58963 00427<->27539  image/jpeg text/80 
1441315779-1441315779 216.39.55.12    00080 139.182.52.108  59110 03123<->01094  text/80     
1441315542-1441315542 69.89.76.20     00080 139.182.52.108  58397 00847<->00638  text/80     
1441315727-1441315727 54.193.88.38    00080 139.182.52.108  58882 00425<->00043  image/gif text/80 
1441316117-1441316444 208.89.14.135   00080 139.182.52.108  57990 66778<->204389  application/x-javascript application/javascript text/80 
...
#Returned 395 rows
#Return code=1
#pcap=/mnt/hgfs/logs/carver2//assets/0_139.182.52.108_80_0_0_1441315636.pcap (14073616 Bytes)
#Fetching pcap file 0_139.182.52.108_80_0_0_1441315636.pcap
#Query took 224 seconds

getflows.pl -u api1:xxxx -s 8.8.8.8 -p 53 -w 3600 -X

This query show the display of content for a set of DNS queries made to dns server 8.8.8.8 in the last hour.

#Query parameters:
#Source address=8.8.8.8
#Destination Address=0.0.0.0
#Source port=53
#Destination port=0
#Time: Thu Sep  3 14:40:45 2015 +/- 3600 seconds
#Return a maximum of 100000 packets
#-------------------------------------------------------------------------------------
#Query optimized for both carved content and raw flow data
#-------------------------------------------------------------------------------------
1441315617 8.8.8.8         00053 139.182.19.3    60926     126 !Ch text/53                        --...G............	ui-dls360.autodesk.com................F.ns-1033	awsdns-01.org..awsdns-hostmaster.amazon......... ......u...Q.-- 
1441315617 139.182.19.3    60926 8.8.8.8         00053      44 !Ch text/53                        --.0..............	ui-dls360.autodesk.com.....-- 
1441315628 8.8.8.8         00053 139.182.89.18   52541     106 !Ch text/53                        --.n.i.............e673.e9.akamaiedge.net..............L.2.n0e9...hostmaster.akamai.com.U...................-- 
1441315628 139.182.89.18   52541 8.8.8.8         00053      44 !Ch text/53                        --.0...............e673.e9.akamaiedge.net.....-- 
1441315636 8.8.8.8         00053 139.182.235.53  56932     116 !Ch text/53                        --.xJ..............www.google.com..............1..J}...........1..J}...........1..J}...........1..J}...........1..J}..-- 
1441315654 139.182.18.164  46362 8.8.8.8         00053      36 !Ch text/53                        --.(... ...........www.google.com.....-- 
1441315654 8.8.8.8         00053 139.182.18.164  46362      52 !Ch text/53                        --.8.!. ...........www.google.com..................:..-- 
1441315665 8.8.8.8         00053 139.182.241.246 52584      96 !Ch text/53                        --.dt..........................p.@.a.root-servers.net..nstld.verisign-grs.com.x..}.........	:...Q.-- 
gridmaster-ib...hostmaster..w..0..............Q.-- 745      92 !Ch text/53                        --.`_..............apple.com..............l.1
1441315682 8.8.8.8         00053 139.182.67.210  61669     101 !Ch text/53                        --.i@.}l...........api.smoot.apple.com................ .api.smoot-apple.com.akadns.net..1.......+....).-- 
1441315682 8.8.8.8         00053 139.182.67.210  50400     282 !Ch text/53                        --....q............helposx.apple.com.............&....helposx.apple.com.edgekey.net../...........e3167.e9.akamaiedge.I.`........./.n0e9.c.hostmaster.akamai..U....................o...............e3167.e9.akamaiedge.net................2.n0e9...hostmaster.akamai.com.U...................-- 
1441315687 139.182.185.241 01243 8.8.8.8         00053      45 !Ch text/53                        --.1...............alt2.aspmx.l.google.com.....-- 
1441315697 139.182.18.164  40971 8.8.8.8         00053      36 !Ch text/53                        --.(..=............www.google.com.....-- 
1441315697 8.8.8.8         00053 139.182.18.164  40971      52 !Ch text/53                        --.8..=............www.google.com..................:..-- 
1441315705 139.182.159.20  58225 8.8.8.8         00053      28 !Ch text/53                        --. [.0T...........server.....-- 
1441315754 139.182.185.241 05944 8.8.8.8         00053      38 !Ch text/53                        --.*?5u............auth40.ns.uu.net.....-- 
1441315754 8.8.8.8         00053 139.182.185.241 05944      54 !Ch text/53                        --.:/.u............auth40.ns.uu.net.....................-- 
...
#Returned 1000 rows
#Return code=1
#pcap=/mnt/hgfs/logs/carver2//assets/0_8.8.8.8_53_0_0_1441316445.pcap (289633 Bytes)
#Fetching pcap file 0_8.8.8.8_53_0_0_1441316445.pcap
#Query took 34 seconds

getflows.pl -u api1:xxxx -B

This query returns the content that was found to be infected by either VirusTotal or the MetaFlows sandbox in the last hour. Combining this with the -D switch would download all these samples to the local system for further analysis.

#Query parameters:
#Source address=0.0.0.0
#Destination Address=0.0.0.0
#Source port=0
#Destination port=0
#Time: Thu Sep  3 14:46:12 2015 +/- 3600 seconds
#Return a maximum of 100000 packets
#-------------------------------------------------------------------------------------
#Query optimized for bad content only. Use -X to see a preview or -D to download the carved files.
#-------------------------------------------------------------------------------------

#ts         srca           srcp  dsta            dstp  Bytes   VT Type                           Sha1                                               Request                                                                
1441207878 93.40.5.132    51690 139.182.2.20    00025   56320 bad application/msword             a658791b213b7d1fb81e05d06e9eb28beea7f32f vpc.altitude-arena.com/check_392013.doc "Janet Anderson" <sales@integratedmodular.net>→mvaziri@csusb.edu:stop payment 
1441235164 204.102.114.41 00080 139.182.97.219  62339  128240 bad application/octet-stream       c75ff733b9e370f62cca252a8961e721bc52008d //mcd                                              
1441237526 52.16.99.27    00080 139.182.16.95   49375  161280 bad application/octet-stream       4cec192586e8f2a5520715ce67fdac4d5f1abbdb //dl/fa8fb870a266431aa75ce663bb584f47/wseliwnu/enneilbo 
1441238467 103.21.180.38  51781 139.182.2.20    00025  438154 bad application/zip                8412b184a4b468064a9f0eac79af7be29d766b96 clients1.google.com/Invoice09875.zip info@juah.com.my→:PAYMENT SLIP 
1441245659 54.231.19.176  00080 139.182.230.100 65006 3390776 bad application/octet-stream       bb862d45ef08a85d754ff449225dc4b7f980bd80 downloads.earthnetworks.com.s3.amazonaws.com//DesktopApp10/Installer/Latest/WeatherBugSetup.exe 
1441248430 52.19.28.2     00080 139.182.117.234 64713  158208 bad application/octet-stream       7d96a4720c762c79bad89239dbb2eb6d53e7cc13 dl.advapplications.com//dl/d5584ec812d5442ca8a7523c27f5cbf5/ehuansop/vewneilg 
1441258225 204.102.114.42 00080 139.182.230.231 49179  128200 bad application/octet-stream       bcb0bab13e3ba4221a1ad66168da40d311033c65 //mcd                                              
1441267539 162.209.3.41   58394 139.182.2.24    00025  976392 bad application/x-zip-compressed   11411e3a701ab3bb44606203cbe0802f8b91b879 jmn.jangonetwork.com/Documents.zip "DHL Express"<maria@customerservice.com>→:DHL AWB# 3382954164 /shipment 
1441269458 204.102.114.41 00080 139.182.97.219  64014  128240 bad application/octet-stream       2dd5be7a942923326a6526258b43e3e931e87f02 //mcd                                              
1441280620 86.98.11.210   30968 139.182.2.20    00025   15783 bad application/zip                374518ec294a46865871f7394e4b33fc076310ae video.bnmla.com/ "Accounts" <message-service@post.xero.com>→ramosl1@csusb.edu:Invoice INV-28948 from St. Paul Cos. for csusb.edu (5522) 
1441292927 1.167.102.133  17592 139.182.2.20    00025   56832 bad application/msword             4392df9c942cd175fe4785a4d3f105924bf5e2d1 search.officemax.com/you_are_cordially_invited.doc "Melissa Anderson" <support@pluginpos.com>→fernando@csusb.edu:Re: party invite 
1441293329 179.24.241.114 01780 139.182.2.24    00025   56832 bad application/msword             4392df9c942cd175fe4785a4d3f105924bf5e2d1 bid.g.doubleclick.net/you_are_cordially_invited.doc "Melissa Anderson" <support@pluginpos.com>→dmaynard@csusb.edu:Re: party invite 
1441294406 81.202.99.189  02232 139.182.2.20    00025   56832 bad application/msword             4392df9c942cd175fe4785a4d3f105924bf5e2d1 www.pandora.com/you_are_cordially_invited.doc "Melissa Anderson" <support@pluginpos.com>→jalmonte@csusb.edu:Re: party invite 
1441294816 93.49.177.34   04835 139.182.2.20    00025   56832 bad application/msword             4392df9c942cd175fe4785a4d3f105924bf5e2d1 images.nike.com/you_are_cordially_invited.doc "Melissa Anderson" <support@pluginpos.com>→sroberts@csusb.edu:Re: party invite 
1441295653 219.85.227.79  02620 139.182.2.20    00025   56832 bad application/msword             4392df9c942cd175fe4785a4d3f105924bf5e2d1 dw.cbsi.com/you_are_cordially_invited.doc "Melissa Anderson" <support@pluginpos.com>→demarco@csusb.edu:Re: party invite 
1441300109 204.102.114.49 00080 139.182.97.219  61999  128240 bad application/octet-stream       cc52e60c92c4048e0c5c0fc9822c6fb041e12cdb //mcd                                              
1441303153 23.235.47.249  00080 139.182.235.85  17497  800200 bad application/octet-stream       e7af846bae7624c76db091dab09e4cff5ebff6b9 //download/56/50001/DNSUnlocker/setup.exe          
1441314621 107.167.87.194 00080 139.182.117.95  41512  146360 bad application/vnd                f74ab350dc069d35b1ad065c0a7033db4785cd9f rockandsexz.com//dl/5d0db2bb7e4194e8               
#Returned 18 rows
#Return code=1
#Query took 16 seconds
You have mail in /var/spool/mail/root

Historical Event Queries

The getflows.pl CLI tool can also query the event database if the -E option is specified.

Syntax

The general command syntax is below:

getflows.pl -u api*:password -E [ -s <IP Address> ] [ -d <IP Address> ] [ -p <port> ] [ -q <port> ] [ -t <UTC time> ] [ -w  ] [ W any|snort|log|service|cor ] [ -H ] [ -I <snort id> ] [ -Q <search string> ] -J -R

-u and -E are the only required options.

Default values are below: -s 0.0.0.0 (any address); -d 0.0.0.0 (any address); -p 0 (any port); -q 0 (any port); -t <current time>; -w 3600 (seconds); -l 100000 events; -W any

-s and -d options: These are 32-bit IPv4 addresses for which it is desired to get events. Only specifying one address, will return all events for that address to any other address.
-p and -q options: These are integers between 1-65535 which identify the port numbers. Both TCP and UPD events are returned for a given port. Only specifying one port will return all events for that port to any other port.
-t option: This specifies the time in UTC seconds around which the query is to be executed.
-w option: This specifies the +/- time window around the UTC time for which the events are requested.
-l option: specifies the maximum number of events to return.
-W option: This specifies if to return (1) ids (Snort), (2) system logs/MetaFlows reports, (3) passive discovery host information, or (4) IDS events with any other event type that is time correlated with it.
-H switch: Tis returns high priority events only.
-I option: This returns only events matching this specific event IDS ID.
-Q option: This returns only events which match the given query string; the string will also match partial IP addresses such as 1.1.1.
-J switch: This produces JSON output rather than text output.
-R switch: This resolves IP addresses to host names if possible.

Examples

Some example queries are listed below.

getflows.pl -E -u api1:xxx -s 139.182.52.108 -w 360000

This query returns all events associated with the IP address 139.182.52.108 in the last 100 hours. Notice that it returned several types of events.

#Query parameters:
#Source address=139.182.52.108
#Destination Address=0.0.0.0
#Source port=0
#Destination port=0
#Time: Thu Sep  3 15:12:23 2015 +/- 360000 seconds
#time      type  cc:srca            srcp  cc:dsta           dstp  cnt gpr/pr  class            msg
1441289933 log   US:74.125.224.131  443   US:139.182.52.108 49206 1   -25/0   system_daemons   Rank/Dorms:US:74.125.224.131:443 ← US:139.182.52.108:49206 and Wireless rank/74_125_224_131: [1:2019416:] ET POLICY SSLv3 outbound connection from client vulnerable to POODLE attack
1441223167 log   **:50.18.78.78     80    US:139.182.52.108 61541 1   -25/0   system_daemons   Rank/Dorms:**:50.18.78.78:80 ← US:139.182.52.108:61541 and Wireless rank/50_18_78_78: 3:5 File-inbound/application-x-msdownload /V3/01/1.OhefgvatCvcr-2snqFreire-2rof.of.freivat-flf.pbz.m/  sha=c770df46de538d0432f355c3bbf5212e55cf6939
1441223013 log   **:50.18.78.78     80    US:139.182.52.108 61541 1   -25/0   system_daemons   File-inbound/application-x-msdownload:**:50.18.78.78:80 ← US:139.182.52.108:61541 /V3/01/1.OhefgvatCvcr-2snqFreire-2rof.of.freivat-flf.pbz.m/  sha=c770df46de538d0432f355c3bbf5212e55cf6939
1441223013 log   **:50.18.78.78     80    US:139.182.52.108 61541 1   -25/0   system_daemons   File-inbound/application-x-msdownload:**:50.18.78.78:80 ← US:139.182.52.108:61541 /V3/01/1.nqfreire-2s1cvkry-2rtvs.pber.vafvtugrkcerffnv.pbz.m/  sha=4c3d808e02078dec113aa6b6865816aeca96c1ba
1441223013 log   **:50.18.78.78     80    US:139.182.52.108 61541 1   -25/0   system_daemons   File-inbound/application-x-msdownload:**:50.18.78.78:80 ← US:139.182.52.108:61541 /V3/01/1.jjj-2sqryvirel-2snse-2rcuc.qryvirelratvar.pyrnepunaary.nqfjvmm.pbz.m/  sha=32daf391aa85fc9fd9abf36539f38a6f505d0819
1441214570 log   **:46.30.46.117    80    US:139.182.52.108 58864 1   -25/0   system_daemons   Rank/Dorms:**:46.30.46.117:80 ← US:139.182.52.108:58864 and Wireless rank/46_30_46_117: [1:2003492:] ET MALWARE Suspicious Mozilla UserAgent  Likely Fake (Mozilla/4.0)
1441214570 log   **:46.30.46.117    80    US:139.182.52.108 58864 1   75 /0   system_daemons   Tracker/trojan_activity1/139_182_52_108:**:46.30.46.117:80 ← US:139.182.52.108:58864 https://nsm.metaflows.com/reports/track/aaf7c3e928f671b5690e354f9be5ccd/trackreport.1441212478.38.pdf
1441212254 log   **:104.68.119.60   80    US:139.182.52.108 51879 1   -25/0   system_daemons   Rank/Dorms:**:104.68.119.60:80 ← US:139.182.52.108:51879 and Wireless rank/104_68_119_60: [1:2018959:] ET POLICY PE EXE or DLL Windows file download HTTP
1441211079 log   US:198.189.255.137 80    US:139.182.52.108 54694 1   -25/0   system_daemons   Rank/Dorms:US:198.189.255.137:80 ← US:139.182.52.108:54694 and Wireless rank/198_189_255_137: 3:5 File-inbound/application-vnd /support/templates/en-us/tp02835264.cab  sha=ac865ed55b914416b18bfa665f377cdbd9ed45a7
1441211003 log   US:198.189.255.137 80    US:139.182.52.108 54699 1   -26/0   system_daemons   File-inbound/application-vnd:US:198.189.255.137:80 ← US:139.182.52.108:54699 /support/templates/en-us/tp03457444.cab  sha=44ea82b7696008357f5c58afa14ed2b35ffa79bb
1441208412 log   **:98.138.79.73    80    US:139.182.52.108 51322 1   -25/0   system_daemons   Rank/Dorms:**:98.138.79.73:80 ← US:139.182.52.108:51322 and Wireless rank/98_138_79_73: [1:2006380:] ET POLICY Outgoing Basic Auth Base64 HTTP Password detected unencrypted
1441147917 log   **:199.27.79.65    80    US:139.182.52.108 51550 1   -25/0   system_daemons   Rank/Dorms:**:199.27.79.65:80 ← US:139.182.52.108:51550 and Wireless rank/199_27_79_65: 3:5 File-inbound/application-x-msdos-program iscale.iheart.com/catalog/track/30868387  sha=69bb06b34f3c24af5c80ec4fc0d17ccdb255f922
1441147805 log   **:199.27.79.65    80    US:139.182.52.108 51550 1   -25/0   system_daemons   File-inbound/application-x-msdos-program:**:199.27.79.65:80 ← US:139.182.52.108:51550 iscale.iheart.com/catalog/track/30868387  sha=69bb06b34f3c24af5c80ec4fc0d17ccdb255f922
1441289933 ids   US:74.125.224.131  443   US:139.182.52.108 49206 1   -25/0   policy-violation snort-policy-violation/policy:1.2019416:ET POLICY SSLv3 outbound connection from client vulnerable to POODLE attack  http://blog.fox-it.com/2014/10/15/poodle/
1441214570 ids   **:46.30.46.117    80    US:139.182.52.108 58864 1   375/375 trojan-activity  snort-trojan-activity/current_events:1.2021725:ET CURRENT_EVENTS Cryptowall docs campaign Aug 2015 encrypted binary 1  
1441214570 ids   **:46.30.46.117    80    US:139.182.52.108 58864 1   -25/0   trojan-activity  snort-trojan-activity/emerging-malware:1.2003492:ET MALWARE Suspicious Mozilla User-Agent - Likely Fake Mozilla/4.0  http://doc.emergingthreats.net/2003492
1441212254 ids   **:104.68.119.60   80    US:139.182.52.108 51879 1   -25/0   policy-violation snort-policy-violation/policy:1.2018959:ET POLICY PE EXE or DLL Windows file download HTTP  http://doc.emergingthreats.net/bin/view/Main/2000419
1441208624 ids   **:98.138.79.73    80    US:139.182.52.108 51528 1   -25/0   policy-violation snort-policy-violation/policy:1.2006380:ET POLICY Outgoing Basic Auth Base64 HTTP Password detected unencrypted  http://doc.emergingthreats.net/bin/view/Main/2006380
1441208412 ids   **:98.138.79.73    80    US:139.182.52.108 51322 1   -25/0   policy-violation snort-policy-violation/policy:1.2006380:ET POLICY Outgoing Basic Auth Base64 HTTP Password detected unencrypted  http://doc.emergingthreats.net/bin/view/Main/2006380
#Query took 2 seconds

getflows.pl -E -u api1:xxx -w 3600 -W service -Q 139.182. | grep -v service\/http\/80 | grep -v service\/ssl\/443 | grep -v skypetoskype

This query finds all network services detected in the last hour on subnet 139.182.0.0/16. The query is then filtered using grep to remove uninteresting services. ’’’Note’’’ that the client port is always 0 for service discovery events.

#Query parameters:
#Source address=0.0.0.0
#Destination Address=0.0.0.0
#Source port=0
#Destination port=0
#Time: Thu Sep  3 15:22:30 2015 +/- 3600 seconds
#time      type  cc:srca            srcp  cc:dsta            dstp  cnt    gpr/pr  class                      msg
1441320286 service US:139.182.55.128  50479 **:173.252.102.97  0     1      -25/0   service/tsp/50479          service/tsp/50479:US:139.182.55.128:50479←**:173.252.102.97 
1441320273 service US:139.182.127.185 161   US:139.182.200.13  0     14560  0  /0   service/nbns/161           service/nbns/161:US:139.182.127.185:161←US:139.182.200.13 
1441320263 service US:139.182.151.150 123   US:171.66.97.126   0     11     0  /0   service/qq/123             service/qq/123:US:139.182.151.150:123←US:171.66.97.126 
1441320246 service US:139.182.118.215 7767  US:198.188.130.157 0     8457   0  /0   service/ssl/7767           service/ssl/7767:US:139.182.118.215:7767←US:198.188.130.157 
1441320243 service US:139.182.35.228  55745 DE:217.91.218.77   0     1      -25/0   service/bittorrent/55745   service/bittorrent/55745:US:139.182.35.228:55745←DE:217.91.218.77 
1441320241 service US:139.182.127.184 161   US:139.182.200.13  0     14930  0  /0   service/nbns/161           service/nbns/161:US:139.182.127.184:161←US:139.182.200.13 
1441320225 service US:139.182.112.107 123   US:17.253.12.253   0     56     0  /0   service/qq/123             service/qq/123:US:139.182.112.107:123←US:17.253.12.253 
1441320204 service US:139.182.127.181 161   US:139.182.200.13  0     14924  0  /0   service/nbns/161           service/nbns/161:US:139.182.127.181:161←US:139.182.200.13 
1441320203 service US:139.182.2.24    25    US:63.99.4.4       0     1      0  /0   service/smtp/25            service/smtp/25:US:139.182.2.24:25←US:63.99.4.4 
1441322554 service US:139.182.118.215 7768  US:198.188.130.155 0     7612   0  /0   service/http/7768          service/http/7768:US:139.182.118.215:7768←US:198.188.130.155 Server:Jetty(8.1.16.v20140903)
1441322554 service US:139.182.2.20    0     US:209.234.210.39  0     1      0  /0   service/http/7768          user/unix:US:139.182.2.20←US:209.234.210.39 GroundbreakingNervePainSecret@craftysmarts.website
...
#Query took 1 seconds

getflows.pl -E -u api1:xxx -w 3600 -W snort -Q trojan -p 80

This query finds all IDS events on port 80 involving the keyword “trojan”.

#Query parameters:
#Source address=0.0.0.0
#Destination Address=0.0.0.0
#Source port=80
#Destination port=0
#Time: Thu Sep  3 15:26:31 2015 +/- 3600 seconds
w=snort&I=%&gid=any&ranked=0&F=0&aquery=trojan'
#time      type  cc:srca            srcp  cc:dsta            dstp  cnt    gpr/pr  class           msg
1441318980 ids   **:115.159.15.29   80    US:139.182.235.44  53290 1      -25/0   trojan-activity snort-trojan-activity/emerging-trojan:1.2016950:ET TROJAN Possible Win32/Hupigon ip.txt with a Non-Mozilla UA  
1441318980 ids   EU:140.207.217.32  80    US:139.182.235.44  53288 1      -25/0   trojan-activity snort-trojan-activity/emerging-trojan:1.2016950:ET TROJAN Possible Win32/Hupigon ip.txt with a Non-Mozilla UA  
1441318980 ids   CN:221.130.190.225 80    US:139.182.235.44  53289 1      -25/0   trojan-activity snort-trojan-activity/emerging-trojan:1.2016950:ET TROJAN Possible Win32/Hupigon ip.txt with a Non-Mozilla UA  
1441318979 ids   **:114.80.165.113  80    US:139.182.235.44  53287 1      -25/0   trojan-activity snort-trojan-activity/emerging-trojan:1.2016950:ET TROJAN Possible Win32/Hupigon ip.txt with a Non-Mozilla UA  
1441317521 ids   US:54.230.87.85    80    US:139.182.116.150 56742 1      -25/0   trojan-activity snort-trojan-activity/trojan:1.2020031:ET TROJAN Trojan.Nurjax Retrieving Domains via JS  
#Query took 1 seconds

getflows.pl -E -u api1:testkey -w 360000 -Q modsec_out | grep CRITICAL | grep :CN

This query finds all outbound ModSec scans of type CRITICAL destined to China.

1441150992 log   CN:202.108.23.29   80    US:139.182.131.94  7702  1      -25/0   system_daemons Rank/Dorms:CN:202.108.23.29:80 ← US:139.182.131.94:7702 and Wireless rank/202_108_23_29: 3:5 modsec_out/CRITICAL/960010 Request content type is not allowed by policy -- pan.baidu.com/rest/2.0/dss/online -- application/x-www-form-urlencoded
1441150762 log   CN:123.125.115.225 80    US:139.182.90.207  63017 1      -25/0   system_daemons Rank/Dorms:CN:123.125.115.225:80 ← US:139.182.90.207:63017 and Wireless rank/123_125_115_225: 3:5 modsec_out/CRITICAL/960010 Request content type is not allowed by policy -- update.pan.baidu.com/statistics -- multipart/form-data
1441150747 log   CN:202.108.23.29   80    US:139.182.90.207  63013 1      -25/0   system_daemons Rank/Dorms:CN:202.108.23.29:80 ← US:139.182.90.207:63013 and Wireless rank/202_108_23_29: 3:5 modsec_out/CRITICAL/960010 Request content type is not allowed by policy -- pan.baidu.com/api/filemetas -- application/x-www-form-urlencoded
1441150636 log   CN:221.228.204.31  80    US:139.182.163.79  51406 1      -25/0   system_daemons Rank/Dorms:CN:221.228.204.31:80 ← US:139.182.163.79:51406 and Wireless rank/221_228_204_31: 3:5 modsec_out/CRITICAL/960010 Request content type is not allowed by policy -- rq.lbcct.cloud.duba.net/query -- application/x-www-form-urlencoded
1441150613 log   CN:59.151.113.106  80    US:139.182.16.71   52921 1      -25/0   system_daemons Rank/Dorms:CN:59.151.113.106:80 ← US:139.182.16.71:52921 and Wireless rank/59_151_113_106: 3:5 modsec_out/CRITICAL/960010 Request content type is not allowed by policy -- elearning.chinaacc.com/cware/video/videoPlay/service/saveCwareKcjyTimeLog.shtm -- application/x-www-form-urlencoded
1441150583 log   CN:221.228.204.31  80    US:139.182.91.134  50621 1      -25/0   system_daemons Rank/Dorms:CN:221.228.204.31:80 ← US:139.182.91.134:50621 and Wireless rank/221_228_204_31: 3:5 modsec_out/CRITICAL/960010 Request content type is not allowed by policy -- rq.drcct.cloud.duba.net/query -- application/x-www-form-urlencoded
1441150569 log   CN:221.228.204.36  80    US:139.182.163.79  51429 1      -25/0   system_daemons modsec_out/CRITICAL/960010:CN:221.228.204.36:80 ← US:139.182.163.79:51429 Request content type is not allowed by policy -- wq.cloud.duba.net/content_push -- application/x-www-form-urlencoded
1441150569 log   CN:221.228.204.36  80    US:139.182.163.79  51427 1      -25/0   system_daemons modsec_out/CRITICAL/960010:CN:221.228.204.36:80 ← US:139.182.163.79:51427 Request content type is not allowed by policy -- wq.cloud.duba.net/content_push -- application/x-www-form-urlencoded
1441150562 log   CN:59.151.113.106  80    US:139.182.16.71   52955 1      -25/0   system_daemons modsec_out/CRITICAL/960010:CN:59.151.113.106:80 ← US:139.182.16.71:52955 Request content type is not allowed by policy -- elearning.chinaacc.com/cware/video/videoPlay/service/saveFlashOperateLog.shtm -- application/x-www-form-urlencoded
...


Previous Chapter	Next Chapter

Navigation

Search

Command Line Interface

Contents

Initialization

Remote Execution

Re-Initialization

API Key Revocation

Using Windows for remote queries

Historical Flow and Payload Data Queries

Query Syntax

Examples

getflows.pl -u api1:xxxx -s 139.182.52.108 -w 3600 -N

getflows.pl -u api1:xxxx -s 139.182.52.108 -w 3600 -N -f

getflows.pl -u api1:xxxx -s 139.182.52.108 -p 80 -w 3600 -c -f -X

getflows.pl -u api1:xxxx -s 139.182.52.108 -p 80 -w 3600 -c -f

getflows.pl -u api1:xxxx -s 8.8.8.8 -p 53 -w 3600 -X

getflows.pl -u api1:xxxx -B

Historical Event Queries

Syntax

Examples

getflows.pl -E -u api1:xxx -s 139.182.52.108 -w 360000

getflows.pl -E -u api1:xxx -w 3600 -W service -Q 139.182. | grep -v service\/http\/80 | grep -v service\/ssl\/443 | grep -v skypetoskype

getflows.pl -E -u api1:xxx -w 3600 -W snort -Q trojan -p 80

getflows.pl -E -u api1:testkey -w 360000 -Q modsec_out | grep CRITICAL | grep :CN