pcap2file using Suricata in windows/linux

There is no easy way to automatically extract the files from the PCAP(not PCAP-NG). We can use Suricata to extract the files (atleast whatever files transferred over HTTP). Here are the steps to configure and extract the files from the PCAP. Some of the changes we do here is to just reduce the errors/warning from the Suricata. You can download the windows binaries from the Suricata website (http://suricata-ids.org/download/)

(Only Edit the lines that are marked in RED)

Step 1: List of changes needs to be done in C:\Program Files\Suricata\suricata.yaml
  # Extensible Event Format (nicknamed EVE) event log in JSON format
  – eve-log:
      enabled: no
  # a line based log of TLS handshake parameters (no alerts)
  – tls-log:
      enabled: yes  # Log TLS connections.
  # a line based log of DNS requests and/or replies (no alerts)
  – dns-log:
      enabled: yes
  # File extraction depends on a lot of things to be fully done:
  # – stream reassembly depth. For optimal results, set this to 0 (unlimited)
  # – http request / response body sizes. Again set to 0 for optimal results.
  # – rules that contain the “filestore” keyword.
  – file-store:
      enabled: yes       # set to yes to enable
  # output module to log files tracked in a easily parsable json format
  – file-log:
      enabled: yes
      filename: files-json.log
  pcap-file:
  # Possible values are:
  #  – yes: checksum validation is forced
  #  – no: checksum validation is disabled
  #  – auto: suricata uses a statistical approach to detect when
  #  checksum off-loading is used. (default)
  # Warning: ‘checksum-validation’ must be set to yes to have checksum tested
  checksum-checks: no
default-rule-path: C:\\Program Files\\Suricata\\rules\\
rule-files:
# – botcc.rules
# – ciarmy.rules
# – compromised.rules
# – drop.rules
# – dshield.rules
# – emerging-activex.rules
# – emerging-attack_response.rules
# – emerging-chat.rules
# – emerging-current_events.rules
# – emerging-dns.rules
# – emerging-dos.rules
# – emerging-exploit.rules
# – emerging-ftp.rules
# – emerging-games.rules
# – emerging-icmp_info.rules
# – emerging-icmp.rules
# – emerging-imap.rules
# – emerging-inappropriate.rules
# – emerging-malware.rules
# – emerging-misc.rules
# – emerging-mobile_malware.rules
# – emerging-netbios.rules
# – emerging-p2p.rules
# – emerging-policy.rules
# – emerging-pop3.rules
# – emerging-rpc.rules
# – emerging-scada.rules
# – emerging-scan.rules
# – emerging-shellcode.rules
# – emerging-smtp.rules
# – emerging-snmp.rules
# – emerging-sql.rules
# – emerging-telnet.rules
# – emerging-tftp.rules
# – emerging-trojan.rules
# – emerging-user_agents.rules
# – emerging-voip.rules
# – emerging-web_client.rules
# – emerging-web_server.rules
# – emerging-web_specific_apps.rules
# – emerging-worm.rules
# – tor.rules
# – decoder-events.rules # available in suricata sources under rules dir
# – stream-events.rules  # available in suricata sources under rules dir
# – http-events.rules    # available in suricata sources under rules dir
# – smtp-events.rules    # available in suricata sources under rules dir
# – dns-events.rules     # available in suricata sources under rules dir
# – tls-events.rules     # available in suricata sources under rules dir
  – files.rules

Step 2: Changed in C:\Program Files\Suricata\rules\files.rules
Uncomment this line:
alert http any any -> any any (msg:”FILE store all”; filestore; noalert; sid:15; rev:1;)

Once you are done with that, you can run the Suricata to extract the files from the PCAP files.
    “c:\Program Files\Suricata\suricata.exe” -r C:\pcap2file\pcapfile.pcap -l C:\pcap2file

Once you executed the above command, you can find the files in C:\pcap2file\files . This folder will have two files for each and every file transferred over HTTP. One file with the original content and another with the meta data about the file.

              image

Update: Python script to convert these files into a directory structure that is easy to navigate.

SuricataExtract2Dir.py:

import sys
import random
import shlex
import os
import struct
import subprocess
import re
import string
import sys
import ctypes
import time
import datetime
import shutil
import re

def usage():
    print ”’
Usage: %s <extract full path> <output full path>
    ”’.strip() % sys.argv[0]
def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
    return ”.join(random.choice(chars) for _ in range(size))

def parsemetafile(metafilepath):
    inputfilehandle = open(metafilepath, ‘rb’)
    srcdata = inputfilehandle.read()
    inputfilehandle.close()
    m = re.search(r”HTTP HOST:[ ]*(.*)[\r\n]”, srcdata)
    http_host= m.group(1)
    m = re.search(r”FILENAME:[ ]*(.*)[\r\n]”, srcdata)
    filename=  m.group(1)
    if filename[-1] == “/”:
        #default one
        filename= “\\Index.html”
    else:
        filename= filename.replace(‘/’,’\\’, 1000000);
    if filename[0] != “\\”:
        filename= “\\” + filename;
    return (metafilepath[0:-5],http_host + filename)
def copythisextractedfile(srcfilepath,targetfilepath):
    inputfilehandle = open(srcfilepath, ‘rb’)
    outputfilehandle = open(targetfilepath, ‘wb’)
    srcdata = inputfilehandle.read()
    outputfilehandle.write(srcdata)
    outputfilehandle.close()
    inputfilehandle.close()

def mkdirs(newdir):
    “”” Create a directory and all parent folders.
        Features:
        – parent directoryies will be created
        – if directory already exists, then do nothing
        – if there is another filsystem object with the same name, raise an exception
    “””
    if os.path.isdir(newdir):
        pass
    elif os.path.isfile(newdir):
        raise OSError(“cannot create directory, file already exists: ‘%s'” % newdir)
    else:
        head, tail = os.path.split(newdir)
        if head and not os.path.isdir(head):
            mkdirs(head)
        if tail:
            os.mkdir(newdir)

if len(sys.argv) < 3 or os.path.isfile(sys.argv[1]) or os.path.isfile(sys.argv[2]):
    #print len(sys.argv)
    usage()
    sys.exit()
extractpath= sys.argv[1];
outputpath = sys.argv[2];

fileList = []
for root, subFolders, files in os.walk(extractpath):
    for file in files:
        if re.search(‘.meta$’, file, re.IGNORECASE):
            fileList.append(os.path.join(root,file))

url2filemapping= {}
for file in fileList:
    (datafilepath, srcurl)= parsemetafile(file)
    #print “%s  ==> %s” % (datafilepath, srcurl)
    if(url2filemapping.has_key(srcurl)):
        srcurl= srcurl + “_RANDOM_” + id_generator()
    url2filemapping[srcurl]= datafilepath;

errorcount=0

for file in url2filemapping:
    srcFilePath= url2filemapping[file]
    destFilePath= “\\\\?\\%s\\%s”%(outputpath,file);
    print “srcFilePath= (%32s) destFilePath=(%s) ” % (srcFilePath,destFilePath);
    destFilePath= destFilePath[:7] + destFilePath[7:].replace(‘:’,”);
    destFilePath= destFilePath.replace(‘\a’,’\\a’);
    destFilePath= destFilePath.replace(‘\b’,’\\b’);
    destFilePath= destFilePath.replace(‘\f’,’\\f’);
    destFilePath= destFilePath.replace(‘\n’,’\\n’);
    destFilePath= destFilePath.replace(‘\r’,’\\r’);
    destFilePath= destFilePath.replace(‘\t’,’\\t’);
    destFilePath= destFilePath.replace(‘\v’,’\\v’);
    mkdirs(os.path.dirname(destFilePath));
    try :
        copythisextractedfile(srcFilePath, destFilePath);
    except (IOError, OSError) as exc:
        print “Copy Error(%d). %s(%s)” %(exc.errno, destFilePath,srcFilePath)
        errorcount= errorcount + 1
        pass

if errorcount:       
    print “Number of file copy errors     =%d” % (errorcount);
else:
    print “Finished copying”

============== End of Script =====================

Once you executed this script, you will get a output like this.

                 image

Advertisements
This entry was posted in Tools, Web, Windows and tagged , , , , , . Bookmark the permalink.

One Response to pcap2file using Suricata in windows/linux

  1. Pingback: pcap2file using Suricata /Windows/Linux* | Adva...

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s