D 2010-12-08T03:39:50
L dvessey-misc-bro
P 579b807bc7b6480a8d3880c9e2f0331e90a5451c
U dvessey
W 3455
<h3>Misc notes</h3>
  *  To change the unix time stamp to put a bar between seconds and microseconds:
<verbatim>
cat FILENAME | sed 's/\(^[0-9]*\)\./\1|/g'
</verbatim>
  *  The run_bro.sh shell script replaces all timestamps with two fields - seconds and microseconds

  *  On a 10.1GB PCAP on machine with 6 proc, 8GB RAM, running the various bro policy scripts as separate processes takes 13:41, maxes out approx 3GB RAM. RAM usage seems to run up and down as different policy scripts finish. Running all of them in one instance takes 20:11, takes up max of approx 3GB RAM
  *  In the same test, it looks like ftp, dns and ssl all finish in approx 3:42, bt-tracker after 5:41, which should help free up some resources. conn-dv finishes after 10:27, the slowest one is http. This could just be a symptom of the type of traffic being examined

<p>
<h3>Output formats</h3>
  *  conn-dv.bro output format to conn-dv.log:
<verbatim>
start time|duration|orig IP|resp IP|service|orig port|resp port|transport layer proto|orig bytes|resp bytes|content 1g entropy|connection state|flags
</verbatim>


  *  dns-dv.bro output format to dns-dv.log
<verbatim>
time|session_id|orig IP|orig port|resp IP|resp port|DNS details string
</verbatim>
  *  dns-dv.bro output to dns_a_responses.log
<verbatim>
time|session_id|orig IP|orig port|resp IP|resp port|A|host name|IP of host|Other DNS details
</verbatim>
** NOTE: Need to do a grep -v "<query addl" to clean up the output of dns_a_responses.log

  *  ftp-dv.bro output format to ftp-dv.log:
<verbatim>
start time|ftp session id|orig IP|orig port|resp IP|resp port|FTP message
</verbatim>


  *  http-dv.bro output - http_req.log - REQUESTS (triple quotes used to escape any weird GET requests)
<verbatim>
time~~~~~~session id~~~~~~orig IP~~~~~~orig port~~~~~~resp IP~~~~~~resp port~~~~~~method~~~~~~URI~~~~~~1g entropy
</verbatim>

  *  http-dv.bro output - http_rep.log - REPLIES (triple quotes used to escape any weird GET requests)
<verbatim>
time~~~~~~session id~~~~~~orig IP~~~~~~orig port~~~~~~resp IP~~~~~~resp port~~~~~~request~~~~~~request 1g entropy~~~~~~code~~~~~~reason~~~~~~content length~~~~~~body length~~~~~~interrupted?~~~~~~Content gap (bytes)
</verbatim>

  *  http-dv.bro output - http_headers.log - HEADERS  (triple quotes used to escape any weird header values)
<verbatim>
time~~~~~~session id~~~~~~orig IP~~~~~~orig port~~~~~~resp IP~~~~~~resp port~~~~~~direction~~~~~~header name~~~~~~header name 1g entropy~~~~~~header value~~~~~~header value 1g entropy
</verbatim>

  *  os-fingerprint-dv.bro ouput to software-dv.log
<verbatim>
time|client IP|message about software being used
</verbatim>
TODO: Right now, 'client IP' field also has the word 'client' at the end.. don't know why. Need to figure out a way to clean that up

  *  bt-tracker-dv.bro ouput to bt-tracker-dv.log
<verbatim>
time|tracker ID|orig IP|orig port|resp IP|resp port|tag|direction|Bittorrent message
</verbatim>
NOTE: The 'infohash' (in some of the bittorrent messages) is actually the hash of the torrent, you can google it and find out what was downloaded

  *  ssl-dv.bro ouput to ssl-dv-conns.log
<verbatim>
time|session ID|orig IP|orig port|resp IP|resp port|start
</verbatim>

  *  ssl-dv.bro ouput to ssl-dv.log
<verbatim>
time session_ID SSL_msg
</verbatim>
NOTE: need to do some post processing on this because bro's output only has spaces

Z 5660044bc776efda9be05a9da3bf3b0b