High-precision NFDUMP data visualization

From braindump
Revision as of 13:41, 22 June 2014 by Uroesch (talk | contribs) (→‎Produce graph with R)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

I was given a similar mandate for as with this task but this time around I did have a lot of historical data already by means of netflow data collected by NFDUMP a high-performance open source netflow collector. The only problem the data as such could not be used for my purpose without a bit of massaging. And here is how I did it.

Goal

Create a graph from collected netflow data in the NFDUMP format with a precision of one second.

Note: For lower precision graphs the NfSen visualization package should suffice.

Prerequisites

Howto

Extract the netflow data

In order to process the NFDUMP binary format with R the data has to be converted into something parsable. The best option is the -o pipe option of the nfdump command. Additionally to suppress statistical data at the end of the output the quiet -q option is to be set. One drawback of the pipe format output is the missing of a descriptive header. To include one the nfdump command is perpended by a simple echo for readability split into two lines.

( echo -n "N1|tstart|mstart|tend|mend|proto|N2|N3|N4|srcip|srcp|"; \
  echo    "N5|N6|N7|dstip|dstp|N8|N9|N10|N11|flags|tos|packets|bytes"; \ 
  nfdump -R /path/to/nfdump/data -q -o pipe \
) > netflow.data

The fields starting with a capital N are later discarded of by the R script.

Produce graph with R

WIP