Network/High-precision NFDUMP data visualization

I was given a similar mandate for as with this task but this time around I did have a lot of historical data already by means of netflow data collected by NFDUMP a high-performance open source netflow collector. The only problem the data as such could not be used for my purpose without a bit of massaging. And here is how I did it.

Goal

Create a graph from collected netflow data in the NFDUMP format with a precision of one second.

Note: For lower precision graphs the NfSen visualization package should suffice.

Prerequisites

• Collected NFDUMP netflow data.
• NFDUMP
• R
• ImageMagick's montage [optional]

Howto

Extract the netflow data

In order to process the NFDUMP binary format with R the data has to be converted into something parsable. The best option is the -o pipe option of the nfdump command. Additionally to suppress statistical data at the end of the output the quiet -q option is to be set. One drawback of the pipe format output is the missing of a descriptive header. To include one the nfdump command is perpended by a simple echo for readability split into two lines.

( echo -n "N1|tstart|mstart|tend|mend|proto|N2|N3|N4|srcip|srcp|"; \
  echo    "N5|N6|N7|dstip|dstp|N8|N9|N10|N11|flags|tos|packets|bytes"; \ 
  nfdump -R /path/to/nfdump/data -q -o pipe \
) > netflow.data

The fields starting with a capital N are later discarded of by the R script.

Produce graph with R

WIP