Difference between revisions of "Network/High-precision NFDUMP data visualization"

From braindump
Jump to navigation Jump to search
 
(2 intermediate revisions by the same user not shown)
Line 13: Line 13:
* [http://imagemagick.org ImageMagick's montage] [optional]
* [http://imagemagick.org ImageMagick's montage] [optional]


== Howto ==
=== Extract the netflow data ===
In order to process the NFDUMP binary format with <tt>R</tt> the data has to be converted into something parsable. The best option is the <tt>-o pipe</tt> option of the <tt>nfdump</tt> command. Additionally to suppress statistical data at the end of the output the quiet <tt>-q</tt> option is to be set. One drawback of the <tt>pipe</tt> format output is the missing of a descriptive header. To include one the <tt>nfdump</tt> command is perpended by a simple <tt>echo</tt> for readability split into two lines.
( echo -n "N1|tstart|mstart|tend|mend|proto|N2|N3|N4|srcip|srcp|"; \
echo "N5|N6|N7|dstip|dstp|N8|N9|N10|N11|flags|tos|packets|bytes"; \
nfdump -R <span class="input">/path/to/nfdump/data</span> -q -o pipe \
) > <span class="input">netflow.data</span>

The fields starting with a capital '''N''' are later discarded of by the <tt>R</tt> script.

=== Produce graph with <tt>R</tt> ===
WIP


[[Category: Network]]
[[Category: Network]]
[[Category: R]]

Latest revision as of 13:41, 22 June 2014

I was given a similar mandate for as with this task but this time around I did have a lot of historical data already by means of netflow data collected by NFDUMP a high-performance open source netflow collector. The only problem the data as such could not be used for my purpose without a bit of massaging. And here is how I did it.

Goal

Create a graph from collected netflow data in the NFDUMP format with a precision of one second.

Note: For lower precision graphs the NfSen visualization package should suffice.

Prerequisites

Howto

Extract the netflow data

In order to process the NFDUMP binary format with R the data has to be converted into something parsable. The best option is the -o pipe option of the nfdump command. Additionally to suppress statistical data at the end of the output the quiet -q option is to be set. One drawback of the pipe format output is the missing of a descriptive header. To include one the nfdump command is perpended by a simple echo for readability split into two lines.

( echo -n "N1|tstart|mstart|tend|mend|proto|N2|N3|N4|srcip|srcp|"; \
  echo    "N5|N6|N7|dstip|dstp|N8|N9|N10|N11|flags|tos|packets|bytes"; \ 
  nfdump -R /path/to/nfdump/data -q -o pipe \
) > netflow.data

The fields starting with a capital N are later discarded of by the R script.

Produce graph with R

WIP