Re: [Nfsen-discuss] [ask] report with asn number

[Brian C. <b.c...@po...>]

On 25/08/2021 09:44, Peter Haag wrote:
> On 18.08.21 11:49, Brian Candler wrote:
>> On 18/08/2021 09:45, Peter Haag wrote:
>>> nfsen is *very* old code, and actually I wanted to rewrite a new version, which got delayed and finally aborted for a number of reasons.
>>> I could consider to give it a go again, but I am still uncertain if it would still be used these days. Feedback would be appreciated.
>>> If there would be a new NfSen incarnation, I would write it in Golang as it provides far more flexibility and speed. I am not a exceptionally gifted
>>> GUI/Web developer - maybe someone would be willing to support. However, also for a new NfSen, I could prefer KISS
>>> - easy of use and - hopefully - fast.
>> A maintained replacement for NFSen would be extremely welcome!
> I think a careful design can help to prevent making the new interface unusable and integrate
> those assets, which users on a large scale would like.
>
> Actually I like the idea using Prometheus, as a lot of things are already integrated.
> Using Prometheus would naturally result in using Grafana as frontend. The drill
> down and listing flows needs to be implemented somehow. Up to now I have no experience in
> developing grafana apps - I need to study this first.
>
> Would you be comfortable with Grafana graphs and the way to zoom in to spot the points of
> interest?

Yes I think so.  Selecting time ranges in Grafana is OK, although 
zooming out once you've already selected a range is a bit painful.  The 
key thing of course would be mapping the time range to the nfdump 
command line.  Checking how grafana deals with loki logs may be helpful 
here.  Maybe it's even possible to integrate with the "Explore" 
functionality. (*)

I would still ideally like to be able to select a single point of 
interest and get the netflow logs from that "bucket" - i.e. the "select 
single" function in nfsen.  It makes it easy to move back and forth 
around a point of interest.

(*) Aside: a completely different approach would be to convert netflow 
records into JSON and write them to loki, with certain fields extracted 
as labels.  This would then use loki's query language for analyzing 
netflow, and make most of nfdump redundant.  I don't think loki's query 
engine is featureful enough to do this (e.g. grouping by subnet); I also 
suspect that even compressed JSON would be less efficient storage-wise 
than nfdump's format.



>
> Creating profile as possible so far in NfSen could be a challenge for Prometheus, as importing
> historical data ( which filtering for a profile is nothing else ) seems not what Prometheus was
> intended for.

Prometheus has fairly recently (v2.24.0) gained the ability to backfill 
historical data:

https://github.com/prometheus/prometheus/releases/tag/v2.24.0

https://github.com/prometheus/prometheus/pull/8230/files

>
> Any remarks opinions and ideas are welcome in order to draft ideas for a modern interface and
> workflow with netflow data.
>
> Meanwhile I try to write a nfdump netflow_exporter for Prometheus to play and experiment with.
> It's almost done.
>
> I would put the exporter on a new repo on Github, so discussion could continue there as an issue,
> or here on the ML.

Sounds excellent, I look forward to playing with this!

Regards,

Brian.


>
> Regards
>
> 	- Peter
>
>> I see two aspects to this:
>>
>> 1. the web UI for selecting time ranges and drilling down to netflow queries
>> 2. the creation and storage of graphs of flow data (i.e. summaries by protocol, and user-defined channels)
>>
>> As regards (1): whilst the UI of NFSen is very much web-1.0, it is straightforward, practical and usable in real world scenarios.
>>
>> Attempts to "modernise" the UI, as in nfsen-ng, have made it unusable.  Just try selecting time ranges: things zoom horizontally, making it very
>> difficult to change to a wider time range.  In my opinion, it's a complete fail.  Selecting time ranges is the main /raison d'être/ of NFSen.
>>
>> As regards (2): I would be very happy to see rrdtool dropped entirely and replaced by something modern, like Prometheus.  This could be very simple to
>> implement: the nfsen backend simply filters and counts flow records, and exposes the counters as prometheus metrics.  Then prometheus scrapes them.
>> This could be done at a finer granularity than today, e.g. 1 minute or better. (In fact, since prometheus considers timeseries "stale" after 5 minutes
>> of no data, it's recommended that you scrape *at least* every 2 minutes.  And the cost of doing so is much lower than rrdtool).  Apart from being
>> efficient and scalable, the Prometheus ecosystem opens up a whole range of possibilities around dashboards and alerting - and it means there's no need
>> to re-implement alerts in NFSen.
>>
>> nfsen-ng missed this opportunity too.
>>
>> Regards,
>>
>> Brian.
>>
>>
>>
>>
>> _______________________________________________
>> Nfsen-discuss mailing list
>> Nfs...@li...
>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>>
>