Sorry for the lack of activity lately, but I've been busy on other projects. I do have some good news on performance to report. We have recently migrated to a new 8core server with 32GB of ram. I have migrated NetDB to a new MariaDB instance and tweaked some indexes and settings. Using concurrent imports which I will include in the new version, here's the impact on our imports:
2016-04-11 18:43:08: netdbctl(21527): Importing 56842 INT Entries in to switchstatus table with 16 Processes
2016-04-11 18:43:12: netdbctl(21527): Importing 38726 MAC Entries in to switchports table with 16 Processes
2016-04-11 18:43:18: netdbctl(21527): Importing 55009 ARP Entries in to ipmac table with 16 Processes
2016-04-11 18:43:28: netdbctl(21527): NetDB update complete
20seconds
Here’s the old server:
2016-04-11 18:40:13: netdbctl(18890): Importing 56842 intstatus entries in to switchstatus table
2016-04-11 18:40:53: netdbctl(18890): Importing 34156 MAC entries in to switchports table
2016-04-11 18:42:58: netdbctl(18890): Importing 46307 ARP Entries in to ipmac table
2016-04-11 18:45:14: netdbctl(18890): NetDB update complete
5:01min
That's a dramatic improvement. I am also going to see what I can do to improve the scraping performance so we can do faster imports. We're hoping to maintain 5min updates when we're done, which shouldn't be hard at this point.
Queries are also dramatically improved on large databases, especially Hostname searches. Once I wrap this up, I plan to push out a new Ubuntu 14.04 VM Image with an optimized version of MariaDB preinstalled.
Here are our current stats:
NetDB Statistics
---------------------------------
MAC Entries: 1072358
ARP Entries: 2236562
Switch Entries: 358006
WiFi Entries: 8218005
Status Entries: 57236
Registrations: 135324
DB Transactions: 254088
Total Rows in DB: 12331579
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Glad you're still enjoying NetDB. I've been hard at work on a new Python project called NetGrph. I'm waiting on an internal code review, but once I get it approved for release, I plan to release both a NetDB and NetGrph hybrid VM on either Ubuntu 14.04 or 16.04 depending on the timing. NetGrph is not out yet, but if you want to take a look, here's the overview:
Thanks for waiting, I think the two should complement each other well. NetDB already knows about everything about the edge of the network, and NetGrph knows everything about the internals.
Yes, they will be very complement.
For fun, my stats this day :
NetDB Statistics
MAC Entries: 47617
ARP Entries: 260461
Switch Entries: 68099
WiFi Entries: 92011
Status Entries: 49959
Registrations: 0
DB Transactions: 1659268
Total Rows in DB: 2177415
You win :)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
At this time, I'm still working on NetGrph integration with NetDB, which is taking longer than I would like. If you would like to work with the latest NetDB code, you can follow these instructions against a fresh Ubuntu 16.01 image, and instead of running the NetGrph playbooks, run the netdb.yml playbook in netdb/extra/playbooks/ on the trunk repository against localhost.
cd /tmp/netdb/extra/playbooks/
ansible-playbook netdb.yml
You should be able to restore your current database with netdbctl -re to try it out. This will install the latest MariaDB and includes all of the performance increases.
Currently the password for the web interface is adminchangeme3742, and you can update the password with htpasswd /etc/apache2/htpasswd netdb
Hopefully I will have a new VM out soon, but I want to wait until NetGrph is integrated out of the box since maintaining VM images is not easy.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry for the lack of activity lately, but I've been busy on other projects. I do have some good news on performance to report. We have recently migrated to a new 8core server with 32GB of ram. I have migrated NetDB to a new MariaDB instance and tweaked some indexes and settings. Using concurrent imports which I will include in the new version, here's the impact on our imports:
2016-04-11 18:43:08: netdbctl(21527): Importing 56842 INT Entries in to switchstatus table with 16 Processes
2016-04-11 18:43:12: netdbctl(21527): Importing 38726 MAC Entries in to switchports table with 16 Processes
2016-04-11 18:43:18: netdbctl(21527): Importing 55009 ARP Entries in to ipmac table with 16 Processes
2016-04-11 18:43:28: netdbctl(21527): NetDB update complete
20seconds
Here’s the old server:
2016-04-11 18:40:13: netdbctl(18890): Importing 56842 intstatus entries in to switchstatus table
2016-04-11 18:40:53: netdbctl(18890): Importing 34156 MAC entries in to switchports table
2016-04-11 18:42:58: netdbctl(18890): Importing 46307 ARP Entries in to ipmac table
2016-04-11 18:45:14: netdbctl(18890): NetDB update complete
5:01min
That's a dramatic improvement. I am also going to see what I can do to improve the scraping performance so we can do faster imports. We're hoping to maintain 5min updates when we're done, which shouldn't be hard at this point.
Queries are also dramatically improved on large databases, especially Hostname searches. Once I wrap this up, I plan to push out a new Ubuntu 14.04 VM Image with an optimized version of MariaDB preinstalled.
Here are our current stats:
Hi jonathan,
what about the new vmware image,
Are you still working about it ?
I look forward to this version.
Thanks a lot again for this beautiful project.
Tartan,
Glad you're still enjoying NetDB. I've been hard at work on a new Python project called NetGrph. I'm waiting on an internal code review, but once I get it approved for release, I plan to release both a NetDB and NetGrph hybrid VM on either Ubuntu 14.04 or 16.04 depending on the timing. NetGrph is not out yet, but if you want to take a look, here's the overview:
https://notehub.org/cj2td
I'm really excited about this project, and I'm just waiting to get it released and tested before I release a new VM with both programs included.
Jonathan
What a nice project too, i'm interesting.
it was worth the wait.
....waiting with smile.
Thanks for waiting, I think the two should complement each other well. NetDB already knows about everything about the edge of the network, and NetGrph knows everything about the internals.
Jonathan
Yes, they will be very complement.
For fun, my stats this day :
NetDB Statistics
MAC Entries: 47617
ARP Entries: 260461
Switch Entries: 68099
WiFi Entries: 92011
Status Entries: 49959
Registrations: 0
DB Transactions: 1659268
Total Rows in DB: 2177415
You win :)
Hah, except for your transaction count ;)
If you're doing automated queries against the database, you can always use netdb -nt to filter those out of the transaction table.
Stay tuned...
Can you share to use when v1.14 will be available?
Christophe,
At this time, I'm still working on NetGrph integration with NetDB, which is taking longer than I would like. If you would like to work with the latest NetDB code, you can follow these instructions against a fresh Ubuntu 16.01 image, and instead of running the NetGrph playbooks, run the netdb.yml playbook in netdb/extra/playbooks/ on the trunk repository against localhost.
Follow Setting Up Ansible to Run Against Localhost:
http://netgrph.readthedocs.io/en/dev/playbooks/README/
Then Clone the repo to tmp:
svn checkout https://svn.code.sf.net/p/netdbtracking/code/trunk netdb
Then run the playbook:
cd /tmp/netdb/extra/playbooks/
ansible-playbook netdb.yml
You should be able to restore your current database with netdbctl -re to try it out. This will install the latest MariaDB and includes all of the performance increases.
Currently the password for the web interface is adminchangeme3742, and you can update the password with
htpasswd /etc/apache2/htpasswd netdbHopefully I will have a new VM out soon, but I want to wait until NetGrph is integrated out of the box since maintaining VM images is not easy.