Re: [Netdisco] Netdisco auto discovery tasks suddenly stopped working

[Jethro B. <jet...@st...>]

I'm interested to know how you diagnosed which the bad record was.

Jethro.


.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .

Jethro R Binks, Network Manager,

Information Services Directorate, University Of Strathclyde, Glasgow, UK


The University of Strathclyde is a charitable body, registered in Scotland, number SC015263.

________________________________
From: alcatron <alc...@gm...>
Sent: 19 January 2022 11:44
To: Christian Ramseyer <ram...@ne...>; net...@li... <net...@li...>
Subject: Re: [Netdisco] Netdisco auto discovery tasks suddenly stopped working


Hi Christian, thankyou for the tips.



I found what the problem is, it was crashing and not going past a certain object in the “device_skip” table in the database.



I truncated that field in psql, and let it re-populate and that fixed the automatic discovery and arpnip/macsuck etc.



I have found after a while perhaps 2-3 months something happens in the “device_skip” table and halts these processes then I need to clear it to make it work again. I remember I had this similar issue a few months back, then I remembered what I did.



Muris



From: Christian Ramseyer <ram...@ne...>
Date: Tuesday, 18 January 2022 at 12:20 pm
To: alcatron <alc...@gm...>, net...@li... <net...@li...>
Subject: Re: [Netdisco] Netdisco auto discovery tasks suddenly stopped working

Hi

 >  could not connect to
 > server: No such file or directory/

This would be very concerning, meaning that Postgres is not running at
all. But since you seem to have the web frontend running that is
probably not the case currently, so I wouldn't worry too much. Might be
an old log entry.


 > Failed on request of size 16 in memory context
 > "MessageContext".

That on the other hand might be the issue. Postgres uses all kinds of
memory parameters, if one of them is too small the total GB of RAM
sticks in the server don't matter much.

I had various issues with huge and clogged up discovery queues over the
years, as a first measure I'd try to:

stop netdisco-backend
restart Postgres, connect to the database with "netdisco-do psql" and in
there run a "delete from admin;".
for good measure, also run "reindex table admin;"
restart netdisco-backend

This sounds dangerous but admin is in fact just the queue of actions to
be done, so no important data will be lost.

Also a "select count(*) from admin" first might be interesting, to see
how many rows are in there. If it's an absurdly high number (millions)
you can run e.g. "create table admin_backup as select * from admin;" for
analysis later.

If you're still getting the memory errors afterwards and it still
doesn't work, I'd try to configure the memory parameters with this
assistant, using the "online transaction processing" db type.
https://pgtune.leopard.in.ua/#/about


Cheers
Christian



On 17.01.22 22:03, alcatron wrote:
> Hi all, just wanting to ask your thoughts on what could be causing
> netdisco to suddenly stop performing auto discovery tasks.
>
> Seems only arpnip is working via scheduled tasks, but discovery/macsuck
> has halted to auto perform. If I go manually to the device on web
> interface and trigger the auto discovery/arpnip/macsuck it works fine on
> the device.
>
> Nothing has changed on system, running for a few months now, and
> suddenly the auto discovery is broken partly.
>
> If I go to the backend log I see error like this below. The server is
> running and operational as I can still perform the manual to get
> discovery etc
>
> The server is not out of memory as it has like 16GB and still plenty
> unused not what the messages are indicating..
>
> Thanks for any assistance ??
>
> /DBIx::Class::Schema::Versioned::_on_connect(): Your DB is currently
> unversioned. Please call upgrade on your schema to sync the DB. at
> /home/netdisco/perl5/lib/perl5/DBICx/Sugar.pm line 121/
>
> /DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI
> connect('dbname=netdisco','netdisco',...) failed: could not connect to
> server: No such file or directory/
>
> /            Is the server running locally and accepting/
>
> /            connections on Unix domain socket
> "/var/run/postgresql/.s.PGSQL.5432"? at
> /home/netdisco/perl5/lib/perl5/DBIx/Class/Storage/DBI.pm line 1639. at
> /home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 50/
>
> //
>
> /[25756] error bless( {'msg' =>
> 'DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st
> execute failed: ERROR:  out of memory/
>
> /DETAIL:  Failed on request of size 16 in memory context
> "MessageContext". [for Statement "SELECT me.job, me.entered, me.started,
> me.finished, me.device, me.port, me.action, me.subaction, me.status,
> me.username, me.userip, me.log, me.debug, me.device_key FROM admin me
> WHERE ( me.job = ? ) FOR UPDATE" with ParamValues: 1=\'186421742\'] at
> /home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267/
>
> /'}, 'DBIx::Class::Exception' )/
>
> /[25781] 2022-01-11 01:33:53 error bless( {'msg' =>
> 'DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st
> execute failed: ERROR:  out of memory/
>
> /DETAIL:  Failed on request of size 16 in memory context
> "MessageContext". [for Statement "SELECT me.job, me.entered, me.started,
> me.finished, me.device, me.port, me.action, me.subaction, me.status,
> me.username, me.userip, me.log, me.debug, me.device_key FROM admin me
> WHERE ( me.job = ? ) FOR UPDATE" with ParamValues: 1=\'186420514\'] at
> /home/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267/
>
>
>
> _______________________________________________
> Netdisco mailing list
> net...@li...
> https://sourceforge.net/p/netdisco/mailman/netdisco-users/

--
Christian Ramseyer, netnea ag
Network Management. Security. OpenSource.
https://www.netnea.com
Phone: +41 79 644 77 64