Re: [Netdisco] Urgent Problem with Netdisco new version 2.52.6

[Andy R. <ac...@gm...>]

This is also something that can happen if your system is running low
on resources. I see from your graphs that this might be the case. I
don't really understand what I'm looking at there.

Disk space might also be an issue.

Check the config file for this:

# number of SNMP workers to run in parallel (in netdisco-backend).
# the default is twice the number of CPU cores. increase this if
# your system has few cores and the schedule is taking too long.
# ```````````````````````````````````````````````````````````````
#workers:
#  tasks: 'AUTO * 2'

Try adjusting that. Also:

# this is the schedule for automatically keeping netdisco up-to-date;
# these are good defaults, so only uncomment if needing to change.
# (or set "schedule: null" if you wish to disable the scheduler)
# ````````````````````````````````````````````````````````````````````
#schedule:
#  discoverall:
#    when: '5 7 * * *'
#  macwalk:
#    when:
#      min: 20
#  arpwalk:
#    when:
#      min: 50
#  nbtwalk:
#    when: '0 8,13,21 * * *'
#  expire:
#    when: '30 23 * * *'
#  makerancidconf: null

I find macwalk (macsuck?) and arpwalk (arpnip?) to only be needed once
an hour or even less. I don't do nbtwalk and expire once a day is fine
for me as well.

This will reduce the load on your machine possibly.

If you really have a memory leak try updating the packages on the system.

Andy

On Mon, May 23, 2022 at 4:52 AM Muris <alc...@gm...> wrote:
>
> Im also checking the postgresql log and I have this also showing - /var/lib/pgsql/12/data/log
>
> few strange errors occurring. Any ideas anyone?
>
> 2022-05-23 19:57:04.247 ACST [10664] STATEMENT:  INSERT INTO device_vlan ( description, ip, last_discover, vlan) VALUES ( $1, $2, now(), $3 )
> 2022-05-23 19:57:04.247 ACST [10664] ERROR:  current transaction is aborted, commands ignored until end of transaction block
> 2022-05-23 19:57:04.247 ACST [10664] STATEMENT:  INSERT INTO device_vlan ( description, ip, last_discover, vlan) VALUES ( $1, $2, now(), $3 )
> 2022-05-23 19:57:04.248 ACST [10664] ERROR:  current transaction is aborted, commands ignored until end of transaction block
> 2022-05-23 19:57:04.248 ACST [10664] STATEMENT:  INSERT INTO device_vlan ( description, ip, last_discover, vlan) VALUES ( $1, $2, now(), $3 )
> 2022-05-23 19:57:04.248 ACST [10664] ERROR:  current transaction is aborted, commands ignored until end of transaction block
> 2022-05-23 19:57:04.248 ACST [10664] STATEMENT:  INSERT INTO device_vlan ( description, ip, last_discover, vlan) VALUES ( $1, $2, now(), $3 )
> 2022-05-23 19:57:36.171 ACST [10809] ERROR:  canceling autovacuum task
> 2022-05-23 19:57:36.171 ACST [10809] CONTEXT:  automatic vacuum of table "netdisco.public.device_port"
> 2022-05-23 20:05:36.933 ACST [11727] ERROR:  canceling autovacuum task
> 2022-05-23 20:05:36.933 ACST [11727] CONTEXT:  automatic vacuum of table "netdisco.public.device_port"
> 2022-05-23 20:06:36.315 ACST [11812] ERROR:  canceling autovacuum task
> 2022-05-23 20:06:36.315 ACST [11812] CONTEXT:  automatic vacuum of table "netdisco.public.device_port"
> 2022-05-23 20:07:36.741 ACST [11906] ERROR:  canceling autovacuum task
> 2022-05-23 20:07:36.741 ACST [11906] CONTEXT:  automatic vacuum of table "netdisco.public.device_port"
> 2022-05-23 20:09:01.974 ACST [12021] ERROR:  invalid input syntax for type macaddr: "00:00:00:00"
> 2022-05-23 20:09:01.974 ACST [12021] STATEMENT:  SELECT me.ip, me.creation, me.dns, me.description, me.uptime, me.contact, me.name, me.location, me.layers, me.num_ports, me.mac, me.serial, me.chassis_id, me.model, me.ps1_type, me.ps2_type, me.ps1_status, me.ps2_status, me.fan, me.slots, me.vendor, me.os, me.os_ver, me.log, me.snmp_ver, me.snmp_comm, me.snmp_class, me.snmp_engineid, me.vtp_domain, me.last_discover, me.last_macsuck, me.last_arpnip, me.is_pseudo FROM device me WHERE ( mac = $1 )
> 2022-05-23 20:10:00.405 ACST [12179] ERROR:  column c.relhasoids does not exist at character 245
> 2022-05-23 20:10:00.405 ACST [12179] STATEMENT:  select n.nspname, c.relname, a.attname, a.atttypid, t.typname, a.attnum, a.attlen, a.atttypmod, a.attnotnull, c.relhasrules, c.relkind, c.oid, pg_get_expr(d.adbin, d.adrelid), case t.typtype when 'd' then t.typbasetype else 0 end, t.typtypmod, c.relhasoids, attidentity, c.relhassubclass from (((pg_catalog.pg_class c inner join pg_catalog.pg_namespace n on n.oid = c.relnamespace and c.oid = 16535) inner join pg_catalog.pg_attribute a on (not a.attisdropped) and a.attnum > 0 and a.attrelid = c.oid) inner join pg_catalog.pg_type t on t.oid = a.atttypid) left outer join pg_attrdef d on a.atthasdef and d.adrelid = a.attrelid and d.adnum = a.attnum order by n.nspname, c.relname, attnum
> 2022-05-23 20:10:00.407 ACST [12179] ERROR:  column c.relhasoids does not exist at character 245
> 2022-05-23 20:10:00.407 ACST [12179] STATEMENT:  select n.nspname, c.relname, a.attname, a.atttypid, t.typname, a.attnum, a.attlen, a.atttypmod, a.attnotnull, c.relhasrules, c.relkind, c.oid, pg_get_expr(d.adbin, d.adrelid), case t.typtype when 'd' then t.typbasetype else 0 end, t.typtypmod, c.relhasoids, attidentity, c.relhassubclass from (((pg_catalog.pg_class c inner join pg_catalog.pg_namespace n on n.oid = c.relnamespace and c.oid = 16535) inner join pg_catalog.pg_attribute a on (not a.attisdropped) and a.attnum > 0 and a.attrelid = c.oid) inner join pg_catalog.pg_type t on t.oid = a.atttypid) left outer join pg_attrdef d on a.atthasdef and d.adrelid = a.attrelid and d.adnum = a.attnum order by n.nspname, c.relname, attnum
> 2022-05-23 20:10:00.408 ACST [12179] ERROR:  column c.relhasoids does not exist at character 245
> 2022-05-23 20:10:00.408 ACST [12179] STATEMENT:  select n.nspname, c.relname, a.attname, a.atttypid, t.typname, a.attnum, a.attlen, a.atttypmod, a.attnotnull, c.relhasrules, c.relkind, c.oid, pg_get_expr(d.adbin, d.adrelid), case t.typtype when 'd' then t.typbasetype else 0 end, t.typtypmod, c.relhasoids, attidentity, c.relhassubclass from (((pg_catalog.pg_class c inner join pg_catalog.pg_namespace n on n.oid = c.relnamespace and c.oid = 16405) inner join pg_catalog.pg_attribute a on (not a.attisdropped) and a.attnum > 0 and a.attrelid = c.oid) inner join pg_catalog.pg_type t on t.oid = a.atttypid) left outer join pg_attrdef d on a.atthasdef and d.adrelid = a.attrelid and d.adnum = a.attnum order by n.nspname, c.relname, attnum
> 2022-05-23 20:10:00.409 ACST [12179] ERROR:  column c.relhasoids does not exist at character 245
>
>
> 2022-05-23 20:50:02.381 ACST [15274] STATEMENT:  select n.nspname, c.relname, a.attname, a.atttypid, t.typname, a.attnum, a.attlen, a.atttypmod, a.attnotnull, c.relhasrules, c.relkind, c.oid, pg_get_expr(d.adbin, d.adrelid), case t.typtype when 'd' then t.typbasetype else 0 end, t.typtypmod, c.relhasoids, attidentity, c.relhassubclass from (((pg_catalog.pg_class c inner join pg_catalog.pg_namespace n on n.oid = c.relnamespace and c.oid = 16405) inner join pg_catalog.pg_attribute a on (not a.attisdropped) and a.attnum > 0 and a.attrelid = c.oid) inner join pg_catalog.pg_type t on t.oid = a.atttypid) left outer join pg_attrdef d on a.atthasdef and d.adrelid = a.attrelid and d.adnum = a.attnum order by n.nspname, c.relname, attnum
> 2022-05-23 20:50:02.382 ACST [15274] ERROR:  column c.relhasoids does not exist at character 245
> 2022-05-23 20:50:02.382 ACST [15274] STATEMENT:  select n.nspname, c.relname, a.attname, a.atttypid, t.typname, a.attnum, a.attlen, a.atttypmod, a.attnotnull, c.relhasrules, c.relkind, c.oid, pg_get_expr(d.adbin, d.adrelid), case t.typtype when 'd' then t.typbasetype else 0 end, t.typtypmod, c.relhasoids, attidentity, c.relhassubclass from (((pg_catalog.pg_class c inner join pg_catalog.pg_namespace n on n.oid = c.relnamespace and c.oid = 16405) inner join pg_catalog.pg_attribute a on (not a.attisdropped) and a.attnum > 0 and a.attrelid = c.oid) inner join pg_catalog.pg_type t on t.oid = a.atttypid) left outer join pg_attrdef d on a.atthasdef and d.adrelid = a.attrelid and d.adnum = a.attnum order by n.nspname, c.relname, attnum
> 2022-05-23 20:50:02.383 ACST [15274] ERROR:  column c.relhasoids does not exist at character 245
> 2022-05-23 20:50:02.383 ACST [15274] STATEMENT:  select n.nspname, c.relname, a.attname, a.atttypid, t.typname, a.attnum, a.attlen, a.atttypmod, a.attnotnull, c.relhasrules, c.relkind, c.oid, pg_get_expr(d.adbin, d.adrelid), case t.typtype when 'd' then t.typbasetype else 0 end, t.typtypmod, c.relhasoids, attidentity, c.relhassubclass from (((pg_catalog.pg_class c inner join pg_catalog.pg_namespace n on n.oid = c.relnamespace and c.oid = 16405) inner join pg_catalog.pg_attribute a on (not a.attisdropped) and a.attnum > 0 and a.attrelid = c.oid) inner join pg_catalog.pg_type t on t.oid = a.atttypid) left outer join pg_attrdef d on a.atthasdef and d.adrelid = a.attrelid and d.adnum = a.attnum order by n.nspname, c.relname, attnum
> 2022-05-23 20:50:02.384 ACST [15274] ERROR:  column c.relhasoids does not exist at character 245
> 2022-05-23 20:50:02.384 ACST [15274] STATEMENT:  select n.nspname, c.relname, a.attname, a.atttypid, t.typname, a.attnum, a.attlen, a.atttypmod, a.attnotnull, c.relhasrules, c.relkind, c.oid, pg_get_expr(d.adbin, d.adrelid), case t.typtype when 'd' then t.typbasetype else 0 end, t.typtypmod, c.relhasoids, attidentity, c.relhassubclass from (((pg_catalog.pg_class c inner join pg_catalog.pg_namespace n on n.oid = c.relnamespace and c.oid = 16525) inner join pg_catalog.pg_attribute a on (not a.attisdropped) and a.attnum > 0 and a.attrelid = c.oid) inner join pg_catalog.pg_type t on t.oid = a.atttypid) left outer join pg_attrdef d on a.atthasdef and d.adrelid = a.attrelid and d.adnum = a.attnum order by n.nspname, c.relname, attnum
> 2022-05-23 20:50:02.385 ACST [15274] ERROR:  column c.relhasoids does not exist at character 245
> 2022-05-23 20:50:02.385 ACST [15274] STATEMENT:  select n.nspname, c.relname, a.attname, a.atttypid, t.typname, a.attnum, a.attlen, a.atttypmod, a.attnotnull, c.relhasrules, c.relkind, c.oid, pg_get_expr(d.adbin, d.adrelid), case t.typtype when 'd' then t.typbasetype else 0 end, t.typtypmod, c.relhasoids, attidentity, c.relhassubclass from (((pg_catalog.pg_class c inner join pg_catalog.pg_namespace n on n.oid = c.relnamespace and c.oid = 16525) inner join pg_catalog.pg_attribute a on (not a.attisdropped) and a.attnum > 0 and a.attrelid = c.oid) inner join pg_catalog.pg_type t on t.oid = a.atttypid) left outer join pg_attrdef d on a.atthasdef and d.adrelid = a.attrelid and d.adnum = a.attnum order by n.nspname, c.relname, attnum
>
>
> On 23/5/2022, 08:39, "Muris" <alc...@gm...> wrote:
>
>     Thanks Andy I did try that but didn’t work still. I actually had to reboot the box, looks like something was stuck in memory or something it didn’t like.
>
>     I track the box cpu/memory, and this is a before and after some screens I took after the reboot (see attached)
>
>     Seems semi normal now.. but discoveries seem a bit hit and miss with the workers, I think it might be getting stuck on something.
>
>     Those errors have cleared now from the logs after a reboot - box was up for 181 days.. but I see these new ones popup
>
>     Use of uninitialized value in sprintf at /home/netdisco/perl5/lib/perl5/App/Netdisco/Worker/Plugin/Discover/Entities.pm line 108.
>     Use of uninitialized value in sprintf at /home/netdisco/perl5/lib/perl5/App/Netdisco/Worker/Plugin/Discover/Entities.pm line 108.
>     Use of uninitialized value in sprintf at /home/netdisco/perl5/lib/perl5/App/Netdisco/Worker/Plugin/Discover/Entities.pm line 108.
>     Use of uninitialized value in sprintf at /home/netdisco/perl5/lib/perl5/App/Netdisco/Worker/Plugin/Discover/Entities.pm line 108.
>     Use of uninitialized value in sprintf at /home/netdisco/perl5/lib/perl5/App/Netdisco/Worker/Plugin/Discover/Entities.pm line 108.
>     Argument "AirOS" isn't numeric in numeric eq (==) at /home/netdisco/perl5/lib/perl5/SNMP/Info/Layer2/Ubiquiti.pm line 101.
>     Argument "AirOS" isn't numeric in numeric eq (==) at /home/netdisco/perl5/lib/perl5/SNMP/Info/Layer2/Ubiquiti.pm line 101.
>     Use of uninitialized value in sprintf at /home/netdisco/perl5/lib/perl5/App/Netdisco/Worker/Plugin/Discover/Entities.pm line 108.
>     Argument "AirOS" isn't numeric in numeric eq (==) at /home/netdisco/perl5/lib/perl5/SNMP/Info/Layer2/Ubiquiti.pm line 101.
>     Argument "AirOS" isn't numeric in numeric eq (==) at /home/netdisco/perl5/lib/perl5/SNMP/Info/Layer2/Ubiquiti.pm line 101.
>     Argument "AirOS" isn't numeric in numeric eq (==) at /home/netdisco/perl5/lib/perl5/SNMP/Info/Layer2/Ubiquiti.pm line 101.
>
>     On 22/5/2022, 21:32, "Andy Ruhl" <ac...@gm...> wrote:
>
>         "Cannot allocate memory" is generally not good, you need to
>         investigate that. That's the back end which doesn't affect the web
>         front end directly, but if the system is out of memory that certainly
>         would.
>
>         Maybe try shutting off the backend and see if the front end works?
>
>         netdisco-backend stop
>
>         Andy
>
>         On Sat, May 21, 2022 at 10:31 PM Muris <alc...@gm...> wrote:
>         >
>         > I had a look at the backend and front end logs and it shows this, I don’t know if this is having any impact.
>         >
>         >
>         >
>         > [16386] 2022-05-22 05:26:04  warn App::Netdisco 2.052006 backend
>         >
>         > MCE::_dispatch_child: Failed to spawn worker 17: Cannot allocate memory at /home/netdisco/perl5/bin/netdisco-backend-fg line 64.
>         >
>         >
>         >
>         > 2022/05/22-14:56:09 Starman::Server (type Net::Server::PreFork) starting! pid(16414)
>         >
>         > Resolved [*]:5000 to [::]:5000, IPv6
>         >
>         > Not including resolved host [0.0.0.0] IPv4 because it will be handled by [::] IPv6
>         >
>         > Binding to TCP port 5000 on host :: with IPv6
>         >
>         > 2022/05/22-14:56:09 Can't connect to TCP port 5000 on :: [Address already in use]
>         >
>         >   at line 64 in file /home/netdisco/perl5/lib/perl5/Net/Server/Proto/TCP.pm
>         >
>         > 2022/05/22-14:56:09 Received QUIT. Running a graceful shutdown
>         >
>         > 2022/05/22-14:56:09 Worker processes cleaned up
>         >
>         > 2022/05/22-14:56:09 Server closing!
>         >
>         >
>         >
>         >
>         >
>         > From: Muris <alc...@gm...>
>         > Date: Sunday, 22 May 2022 at 14:49
>         > To: "net...@li..." <net...@li...>
>         > Subject: Urgent Problem with Netdisco new version 2.52.6
>         >
>         >
>         >
>         > Hi All,
>         >
>         >
>         >
>         > Have upgraded to netdisco to the new version on test box without issues on 2.52.6, applied the same on production, and now any item when you click on the device the page comes up blank,
>         >
>         > I can see the side bar to the right loads then quickly disappear and the page is turning up blank when click on a device.
>         >
>         >
>         >
>         > The only thing that shows up is the ip address to the right, and a CSV icon, that’s all , all rest of the tabs are missing/and do not load.
>         >
>         >
>         >
>         > Any advice?
>         >
>         >
>         >
>         > Thanks
>         >
>         >
>         >
>         >
>         >
>         >
>         >
>         >
>         >
>         > _______________________________________________
>         > Netdisco mailing list
>         > net...@li...
>         > https://sourceforge.net/p/netdisco/mailman/netdisco-users/
>
>
>
>
>
> _______________________________________________
> Netdisco mailing list
> net...@li...
> https://sourceforge.net/p/netdisco/mailman/netdisco-users/