Within the last week, I have been getting a failure to restart on ncidd. This happened after my most recent update of the blacklist file using the FCC database. On my system and after the most recent update, the blacklist file size (which includes the numbers I have manually entered) was 587626. That file size is about 10X larger than it was in December 2015. I tried to use the immediately preceeding version of the blacklist file, (size about 567692) but that still didn't work. So, I stripped out all the FCC numbers, leaving only the numbers I have manually entered, and I was able to start/restart. The file size of the stripped blacklist is now only about 60000. So, somehow I think we are running up against some type of size limit, and as the FCC file gets larger and larger, more people will start having problems.
Chris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm running on a Raspberry Pi 1 and have no issue. However, I have not
restarted it in a long time so I should give it a try. I was also thinking
of adding another criteria to throw out all complaint over one year old and
then digesting that data. If that doesn't happen it may be that, one day,
all numbers will be on the list as the telemarketers just randomly put in
numbers.
Hi Chris, Does ncidd.log indicate an abort? Or does the RPi just hang?
The entire alias, blacklist, and whitelist files are read into memory so you could have a memory problem.
The Raspberry Pi has 1 GB RAM but uses virtual memory. The CPU pages into 1 or more swap files which can be up to 2GB each. It could be the swap file is too small for the large FCC file. Try using the free command to display amount of free and used memory.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I apologize for "being away"...I didn't know NCID was being so actively maintained (I think I have my RPi running a modified v1.4)!
I glanced at the new source for 1.7 (and will probably get back to upgrading to that soon), but I just wanted to offer that I have an "fcc downloader" script I either wrote or modified from somewhere (I'm honestly not sure) that downloads the latest from the fcc site, combines it with your local blacklist, etc. and then restarts ncidd.
Something I didn't notice discussed - and I don't know where you guys are now downloading your fcc list - is that the fcc list needs to be de-dup'ed before using it. Happy to share it with anyone who wants it if there's not an official one in the latest distro.
BTW: I had been using the opendata.fcc.gov with the /resource/ URI, but I found that it was not downloading anything past the first page, so I switched to the raw download of the csv and then massage that.
-AJ
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ah, that's good info. I didn't notice that in the User Manual (I obviously did not re-RTFM).
So the difference between what I'm generating and what you have seems to be that you are "giving the benefit of the doubt" and only including those that are reported at least 3 times. That reduces false-positives, for sure.
Thanks for pointing me in the right direction!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Within the last week, I have been getting a failure to restart on ncidd. This happened after my most recent update of the blacklist file using the FCC database. On my system and after the most recent update, the blacklist file size (which includes the numbers I have manually entered) was 587626. That file size is about 10X larger than it was in December 2015. I tried to use the immediately preceeding version of the blacklist file, (size about 567692) but that still didn't work. So, I stripped out all the FCC numbers, leaving only the numbers I have manually entered, and I was able to start/restart. The file size of the stripped blacklist is now only about 60000. So, somehow I think we are running up against some type of size limit, and as the FCC file gets larger and larger, more people will start having problems.
Chris
I left out an important piece of information. I am running on a RaspberryPi (4 processor version) which I think is an RPI3.
Hi Chris,
I'm running on a Raspberry Pi 1 and have no issue. However, I have not
restarted it in a long time so I should give it a try. I was also thinking
of adding another criteria to throw out all complaint over one year old and
then digesting that data. If that doesn't happen it may be that, one day,
all numbers will be on the list as the telemarketers just randomly put in
numbers.
Mike
On Thu, Oct 6, 2016 at 3:03 PM, Chris Holm holmchris@users.sf.net wrote:
Hi Chris, Does ncidd.log indicate an abort? Or does the RPi just hang?
The entire alias, blacklist, and whitelist files are read into memory so you could have a memory problem.
The Raspberry Pi has 1 GB RAM but uses virtual memory. The CPU pages into 1 or more swap files which can be up to 2GB each. It could be the swap file is too small for the large FCC file. Try using the free command to display amount of free and used memory.
Problem solved.. I had another process running as well. Once that was stopped, things seem to be back to normal
I apologize for "being away"...I didn't know NCID was being so actively maintained (I think I have my RPi running a modified v1.4)!
I glanced at the new source for 1.7 (and will probably get back to upgrading to that soon), but I just wanted to offer that I have an "fcc downloader" script I either wrote or modified from somewhere (I'm honestly not sure) that downloads the latest from the fcc site, combines it with your local blacklist, etc. and then restarts ncidd.
Something I didn't notice discussed - and I don't know where you guys are now downloading your fcc list - is that the fcc list needs to be de-dup'ed before using it. Happy to share it with anyone who wants it if there's not an official one in the latest distro.
BTW: I had been using the opendata.fcc.gov with the /resource/ URI, but I found that it was not downloading anything past the first page, so I switched to the raw download of the csv and then massage that.
-AJ
Hi AJ,
Welcome back. :)
There have been improvements on how local NCID instances can access a trimmed down version of the FCC data.
You can read more about it here:
CallerID Blacklist from FCC Data (main section in User Manual)
How FCC Data Retrieval Works
Ah, that's good info. I didn't notice that in the User Manual (I obviously did not re-RTFM).
So the difference between what I'm generating and what you have seems to be that you are "giving the benefit of the doubt" and only including those that are reported at least 3 times. That reduces false-positives, for sure.
Thanks for pointing me in the right direction!