[NR-devel] Successful Install of NR 4.0 on *ubuntu 16.04

[Dustin M. <du...@gm...>]

The following is an account of my successful installation of the
unstable/development NR 4.0 ('default' branch compressed file
downloaded from BitBucket on/around 3 Jul 2017) on a Lubuntu 16.04.2
LTS system that did not previously have any version of NR installed on
it. I haven't tested it, but I'm guessing it's highly likely this
should work for any *ubuntu 16.04.x system and perhaps earlier/later
versions that use 'systemd' and for which there is a compatible
version of the POCO libs available.

Short Version: (What I'm 99% sure I should have done, but my actual
process was much more convoluted because of trial-and-error
troubleshooting.)

*** Made sure the following general build-related packages (+depends,
if any) were installed ( all in 16.04 repos):
build-essential
autoconf
libtool

*** Made sure the following POCO build-related packages were installed:
OpenSSL (in 16.04.2 by default, 'openssl')
???MySQL Client??? (16.04 repos: 'libmysqlclient-dev') (90% sure this
isn’t needed.)

*** Installed the following NR build-related packages:
libpcap-dev (16.04 repos)
libssl-dev (16.04 repos) (may also be necessary for POCO build, not sure)

*** Compiled/installed POCO C++ libraries (‘Complete Edition’) (v
1.7.8p3) from source:
./configure --shared --no-tests --no-samples
make
sudo make install

*** Compiled/installed NR 4.0 from source:
autoreconf -i      # [DM note: slight chance running w/ options '-vfi'
is needed, not sure.]
./configure --prefix=/usr
make
sudo make install

* Followed prompts for entering NR username and password to complete install.
* Performed a few tests to ensure it was functioning as it should.
Still some bugs, but these don't hinder basic functioning AFAIK.

******************************

Long Version: (This includes more detailed info/notes that may be
helpful for users, testers and/or developers.)

Background Notes:
* Currently v3.0.3 is the latest stable version but it no longer works
(can't send emails, etc.), so the unstable development v4.0 is the
only working one available.
* For 16.04 (and presumably other releases using 'systemd') you want
to use the “default” branch from the repo
(https://bitbucket.org/netresponsibilityteam/net-responsibility/overview).
You can clone the repo to get it, or you can get a compressed file via
“Download Repository” link
(https://bitbucket.org/netresponsibilityteam/net-responsibility/downloads/?tab=downloads).
* I decided to do my testing of this install on a non-production test
system so I would feel more comfortable doing any trial-and-error
troubleshooting I desired.
* Currently neither the regular NR PPA (ppa:roggan87/nr) nor the
development one (ppa:roggan87/nrdev) can be used to successfully
install the POCO libs or NR 4.0.)
* If you are using a version of NR prior to v4.0, you may want to
uninstall/remove that before installing NR 4.0, although I'm not sure
it's necessary.
* A handy feature in some *ubuntu releases is a function key ('F4' in
Lubuntu 16.04) or a right-click context menu option (i.e. 'Open in
Terminal') that will open the active file manager directory (folder)
in the terminal. This may be available by default or you may need to
install a package from the repos (e.g. a nautilus plugin for Ubuntu)
to get this functionality.

POCO build notes:
* For OpenSSL depends, it's possible/probable that besides 'openssl'
you also need to have 'libssl-dev' installed. If that's the case, then
it would make more sense to include the 'libssl-dev' install in the
POCO build section in my 'Short Version' above. Instead of testing
this, I just made sure its install was listed before the POCO build
was.
* It seems that if the configure script is not able to find the
necessary external dependencies (installed or not) for certain POCO
parts, then it omits these from the build by default, so no need to
specify their omission unless you have the depends installed in a way
that it will find them.
* I'm not sure if there is a command to run that reverses the effects
of the './configure' command, but it seems that simply running
'./configure' again will make any desired changes.
* I wasn't able to easily find a command that uninstalls the POCO
libs. ('sudo make u‌ninstall' doesn't work.) (If there isn't one, then
I guess the best workaround is manual removal of files.)
* Need to make sure you have the 'Complete Edition' not just the
'Basic Edition'.
* Basic process: Download compressed file, extract its contents, and
run the build commands (see 'Short Version' above) from within the
trunk/source directory. (If you want to redo the compile/build, you
can run ‘make clean’ to clean up the source code from the effects of
the previous ‘make’ command you ran.) (To change the configuration
options, I think you just run './configure' again with the desired
options and it will change things as necessary.)
* In order to avoid potential permissions errors, probably the most
pain-free approach is to clone/extract the source code to a dir within
your home dir and then run the build/install commands fr within that
dir. (If that sounds odd to you or you disagree, then good chance
you're quite capable of dealing with any permissions errors that might
arise. :)  )

nrdev PPA POCO install notes: (ultimately unsuccessful for NR 4.0 install)
* Depending on what's currently in the PPA and what version of *ubuntu
you're trying to install 'libpoco-dev' on, you may run into some
depends issues. Following is what I needed to do to install
libpoco-dev (v 1.6.1-11~2209~ubuntu14.04.1) on Lubuntu 16.04.2 LTS.
(One thing to note is that if Synaptic is not acting right or giving
ambiguous errors, you can try apt-get on the CLI and it might give you
more info.)
* The 'libpoco-dev' pkg depends on the packages
'libpocodatamysql-31.dbg' AND 'libpocodatamysql-31.rel' and these
packages depend on 'libmysqlclient18' (>= 5.5.24+dfsg-1), and a search
for libmysqlclient in Synaptic showed I had 'libmysqlclient20' v
5.7.18-0ubuntu0.16.04.1, as well as a dev version of it, available for
install.
* The solution I used: Got a 'libmysqlclient18' deb pkg from
(https://www.ubuntuupdates.org/package/core/trusty/main/updates/libmysqlclient18
[accessed 7/3/17].) (Looks like it was in the 14.04 repos but the
16.04 repos only have v20.) Site seems legit. Checked depends and
pre-depends and I had everything installed already or available in the
repos to allow this to install. Once I had installed it, I was able to
install libpoco-dev and its depends w/ Synaptic w/o any errors or
complaining.
* An alternative solution I didn't try was to force the install of
libpoco-dev w/ 'apt-get -f install libpoco-dev' (might complain, but
things might work if you don't actually need those dependencies).
* Unfortunately, in spite of my 'solution' above, the install of these
v 1.6.x POCO libs was 'broken' somehow in the sense that it didn't
allow for building NR 4.0.
* Ultimate solution: Compile/install the latest stable release of the POCO libs.

NR 4.0 build notes: (A couple of the generic notes from the POCO build
section apply here as well.)
* For the 'autoreconf' command I'm pretty sure that using only the
option '-i' should work. The option '-v' (verbose) just gives more
output and the option '-f' (force) considers "all files obsolete"
which presumably does a complete refresh of the relevant dir contents,
which may be necessary in some instances.
* When I ran 'make' while trying to use POCO v1.6.x from the nrdev
PPA, it gave lots of errors about undefined references. Tried w/
'autoreconf -vfi' but didn’t help. Restarting system didn’t help.
Purging/reinstalling poco v1.6.x didn’t help. Running the sequence
[make clean  //  ./configure  //  make] didn’t help. Running the
sequence [make clean  //  ./configure --prefix=/usr  //  make] didn’t
help. It finally worked when I used the latest POCO libs (7/3/17: v
1.7.8p3) compiled from source.
* When I ran 'sudo make install' after having used the './configure'
command without specifying the prefix with the option '--prefix=/usr',
this gave errors related to the shared library ‘libPocoNet.so.48’
being missing. I initially thought this might be caused by some parts
of POCO not being installed but it turned out that wasn't the problem.
* I ran 'locate libPocoNet.so.48', but it didn't find anything,
meaning the file wasn't on my system (turns out it actually was), the
link to it was broken (?), or that it couldn't find the location of
the file for some reason (likely?). I did confirm then that the file
was in poco’s (trunk) /lib/Linux/i686 dir, as well as a copy of it in
/usr/local/lib.
* I thought perhaps that this might be a problem related to the
library or library path not being included in the shared library
cache, but printing the cache by running ‘ldconfig -p’ showed that
they were both present there.
* Running the sequence [make clean  //  ./configure --prefix=/usr  //
make  //  sudo make install] finally got rid of these errors and
allowed me to successfully install NR.
* After entering my username and password, it seemed to be able to
configure and set up things okay, except for the following error
output (which likely indicates a bug but doesn't seem to hinder the
successful install/functioning of NR 4.0):

terminate called after throwing an instance of 'Poco::SystemException'
  what():  System exception ./scripts/postinst: line 68: 21686
Aborted                 (core dumped) net-responsibility --install

NR 4.0 testing notes:
*The “Check NR” script that worked with some previous versions doesn't
work with v4.0.
* Confirmed that I received the install report email (generated during install).
* Running 'ps -ef|grep net-responsibility' tested OK. (If the only
line of output you see contains only 'grep', then NR is not running as
it should. However, if you see an additional line that contains
'net-responsibility --daemon' then it is probably running as it
should.)  (Note that this may not be a fool-proof test for every type
of problem/bug.)
* Manually sending a report by running 'sudo net-responsibility -r'
worked OK before a restart of the system. Also worked after a restart,
except for the following abnormal/error output coming right after the
‘Report finished’ line (I'll refer to this later as "#1 POCO
exception"):

Another instance of the sniffer is already running
terminate called after throwing an instance of 'Poco::SystemException'
  what():  System exception
Aborted (core dumped)

This also triggered a Lubuntu pop-up window with the message “System
program problem detected”, which I think also occurred when the NR
install threw a POCO exception.
* Running tests for configuration, reports, and the sniffer by running
'sudo net-responsibility -t' worked worked OK before a restart of the
system. It also partially worked after a restart, except for not
receiving the test report and getting the following abnormal/error
output coming right after the ‘Connecting to database’ line and w/ no
subsequent output (I'll refer to this later as "#2 POCO exception"):

“Aborted (core dumped)”

So I tried testing just sending the test report by running 'sudo
net-responsibility --test-report' (option '-tr' won't work because
it's "ambiguous"). This gave the #1 POCO exception error output but I
also received the test report email this time.
* Note: For tests including the sending of a report, full success
implies that the report was also received by email.
* Changed my online config, ran a config update ('sudo
net-responsibility --update-files') (option '-uf' won't work because
it's "ambiguous") and confirmed my changes were implemented:
-- Went to online settings, ‘Report parts’, [un]checked box of 2nd
‘Shutdowns’ entry, and saved settings.
-- Updated config ('sudo net-responsibility --update-files') (this
throws the #1 POCO exception).
-- Sent a manual report ('sudo net-responsibility -r'), and looked at
attached part of the manual report I received by email to see if the
'Shutdowns' section was absent/present.
* After changing my config, confirmed that I received the config change email.
* Restarted the system and did/redid some tests (see notes above).
* Note that on restart (likely whenever system starts up) I also get
the Lubuntu pop-up window w/ the message “System program problem
detected”, likely caused by a POCO exception being thrown when NR
starts automatically. (BTW, this is probably the reason for similar
error[s] I'm getting on startup of Ubuntu16.04 on another box after
upgrading from 14.04.)
* I'm in the process of checking to make sure the report frequency is
happening right:
-- In online settings, ‘Tweaks’, changed ‘Report frequency’ from ‘7’
to ‘1’ (days) for quicker results.
-- I'm guessing it will work OK. I haven't run into the problem of
reports being sent every 10 minutes that some have experienced on this
and/or previous versions.
* I think if you run ‘sudo net-responsibility and it returns ‘Other
instance found’ then that likely means it is already running okay.

Possible/Definite Bugs in NR 4.0:
* The CLI options '-tr' and '-uf' won't work because they are
"ambiguous". The workaround is to use the longer forms '--test-report'
and '--update-files'.
* Whatever is causing the various POCO exceptions referenced above.
* In the attached report, for domains with more than two periods in
them (eg. www.google.com.bd), they are over-truncated in the
folder/dir name (eg. 'com.bd' instead of the expected and preferred
'google.com.bd').

Let me know if you have any questions.

For developers, let me know if you want me to file any of the
above-mentioned bugs in the bug-tracker, and I'll try to get to that
sometime.

For those who have more knowledge and experience on any of the things
I've covered here, feel free to chime in with any helpful corrections
or clarifications.

Glad to finally have a successful install, :)
Dustin