#8 [support] getting segmentation fault with mod_countm

[Robin Cook]

Ok have worked a bit more....

I have taken the simplest config and used it.

When I try to access the test counter page I created
from any machine but the local it doesn't run the query
url and gets a broken image.

When I run from the local machine it accesses the query
URL and sometimes displays the count sometimes it
doesn't. When it doesn't display the count image the
log shows a segmentation fault.

So question is what do I need to allow the query from
any computer

And the other problem is the segmentation fault.

During the segmentation faults are also where there can
be a double count for an access.

[Byron Young]

Logged In: YES
user_id=151326

>Ok have worked a bit more....

>I have taken the simplest config and used it.

The simplest, IE least restrictive access privileges, are:

BEGIN HTTPD.CONF SNIPPET
LoadModule countm_module /usr/lib/httpd/modules/mod_countm.so

CountmFontDir "/var/countm/fonts"
CountmDefFont "FreeMono"
CountmEnvironmentHome "/var/countm/dbase"
CountmDB "countm.db"
CountmDBLockFile "/var/countm/countm.lock"
CountmBgfileDir "/var/countm/bgfiles"
CountmActivate ON

<Location /countm>
SetHandler mod-countm
HostNameLookups ON
CountmAllowFontsQuery ON
CountmAllowReset ON
CountmDBPriority ON
Order allow, deny
Allow from all
</Location>
END HTTPD.CONF SNIPPET

An HTML 4.0 <FORM> Element Attribute ACTION URL PATH_INFO to
enter the counter into the database would be /countm.

Example:
<form action=http://my.domain.com/countm method=GET
enctype="x-www-form-urlencoded">
Somewhere in the form, an HTML 4.0 INPUT Element with an
attribute NAME link attribute VALUE linkname would set the
?link=linkname portion.
In some rare cases, a / trailing countm may be necessary.

Once the counter is entered into the database, it may be
accessed by the page being counted using a HTML 4.0 IMG
element "Query URL". For this example,

<img src="http://my.domain.com/countm?link=linkname">
Generally, only one countm IMG element per page works.

If the link linkname is not in the database (has not been
entered), then an image "ELink" should be returned, not a
broken image.

An IMG element attribute SRC value using the localhost (or
127.0.0.1) as the domain name will always return a broken
image when the accessing client is not on the local machine.

Using the IPV4 address of the server in place of any
hostnames (ie 127.0.0.1 instead of localhost) would rule out
any problematic resolver issues.

>When I run from the local machine it accesses the query
>URL and sometimes displays the count sometimes it
>doesn't. When it doesn't display the count image the
>log shows a segmentation fault.

>So question is what do I need to allow the query from
>any computer

>And the other problem is the segmentation fault.

>During the segmentation faults are also where there can
>be a double count for an access.

The countm module will add debugging info to the httpd
server logs. This may be activated by setting the Apache2
directive LogLevel to Debug. The Apache2 directive ErrorLog
normally points to the error log file. The httpd access log
for the query will also give a better idea of the requesting
URL formation.

In earlier tests, db4 was segfaulting due to the way countm
was (or rather was not) handling threads on the threaded
MPMs. Since correcting that minor (right) issue, the module
has not segfaulted. All testing up to this point has been
with StartServers set to around 8, but theoretically that
should be irrelevant.

BTW, I just saw the logs tar file you submitted. Looking
now.....

[Byron Young]

Logged In: YES
user_id=151326

Setting your access_log to combined will add a little extra
information.
Deactivate all experimental modules. Use httpd -l to see if
any are compiled into the server. If so, then use
server-info to check that the default values make sense.
With the server stopped, erase all the database information
in /var/countm/dbase with 'rm -f *'.
Restart the server and enter the link into the counter using
the mod_countm test page from the localhost.
Access the counter using the test page below. A modern
HTML/1.1 User Agent will be required as the page source
function lists two headers and two endings for the returned
page. Obviously, the hostname must be altered.
Stop the server.

All of the above was tested with FC2 libphp4.so php 4.3.4.
Something may have changed in 5.0.2.

BEGIN test.php
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>

<meta content="text/html; charset=ISO-8859-1"
http-equiv="content-type">
<title>mod_countm PHP test page</title>

</head><body>
<br>
<? phpinfo(); ?>
<br>
<img src="http://www.localzone/countm?link=linkname"><br>
</body></html>
END test.php

[Robin Cook]

Logged In: YES
user_id=5208

Ok have done all that was suggested and I am still getting
the same results.

From browser on same machine that apache is running. It
will so sometime and give a broken image on others and when
it gives the broken image the log indicates a segmentation
fault. Browsers used were Mozilla 1.7.2, Galeon 1.3.18, and
epiphany 1.4.4.

With broswers on a different machine, it just indicates a
broken image. Browsers used on second computer Firefox 1.0
and IE 6.

I am going to attach the apache log files, the test html
files. Below lists the countm config file as well as the
modules loaded in apache.

=============================================================================
mod_countm.conf

LoadModule countm_module /usr/libexec/mod_countm.so

#Global server settings
CountmFontDir "/var/countm/fonts"
CountmDefFont "FreeMono"
CountmEnvironmentHome "/var/countm/dbase"
CountmDB "countm.db"
CountmDBLockFile "/var/countm/countm.lock"
CountmBgfileDir "/var/countm/bgfiles"
CountmActivate ON

<Location /countm>
SetHandler mod-countm
HostNameLookups OFF
CountmAllowFontsQuery ON
CountmAllowReset ON
CountmDBPriority ON
Order allow,deny
Allow from all
</Location>

<IfModule mod_info.c>
AddModuleInfo mod_countm.c
"Package:mod_countm-pre5.5.0.tar.gz"
AddModuleInfo mod_countm.c "Summary:Apache2 mod_countm
DSO module pre5.5.0"
AddModuleInfo mod_countm.c "Version:pre5.5.0.tar.gz"
</IfModule>

=============================================================================

From httpd -l

Compiled in modules:
core.c
worker.c
http_core.c
mod_so.c

=============================================================================

Modules being loaded from httpd.conf

LoadModule access_module libexec/mod_access.so
LoadModule auth_module libexec/mod_auth.so
LoadModule auth_anon_module libexec/mod_auth_anon.so
LoadModule auth_dbm_module libexec/mod_auth_dbm.so
LoadModule example_module libexec/mod_example.so
LoadModule include_module libexec/mod_include.so
LoadModule log_config_module libexec/mod_log_config.so
LoadModule log_forensic_module libexec/mod_log_forensic.so
LoadModule env_module libexec/mod_env.so
LoadModule mime_magic_module libexec/mod_mime_magic.so
LoadModule cern_meta_module libexec/mod_cern_meta.so
LoadModule expires_module libexec/mod_expires.so
LoadModule headers_module libexec/mod_headers.so
LoadModule usertrack_module libexec/mod_usertrack.so
LoadModule unique_id_module libexec/mod_unique_id.so
LoadModule setenvif_module libexec/mod_setenvif.so
LoadModule mime_module libexec/mod_mime.so
LoadModule dav_module libexec/mod_dav.so
LoadModule status_module libexec/mod_status.so
LoadModule autoindex_module libexec/mod_autoindex.so
LoadModule asis_module libexec/mod_asis.so
LoadModule info_module libexec/mod_info.so
LoadModule cgid_module libexec/mod_cgid.so
LoadModule cgi_module libexec/mod_cgi.so
LoadModule dav_fs_module libexec/mod_dav_fs.so
LoadModule vhost_alias_module libexec/mod_vhost_alias.so
LoadModule negotiation_module libexec/mod_negotiation.so
LoadModule dir_module libexec/mod_dir.so
LoadModule imap_module libexec/mod_imap.so
LoadModule actions_module libexec/mod_actions.so
LoadModule speling_module libexec/mod_speling.so
LoadModule userdir_module libexec/mod_userdir.so
LoadModule alias_module libexec/mod_alias.so
LoadModule rewrite_module libexec/mod_rewrite.so

php5 and mod_countm are being loaded from include files in
/etc/httpd/conf.d

[Byron Young]

Logged In: YES
user_id=151326

From php-5.0.2/INSTALL

Do not use Apache 2.0.x and PHP in a production
environment neither on
Unix nor on Windows. For information on why, read the
following FAQ
entry

Note: To build a multithreaded version of Apache your
system must
support threads. This also implies to build PHP with
experimental
Zend Thread Safety (ZTS). Therefore not all extensions
might be
available. The recommended setup is to build Apache
with the
standard prefork MPM-Module.

Fedora released a development php5 rpm, encouraging the
ignoring of the above ominous predictions. After some spec
file tinkering, the rpm built and installed, running the
test.php without problem. After rebuilding with
--enable-maintainer-zts and switching to the worker MPM,
much to my relief, test.php ran without errors on the FC2
test system. The php folks deserve credit for coercing all
the dependent packages under the thread umbrella. FC3
enables se-linux by default. Any policy that denys
mod_countm/httpd access to database files will result in
unforseen errors, which may or may not result in broken
images. After upgrading somtime in the future, those
possible problems will be sorted out.

[Robin Cook]

Logged In: YES
user_id=5208

I'm not running this in a production envrionment at the
moment and I don't run FC so the se-linux won't be a problem
either. I run a source based distribution called
SourceMage. But I will tinker with the php build and see if
adding the --enable-maintainer-zts fixes the problems I am
seeing. If I have to I will look at rebuilding apache
with the prefork though I would rather not if I don't have to.

[Robin Cook]

Logged In: YES
user_id=5208

Ok tried with the --enable-maintainer-zts and it still had
the same results.
I then recompiled apache 2.0.52 with the pre-fork and php
without the --enable-maintainer-zts and I still have the
same problem with the segmentation fault.

[Robin Cook]

Logged In: YES
user_id=5208

Ok I have tried the same thing with PHP 4.3.9.
Apache 2.0.52 with MPM-Worker and PHP 4.3.9
--enable-maintainer-zts: Same results
Apache 2.0.52 with Pre-Fork and PHP 4.3.9 without zts: same
results.

It sometimes displays the count sometimes not and when it
doesn't it looks like it is segfaulting. Also it only
works with a browser on the same machine as the web server.
From a different machine it never displays the counter.

[Byron Young]

Logged In: YES
user_id=151326

Automated response: Project countm.

An automated response is generated when a countm support request
is received for an unsupported system.

The countm project is developed, tested, and packaged for two
supported systems; Fedora Core 2 (i386) and FreeBSD 5.3-RELEASE
(i386).

The coding style, practices and intent is to maintain
compatibility with any system that may run the Apache2 server.

Direct attempts to resolve issues concerning unsupported systems
will be limited to verifying the general configuration is
correct, and, assuming the necessary information is supplied,
attempting to reproduce the issue on the test systems. Once the
general countm configuration is proved to be correct, and the
same configuration, packages, and associated web-pages do not
reproduce problems on the test systems, the options to resolve
the errors and issues are greatly diminished.

Many errors are not countm problems, but are problems that effect
countm. These are usually thread issues, dependent package build
configuration issues, and/or (but by no means limited to) C
library implementation issues. Integrating solutions into countm
for even easily solved issues of these types require developer
access to the systems experiencing the problems. As an example,
the RedHat 9 i586 system required a non native threads posix
linux (non-ntpl) version of DB4, but RedHat 9 i686 systems
required a ntpl version of DB4. For countm to run properly on
RedHat 9 i586 systems, DB4 must be rebuilt. A sample spec file
patch to do this is included in the countm manual. The solution
required access to log files, access to database files, access to
httpd configuration files, access and knowledge of the system
packaging mechanism, editing, rebuilding and reinstalling a
distribution package, and, building and running a debugger
version of Apache2. For this particular example, all of this was
required, without the need to alter one line of countm code.

On the other hand, many errors are countm problems. Earlier
versions of countm incorrectly assumed the Apache2 API hide all
thread issues from the DSO module. Although the Apache2 API hides
many thread issues related to Apache2, it does not hide, and
probably never may hide, thread issues for dependent packages and
libraries. The solution to this issue required rewriting large
sections of countm code, reassessing and altering the entire
countm module model, redesigning and rewriting the countm DB4
interface implementation, and adding extra countm code to support
libgd. Most of this was added to support the threaded worker MPM
and tests correctly on the supported development systems.

Errors that are not countm, or package dependency, or threads, or
C library implementation issues, but affect countm may be other
system configuration issues, including, but not limited to,
gateways (as defined by HTTP 1.1), gateways (forwarders,
routers), firewalls, DNS settings and resolver settings.

For all tests, set the Apache2 logging to debug and combined.

Test1
See the manual, and verify the ex_*.html tests work properly.
Save the log output.

Test2
From a remote client, perform a countm fonts query. If the list
of installed fonts is returned, then you may rule out most
configuration issues.

Test3
With the server stopped, erase the countm db4 database files, and
restart the server. From the localhost using ex_enter.html enter
a link into the database, and, from a remote client, access a
page on the localhost (the server running countm) that is being
counted.

Test4
From the localhost, using ex_enter.html, enter another link into
the database, and, from a remote client, access a page on another
server (a server not running countm) that is being counted (by
the server running countm).

If the above tests properly, and errors with other link entry
pages and query pages are encountered, then, the countm project
will accept, via posting to the countm support forum the
associated documents. Please mention in the posing that
additional information was attached and submitted.

The access_log showing the counter entry information.
The access_log showing the counter query information.
The page used to enter the link into the database.
The page used to query the counter.

NOTE: All information submitted should have the IP addresses and
hostnames changed. Example:
sed -i -e 's/121.212.121.212/111.111.111.111/g' log_file
sed -i -e 's/my.domain.com/www.example.com/g' log_file

NOTE: All information is submitted to the public domain. Any
person may download it and use it for their own purposes. It may
be added into the countm manual as examples and/or solutions to
problems.

If the submitted information does not reproduce the problem on
the supported test systems, then the only other option is to
allow direct root access to your systems. This is not guaranteed
to yield results and is a decision that should be fully
understood prior to being made.

Many issues must be resolved prior developer access to your
unsupported system.

The system must be a recent 'stock' developer (preferably
everything) installation of a mainstream publicly downloadable
distribution. It is recommended that the system be a new
installation that a #( cd /; rm -rf *; ) may be run on and no
important information be lost.

The system must be connected to the public Internet and allow
downloading and installing of system distribution packages. If a
countm dependent package is not available as a system
distribution package, then problem resolution must be stopped.

The countm distribution must be installed in such a manner that
the ex_*.html example, as outlined in the manual, is fully
operational. If the ex_*.html test/example does not work, then
problem resolution will be stopped.

The httpd server must be accessible from the public Internet and
a http://host/manual query must return the default installed
manual. In this case, host may be a dynamic IP address, static IP
address, or a registered domain name.

Any material developed or created must be allowed to be relocated
from the system and may or may not be integrated into the countm
distribution. This includes backups.

The system must be available for use at times convenient for
developers for as long as necessary to resolve the issue.

If during problem resolution the system becomes broken or
unusable, a new installation must be provided.

After the problem resolution is stopped, the system must be
erased and reinstalled.

If, during problem resolution, either developers or system
providers wish to stop problem resolution for whatever reason,
the developers will signal intent to do so by quitting problem
resolution, and the system providers will signal intent to do so
by denying developers access to the system.

The system must be owned and operated directly by the individual
attempting to solve the problem, or, have permission from the
owners to allow outside developer access.

The above is some of the more important issues to consider.
Basically, the countm developers are interested in resolving all
issues related to Linux distributions and are willing to work
towards a solution.

[Byron Young]

Logged In: YES
user_id=151326

To understand the logs, I'll need some more info.

The pages test.php and test2.php are physically located on
cuzndragon.in.eisd.net in the root webspace directory? What
is the IP address for cuzndragon.in.eisd.net as seen by
remote clients on the local intranet?

The public IP address 10.2.0.3 and 10.2.6.23 are aliases on
an cuzndragon.in.eisd.net interface which is running the
counter.

The apache2 server running the counter and generating the
cuzndragon-error_log is generating no errors, and is running
on the cusndragon.in.eisd.net host.

What machine (local intranet ip address) is generating
access_log and error_log? That is the machine generating the
segfaults.

Generally, it seems like your configuration is correct, but
the location of the counter, and the page being counted may
be getting confused by the requesting client. Although, this
should not cause a segfault, on either the server running
the counter or serving the page being counted, which is
puzzling.

Also, I am testing my new canned response for unsupported
systems. Please ignore it, as it does not apply to this thread.

[Robin Cook]

Logged In: YES
user_id=5208

Yes the test pages are in /var/www/html which is the root
for cuzndragon and default vitural server. The IP address
is 10.2.0.3 for the server the 10.2.6.23 is the windows
system that I also tried testing the counter (the one with
IE and Firefox.)

I have error_log as the default log and the other log is the
one in the virtual server section.

The 10.2.0.3 the cuzndragon.in.eisd.net that is running
apache2 is the one doing the segfaulting. The only logs
with something in them are the ones I uploaded. I move the
old logs out and restart apache2 before I start doing any
testing.

[Byron Young]

Logged In: YES
user_id=151326

A setup similar to the faulting system has been setup as
follows:
The global server defines the counter databases via
mod_countm.conf (simplest)
The global and virtual servers use the global counter.
The default virtual server (using * in the directives) uses
the global counter. This could be altered to no '*' in the
directives, which would change the default virtual server.
The logs are split. The global server uses access_log and
error_log, and the virtual servers use virt-access_log and
virt-error_log.
The default user access (~users) is activated.
The test system has two ethernet interfaces, without any
aliases.
The Apache2 server listens on all interfaces (for testing
purposes.)

A remote client query from a machine on the subnet of the
first ethernet interface to global server test.php and
test2.php (modified to use the ip address of the interface;
In your case, this probably means 10.2.6.23) causes no
faults. When test.php and test2.php are moved to a user
public_html directory, it causes no faults, when moved to
any other virtual server roots, it causes no faults.

Theoretically, only the number of 'unique databases' should
matter. For this setup, only one is defined. The number of,
and the name of, the global and virtual servers should be
irrelevant.

The problem may be with the link entry. The command
line/query line parsing may be setting some values
incorrectly, which are causing segfaults. Submitting an
abbreviated script/html page and the appropriate log
snippets, especially the access_log showing the entry of the
link into the counter, may help. If you are using telnet to
enter the link, send a session capture.

[Robin Cook]

Logged In: YES
user_id=5208

There is definately some interaction between php and
mod_countm. When I have PHP commented out of the httpd.conf
mod_countm works perfectly. When I load PHP (php 4 or php
5, pre_fork, or threaded compiled) mod_countm starts having
problems. Even on test files that don't even have php code
in it, mod_countm will randomly multiple count a single
access or segfault.

But all the other PHP pages and apps I have work as normal.
I am stumped.

Not sure what else could be the problem.

linux - 2.6.8.1
gcc - 3.3.3
glibc - 2.3.3 with NPTL enabled
apr - 0.9.5 from CVS
apu - 0.9.5 from CVS
apache - 2.0.52
php - 5.0.2
gd - 2.0.33

[Robin Cook]

httpd log files from last test.

[Robin Cook]

httpd & countm confs, html files and phpinfo page

[Byron Young]

Logged In: YES
user_id=151326

I'll consider this a "Seems to test okay, but may be
incompatible when integrated into mod_php PHP code." bug
report. My server runs phpGedView, poplargedcomviewer (both
PHP apps) and mod_countm. However, I've never actually
tested mod_countm with mod_countm interaction embedded into
PHP code. Once I extract myself from this SELinux quagmire
of 330,000+ rules and 1400+ avc errors, I'll investigate
further.

[Byron Young]

Logged In: YES
user_id=151326

The Current_5_4 update solves a similar segfault condition
detected on x86_64 systems.