[VuFind-Tech] alphabetic browse indexing: not finding SQLite library

[Tod O. <to...@uc...>]

vufind-tech,

I have a problem with one machine that's unable to create the SQLite indexes because the browse-indexer can't find the SQLite code. The problem is unique to this host, other local are building the browse indexes just fine. The jar file being executed is a local version of the browse indexer, and it worked just fine on this host a few weeks ago. Clearly something has changed on this host, but I've been unable to identify what, so I turn to the list.

Here's the failure message that we see on the one host:

diamond% java -Dfile.encoding=UTF-8 -cp 'browse-indexing-multi-title.jar:../solr/lib/*' CreateBrowseSQLite sorted-title.tmp title_browse.db                        
Exception in thread "main" java.sql.SQLException: Error opening connection
	at org.sqlite.Conn.open(Conn.java:165)
	at org.sqlite.Conn.<init>(Conn.java:87)
	at org.sqlite.JDBC.createConnection(JDBC.java:113)
	at org.sqlite.JDBC.connect(JDBC.java:87)
	at java.sql.DriverManager.getConnection(DriverManager.java:620)
	at java.sql.DriverManager.getConnection(DriverManager.java:222)
	at CreateBrowseSQLite.create(CreateBrowseSQLite.java:128)
	at CreateBrowseSQLite.main(CreateBrowseSQLite.java:154)
Caused by: java.lang.Exception: Error loading native library: /org/sqlite/native/FreeBSD/amd64/libsqlitejdbc.so
	at org.sqlite.SQLiteJDBCLoader.loadSQLiteNativeLibrary(SQLiteJDBCLoader.java:241)
	at org.sqlite.SQLiteJDBCLoader.initialize(SQLiteJDBCLoader.java:63)
	at org.sqlite.NativeDB.load(NativeDB.java:50)
	at org.sqlite.Conn.open(Conn.java:161)
	... 7 more

Can't find the SQLite native library code, that seems straightforward. 

On successful host magma, truss shows that java finds the SQLite JDBC and native object code here:

	/data/magma/vufind2/solr/lib/sqlitejdbc-native.jar
	/data/magma/vufind2/solr/lib/sqlite-jdbc-3.7.15-SNAPSHOT.jar
	/usr/local/lib/libsqlitejdbc.so
	/usr/local/lib/libsqlite3.so.8

These files all exist on the unsuccessful host, with the same checksums. Same for browse-indexing-multi-title.jar. So both hosts have the same object code. Java is at the same version, OpenJDK Runtime Environment (build 1.6.0_32-b27).

What _is_ different is that on the unsuccessful host, according to truss:

1. java opens the sqlite jar files in the opposite order, and
2. _never_ stats for libsqlitejdbc.so.

That second bit is what gets me. The successful host stats for the shared object file in the various java and system libraries, the unsuccessful host NEVER call stat() or access() or in any way looks for libsqlitejdbc.so.

Looking at the dynamic linking on both systems, LD_LIBRARY_PATH is not set, setting it does not seem to make a difference, and both systems list /usr/local/lib in "search directories" /var/run/ld-elf.so.hints. So I _think_ that the load library configuration is _not_ to blame here.

So what does that leave? Some conditional in the Java code? a config file I'm not thinking of? I'm almost tempted to walk through the command-line debugger, so any ideas would be most welcome.

Best,

-Tod