Query failure when comparing with non materialized value

[gjdev]

The following query fails when the query binding of arg is set to a value that is not available as a term in the database:

SELECT ?subj WHERE {
  ?subj <os:prop> ?val .
  FILTER(STR(?val) != ?arg)}

I discovered this because the logs of our CI are filled with com.bigdata.rdf.internal.NotMaterializedException? reports. But on queries that compare for equality (=) instead of !=, and those return the correct value even after the ConditionalRoutingOp? is shut down because of this exception.

Other than the obvious query result failure, it actually worries me that queries will silently (only logging) ignore unexpected exceptions and return possibly incorrect results instead of reporting the unexpected failure as a SailException? to the caller. Instead of shutting down this bop after catching an exception, the exception should have propagated to the thread that started the query in the first place.

Example of logged stacktraces:

WARN : 3309      com.bigdata.journal.Journal.executorService3 com.bigdata.util.concurrent.Haltable.logCause(Haltable.java:418): com.bigdata.util.concurrent.Haltable@de16483 : isFirstCause=true : java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: DummyIV
java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: DummyIV
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1106)
	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:704)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:617)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: DummyIV
	at com.bigdata.rdf.internal.constraints.SPARQLConstraint.accept(SPARQLConstraint.java:128)
	at com.bigdata.bop.bset.ConditionalRoutingOp$ConditionalRouteTask.call(ConditionalRoutingOp.java:168)
	at com.bigdata.bop.bset.ConditionalRoutingOp$ConditionalRouteTask.call(ConditionalRoutingOp.java:1)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1105)
	... 9 more
Caused by: com.bigdata.rdf.internal.NotMaterializedException: DummyIV
	at com.bigdata.rdf.internal.AbstractIV.getValue(AbstractIV.java:659)
	at com.bigdata.rdf.internal.constraints.CompareBOp.accept(CompareBOp.java:154)
	at com.bigdata.rdf.internal.constraints.XSDBooleanIVValueExpression.get(XSDBooleanIVValueExpression.java:60)
	at com.bigdata.rdf.internal.constraints.XSDBooleanIVValueExpression.get(XSDBooleanIVValueExpression.java:1)
	at com.bigdata.rdf.internal.constraints.SPARQLConstraint.accept(SPARQLConstraint.java:111)
	... 14 more
WARN : 3310      com.bigdata.journal.Journal.executorService3 com.bigdata.util.concurrent.Haltable.logCause(Haltable.java:418): com.bigdata.util.concurrent.Haltable@de16483 : isFirstCause=false : java.lang.InterruptedException
java.lang.InterruptedException
	at com.bigdata.util.concurrent.Haltable.cancel(Haltable.java:220)
	at com.bigdata.bop.engine.AbstractRunningQuery.cancel(AbstractRunningQuery.java:1024)
	at com.bigdata.bop.engine.AbstractRunningQuery.halt(AbstractRunningQuery.java:995)
	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1108)
	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:704)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:617)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

[gjdev]

Testcase that demonstrates the issue. Run in quads mode.

[thompsonbry]

The QueryEngine? notes the first cause of an error on a query. The logged messages indicate this first cause with

<pre>
isFirstCause=true
</pre>

For the stack trace above, it appears that the nested exception was caught by the SPARQLConstraint, which is providing the effective boolean value (EBV) of the value expression. The try/catch logic there is:

	public boolean accept(final IBindingSet bs) {

		try {

			// evaluate the EBV operator
			final XSDBooleanIV iv = get(0).get(bs);
			
			return iv.booleanValue();

		} catch (Throwable t) {

			if (InnerCause.isInnerCause(t, SparqlTypeErrorException.class)) {

				// trap the type error and filter out the solution
				if (log.isInfoEnabled())
					log.info("discarding solution due to type error: " + bs
							+ " : " + t);

				return false;

			}

			throw new RuntimeException(t);

		}

	}

That logic appears to be correct. The stack trace also shows that the exception was rethrown out of SPARQLConstraint and caught in the ChunkedRunningQuery?. The relevant code block there is:

        public Void call() throws Exception {
            if (log.isDebugEnabled())
                log.debug("Running chunk: " + this);
			try {
	            ft.run(); // run
				ft.get(); // verify success
			} catch (Throwable t) {
				halt(t);  // ensure query halts.
				if (getCause() != null) {
					// abnormal termination - wrap and rethrow.
					throw new Exception(t);
				}
				// otherwise ignore exception (normal completion).
			}
            // Done.
            return null;
        } // call()

    } // class ChunkTask

The logic in halt(cause) should be such that getCause() will report non-null iff this is an abnormal termination (various exceptions whose root cause indicates that the query was interrupted are treated as normal termination. Such interrupts are generated when Sesame closes an iterator reading solutions from the query and do not represent an error in the query evaluation.)

Running the unit test, it appears that the exception was trapped in the ChunkedRunningQuery? and set on the query. However, it was not reported back to the thread draining the query solutions. I will look into why. I have filed a separate issue for this as the failure to report the error is a different problem from the NotMaterializedValue? exception.

The attachment has been incorporated into CI in the 1.0.0 release branch and the development branch as TestTicket?355.

A separate issue has been filed for the failure to report the error back to the thread draining the query solutions. See [1].

[1] https://sourceforge.net/apps/trac/bigdata/ticket/356

Committed revision r4945.