#198 RegularExpressionAnalyser cannot be correctly applied on headers field

[Guido]

Dear Sir/Madam,

We have documents in a database, which we serve on our website. We would like to index and store the database id of these documents by supplying the database id as a "X-OSS-ItemKey" response header, but we run into multiple issues. Either I don't understand how to configure this process or I ran into some bugs. I would be grateful if you could have a look at my workflow and explain if I'm doing anything incorrect and perhaps I found some bugs, which you could fix to improve your product.

The summary of the problem:
I'm creating a field into which I store all the response headers, this works correctly. Then I create a second field, into which I will store the database id. The ID is captured by reading the headers field, then applying a regular expression analyser. I cannot get the regular expression analyser to be correctly applied over all the headers field.

For step-by-step instructions on how to reproduce this problem, see below.

As I mentioned before, any help or instructions would be greatly appreciated.

Yours sincerely,
Guido Senff

1. Define a field that stores and indexes all headers
Schema
    Fields
        name: headerinformation
        indexed: yes
        stored: yes
        termvector: no
        analyser: none
        copy of: none

2. index all header information into this field (unfortunately, no regular expression filter can be applied on the headers at this stage. Feature request?)
Crawler
    Web
        Field mapping
            URL field: headers
            Index field: headerinformation

3. Create an analyser to subtract the part of the headers in which we are actually interested, the number after X-OSS-ItemKey:
Schema
    Analyser
        Header Analyser - ItemKey
            + Language undefined
            + index: KeywordTokenizer
            + query: KeywordTokenizer
            + Filter list:
                RegularExpressionFilter, Query and indexation, Pattern: X-OSS-ItemKey: (\d+)

4. Test this analyser on some sample header data, the results seem good! We see the 12345 key that we are interested in!
Analyser test
    Enter a text to analyze:
        X-OSS-Company:test
        X-OSS-DocumentType:Overig
        X-OSS-ItemKey: 12345
        X-OSS-Keywords:
        X-OSS-SearchType:

    Result:
        KeywordTokenizer: X-OSS-Company:test X-OSS-DocumentType:Overig X-OSS-ItemKey: 12345 X-OSS-Keywords: X-OSS-SearchType: [0,99 - 1]
        RegularExpressionFilter:    1009160 [0,104 - 1]

5. Now create a field, into which our "X-OSS-ItemKey"-value is copied, that is, copied from field "headerinformation", but with our regular expression applied from the "Header Analyser - ItemKey"-analyser
Schema
    Fields
        name: ItemKeyHeader
        indexed: yes
        stored: yes
        termvector: no
        analyser: Header Analyser - ItemKey
        copy of: headerinformation

6. Do a manual crawl on a webpage, to test our newly added header "X-OSS-ItemKey"-value parser.

Crawler
    Web
        Manual crawl
            URL to crawl: <our internal URL>

    Crawl Result
        Index document
            ItemKeyHeader:
                X-OSS-SearchType:
                X-OSS-Company: test
                X-OSS-ItemKey: 12345
                X-OSS-DocumentType: Overig
                X-OSS-Keywords:
                .. all other headers ..

            headerinformation:
                X-OSS-SearchType:
                X-OSS-Company: test
                X-OSS-ItemKey: 12345
                X-OSS-DocumentType: Overig
                X-OSS-Keywords:
                .. all other headers ..

BUG? This is not what I expected? Field "headerinformation" seems correct, it contains all headers. But "ItemKeyHeader" also contains all headers! I would expect to see "X-OSS-ItemKey"-value here, which is: 12345

7. Checking the actual index with Luke, lucine index toolbox
    Luke - Lucine index toolbox
        Search
            Enter search expression: *:*

        Results:
            Row with my document
                ItemKeyHeader: X-OSS-ItemKey: 12345
                headerinformation: .. all headers ..
                .. other indexed fields ..

BUG? This appears incorrect, I would expect to see 12345 here. Did OSS ignore my group and matched the entire pattern?

I also tried the RegularExpressionReplace filter, to just replace it into $1. This does not seem to affect the input string at all, all the headers are returned as if there is no regular expression.

Side information:
For non-binary documents, we supply the database id information as html meta header. We use the follow configuration to catch the
meta data element matching "ItemKey", we also use a group in this regular expression to capture the database id and
it works at this configuration.

Schema
    Parser list
            Field mapping
                HTML
                    htmlSource ItemKey <meta\s+(?:name="ItemKey"|content="(.[^"]*)"){1}\s+(?:name="ItemKey"|content="(.[^"]*)"){1}\s*/?> false