Menu
▾
▴
Re: [htdig] Excerpts vanish after running rundig.sh (was: Re: pdf-parser)
|
From: Ted Stresen-R. <bow...@ho...> - 2002-09-03 20:10:03
|
D'OH!!!!!!!!!!! After researching and testing this like crazy, I went back to the htdig site to see where I got this rundig.sh script to begin with and I realized that there was another script called rundig.3.2.sh... Perhaps I should be running this script since I have 3.2 installed rather than running a script that was inteded for 3.1.6... God do I feel stupid!!! Ted (for a good laugh, here's what I had done to reach this moment of clarity) For those of you just joining us, Jim Cole is trying to help me figure out why my rundig.sh script appears to delete excerpts from my databases. More testing... Following the good advice of running the commands from the command line, I discovered that the htmerge command appeared to be failing. Rather than producing any kind of output, it just appeared to do nothing. I suspect that it was in fact doing nothing because, from its point of view, nothing needed to be merged, everything was up to date. So... I deleted all the databases and work files from the db directory and ran the commands from the command line from scratch. After running /var/htdig/bin/htdig -v -s -a -c /var/htdig/conf/hci.conf The report read: htdig: Run complete htdig: 1 server seen: htdig: inside.hinshawlaw.com:80 2606 documents HTTP statistics =============== Persistent connections : Yes HEAD call before GET : No Connections opened : 2698 Connections closed : 2698 Changes of server : 0 HTTP Requests : 2698 HTTP KBytes requested : 195378 HTTP Average request time : 0.935878 secs HTTP Average speed : 77.3773 KBytes/secs ht://dig End Time: Tue Sep 3 14:33:46 2002 And I was left with -rw-r--r-- 1 root root 712704 Sep 3 14:33 db.docdb.work -rw-r--r-- 1 root root 598016 Sep 3 14:33 db.docs.index.work -rw-r--r-- 1 root root 6348800 Sep 3 14:33 db.excerpts.work -rw-r--r-- 1 root root 2646016 Sep 3 14:33 db.words.db.work -rw-r--r-- 1 root root 16384 Sep 3 14:33 db.words.db.work_weakcmpr So I then ran /var/htdig/bin/htmerge -s -a -c /var/htdig/conf/hci.conf And it appeared to do nothing. It returned nothing to STDOUT. So I ran /var/htdig/bin/htmerge -v -s -a -c /var/htdig/conf/hci.conf And still got no ouptut from the command. I double checked permissions for htmerge to see if the execute bit was set properly (and it was/is) and that the current user (root) had permission to execute it (and it did/does, root was/is the owner). The documentation for htmerg says, "-a Use alternate work files. Tells htdig to append .work to database files, causing a second copy of the database to be built. This allows the original files to be used by htsearch during the indexing run." My read of this is that htmerge expects to find valid database documents, but I'm not sure which documents it expects to find, so I made copies of the work files (renaming each db document by truncating ".work" along the way) one by one and tried the command again until some results were returned. At no point did htmerge return any results. Prior to making the new files, htsearch was failing complaining it could not find the words.db file (not surprising, it was named db.words.db.work). After copying and renaming the files, htsearch worked perfectly. Before proceeding, here is what I currently have in my db directory: -rw-r--r-- 1 root root 712704 Sep 3 14:33 db.docdb -rw-r--r-- 1 root root 712704 Sep 3 14:33 db.docdb.work -rw-r--r-- 1 root root 598016 Sep 3 14:33 db.docs.index -rw-r--r-- 1 root root 598016 Sep 3 14:33 db.docs.index.work -rw-r--r-- 1 root root 6348800 Sep 3 14:33 db.excerpts -rw-r--r-- 1 root root 6348800 Sep 3 14:33 db.excerpts.work -rw-r--r-- 1 root root 2646016 Sep 3 14:33 db.words.db -rw-r--r-- 1 root root 16384 Sep 3 14:33 db.words.db_weakcmpr -rw-r--r-- 1 root root 2646016 Sep 3 14:33 db.words.db.work -rw-r--r-- 1 root root 16384 Sep 3 14:33 db.words.db.work_weakcmpr On 8/30/02 8:32 PM, "Jim Cole" <gre...@yg...> wrote: > Ted Stresen-Reuter's bits of Fri, 30 Aug 2002 translated to: > >> Yes, $BASEDIR points to /var/htdig and the related files are there. > > What about $CONF ? Is it set to hci.conf as you used when you ran > htdig directly? Whether the programs are run from a script or the > command line shouldn't have any effect on the output, so it seems > like the path used for the program or the config file is suspect. > > You might also want to take a close look at the db.docdb and > db.docdb.work files. Can you verify that db.docdb is actually > being updated when you run the script? Is it possible that there > is some sort of permission problem that is causing an old file to > remain stuck in the database directory? I notice that your script > uses cp for that file, but mv for all the others. Is there some > reason for this? > > The only other thing I can think of to suggest is that you try > running each line of the script manually from the command line. > Copy, paste, replace variables, and check the output one step at > a time. > >> I just finished some more testing. I found that changing the line >> >> mv $BASEDIR/db/db.wordlist.work $BASEDIR/db/db.wordlist > > I would have been surprised if this made any difference. I don't > think db.wordlist is even used after the index is created (except > for update digs). > > Jim > > > > ------------------------------------------------------- > This sf.net email is sponsored by: OSDN - Tired of that same old > cell phone? Get a new here for FREE! > https://www.inphonic.com/r.asp?r=sourceforge1&refcode1=vs3390 > _______________________________________________ > htdig-general mailing list <htd...@li...> > To unsubscribe, send a message to > <htd...@li...> with a subject of unsubscribe > FAQ: http://htdig.sourceforge.net/FAQ.html > |