Question about BDB ISAM I/O - when does the named file get updated?

[Gregory A Failing]

Here is the scenario ...
1) the client program initializes an BDB ISAM with no records in it (this is verified with 'db_stat')
2) client program contacts a server
3) server send a few hundred transactions each a 'send and wait for answer' style exchange
4) the client processes each transaction and writes it to the BDB ISAM file (no errors observed/logged)
5) server receives each response and logs it
6) server send EOJ message to client and resumes listening
7) client program CLOSEs any open files (usually none) and exits.

Now I use the 'db_stat' utility to acquire record count; it reports zero records in the BDB ISAM file.

So the questions:
where is the client data?
If the data is held in a work file somewhere, when does it get added to my ISAM file?
How can I force BDB to update my ISAM file immediately? (a BDB option?)
Is there some combination of OPEN / SHARE or [NO] LOCK needed?

Other info:
'DB_HOME' is established for the working folder.
'COB_FILE_PATH' is correct for the client environment.

I have related issues with some other programs ie: a common file (used by many processes) has maintenance to a field performed but observing the file after exiting the maintenance program closes shows no change the the data.

Any help appreciated ...

Thanks.

[Vincent (Bryan) Coen]

When I tried to use DB-HOME I had some funny (well to me anyway) happenings.

For the ISAM data file look where it is pointing but also at your
current directory.

Better still do a search using : -

find / -iname ISAM-filename

just in case :)

Vince

Mod-Edit: removed reply-to

[Gregory A Failing]

Vincent,

A 'find' produces only one name and it is in the correct folder.

G

[Vincent (Bryan) Coen]

Blows that idea out then :)

On 06/06/16 19:00, Gregory A Failing wrote:

Vincent,

A 'find' produces only one name and it is in the correct folder.

G

---

[Simon Sobisch]

Record locking between multiple processes for BDB-configured ISAM needs DB_HOME set, therefore you have the first issue met already.
It could help to check the current lock modes. I assume all writing I/O to these files is done via libcob.

You can force GnuCOBOL to change the update mode via COB_SYNCH which is a global setting for current version, allows a per-file setting in rw-branch already.

Which version of BDB are you using?
Did you tried VBISAM yet (=> same issue = something is totally wrong / working fine = BDB is borked)?

[Gregory A Failing]

Simon,

It could help to check the current lock modes.
I am not sure what you want here ... ?

I assume all writing I/O to these files is done via libcob.
Yes.

BDB version=6.2.23. While BDB could be 'borked' I tend to not think so ... or there is something else very wrong. I have observed the BDB behaviour since v4.x.x.

OC version = 2.0 branch 880 with a small mod to libcob (see wish list #131 - diff attached). The mod is to allow 'COB_FILE_PATH' to contain several folder names that are allowed to prefix a file name (this is a RM-Cobol behaviour - works good). Removing this small patch does not alter the observed BDB results.

VB-ISAM not tried ... does it do record locking etc?

[Simon Sobisch]

It could help to check the current lock modes.
I am not sure what you want here ... ?

I've referred to:

Is there some combination of OPEN / SHARE or [NO] LOCK needed?

And wanted to ask what modes you use (you can disable locking if you set everything "correct" - which I assume you did not do).

If BDB isn't borked then the BDB part of fileio may be borked...

VB-ISAM not tried ... does it do record locking etc?
Of course VB-ISAM supports locking and etc. [ = Important ISAM stuff] :-)

In any case: Can you please try to come up with a small test suite where you can reproduce the issue (you could try to start the COBOL test part asynchron multiple times)? You could both check easily with other configuration options / small changes to the share/lock modes and others could do the same.

In any case this thread is a good reminder for rechecking the patch in the near future...

Simon

BTW: 2.0 branch 880 is a GC (GnuCOBOL) version, not an OC (OpenCOBOL) one ;-)

[Gregory A Failing]

Heh ... sorry about the OC/GC mix-up ...

So where can I acquire a copy of VB-ISAM that makes GC happy? I have seen references to it in other posts but no link.

Thanks.

[Brian Tiffin]

I use the VBISAM tarball from https://github.com/opensourcecobol/opensource-cobol, in a subdir of that tree, Gregory.

And a normal

./configure ; make ; sudo make install

dance, then

./configure --with-vbisam

for the GnuCOBOL build.

Brian

[Brian Tiffin]

Not really an informed post; there is COB_SYNC, set to Y in the enviroment, it'll sync flush every write. Performance drops considerably, but it might be worth testing in this case.

Brian

[Simon Sobisch]

In any case, setting COB_SYNC will lead to a performance penalty.
@Brian: Are you sure your descriptions doesn't map COB_SYNC=P (paranoid)?

[Brian Tiffin]

Oh, is it P? Sorry, didn't look close enough.

[Simon Sobisch]

There are both 'P' and 'Y', with a different behaviour and 'P' don't being useful for most production environments as it syncs too often.
I encourage you to recheck this in the libcob sources and add an entry in the FAQ if there isn't an entry yet.

[Gregory A Failing]

Interesting development ... the addition of 'COB_SYNCH=Y' to the calling script seems to have helped with the missing maintenance that I mentioned but it did nothing for the client/server app. The transaction ISAM file remains empty after a short session. Odd ...

I will review the client program to be sure that the OPENs and READs are using the LOCK phrase properly.

Adding COB_SYNCH should not be too much of a performnce issue because in the real world each client will be receiving items sporadically ... sometimes several per second, usually much slower. Taking a few millisecs to guarantee the data is captured where I want it is a acceptable trade-off.

[Gregory A Failing]

OK ... I have VBisam installed and it all seems to work at least as far as preliminary testing is concerned although my RMCobol extensions fail because of the vbisam file names.

I do have a few questions ...

1) can vbisam be configured to use a combined idx/dat file so there is only one file name involved?
2) I have noticed that the dat file seems to have LF terminated records (unix text) inside ... does this preclude using binary (comp-?) data within the records for a file? (I doubt it but asking is prudent)
3) how is 'libcob:fileio' aware that it is talking to vbisam? (I see no #ifdefs for vbisam in the code)

More to come later ...

Thanks.

[Simon Sobisch]

can vbisam be configured to use a combined idx/dat file so there is only one file name involved?

No - this is the same for all ISAM implementations if I'm not wrong (your RM extensions [=patch] should take this into account). Hint: GnuCOBOL's Implementation with BDB has the main file + one for each additional key [this could be changed in fileio.c but nobody did this].

does this preclude using binary (comp-?) data within the records for a file?

No, everything is fine (you can try this by writing x'00...FF' to a record.

how is 'libcob:fileio' aware that it is talking to vbisam? (I see no #ifdefs for vbisam in the code)

::c
#ifdef  WITH_DB

#include <db.h>

#elif   defined(WITH_CISAM) || defined(WITH_DISAM) || defined(WITH_VBISAM)

#define WITH_ANY_ISAM

[Gregory A Failing]

Simon, thanks for getting back to me ...

"No - this is the same for all ISAM implementations " Actually the default RMCobol ISAM file is a combined unit (header block + index tree + data). It can be configured to produce separate files but I have not seen this done for years.

I will need the "WITH_VBISAM" reference to modify my patch to look for "*.dat" files. Thanks.

G