My application creates the data file only once and will never modify during process up and running.
But I do a several millions of indexed read operation on these data files to get the data.
I have source code to open the file in INPUT MODE without any LOCKS.
But VBISAM hardcoded the file open mode to READ-WRITE(isopen()->ivbopen(O_RDWR) )and enabled WRITE locks(ivbenter()->ivblock(VBWRLOCK)) by default.
As there are huge read operations, there is a performance degradation while running single instance or multiple instances of the application.
We have ran perf profiler for the process and observed fcntl locks are adding a considerable overhead to the process.
I feel it’s better to differentiate the INPUT ONLY operations from INPUT-OUTPUT operations to improve the performance.
Do you have any options to avoid this locking as of now.
Please help me out.
Thanks and Regards,
Naresh
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
At first glance, I can't say there is an easy way, Naresh, sorry. Will keep looking. It'll be a worthy pursuit for an enhancement.
We just have to avoid finding more things that will delay an official 2.2 release over the next few days. There will always be yet another reason to delay, and 6 years is too many years now. Won't matter how perfect we can make GnuCOBOL if no one ever sees it. :-)
Cheers,
Brian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, I have made change to open the file(ivbopen()) in READ ONLY and commented fcntl locks(in vblowlevel.c->ivblock()). It worked fine and gave 8% improvement when 10 processes ran in parallel.
Thanks and Regards,
Naresh
Last edit: NARESH NADIGOTI 2017-06-27
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
@Simon, Can I deliver this modified VBISAM code to my client? If yes , how?
I am not aware of the licensing policies of the VBISAM. Can you please help me on this.
Last edit: NARESH NADIGOTI 2017-06-27
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
VBISAM is LGPL. And you are allowed to change it as much as you like Naresh, we all are as part of the freedoms. You just need to publish a link to allow your customers to get those sources. And anyone else that may want/need a copy. You can keep the changed copy between yourself and any individuals you share with, it seems. For LGPL.
You can have them compile VBISAM on site or deliver a build of GnuCOBOL with it built in. You don't have to deliver COBOL sources that use that library, but as a GNU project we really do like to endorse and recommend free software.
You can and should add a copyright notice for the changes. If there is ever a day when we add VBISAM as part of the source tree, all official project sources will go through a copyright reassigment phase. If we can't find all the authors, then it'll have to be just another contribution, and be separate from the product or the changes by authors that can't be tracked down will have to be pulled out.
If you just use VBISAM stock, then you only need to point to the link that you got the code from, and don't have to provide hosting of any changes, seeing as you didn't make any changes, and the original copy is hosted somewhere else.
The four GPL freedoms are mainly focused on end users - they (/we) have the freedom and the right to swap out free software components and personalize them to suit needs. And the GPL means you can share and not get in trouble, and can't stop anyone else from sharing.
If this was BDB or a GPL library, you'd also have to provide sources of the code that uses the GPL code (or third clause BDB), but VBISAM is Lesser GPL. That means, deliver changes to the tool, but you and/or your customers can then write more COBOL that use that library and keep the COBOL part closed (not recommended), they just need to ensure that their customers can get the sources to the VBISAM that they used, and so on and so on.
If they make further mods to VBISAM along with COBOL programs, you have the right to that modified VBISAM as well, but maybe not the application COBOL source if they license those bits a little less freely.
Again, first recommendation is free software. Give the people using programs the right to customize and to share the customizations.
Cheers,
Brian
Edits for facts from Simon, below
Last edit: Brian Tiffin 2017-06-27
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If they make further mods to VBISAM along with COBOL programs, you have the right to that modified VBISAM as well
I have to admit that I needed a long time to get this right in my head - it isn't the case (Richard Stallman pointed this out to me): the license applies on the use and further distribution. Given "person X gets a LPGL library and the code and changes it":
he only uses it himself: he is the only person that has the right to get the LGPL code
he distributes his changes to company Y (no matter if he's part of the company, if he got paid for it or not, ...) and Y uses it company wide: Y got the library and has the right to get the LGPL code, too - note: the people working at Y doesn't have the right, even if they use the binary - it is Y's right only
Y distributes the library via a hosting site: everybody that got the library under LGPL from there has the right to get the LGPL code, too
@Nathan: The simplest way of delivering a modified LGPL library is always to make sure the official sources get the changes in - this way you don't deliver a modified library and don't have to take care of this at all (that is actually the reason we got the CPP version of GnuCOBOL - the main author distributed the library and had to make the sources available). The only thing you have to make sure is to ship the library with its license.
In every other case you have to ship the binary and the license and allow the customer to get the source code in a reasonable manner (nowadays this mostly means hosting the source [which could be in a locked down client-only site, too]) - but keep in mind that the customer would have the right to distribute the library and its code free to everybody else (which is the reason that you normally don't find LGPLed sources "locked down").
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So it really is Lesser then. If X gives to an individual (that being person or company) it doesn't have to be shared further. Hmm. So X can give to Y and Z and still stays tight. Only when X (or Y or Z...) decide to make a public copy does it go to public rights, Nice. Kinda, nice is the wrong word. Corporate concession, nice.
But does X get a say if Y or Z decides to post public? Or does Y have some power to restrict X from further distribution (say to Z, in the first place)? When X makes the change, who gets to decide to post public? I hope Y can't restrict X, or Z for that matter (assuming sharing with Z was a right of X in the first place).
Heady.
Cheers,
Brian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
But does X get a say if Y or Z decides to post public? Or does Y have some power to restrict X from further distribution (say to Z, in the first place)? When X makes the change, who gets to decide to post public? I hope Y can't restrict X, or Z for that matter (assuming sharing with Z was a right of X in the first place).
No. Neither "X" nor "Y" can do anything as the LGPL explicit allows this. The only way to remove the "restriction" that anybody can share the library and the code is to change the license - which none of these can do as "X" (or "Y" if "X" was paid to do these changes) has only the Copyright for the change they did, not for the complete library. And as the code is in the original files "X" can't do anything.
It would be up to a court (or at least a lawyer) to say if it is possible to add parts in a different file with a restrictive license that is linked to the LGPL library - the (likely optional via #ifdef) code parts in the LGPL library that link to restrictive library are under LGPL for sure, the proprietary library itself may be not.
Note: This applies to the GPL the same way in all points but the linking - if you link code to GPL you always have to use a compatible license for this - which is in most cases GPL only.
Note: This is exactly what CIT (pronounce it s-h-...) did: cobc stays GPL, external tools it calls (preparsers etc) are called on a system level (not linked at all), the calls cobc generate are to be found in a non-LGPL library. It maybe is OK to do so, while surely not very ethic.
Anybody that receives the code of this modified cobc and libcob is free to share it under GPL/LGP but not the parts (if the sources are available at all) that are in the external tools/library. The client is free to remove any calls to the non-free code - if the program and library works afterwards is a different issue that is not necessarily under control of the original license.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
In the GNU COBOL builds that I'm offering, the 'configure' reports
Use gettext for international messages: yes
Use fcntl for file locking: no
Use ncurses/pdcurses/curses for screen I/O: pdcurses
Use VBISAM for ISAM I/O: yes
Because of the 'no' above, I'm not sure if these builds would
help the OP.
If I try another build with the latest files, the 'configure' of
mpir-3.0.0 fails with the warning
error: no system-wide yasm found
With the older mpir-2.7.2, the gnu-cobol 'make' fails with the
warning
fcntl is used if it is found during configure, if it isn't you only could change the underlying system (libraries, headers, compiler, maybe even OS) to activate it...
yasm is the assambler compiler that MPIR uses. If you want to use MPIR it is reasonable to download it and point MPIRs configure script to its path: --with-yasm=/path/to/yasm.
GnuCOBOL failing with an unrecognized GCC option has nothing to do with MPIR. I suggest to try to pass --disable-rpath to GnuCOBOL's configure-script which removes the option completely.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
And yes: as it currently is the only option other than fixing VBISAM seems to either use libdb (free software but not necessarily "gratis"free-to-use) or the D-ISAM library (non-free software).
VBISAM will surely be fixed in the long term but the fastest way would be to do it yourself (or let some other person do it) and provide the patches for VB-ISAM to be incorporated in the next version (which will come but it is unlikely that this happens soon).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
There are no complaints about 'yasm' in mpir-2.7.2, so I just used that.
Adding '--disable-rpath' suppresses the complaint about the unsupported '-R' option.
The start of the 'configure' says 'yes' to both the presence and usability of
'fcntl.h'. But the end of the 'configure' still says that 'fcntl' is not used.
So I have three successful builds but none have a 'fcntl' to help the OP.
I have other builds with WinXP. How can I check if these have 'fcntl'?
The command 'cobc -i' doesn't help and I no longer have the XP machine
to make another build and check.
Anon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi All,
My application creates the data file only once and will never modify during process up and running.
But I do a several millions of indexed read operation on these data files to get the data.
I have source code to open the file in INPUT MODE without any LOCKS.
But VBISAM hardcoded the file open mode to READ-WRITE(isopen()->ivbopen(O_RDWR) )and enabled WRITE locks(ivbenter()->ivblock(VBWRLOCK)) by default.
As there are huge read operations, there is a performance degradation while running single instance or multiple instances of the application.
We have ran perf profiler for the process and observed fcntl locks are adding a considerable overhead to the process.
I feel it’s better to differentiate the INPUT ONLY operations from INPUT-OUTPUT operations to improve the performance.
Do you have any options to avoid this locking as of now.
Please help me out.
Thanks and Regards,
Naresh
At first glance, I can't say there is an easy way, Naresh, sorry. Will keep looking. It'll be a worthy pursuit for an enhancement.
We just have to avoid finding more things that will delay an official 2.2 release over the next few days. There will always be yet another reason to delay, and 6 years is too many years now. Won't matter how perfect we can make GnuCOBOL if no one ever sees it. :-)
Cheers,
Brian
This looks like a real issue in VBISAM. Did you tried already to change the mode (I assume you talk about isopen.c)?
Hi Simon,
Yes, I have made change to open the file(ivbopen()) in READ ONLY and commented fcntl locks(in vblowlevel.c->ivblock()). It worked fine and gave 8% improvement when 10 processes ran in parallel.
Thanks and Regards,
Naresh
Last edit: NARESH NADIGOTI 2017-06-27
@Naresh: Please create a patch file including a Changelog, this helps to easy the integration.
@Simon, Can I deliver this modified VBISAM code to my client? If yes , how?
I am not aware of the licensing policies of the VBISAM. Can you please help me on this.
Last edit: NARESH NADIGOTI 2017-06-27
VBISAM is LGPL. And you are allowed to change it as much as you like Naresh, we all are as part of the freedoms.
You just need to publish a link to allow your customers to get those sources. And anyone else that may want/need a copy.You can keep the changed copy between yourself and any individuals you share with, it seems. For LGPL.You can have them compile VBISAM on site or deliver a build of GnuCOBOL with it built in. You don't have to deliver COBOL sources that use that library, but as a GNU project we really do like to endorse and recommend free software.
You can and should add a copyright notice for the changes. If there is ever a day when we add VBISAM as part of the source tree, all official project sources will go through a copyright reassigment phase. If we can't find all the authors, then it'll have to be just another contribution, and be separate from the product or the changes by authors that can't be tracked down will have to be pulled out.
If you just use VBISAM stock, then you only need to point to the link that you got the code from, and don't have to provide hosting of any changes, seeing as you didn't make any changes, and the original copy is hosted somewhere else.
The four GPL freedoms are mainly focused on end users - they (/we) have the freedom and the right to swap out free software components and personalize them to suit needs. And the GPL means you can share and not get in trouble, and can't stop anyone else from sharing.
If this was BDB or a GPL library, you'd also have to provide sources of the code that uses the GPL code (or third clause BDB), but VBISAM is Lesser GPL. That means, deliver changes to the tool, but you and/or your customers can then write more COBOL that use that library and keep the COBOL part closed (not recommended), they just need to ensure that their customers can get the sources to the VBISAM that they used, and so on and so on.
If they make further mods to VBISAM along with COBOL programs,
you have the right to that modified VBISAM as well, but maybe not the application COBOL source if they license those bits a little less freely.Again, first recommendation is free software. Give the people using programs the right to customize and to share the customizations.
Cheers,
Brian
Edits for facts from Simon, below
Last edit: Brian Tiffin 2017-06-27
@Brian:
I have to admit that I needed a long time to get this right in my head - it isn't the case (Richard Stallman pointed this out to me): the license applies on the use and further distribution. Given "person X gets a LPGL library and the code and changes it":
@Nathan: The simplest way of delivering a modified LGPL library is always to make sure the official sources get the changes in - this way you don't deliver a modified library and don't have to take care of this at all (that is actually the reason we got the CPP version of GnuCOBOL - the main author distributed the library and had to make the sources available). The only thing you have to make sure is to ship the library with its license.
In every other case you have to ship the binary and the license and allow the customer to get the source code in a reasonable manner (nowadays this mostly means hosting the source [which could be in a locked down client-only site, too]) - but keep in mind that the customer would have the right to distribute the library and its code free to everybody else (which is the reason that you normally don't find LGPLed sources "locked down").
So it really is Lesser then. If X gives to an individual (that being person or company) it doesn't have to be shared further. Hmm. So X can give to Y and Z and still stays tight. Only when X (or Y or Z...) decide to make a public copy does it go to public rights, Nice. Kinda, nice is the wrong word. Corporate concession, nice.
But does X get a say if Y or Z decides to post public? Or does Y have some power to restrict X from further distribution (say to Z, in the first place)? When X makes the change, who gets to decide to post public? I hope Y can't restrict X, or Z for that matter (assuming sharing with Z was a right of X in the first place).
Heady.
Cheers,
Brian
No. Neither "X" nor "Y" can do anything as the LGPL explicit allows this. The only way to remove the "restriction" that anybody can share the library and the code is to change the license - which none of these can do as "X" (or "Y" if "X" was paid to do these changes) has only the Copyright for the change they did, not for the complete library. And as the code is in the original files "X" can't do anything.
It would be up to a court (or at least a lawyer) to say if it is possible to add parts in a different file with a restrictive license that is linked to the LGPL library - the (likely optional via
#ifdef) code parts in the LGPL library that link to restrictive library are under LGPL for sure, the proprietary library itself may be not.Note: This applies to the GPL the same way in all points but the linking - if you link code to GPL you always have to use a compatible license for this - which is in most cases GPL only.
Note: This is exactly what CIT (pronounce it s-h-...) did:
cobcstays GPL, external tools it calls (preparsers etc) are called on a system level (not linked at all), the callscobcgenerate are to be found in a non-LGPL library. It maybe is OK to do so, while surely not very ethic.Anybody that receives the code of this modified cobc and libcob is free to share it under GPL/LGP but not the parts (if the sources are available at all) that are in the external tools/library. The client is free to remove any calls to the non-free code - if the program and library works afterwards is a different issue that is not necessarily under control of the original license.
View and moderate all "Help getting started" comments posted by this user
Mark all as spam, and block user from posting to "Discussion"
There won't be a further delay than end of June 2017 for the release candidate and likely end of July 2017 for the final release...
Did I understood it correctly that this is about fileio.c? Does anyone checked if rw-branch has the same issue?
I'm guessing this is you, Simon...
It's likely something in VBISAM in particular, but we could run some trials with libdb to be sure. I only glanced to see if there was a quick fix.
Cheers,
Brian
View and moderate all "Help getting started" comments posted by this user
Mark all as spam, and block user from posting to "Discussion"
There are three versions of VBISAM that can be used to build GNU COBOL.
In this thread
https://sourceforge.net/p/open-cobol/discussion/help/thread/90b4ccdc/
I am offering to send Arnold three different builds (one for each VBISAM).
Then you could check each build to see one of the VBISAM will resolve
the problem.
View and moderate all "Help getting started" comments posted by this user
Mark all as spam, and block user from posting to "Discussion"
In the GNU COBOL builds that I'm offering, the 'configure' reports
Use gettext for international messages: yes
Use fcntl for file locking: no
Use ncurses/pdcurses/curses for screen I/O: pdcurses
Use VBISAM for ISAM I/O: yes
Because of the 'no' above, I'm not sure if these builds would
help the OP.
If I try another build with the latest files, the 'configure' of
mpir-3.0.0 fails with the warning
error: no system-wide yasm found
With the older mpir-2.7.2, the gnu-cobol 'make' fails with the
warning
. ../tests/atconfig; . ../tests/atlocal; \
cobc -m -Wall -std=mf -O -o CBL_OC_DUMP.dll CBL_OC_DUMP.cob
gcc: error: unrecognized command line option '-R'
Is there a way to modify the VBISAM build document so that 'fcntl'
is enabled, 'yasm' is found, and the 'gcc -R' option is disabled?
fcntlis used if it is found during configure, if it isn't you only could change the underlying system (libraries, headers, compiler, maybe even OS) to activate it...yasmis the assambler compiler that MPIR uses. If you want to use MPIR it is reasonable to download it and point MPIRs configure script to its path:--with-yasm=/path/to/yasm.GnuCOBOL failing with an unrecognized GCC option has nothing to do with MPIR. I suggest to try to pass
--disable-rpathto GnuCOBOL's configure-script which removes the option completely.And yes: as it currently is the only option other than fixing VBISAM seems to either use libdb (free software but not necessarily "gratis"free-to-use) or the D-ISAM library (non-free software).
VBISAM will surely be fixed in the long term but the fastest way would be to do it yourself (or let some other person do it) and provide the patches for VB-ISAM to be incorporated in the next version (which will come but it is unlikely that this happens soon).
View and moderate all "Help getting started" comments posted by this user
Mark all as spam, and block user from posting to "Discussion"
In all three VBISAM versions:
There are no complaints about 'yasm' in mpir-2.7.2, so I just used that.
Adding '--disable-rpath' suppresses the complaint about the unsupported '-R' option.
The start of the 'configure' says 'yes' to both the presence and usability of
'fcntl.h'. But the end of the 'configure' still says that 'fcntl' is not used.
So I have three successful builds but none have a 'fcntl' to help the OP.
I have other builds with WinXP. How can I check if these have 'fcntl'?
The command 'cobc -i' doesn't help and I no longer have the XP machine
to make another build and check.
View and moderate all "Help getting started" comments posted by this user
Mark all as spam, and block user from posting to "Discussion"
All bug reports regarding VBISAM should explain which VBISAM is being used because results with another version may be different.
For instance, in the Japanese (OSCOBOL) and Master versions of VBISAM-2.0.1, the NIST85 test suite pauses and can't finish the IX module.
Only the original VBISAM-2.0.1 runs all of the tests in a few minutes -- the complete output log is posted as an attachment in this thread:
https://sourceforge.net/p/open-cobol/discussion/help/thread/51d48303/?page=2
Could this explain the OP's problem with slow performance in his database?