#259 Packaging XSB for Guix

[Alvin Hsu]

Hello,

I would like to share my experience packaging XSB for the Guix package manager, with the aim of revealing any bugs that I may have encountered during the process as well as provide an example package for those who would like to package XSB in other formats.

This report covers potentially more than one bug, so if desirable I can break this up into multiple bug reports instead.

Most notably:

The newerthan syspath_op evaluates to true when the modification timestamps of two files are equal. I'm not sure if this is a bug or not, but its use for checking if files need recompiling is an issue particularly for Guix because builds are put in a read-only store where file timestamps are reset to Unix epoch for reproducibility purposes. This means that when these timestamps are set on build completion, XSB will attempt to rebuild again in a read-only environment and crash. I was able to resolve this by patching needs_recompile(newerthan,PFileName,Obj) in syslib/consult.P to not include cases where they are equal, though (see attached).

It also appears that the build option for./configure may not be working. Guix on my system automatically provides --build=x86_64-unknown-linux-gnu, but it seems that ./makexsb ignores this and uses the default x86_64-pc-linux-gnu as if no build option was provided, thus causing files to not be found.

Please feel free to examine the attached package definition for further analysis. This package may still have quirks that I need to sort out, but it is currently in a usable state for me, and I do plan on submitting it to be included in the official repositories once I feel that I have satisfied the packaging guidelines.

I would be happy to provide any further information or assistance.

[Michael Kifer]

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <style id="user-content-bidiui-paragraph-margins" type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>

Hi, Alwin, sorry for the long wait.

There probably are as many arguments for as there are against recompiling xwam with modtime equal to that of its corresponding  .P file.

Perhaps a global flag could be used to indicate that.

As to the --build option, it is not supported.

--

       --- michael


 

On 6/1/23 4:50 PM, Alvin Hsu wrote:

<meta http-equiv="content-type" content="text/html; charset=UTF-8">

---

[bugs:#259] Packaging XSB for Guix

Status: open
Group: Other
Created: Thu Jun 01, 2023 08:50 PM UTC by Alvin Hsu
Last Updated: Thu Jun 01, 2023 08:50 PM UTC
Owner: nobody
Attachments:

• xsb.scm (sourceforge.net) (6.6 kB; text/x-scheme)

Hello,

I would like to share my experience packaging XSB for the Guix package manager, with the aim of revealing any bugs that I may have encountered during the process as well as provide an example package for those who would like to package XSB in other formats.

This report covers potentially more than one bug, so if desirable I can break this up into multiple bug reports instead.

Most notably:

The newerthan syspath_op evaluates to true when the modification timestamps of two files are equal. I'm not sure if this is a bug or not, but its use for checking if files need recompiling is an issue particularly for Guix because builds are put in a read-only store where file timestamps are reset to Unix epoch for reproducibility purposes. This means that when these timestamps are set on build completion, XSB will attempt to rebuild again in a read-only environment and crash. I was able to resolve this by patching needs_recompile(newerthan,PFileName,Obj) in syslib/consult.P to not include cases where they are equal, though (see attached).

It also appears that the build option for./configure may not be working. Guix on my system automatically provides --build=x86_64-unknown-linux-gnu, but it seems that ./makexsb ignores this and uses the default x86_64-pc-linux-gnu as if no build option was provided, thus causing files to not be found.

Please feel free to examine the attached package definition for further analysis. This package may still have quirks that I need to sort out, but it is currently in a usable state for me, and I do plan on submitting it to be included in the official repositories once I feel that I have satisfied the packaging guidelines.

I would be happy to provide any further information or assistance.

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/xsb/bugs/259/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

<link itemprop="url" href="https://sourceforge.net/p/xsb/bugs/259/"> <meta itemprop="name" content="View">

<meta itemprop="description" content="View">

[Alvin Hsu]

Hello Michael,

Admittedly, I think this bug report was a bit premature given my experience, and
I should have taken some more time with it - apologies for that. I'm still
interested in working on getting this upstream, but I'll likely need to look
over this and work more closely with the Guix maintainers to see if some of
these issues should/can be taken care of on the packaging side.

Hopefully I can get back soon on this; for now, perhaps it might be useful for
others looking to package XSB for other formats :)

Thanks for looking into this.

[Alvin Hsu]

Hello again,

I finally got around to bringing the modtime recompilation issue up with others in the Guix community, and it seems like resolving it without a downstream patch would indeed require a change in how XSB decides when recompilation is needed.

Would a build-time flag for determining this be acceptable? An environment variable check might also work (we can wrap xsb to always have it set in this case).

[Michael Kifer]

I see that David changed<= to <.
Will use it for now unless someone complains that this breaks something

Feb 8, 2024 23:27:09 Alvin Hsu alvhsu@users.sourceforge.net:

Hello again,

I finally got around to bringing the modtime recompilation issue up with others in the Guix community, and it seems like resolving it without a downstream patch would indeed require a change in how XSB decides when recompilation is needed.

Would a build-time flag for determining this be acceptable? An environment variable check might also work (we can wrap xsb to always have it set in this case).

---

[bugs:#259][https://sourceforge.net/p/xsb/bugs/259/] Packaging XSB for Guix

Status: open
Group: Other
Created: Thu Jun 01, 2023 08:50 PM UTC by Alvin Hsu
Last Updated: Wed Aug 30, 2023 02:46 AM UTC
Owner: nobody
Attachments:

• xsb.scm[https://sourceforge.net/p/xsb/bugs/259/attachment/xsb.scm] (6.6 kB; text/x-scheme)

Hello,

I would like to share my experience packaging XSB for the Guix package manager[https://guix.gnu.org/], with the aim of revealing any bugs that I may have encountered during the process as well as provide an example package for those who would like to package XSB in other formats.

This report covers potentially more than one bug, so if desirable I can break this up into multiple bug reports instead.

Most notably:

The newerthan syspath_op evaluates to true when the modification timestamps of two files are equal. I'm not sure if this is a bug or not, but its use for checking if files need recompiling is an issue particularly for Guix because builds are put in a read-only store where file timestamps are reset to Unix epoch for reproducibility purposes. This means that when these timestamps are set on build completion, XSB will attempt to rebuild again in a read-only environment and crash. I was able to resolve this by patching needs_recompile(newerthan,PFileName,Obj) in syslib/consult.P to not include cases where they are equal, though (see attached).

It also appears that the build option for./configure may not be working. Guix on my system automatically provides --build=x86_64-unknown-linux-gnu, but it seems that ./makexsb ignores this and uses the default x86_64-pc-linux-gnu as if no build option was provided, thus causing files to not be found.

Please feel free to examine the attached package definition for further analysis. This package may still have quirks that I need to sort out, but it is currently in a usable state for me, and I do plan on submitting it to be included in the official repositories once I feel that I have satisfied the packaging guidelines.

I would be happy to provide any further information or assistance.

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/xsb/bugs/259/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

[Alvin Hsu]

Hi Michael,

Thanks for going through with this! Unfortunately, it appears that the issue is still present at commit [660f0f79b1007f2f0be5bfa4e079abaf12b4f1d9]; building without my patch results in XSB attempting to recompile for equal modtimes as it did before.

I've written a shell script that reproduces this on my end with a standard (non-Guix) build that follows "Installing XSB under UNIX" in the manual:

# Example with xsb_configuration file, but similarly errors out when applied to
# other files

# Run this from $XSB_DIR

# When timestamps are set equal, see that recompilation is triggered
cd config/x86_64-pc-linux-gnu/lib
touch --date='2000-01-01 00:00:00' xsb_configuration.P
touch --date='2000-01-01 00:00:00' xsb_configuration.xwam
cd ../../..
./bin/xsb

# When read-only directory is emulated, see that XSB's attempt to recompile
# fails (normally applied to everything else in $XSB_DIR too, but scope is
# limited for testing here)
cd config/x86_64-pc-linux-gnu/lib
chmod -R -w .
touch --date='2000-01-01 00:00:00' xsb_configuration.P
touch --date='2000-01-01 00:00:00' xsb_configuration.xwam
cd ../../..
./bin/xsb

Here is XSB's output for that last invocation in the script:

[Compiling /home/alvin/git/xsb-code/XSB/config/x86_64-pc-linux-gnu/lib/xsb_configuration]
% Specialising partially instantiated calls to xsb_configuration/2
++Error[XSB/Runtime/P]: [Existence (No file for module xsb_configuration exists)] []
Forward Continuation...
... machine:xsb_backtrace/1  From /home/alvin/git/xsb-code/XSB/syslib/machine.xwam
... loader:load/2  From /home/alvin/git/xsb-code/XSB/syslib/loader.xwam
... loader:load_pred1/2  From /home/alvin/git/xsb-code/XSB/syslib/loader.xwam
... loader:load_pred0/2  From /home/alvin/git/xsb-code/XSB/syslib/loader.xwam
... loader:load_pred/2  From /home/alvin/git/xsb-code/XSB/syslib/loader.xwam
... xsb_configuration:xsb_configuration/2  From /home/alvin/git/xsb-code/XSB/xsb_configuration
... error_handler:translate_system_error/2  From /home/alvin/git/xsb-code/XSB/syslib/error_handler.xwam
... standard:open/3  From /home/alvin/git/xsb-code/XSB/syslib/standard.xwam
... compile:compile1/9  From /home/alvin/git/xsb-code/XSB/cmplib/compile.xwam
... compile:compile_parsed_file/12  From /home/alvin/git/xsb-code/XSB/cmplib/compile.xwam
... standard:call/1  From /home/alvin/git/xsb-code/XSB/syslib/standard.xwam
... consult:compile_f/7  From /home/alvin/git/xsb-code/XSB/syslib/consult.xwam
... consult:reconsult0/3  From /home/alvin/git/xsb-code/XSB/syslib/consult.xwam
... consult:reconsult/1  From /home/alvin/git/xsb-code/XSB/syslib/consult.xwam
... standard:call/1  From /home/alvin/git/xsb-code/XSB/syslib/standard.xwam
... standard:catch/3  From /home/alvin/git/xsb-code/XSB/syslib/standard.xwam

End XSB (cputime 0.02 secs, elapsetime 0.03 secs)

I've also attached the patch I previously used to mitigate this issue which still works.

[Michael Kifer]

Why do you touch xsb configuration.p?
You can see that this causes an error.
Try touching a new empty file

Feb 13, 2024 02:44:27 Alvin Hsu alvhsu@users.sourceforge.net:

Hi Michael,

Thanks for going through with this! Unfortunately, it appears that the issue is still present at commit [660f0f79b1007f2f0be5bfa4e079abaf12b4f1d9]; building without my patch results in XSB attempting to recompile for equal modtimes as it did before.

I've written a shell script that reproduces this on my end with a standard (non-Guix) build that follows "Installing XSB under UNIX" in the manual:

~~~sh

Example with xsb_configuration file, but similarly errors out when applied to

other files

Run this from $XSB_DIR

When timestamps are set equal, see that recompilation is triggered

cd config/x86_64-pc-linux-gnu/lib
touch --date='2000-01-01 00:00:00' xsb_configuration.P
touch --date='2000-01-01 00:00:00' xsb_configuration.xwam
cd ../../..
./bin/xsb

When read-only directory is emulated, see that XSB's attempt to recompile

fails (normally applied to everything else in $XSB_DIR too, but scope is

limited for testing here)

cd config/x86_64-pc-linux-gnu/lib
chmod -R -w .
touch --date='2000-01-01 00:00:00' xsb_configuration.P
touch --date='2000-01-01 00:00:00' xsb_configuration.xwam
cd ../../..
./bin/xsb
~~~

Here is XSB's output for that last invocation in the script:

~~~
[Compiling /home/alvin/git/xsb-code/XSB/config/x86_64-pc-linux-gnu/lib/xsb_configuration]
% Specialising partially instantiated calls to xsb_configuration/2
++Error[XSB/Runtime/P]: [Existence (No file for module xsb_configuration exists)] []
Forward Continuation...
... machine:xsb_backtrace/1  From /home/alvin/git/xsb-code/XSB/syslib/machine.xwam
... loader:load/2  From /home/alvin/git/xsb-code/XSB/syslib/loader.xwam
... loader:load_pred1/2  From /home/alvin/git/xsb-code/XSB/syslib/loader.xwam
... loader:load_pred0/2  From /home/alvin/git/xsb-code/XSB/syslib/loader.xwam
... loader:load_pred/2  From /home/alvin/git/xsb-code/XSB/syslib/loader.xwam
... xsb_configuration:xsb_configuration/2  From /home/alvin/git/xsb-code/XSB/xsb_configuration
... error_handler:translate_system_error/2  From /home/alvin/git/xsb-code/XSB/syslib/error_handler.xwam
... standard:open/3  From /home/alvin/git/xsb-code/XSB/syslib/standard.xwam
... compile:compile1/9  From /home/alvin/git/xsb-code/XSB/cmplib/compile.xwam
... compile:compile_parsed_file/12  From /home/alvin/git/xsb-code/XSB/cmplib/compile.xwam
... standard:call/1  From /home/alvin/git/xsb-code/XSB/syslib/standard.xwam
... consult:compile_f/7  From /home/alvin/git/xsb-code/XSB/syslib/consult.xwam
... consult:reconsult0/3  From /home/alvin/git/xsb-code/XSB/syslib/consult.xwam
... consult:reconsult/1  From /home/alvin/git/xsb-code/XSB/syslib/consult.xwam
... standard:call/1  From /home/alvin/git/xsb-code/XSB/syslib/standard.xwam
... standard:catch/3  From /home/alvin/git/xsb-code/XSB/syslib/standard.xwam

End XSB (cputime 0.02 secs, elapsetime 0.03 secs)
~~~

I've also attached the patch I previously used to mitigate this issue which still works.

Attachments:

• 0001-Avoid-recompiling-when-modification-times-are-equal.patch (sourceforge.net) (922 Bytes; text/x-patch)

---

[bugs:#259] Packaging XSB for Guix

Status: open
Group: Other
Created: Thu Jun 01, 2023 08:50 PM UTC by Alvin Hsu
Last Updated: Fri Feb 09, 2024 04:27 AM UTC
Owner: nobody
Attachments:

• xsb.scm (sourceforge.net) (6.6 kB; text/x-scheme)

Hello,

I would like to share my experience packaging XSB for the Guix package manager, with the aim of revealing any bugs that I may have encountered during the process as well as provide an example package for those who would like to package XSB in other formats.

This report covers potentially more than one bug, so if desirable I can break this up into multiple bug reports instead.

Most notably:

The newerthan syspath_op evaluates to true when the modification timestamps of two files are equal. I'm not sure if this is a bug or not, but its use for checking if files need recompiling is an issue particularly for Guix because builds are put in a read-only store where file timestamps are reset to Unix epoch for reproducibility purposes. This means that when these timestamps are set on build completion, XSB will attempt to rebuild again in a read-only environment and crash. I was able to resolve this by patching needs_recompile(newerthan,PFileName,Obj) in syslib/consult.P to not include cases where they are equal, though (see attached).

It also appears that the build option for./configure may not be working. Guix on my system automatically provides --build=x86_64-unknown-linux-gnu, but it seems that ./makexsb ignores this and uses the default x86_64-pc-linux-gnu as if no build option was provided, thus causing files to not be found.

Please feel free to examine the attached package definition for further analysis. This package may still have quirks that I need to sort out, but it is currently in a usable state for me, and I do plan on submitting it to be included in the official repositories once I feel that I have satisfied the packaging guidelines.

I would be happy to provide any further information or assistance.

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/xsb/bugs/259/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

[Alvin Hsu]

Hi Michael,

The touch commands were meant to emulate an XSB source file having the same modtime as its compiled file. This issue is not unique to xsb_configuration.P; I just used it as an example. Guix sets all file modtimes to the UNIX epoch and installs packages in read-only directories for reproducibility purposes, so this issue should show up with any built-in recompile-able module.

[David S. Warren]

I see I didn't change all the places that file times were checked. I've found and changed more. I will see if it passes the testsuite, and if so, commit the changes. Stay tuned....

[David S. Warren]

I've committed the changes. Alvin, can you try again?

[Alvin Hsu]

Hi David,

It looks like XSB still encounters the same error. The offender seems to be the use of path_sysop/3 on line 227 of consult.P in this case; I've attached a patch that swaps it out for file_strictly_older_than/2 which allows me to start the REPL without an error.

[David S. Warren]

You found another way file times are tested.
OK. I've made changes to the system, as you suggested. Maybe now we've found them all?
Thanks for your patience.

[Alvin Hsu]

I believe I may have found one more case.

Recompilation is triggered for at least a subset of packages in $XSB_DIR/packages. I can get it to reliably reproduce with the following shell script on a fresh build of git-origin (using [wildmatch] as an example, although brief testing of [regmatch] shows the same result):

# In $XSB_DIR

# Attempt to load [wildmatch] package, and see it works fine without
# recompilation.
bin/xsb

# Simulate all timestamps being set as Guix does ($date is arbitrary; just
# has to be constant between the files)

date='2000-01-01 00:00:00'

find . -exec touch --date="$date" {} \;

# Attempt to load [wildmatch] package, and see it gets recompiled.
bin/xsb

This appears to be triggered when config/x86_64-pc-linux-gnu/lib/xsb_wildmatch.xwam and packages/wildmatch/cc/xsb_wildmatch.c have equal timestamps. Unfortunately no patch/trace at the moment.

I have been unable to find any other cases so far, although my testing may not be exhaustive.

[Michael Kifer]

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <style id="user-content-bidiui-paragraph-margins" type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>

I may have found what's wrong with this package. Please try

On 3/1/24 4:36 AM, Alvin Hsu wrote:

<meta http-equiv="content-type" content="text/html; charset=UTF-8">

I believe I may have found one more case.

Recompilation is triggered for at least a subset of packages in $XSB_DIR/packages. I can get it to reliably reproduce with the following shell script on a fresh build of git-origin (using [wildmatch] as an example, although brief testing of [regmatch] shows the same result):

# In $XSB_DIR

# Attempt to load [wildmatch] package, and see it works fine without
# recompilation.
bin/xsb

# Simulate all timestamps being set as Guix does ($date is arbitrary; just
# has to be constant between the files)

date='2000-01-01 00:00:00'

find . -exec touch --date="$date" {} \;

# Attempt to load [wildmatch] package, and see it gets recompiled.
bin/xsb

This appears to be triggered when config/x86_64-pc-linux-gnu/lib/xsb_wildmatch.xwam and packages/wildmatch/cc/xsb_wildmatch.c have equal timestamps. Unfortunately no patch/trace at the moment.

I have been unable to find any other cases so far, although my testing may not be exhaustive.

---

[bugs:#259] Packaging XSB for Guix

Status: open
Group: Other
Created: Thu Jun 01, 2023 08:50 PM UTC by Alvin Hsu
Last Updated: Tue Feb 13, 2024 08:02 PM UTC
Owner: nobody
Attachments:

• xsb.scm (sourceforge.net) (6.6 kB; text/x-scheme)

Hello,

I would like to share my experience packaging XSB for the Guix package manager, with the aim of revealing any bugs that I may have encountered during the process as well as provide an example package for those who would like to package XSB in other formats.

This report covers potentially more than one bug, so if desirable I can break this up into multiple bug reports instead.

Most notably:

The newerthan syspath_op evaluates to true when the modification timestamps of two files are equal. I'm not sure if this is a bug or not, but its use for checking if files need recompiling is an issue particularly for Guix because builds are put in a read-only store where file timestamps are reset to Unix epoch for reproducibility purposes. This means that when these timestamps are set on build completion, XSB will attempt to rebuild again in a read-only environment and crash. I was able to resolve this by patching needs_recompile(newerthan,PFileName,Obj) in syslib/consult.P to not include cases where they are equal, though (see attached).

It also appears that the build option for./configure may not be working. Guix on my system automatically provides --build=x86_64-unknown-linux-gnu, but it seems that ./makexsb ignores this and uses the default x86_64-pc-linux-gnu as if no build option was provided, thus causing files to not be found.

Please feel free to examine the attached package definition for further analysis. This package may still have quirks that I need to sort out, but it is currently in a usable state for me, and I do plan on submitting it to be included in the official repositories once I feel that I have satisfied the packaging guidelines.

I would be happy to provide any further information or assistance.

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/xsb/bugs/259/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

<link itemprop="url" href="https://sourceforge.net/p/xsb/bugs/259/"> <meta itemprop="name" content="View">

<meta itemprop="description" content="View">