nasm-users

[Nasm-users] nasm intermittently fails to assemble a file

[Jakob B. <jb...@wi...>]

Hi, I am having a strange problem with nasm, which has me somewhat
baffled:

When building a specific project (Openssl 1.0.0a for Win64-amd64),
nasm 2.09.02 (and 2.08.02 and 2.08.01 too) fails to assemble one of the
files (x86_64_cpuid.asm) when run on one build machine, but not the
other.

Strangely, manually running the same nasm command with the same values
of all environment variables on the computer where it just failed,
produces the correct output, but running the build script always fails
on that same computer.

When nasm fails in that buildscript on that machine, it randomly
chooses between two different failure modes:

A (most frequent): nasm reports success (no messages and 0 exit code),
but the generated .obj file is empty (not 0-byte), containing no code,
data or symbols, just a valid .obj file header (This obviously leads
to a later linker error).

B (less frequent): nasm aborts claiming that all the local symbols
are undefined.

On the other build machine, nasm just works as expected for the same
files.

What could be causing this and how do I work around this?

P.S.
The machine in question is an automated build server which has no
trouble building millions of other lines of code (none of them with
nasm though).  It runs Win2003 server (i386), cross compiling for
multiple target architectures (including x86_64).

P.P.S
I am new to this list, hope my question is OK.

P.P.P.S
The error messages when failing in mode B are:

	nasm -f win64 -DNEAR -Ox -g -o tmp32\x86_64cpuid.obj
                                                  tmp32\x86_64cpuid.asm
(above is one long line wrapped for e-mail)
tmp32\x86_64cpuid.asm:5: error: symbol `OPENSSL_cpuid_setup' undefined
tmp32\x86_64cpuid.asm:18: error: symbol `L$spin' undefined
tmp32\x86_64cpuid.asm:54: error: symbol `L$intel' undefined
tmp32\x86_64cpuid.asm:65: error: symbol `L$intel' undefined
tmp32\x86_64cpuid.asm:71: error: symbol `L$intel' undefined
tmp32\x86_64cpuid.asm:81: error: symbol `L$done' undefined
tmp32\x86_64cpuid.asm:84: error: symbol `L$done' undefined
tmp32\x86_64cpuid.asm:86: error: symbol `L$done' undefined
tmp32\x86_64cpuid.asm:91: error: symbol `L$nocacheinfo' undefined
tmp32\x86_64cpuid.asm:104: error: symbol `L$notintel' undefined
tmp32\x86_64cpuid.asm:108: error: symbol `L$notintel' undefined
tmp32\x86_64cpuid.asm:112: error: symbol `L$done' undefined
tmp32\x86_64cpuid.asm:115: error: symbol `L$done' undefined
tmp32\x86_64cpuid.asm:120: error: symbol `L$done' undefined
tmp32\x86_64cpuid.asm:136: error: symbol `L$ot' undefined
tmp32\x86_64cpuid.asm:138: error: symbol `L$ret' undefined
tmp32\x86_64cpuid.asm:143: error: symbol `L$ittle' undefined
tmp32\x86_64cpuid.asm:149: error: symbol `L$aligned' undefined
tmp32\x86_64cpuid.asm:153: error: symbol `L$ot' undefined
tmp32\x86_64cpuid.asm:159: error: symbol `L$aligned' undefined
tmp32\x86_64cpuid.asm:161: error: symbol `L$ittle' undefined
NMAKE : fatal error U1077: '!NASM208DIR!\nasm.EXE' : return code '0x1'

(The symbols in question are defined on lines before they are used, the
errors report the lines where they are used).

Re: [Nasm-users] nasm intermittently fails to assemble a file

[Frank K. <fbk...@zy...>]

Jakob Bohm wrote:

 > P.P.S
 > I am new to this list, hope my question is OK.

No. Too difficult. Take it away! :)

Seriously, welcome aboard, Jakob! I've cleared your "moderation" flag, 
so you should be able to post without delay, from now on. [hint: "reply 
all" to copy the list - just "reply" will reply only to me]

> Hi, I am having a strange problem with nasm, which has me somewhat
> baffled:

Me, too! Someone more knowledgeable is probably going to have to bail me 
out on this...

> When building a specific project (Openssl 1.0.0a for Win64-amd64),
> nasm 2.09.02 (and 2.08.02 and 2.08.01 too) fails to assemble one of the
> files (x86_64_cpuid.asm) when run on one build machine, but not the
> other.

So... what differs? Same build of Nasm? Same Perl?

I downloaded the source, but I can't find the file in question (or even 
the tmp32 directory). I deduce that this is built from 
crypto/x86_64cpuid.pl... perhaps with the assistance of 
crypto/perlasm/x86_64-xlate.pl and/or x86nasm.pl? Closest thing I see on 
my system (*old* Linux) after doing "make" is x86cpuid.s, which doesn't 
help. Can you give me a "dummies' guide" to creating this file? (or post 
it?)

Better yet(?), can you confirm that the .asm files on the "good" machine 
and the "bad" machine are identical?

> Strangely, manually running the same nasm command with the same values
> of all environment variables on the computer where it just failed,
> produces the correct output, but running the build script always fails
> on that same computer.

That makes it sound like the problem is external to Nasm, no?

> When nasm fails in that buildscript on that machine, it randomly
> chooses between two different failure modes:

I don't think Nasm includes a random number generator, so this too 
points to something other than Nasm - although I can't imagine what.

> A (most frequent): nasm reports success (no messages and 0 exit code),
> but the generated .obj file is empty (not 0-byte), containing no code,
> data or symbols, just a valid .obj file header (This obviously leads
> to a later linker error).
> 
> B (less frequent): nasm aborts claiming that all the local symbols
> are undefined.

Very strange, no matter what's doing it!!!

> On the other build machine, nasm just works as expected for the same
> files.

Yeah, that's what it's "supposed" to do... :)

> What could be causing this

Dunno.

> and how do I work around this?

If we figure out the first question, we might be able to figure out this 
one. I suppose "don't use that machine" isn't an option...

> P.S.
> The machine in question is an automated build server which has no
> trouble building millions of other lines of code (none of them with
> nasm though).  It runs Win2003 server (i386), cross compiling for
> multiple target architectures (including x86_64).

I'm paranoid enough to think that MS might introduce something that 
would cause random bugs in Nasm. Not paranoid enough to think it's very 
likely, though. :)

So we've got a 32-bit Nasm executable, assembling for "-f win64". That 
"shouldn't" be a problem...

> P.P.S
> I am new to this list, hope my question is OK.

Sure. I'm delighted to have a message I can approve (mostly, my job on 
this list is rejecting spam). I'm looking forward to a question I can 
actually answer!

> P.P.P.S
> The error messages when failing in mode B are:
> 
> 	nasm -f win64 -DNEAR -Ox -g -o tmp32\x86_64cpuid.obj
>                                                   tmp32\x86_64cpuid.asm
> (above is one long line wrapped for e-mail)

I can tell you that "-g" is being silently ignored. Shouldn't be a 
problem. You may want to leave it in, to be ready if/when we *do* add 
debug-info to the win32/win64 output formats.

> tmp32\x86_64cpuid.asm:5: error: symbol `OPENSSL_cpuid_setup' undefined

[snip more-of-same presumably from the same cause...]

> (The symbols in question are defined on lines before they are used, the
> errors report the lines where they are used).

Right. Defined as "extern" in the case of "OPENSSL_cpuid_setup" - just 
labels for the rest of 'em?

But other times, no errors but a "naked" header, right? Exact same .asm 
file, or a new Perl output?

Damn! I can't imagine what this could be, Jakob! I'm inclined to think 
"not a Nasm problem", but I'm not absolutely certain.

Anyone else (perhaps with Windows experience) got any ideas?

In any case, thanks for the feedback!

Best,
Frank

Re: [Nasm-users] nasm intermittently fails to assemble a file

[Jakob B. <jb...@wi...>]

On 13-10-2010 01:40, Frank Kotler wrote:
> Jakob Bohm wrote:
>
>  > P.P.S
>  > I am new to this list, hope my question is OK.
>
> No. Too difficult. Take it away! :)
>
> Seriously, welcome aboard, Jakob! I've cleared your "moderation" flag,
> so you should be able to post without delay, from now on. [hint: "reply
> all" to copy the list - just "reply" will reply only to me]
>
>> Hi, I am having a strange problem with nasm, which has me somewhat
>> baffled:
>
> Me, too! Someone more knowledgeable is probably going to have to bail me
> out on this...
>
>> When building a specific project (Openssl 1.0.0a for Win64-amd64),
>> nasm 2.09.02 (and 2.08.02 and 2.08.01 too) fails to assemble one of the
>> files (x86_64_cpuid.asm) when run on one build machine, but not the
>> other.
>
> So... what differs? Same build of Nasm? Same Perl?
>

I tried copying over the same nasm.exe, same result

Different perl though, but the asm file looks fine, see also below

> I downloaded the source, but I can't find the file in question (or even
> the tmp32 directory). I deduce that this is built from
> crypto/x86_64cpuid.pl... perhaps with the assistance of
> crypto/perlasm/x86_64-xlate.pl and/or x86nasm.pl? Closest thing I see on
> my system (*old* Linux) after doing "make" is x86cpuid.s, which doesn't
> help. Can you give me a "dummies' guide" to creating this file? (or post
> it?)
>

The .asm file is generated by the makefile just before invoking nasm, 
this is upstreams buildsystem, so I have limited control over what it 
does or how.

On a higher level, the sequence of commands is

...\openssl-1.0.0a>perl Configure VC-WIN64A --prefix=C:\somedir
...\openssl-1.0.0a>call ms\do_win64a
...\openssl-1.0.0a>nmake -f ms\nt.mak install


> Better yet(?), can you confirm that the .asm files on the "good" machine
> and the "bad" machine are identical?
I haven't checked, but the file looks normal in a text editor after the
failure, see also below.

>
>> Strangely, manually running the same nasm command with the same values
>> of all environment variables on the computer where it just failed,
>> produces the correct output, but running the build script always fails
>> on that same computer.
>
> That makes it sound like the problem is external to Nasm, no?
>

Yeah, but in this case it IS the exact same .asm file and nasm binary 
that failed a few seconds or minutes before, which makes me wonder what
could induce nasm to behave differently.

>> When nasm fails in that buildscript on that machine, it randomly
>> chooses between two different failure modes:
>
> I don't think Nasm includes a random number generator, so this too
> points to something other than Nasm - although I can't imagine what.
>
Uninitialized memory, process id, system load could be factors if one of 
those might trigger a latent bug.
>> A (most frequent): nasm reports success (no messages and 0 exit code),
>> but the generated .obj file is empty (not 0-byte), containing no code,
>> data or symbols, just a valid .obj file header (This obviously leads
>> to a later linker error).
>>
>> B (less frequent): nasm aborts claiming that all the local symbols
>> are undefined.
>
> Very strange, no matter what's doing it!!!
>
>> On the other build machine, nasm just works as expected for the same
>> files.
>
> Yeah, that's what it's "supposed" to do... :)
>
>> What could be causing this
>
> Dunno.
>
>> and how do I work around this?
>
> If we figure out the first question, we might be able to figure out this
> one. I suppose "don't use that machine" isn't an option...
>
>> P.S.
>> The machine in question is an automated build server which has no
>> trouble building millions of other lines of code (none of them with
>> nasm though). It runs Win2003 server (i386), cross compiling for
>> multiple target architectures (including x86_64).
>
> I'm paranoid enough to think that MS might introduce something that
> would cause random bugs in Nasm. Not paranoid enough to think it's very
> likely, though. :)
>
> So we've got a 32-bit Nasm executable, assembling for "-f win64". That
> "shouldn't" be a problem...
>
>> P.P.S
>> I am new to this list, hope my question is OK.
>
> Sure. I'm delighted to have a message I can approve (mostly, my job on
> this list is rejecting spam). I'm looking forward to a question I can
> actually answer!
>
>> P.P.P.S
>> The error messages when failing in mode B are:
>>
>> nasm -f win64 -DNEAR -Ox -g -o tmp32\x86_64cpuid.obj
>> tmp32\x86_64cpuid.asm
>> (above is one long line wrapped for e-mail)
>
> I can tell you that "-g" is being silently ignored. Shouldn't be a
> problem. You may want to leave it in, to be ready if/when we *do* add
> debug-info to the win32/win64 output formats.
>
Again upstream build system
>> tmp32\x86_64cpuid.asm:5: error: symbol `OPENSSL_cpuid_setup' undefined
>
> [snip more-of-same presumably from the same cause...]
>
>> (The symbols in question are defined on lines before they are used, the
>> errors report the lines where they are used).
>
> Right. Defined as "extern" in the case of "OPENSSL_cpuid_setup" - just
> labels for the rest of 'em?
>
Yep
> But other times, no errors but a "naked" header, right? Exact same .asm
> file, or a new Perl output?
Fresh run of perl each time starting from an empty dir, unpacking source
and running the upstream build system from a batch file.
>
> Damn! I can't imagine what this could be, Jakob! I'm inclined to think
> "not a Nasm problem", but I'm not absolutely certain.
>
> Anyone else (perhaps with Windows experience) got any ideas?
>
> In any case, thanks for the feedback!
>
You're welcome...

Re: [Nasm-users] nasm intermittently fails to assemble a file

[anonymous c. <nas...@us...>]

> The .asm file is generated by the makefile just before invoking nasm
> ...
> Yeah, but in this case it IS the exact same .asm file and nasm binary
> that failed a few seconds or minutes before, which makes me wonder
> what could induce nasm to behave differently.

Does your build system ensure serialization between the Perl script
that generates the .asm file and the invocation of the NASM binary?

Another possibility is bad buffering, i.e. NASM invoked immediately
may see a partial .asm file (not from disk, but a memory buffer), but
NASM invoked a little later may see the proper .asm file (from disk).

You could try to replace the NASM binary with e.g. cat, to show the
content of the .asm file. Though... if that doesn't show a partial .asm
file, then you might need to "work your way backwards" by building a
custom NASM binary that prints out every input line. (Of course that
might change buffering enough to hide/avoid the problem.)

Re: [Nasm-users] nasm intermittently fails to assemble a file

[Jakob B. <jb...@wi...>]

On 13-10-2010 06:15, anonymous coward wrote:
>> The .asm file is generated by the makefile just before invoking nasm
>> ...
>> Yeah, but in this case it IS the exact same .asm file and nasm binary
>> that failed a few seconds or minutes before, which makes me wonder
>> what could induce nasm to behave differently.
>
> Does your build system ensure serialization between the Perl script
> that generates the .asm file and the invocation of the NASM binary?
>

Yep, that was it!  Now I have to figure out how to make the upstream
build system (not my own), ensure the perl run does not exit until it
is done writing the file.  This is a classic non-parallel makefile
system, so this really shouldn't have happened, but it did.

> Another possibility is bad buffering, i.e. NASM invoked immediately
> may see a partial .asm file (not from disk, but a memory buffer), but
> NASM invoked a little later may see the proper .asm file (from disk).
>
Equally possible, given the circumstances above.

> You could try to replace the NASM binary with e.g. cat, to show the
> content of the .asm file. Though... if that doesn't show a partial .asm
> file, then you might need to "work your way backwards" by building a
> custom NASM binary that prints out every input line. (Of course that
> might change buffering enough to hide/avoid the problem.)

I tried that (with a custom clone of cat, since cat is not a standard
binary on Windows and a ported cat would barf at the other commandline
args anyway), and it showed that the .asm file was either empty or
missing when "nasm.exe" started.

Thanks for the help

Jakob Bohm