Re: [FFADO-devel] FFADO-mixer and python

[Jonathan W. <jw...@ju...>]

Hi Vladimir

Thanks for taking the time to work through this.

On Wed, Mar 11, 2020 at 08:01:36AM +0300, Vladimir Sadovnikov wrote:
> I'm using latest scons installed by:
>      zypper rm scons3
>      pip3 install --upgrade scons

Could you confirm the version number by running "scons --version"?

Update: I see from the scons github link you provided later on that scons is
v3.1.2 running under python 3.6.10.

> Let me explain again why this line is illegal.
> Let's look into SConstruct script. Here we're detecting C++ flags to use for
> specific libraries:
>          pkgs = {
>              'libraw1394' : '2.0.5',
>              'libiec61883' : '1.1.0',
>              'libconfig++' : '0'
>              }
> 
>          if env['REQUIRE_LIBAVC']:
>              pkgs['libavc1394'] = '0.5.3'
> 
>          if not env['SERIALIZE_USE_EXPAT']:
>              if conf.CheckPKG('libxml++-3.0'):
>                  pkgs['libxml++-3.0'] = '3.0.0'
>              if not('libxml++-3.0' in pkgs):
>                  pkgs['libxml++-2.6'] = '2.13.0'

I have only peripheral experience with scons and how it works, so my
understanding of how this all fits together is not extensive.

The above code is, I think, primarily about setting minimum version numbers
in "pkgs".  Perhaps conf.CheckPKG() stores flags too - I wouldn't know.

> Then, we're saving these flags into environment variables:
>          for pkg in pkgs:
>              name2 = pkg.replace("+","").replace(".","").replace("-","").upper()
>              env['%s_FLAGS' % name2] = conf.GetPKGFlags( pkg, pkgs[pkg] )
>              print('%s_FLAGS = %s' % (name2, env['%s_FLAGS' % name2].decode())) # This line added by me for debug
>              if env['%s_FLAGS'%name2] == 0:
>                  allpresent &= 0

I think this code both fetches the flags and stores them in "env".  The
flags it fetches take the previously set version numbers into account.

> The code below outputs the line:
>      LIBRAW1394_FLAGS = -lraw1394
> And then we're omitting these flags just by rewriting environment:
>      config_guess = conf.ConfigGuess()
> 
>      env = conf.Finish()

The ConfigGuess() call just fetches the system triple.  I presume you're
refering to the last line - the conf.Finish() call.

The scons 3.1.2 manual describes the Finish() methods as follows.

    This method should be called after configuration is done.  It returns
    the environment as modified by the configuration checks performed. 
    After this method is called, no further checks can be performed with
    this configuration context.  However, you can create a new Configure
    context to perform additional checks.  Only one context should be active
    at a time.

This indicates to me that it is not wrong to call conf.Finish(), and in fact
it appears to be necessary once the Configure instance has been finished
with.  I guess the question is whether the return value should be assigned 
to the "env" variable, since that is the "rewriting" you would be referring
to.

> Then we call the nested src/SConstruct script:
>      subdirs=['src','libffado','support','doc']
>      if env['BUILD_TESTS']:
>          subdirs.append('tests')
> 
>      print("!!!!!!!! %s\n" % (env.Dump())) # Added by me for debug
> 
>      env.SConscript( dirs=subdirs, exports="env" )
> The code shows that there is no LIBRAW1394_FLAGS variable in environment more.

I see.

> And in src/SConstruct we try to reuse it:
>      libenv = env.Clone()
>      #...
>      if not env.GetOption( "clean" ):
>          libenv.MergeFlags( "-lrt -lpthread" )
>          libenv.MergeFlags( env['LIBRAW1394_FLAGS'].decode() ) # BOOM!
> :
> Finally, the build crashes with error:
>      KeyError: 'LIBRAW1394_FLAGS':
>        File "/home/sadko/tmp/lv2/ffado/libffado/SConstruct", line 925:
>          env.SConscript( dirs=subdirs, exports="env" )
>        File "/usr/lib/python3.6/site-packages/scons/SCons/Script/

Certainly if LIBRAW1394_FLAGS isn't in the environment one would expect a
problem here.

What I can't understand at this point is that if it is wrong to assign "env"
to the return of conf.Finish(), how has this evidently been working fine up
until now?  The conf.Finish() line was added in r774 on 24 Dec 2007.  In the
12 years since then it has apparently been running fine until now.  This
same SConstruct script works fine on my development system at home, for
example.  Do you have any ideas?

The other puzzling thing is that given the scons manual page description of
conf.Finish(), I don't understand how things could be working for you if
this call is omitted.  The manual seems to suggest that the Finish() call is
required.

> Here's the related topic in SCons github repository which was immediately
> closed by maintainers:
> https://github.com/SCons/scons/issues/3579

Unfortunately this doesn't give any indication as to what the problem
might be.

> So we have improperly working build script with ridiculous behaviour.

Personally I think that's a bit harsh, given that the script with the line
in question has worked for over 12 years and continues to do so on a
significant number of systems.

Looking a bit closer at your report on the above github issue, I note that
you are specifying a large number of build options manually (reproduced at
the end of this message).  This may go part of the way towards explaining
why no one else has encountered this problem before.  It could well be that
no one has attempted to compile with this precise mix of options before. 

Is there a particular reason why you are specifying compilation-related
options?  In most situations the scons build system correctly determines
what they should be.

In particular, the deprecated COMPILE_FLAGS option is in use.  Although I
don't think it's related to your issue, it would be a good idea to follow
the suggestion from SConstruct and use CFLAGS, CXXFLAGS and CUSTOM_ENV=True.
When using COMPILE_FLAGS, CUSTOM_ENV=True is forced.

With CUSTOM_ENV=True, the build environment will make use of the following
predefined environment variables if they are present in the environment used
to run scons: CC, CXX, CFLAGS, CXXFLAGS, LDFLAGS.  The content of
COMPILE_FLAGS will also be merged.

You have included

  LIBRAW1394_FLAGS=-O2

as a parameter variable when calling scons as well as defining it in the
environment.  The parameter variable won't take any effect because we don't
define such a variable.  Thus any propagation of LIBRAW1394_FLAGS into scons
can only come from the os.environ environment.

At present I cannot see why things are broken on your system but not on any
others.  I suspect it's related to the mix of options being explicitly 
set, but determining the exact cause will take some time.  Perhaps the
external definition of the LIBRAW1394_FLAGS environment variable is
somehow interfering with the operation of conf.GetPKGFlags().

I am not sure when I'll get a chance to look into this.  In the meantime I
would welcome any further insights you might be able to provide.

One thing which would be of particular interest is whether a "normal" build
of FFADO works for you.  In other words, in a fresh shell with a default
environment (without LIBRAW1394_FLAGS or anything else), run

  scons --config=force -j 8

and see what happens.  It may even be worth doing this in a freshly
extracted source tree.

Regards
  jonathan

==== Build script which causes the error on Vladimir's system ====

#!/bin/bash

export SCONS="python3 `which scons`"

echo $SCONS

export LIBRAW1394_FLAGS=-O2

$SCONS -c
$SCONS --config=force -j 8 \
    PREFIX=/usr \
    LIBDIR=/usr/lib64 \
    MANDIR=/usr/share/man \
    UDEVDIR=/usr/lib/udev/rules.d \
    ENABLE_GENERICAVC=yes \
    SERIALIZE_USE_EXPAT=no \
    DEBUG=no \
    ENABLE_ALL=yes \
    PYPKGDIR=/usr/lib/python3.6/site-packages \
    ENABLE_OPTIMIZATIONS=yes \
    ENABLE_SETBUFFERSIZE_API_VER=auto \
    BUILD_TESTS=yes \
    ENABLE_DICE=true \
    COMPILE_FLAGS='-O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -fno-strict-aliasing -ggdb -Wno-deprecated-declarations --std=gnu++11' \
    LIBRAW1394_FLAGS=-O2 \
    CUSTOM_ENV=True