File	Date	Author	Commit
bin	2025-02-07	Walter Eaves	[635033] toml0.py now parses the tool.setuptools.package...
conf	2025-04-01	Walter Eaves	[c61341] -
etc	2025-03-31	Walter Eaves	[398c86] -
.gitmodules	2025-03-31	Walter Eaves	[398c86] -
README.org	2025-04-01	Walter Eaves	[23ad99] -
conf.mk	2025-03-31	Walter Eaves	[f322d4] clean build changes
defs.mk	2025-03-31	Walter Eaves	[f322d4] clean build changes
pkg.mk	2025-03-10	Walter Eaves	[c41014] -

Read Me

#+title:  Pitono
#+author: weaves
#+date:   2025-01-28

* Overview

This top-level provides utilities to build and package the modules that are part
of the "pitono" namespace.

** Modules

The namespace contains the following modules.

  pitono.buildr
    pitono.weaves

      pitono.audio

      pitono.tvdb
        pitono.televida

The indentation indicates the dependencies.

They each have their own README files. This document describes features of the
packaging and my own implemenation.

The audio package provides a usable application to build audiobooks.
The televida package provides a package that can rename TV episodes using
remote queries if needed.

There are also other non-Python modules that build or use the utilities.

** Installation and Build

*** Repositories 

The build utility is pitono and is the top-level directory. On SourceForge
this is:

 eavesw@git.code.sf.net/p/pitono/code

If you git clone that - do *not* use recurse-submodules, because you have to
change the .gitmodules to use the remotes:

 eavesw@git.code.sf.net/p/pitono-buildr/code
 eavesw@git.code.sf.net/p/pitono-weaves/code
 eavesw@git.code.sf.net/p/pitono-tvdb/code
 eavesw@git.code.sf.net/p/pitono-renomo/code
 git@github.com:eepgwde/audiobooks.git
 
Then you have to follow get the submodules

 git submodule init ; git submodule update

This should clone the repositories into their directories.

It is worthwhile going into each directory and checking you don't have a
GIT "Detached HEAD" branches.

*** Configuration

You then have to generate each package's pkg.def file.

 make -f conf.mk all-defs-local

*** Python Environment

You need to have a big Python environment: mamba, conda, poetry or venvs.

There is a list of Python modules from mam ba in the file etc/conda.lst.
This should be complete, but beware of missing dependencies and update the
list if needed. A lot of the packages may not be needed. You may not use
Jupyter or Pandas or numpy or the jedi-language-server.

I build on Debian bookworm which has Python 3.11. This system assumes that
some packages are provided by the host OS. In particular, the ABI packages
like python3-mediainfodll and python3-mutagen are best provided by the OS.

The build system adds /usr/lib/python3/dist-packages to the PYTHONPATH for
this purpose.

*** OS Environment

To use the build system, you will need some extra packages. rlwrap is the
readline wrapper used with unittest.

*** Configuring the Packages

You can then add the common .gitignore file to each package.

  make -B -f conf.mk all-ignore-local

Some packages use a common set of test directories and test configuration
files. This is the pitono.buildr.tests.Envs0 module and envsALT() if you
get errors.
  
  make -B -f conf.mk all-tests-local

*** Building and Testing

You then need to build and install the packages in order. Each relies on it predecessor.
You should then enter the package directory and try to check and install it
locally. The local install is an pip editable install no dependencies.

  make -f ../pkg.mk check and the install-local

More sophisticated debugging methods are given below.

**** buildr

This module needs no external resources and should check correctly.

**** weaves

This needs a web-server address. I currently use theia.site0 you can search
for that in the source or look in

  weaves.tests.test_02_main.Test1.testMethod=test_022

**** audio

This is a final package. It needs some external data and some Python
packages. The test data needed is a directory of .m4a files called media/ and a file
listing them called p1.lst.

You can provide links to these in src/pitono/audio/tests/ or use the
environment variable SDIR.

The p1.lst file needs to be created in the tests/ directory and is created
with absolute paths, so something like this:

 find -L $PWD/media -type f -name '*.m4a' > p1.lst

test_02_chapters.py needs to place a test file somewhere. Search for
walser.m4b in the source code.

It then needs some OS packages. the package lists various sources for
these. The Debian Multi-Media Org is the most useful.

 gpac
 python3-mediainfodll

The key test is test_02_chapters.py::Test::test_51

**** tvdb-v4-python

This uses a few .pkl files in the pkls/ directory. These files appear in
the .gitignore be sure they are there. These are accessed by the TVDB Mock
object. This uses the TOP environment variable to locate the directory.

**** televida-renomo

This requires the transmission-rpc package which has to be installed with pip.

This uses test data in the etc/directory. This is referenced in the
configuration files which are found by the test scripts in the tests/home0
directory. The home0/ directory is maintained in the super-directory
pitono/conf/home0.

** Tooling

I work on a Linux system and make use of GNU make to perform the packaging
operations.

*** toml0.py utility

This is a simple utility in the bin/ directory that is used to take some
important settings from the pyproject.toml file and make them available to the
build system's makefiles. It has a dependency on the toml module, which needs to
be installed for it to run.

The pitono.buildr is the source of this script, but, because that package can't
be built in the same way as the other packages if the script isn't present, a
bootstrap copy has been installed in the bin/ directory.

*** Development and Runtime Environments

The Runtime environment is Debian bookworm. This has many python3 packages.
I don't want to install very large packages (like Jupyter or Pandas) for my
scripts to work. So I expect the operating system or PIP installs to
provide small or specialized modules.

The development environment I use is mamba and to be able to share
packages with Debian bookworm, I have a Python 3.11 environment.

To integrate the runtime environment with the development one, I simply add
the runtime environment to PYTHONPATH see later regarding defs.mk 

*** Using Makefiles

This configuration system uses GNU make and makefiles to reconfigure the packages.

**** Common Goals

There are a number of makefile targets (or goals) that are common to the
makefile. They are listed in defs.mk under .PHONY. Of these, view-local can be
used to list the definitions that the other makefiles will use.

**** Top-level

This top-level project has a Makefile called conf.mk

The conf/ directory has a Makefile called git0.mk that is used to update the
local git repositories and pushing to any remote repositories.

Another Makefile is pkg.mk which is used by the individual packages. This
ensures that all the packages operate in the same way.

This includes the defs.mk Makefile that contains definitions.

**** Package Makefiles

Each package has two small Makefiles: pkg.def and pkg.mk. One is generated and
the other needs to be defined and maintained.

  pkg.def is generated by the top-level makefile conf.mk. It contains some control
  information from the pyproject.toml file of the package presented as Makefile
  variable definitions. It is generated by the toml0.py script.

  defs.mk is specific and defined by the package maintainer. It can add
  additional targets. check-local is a library goal (it uses the :: notation),
  so that additional tests can be run specific to a package. See tvdb-v4-python/
  for an example.

*** Top-level Makefiles

**** conf.mk

***** pkg.def creation using the all-defs-local goal

The most important goal in this makefile must be satisfied for the others to be
to built. This rebuilds the pkg.def files in each package.

  make -B -f conf.mk all-defs-local

***** .gitignore management: all-ignore-local

The .gitignore files of the packages have been centralized. To install them as
hard links, use this goal.

***** test data management: all-tests-local

The packages all use the same test dataset - but each may extend it. In the
source directory of each packages, there is a tests directory and this installs
into it, the directories home0 and home1.

The makefile uses another makefile mkr.mk. The design is a bit involved.

***** Scoping the files: all-cscope-local

cscope(1) still works with Python code. To make use of it, you will need a
script to launch a file within your chosen editor. It currently uses my script
emacs0.sh.

***** Licences: install-local

There are other goals that support adding licences to all the source files.
These make use of my local utility script which is known as the variable H_ in
the makefiles.

***** Conda and PIP: all-packages-local

This record the PIP and Conda packages installed in the lists, .lst file, in
etc/. The file local.lst is locally installed packages.

@TODO
I have to migrate packages from the local.lst to conda.lst. The packages have
requirements.txt files that have installed these packages extra to conda.

** Package Build, Test and Install

Each of the packages uses the makefile in the pitono/ build
super-directory.

  make -f ../pkg.mk check
  make -f ../pkg.mk install-local

You will find that

  make -f ../pkg.mk PYTEST=unittest check

And for a test class that fails

  make -f ../pkg.mk PYTEST=unittest UUT='test_00_*.py' check

For example, tests the classes in the Python test classes in the source
code files test_00_*.py.

And some packages have a further check-local 

  make -f ../pkg.mk PYTEST=unittest check-local

That uses a package defined rule in the defs.mk in the package directory.

** Debugging Support

*** spacemacs and anaconda-mode

I work with _spacemacs_ which supports _anaconda-mode_. This supports
running test scripts with the debugger enabled.

Firstly, you need to pass your Python runtime environment to emacs.

You need to have anaconda-mode loaded in spacemacs. This is loaded by
default with the python layer. You then need to customize the following.

   '(pytest-cmd-format-string "cd '%s' && source pkg.sh && cd $PKG_SRC && %s %s '%s'")

These other specializations are useful.

   '(pytest-cmd-flags "--lfnf=none -x -s ")
   '(python-indent-guess-indent-offset t)
   '(python-shell-completion-native-disabled-interpreters '("pypy" "ipython3"))
   '(python-shell-interpreter "python3")

**** pkg.sh

The pkg.sh script is a shell script version of the pkg.def file.

It should look something like this:

  PKG="pitono-televida"
  PKG_VER="0.9.0"
  PKG_SRC="src/pitono/televida"
  TOP="/misc/build/1/pitono/televida-renomo"
  export PKG PKG_VER PKG_SRC TOP

This file can be generated with the toml0 script available from
pitono.buildr

It can be generated with toml0 -o sh. You have to be in the package
directory, ie. where the pyproject.toml file is.

  toml0 -o sh > pkg.sh

** PyProject Packaging

Some notes on how I have implemented my packaging for these modules using the
latest system from PyPa

  https://packaging.python.org/en/latest/guides/packaging-namespace-packages/

*** pyproject.toml files, pkg.def and toml0

The pyproject.toml files are installed as soft links to a file in the etc/
directory of each package. This allows me to use files that are not so
comprehensive and packages can be quickly installed when testing.

A pyproject.toml file must be in place when the top-level conf.mk satisfies 
the all-defs-local goal.

*** Scripts

These are the most useful part of packages. A caller script is generated by the
PyPa install operation. For a pip --user installation they are placed into a directory
called ~/.local/bin/.

The scripts have a peculiar implementation, see pitono.buildr _toml0.py and
pitono.televida _renomo.py.

Looking at the pitono.televida project, there is a section in a pyproject.toml
called

  [project.scripts]
  renomo = "pitono.televida:renomo"

The script that is generated is very simple:

  #!/misc/conda/envs/local/bin/python
  # -*- coding: utf-8 -*-
  import re
  import sys
  from pitono.televida import renomo
  if __name__ == '__main__':
      sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
      sys.exit(renomo())

So for this script to work, you need to have a method renomo() that can be
that can be imported from the module, so in the __init__.py of the module there is

  from ._renomo import renomo

And in the file _renomo.py there is a function called renomo() that is effectively, the
main() method of the script.

If you look at my example, the renomo() method calls a method call main()
which, after parsing the command line arguments, finally calls the methods of
the module in the function renomo0().


* Postamble

#  Local Variables:
#  mode:org
#  mode:auto-fill
#  fill-column: 75
#  coding: utf-8
#  comment-column:50
#  comment-start: "#  "
#  End: