UsingJenkinsCi

Using Jenkins CI To Build OpenVMS Programs

The Jenkins CI system does not run on OpenVMS and I do not know of any existing ports of it.

It can be run on Linux system to direct builds on an OpenVMS system.

Commercial support for using Jenkins CI for OpenVMS building is available from eCube Systems. This article does not cover the eCube Systems implementation. You can visit their site for information on their product.

This article will eventually document the how I set up and use a Jenkins system running on Fedora Core 22 Linux for doing builds.

The major files and scripts will be stored in the Jenkins Support Mecurial repository.

The overview of this implementation is that the Jenkins server and all the OpenVMS builders share a common NFS mounted disk, and the Jenkins build agents are ssh build agents the same Jenkins system, but separate from master. Each Jenkins build agent is dedicated to a specific OpenVMS host at this time. The Jenkins build agent sends OpenVMS commands to the actual build system.

Jenkins Build Host

The Jenkins build host is currently a 64 bit PC running Fedora Core 22.

Along with Jenkins, this system is also an NFS server, and should also be able to support containers and VMs.

Fedora 22 can be configured to serve NFS v2, which allows me to use it with OpenVMS/VAX 7.3 and the HP TCP/IP version that is bundled with it. I have not yet tried an older version.

I tried to run the Jenkins war file on OpenVMS/IA64 8.4 using Tomcat. I got no output, and did not want to take on the project of debugging it.

The Jenkins Build Host can set environment variables that will be passed to the build jobs. I set environment variables to indicate the local directory on the build agent where the mirrors will be store.

For example: GNV_MIRRORS is for mirrors of the repositories in the GNV project, GNU_MIRRORS is for mirrors of the GNU repositories.

This allows me to change the storage location with out having to change the Jenkins build jobs if I need to reorganize the disk storage.

OpenVMS Build Agents

I tried to use the OpenVMS/IA64 8.4 as a build agent, first as an SSH launched agent. It turns out that the HP TCP/IP SSHD client writes out some banner text that causes Jenkins to reject the connection.

I then tried to have it connect to a Java client started on that OpenVMS system. That also did not work.

So what I ended up doing is starting up SSH connected SSH build agents on the Jenkins build host. This is different than using additional build executors on the Jenkins master build host.

What Jenkins considers a build agent, also known as a Node, is not actually a network node. In the case of an SSH build agent, a node is a combination of a build host, and a unique directory on it.

You can use specialized Jenkins build agents to create rate limited queues to resources used for building.

I use a directory convention of "/home/username/hostname/vmsnode_number". Where the username is the username used by the Jenkins build agent. In my case "jeeves". The hostname is the for the Jenkins build agent host. The vmsnode is the host name of the OpenVMS build agent. And the number is currently set to 1 to allow for adding additional Jenkins build agents to use the same OpenVMS host.

One example in a future might be be to set up a alternate cross build environment where the OpenVMS build environment uses logical names and foreign commands to build for a different version or even a different architecture of OpenVMS.

I currently use one of these build agents dedicated to each OpenVMS system. The build agents are given labels to support the OpenVMS version, platform, and other attributes specific to that system to allow jobs to select them. Currently I have only enabled one build executor per build agent to not overload my systems.

The build agents get a label of "VMS", and a label of "ODS5" if the OpenVMS build agent supports ODS5.
The most important is a label with "VMS_ARCH_VERSION" where ARCH is currently one of VAX, AXP, or IA64 and VERSION is a 4 digit numeric code for the version. 0840 is used for 8.4 for example.

The build agent sets up environment variables that are passed to the build jobs.

• REMOTECMD - Either "ssh" or "rsh -l $USER" for SSH or RSH
• VMS_CMD_DIR - The VMS directory that build commands scripts are in.
• VMSHOST - The VMS hostname for the Jenkins build agent to connect to.

Communicating With the OpenVMS Build Agent

i set up an account named Jeeves on the Jenkins build node and all of the OpenVMS systems.

SSH keys were set up for password less connection.between all the systems.

DECNET proxies were set up for communication between the OpenVMS systems. Currently the build procedure is not taking advantage of that.

While this worked from a Linux shell session, I discovered that a Jenkins Linux build agent build step could not ssh to either my OpenVMS/AXP 8.3 with HP TCP/IP, or with the OpenVMS/VAX 7.3 system.

For those I had to resort to setting up RSH as noted above.

To enable RSH on Fedora, you have to add a special firewall rule.

With HP TCP/IP on OpenVMS you just need to enable RSH and then add RSH communication proxies.

The OpenVMS builder account needs plenty of certain quota. A configure script and make process can get several process levels deep.

A sample OpenVMS builder SYSUAF entry:

Username: JEEVES                           Owner:  Jenkins
Account:  CI                               UIC:    [621,145] ([CI,JEEVES])
CLI:      DCL                              Tables: DCLTABLES
Default:  USER_ROOT:[JEEVES]
LGICMD:
Flags:
Primary days:   Mon Tue Wed Thu Fri
Secondary days:                     Sat Sun
No access restrictions
Expiration:            (none)    Pwdminimum:  6   Login Fails:     0
Pwdlifetime:           (none)    Pwdchange:  20-FEB-2016 13:03
Last Login: 10-JUL-2016 11:31 (interactive), 28-JUL-2016 11:20 (non-interactive)
Maxjobs:         0  Fillm:      4096  Bytlm:       400000
Maxacctjobs:     0  Shrfillm:      0  Pbytlm:           0
Maxdetach:       0  BIOlm:       150  JTquota:      30000
Prclm:          30  DIOlm:       150  WSdef:         2000
Prio:            4  ASTlm:       250  WSquo:         4000
Queprio:         4  TQElm:       100  WSextent:    200000
CPU:        (none)  Enqlm:      2000  Pgflquo:    2300000
Authorized Privileges:
  NETMBX       TMPMBX
Default Privileges:
  NETMBX       TMPMBX
Identifier                         Value           Attributes
  NORMAL_USER_CLASS                %X80010001

Exchanging files with the OpenVMS Build Agent

The build agent host has an NFS directory that is mounted by the OpenVMS hosts. This is used to share data between the build agent and the OpenVMS host.

This directory can be shared in the reverse direction with an OpenVMS system hosting it. Currently though I have a lot more disk space available on the Jenkins host than on the OpenVMS systems.

Note: You should make sure that the the OpenVMS NFS clients are having their time set by the same time standard as the NFS server.

As mentioned earlier, HP TCP/IP on OpenVMS/VAX needs NFS V2, which can be enabled on Fedora 22.

Jenkins Job Chains

I have set up Jenkins jobs for each external repository and branches that I am building for. Currently all these jobs are doing is locally mirroring the repository. In the future for the branches that need it, the job could run the autoconf step as this can be done on Linux, which puts the repository in a state for doing the OpenVMS build.

For some products, there are only zip or compressed tar archives available. For those I just have a job that unpacks the archive, mainly as a place holder. In the future I may look at automation to look for updates on the Internet.

The jobs that do the building monitor the mirror jobs, so will start if a mirror copy gets updated.

Jenkins Build Job Structure

Jenkins Build Agent Job Steps

When Jenkins starts a build job, it sets a number of environment variables, including the ones described earlier.

What I do is use that data to write a file named workspace.ini.

This is so that if I need to troubleshoot or debug the build, I have the same information as Jenkins did.

Jenkins sets a WORKSPACE environment variable with the directory used for the job.

I then launch a more general build script, passing it the value of the WORKSPACE environment variable so that it can find the workspace on the NFS served volume.

printf "PROJECT_SHORT_NAME bash\n" > workspace.ini
printf "REFERENCE_SRC $GNV_MIRRORS/bash/V4.3/reference\n" >> workspace.ini
printf "VMS_SOURCE $GNV_MIRRORS/bash/V4.3/vms_source\n" >> workspace.ini
printf "WORKSPACE $WORKSPACE\n" >> workspace.ini
printf "REPO_SIZE `du --summarize --block-size=512 $GNV_MIRRORS/bash/V4.3 | cut --fields=1`\n" >> workspace.ini
printf "BUILD_DEFAULT PRJ_ROOT:[bash]\n" >> workspace.ini
printf "BUILD_SCRIPT_1 build_bash.com\n" >> workspace.ini
printf "BUILD_SCRIPT_2 stage_bash_install.com\n" >> workspace.ini
printf "BUILD_SCRIPT_3 pcsi_product_gnv_bash.com\n" >> workspace.ini
printf "PCSI_PRODUCER GNV\n" >> workspace.ini
printf "PCSI_PRODUCER_FULL_NAME Eric W. Robertson/John E. Malmberg\n" >> workspace.ini

sleep 10s

$REMOTECMD $VMSHOST @${VMS_CMD_DIR}build_simple_vms_project.com \"$WORKSPACE\"

OpenVMS Directory Structure Used By Build Agents

The directory that is used for GNV should have the logical name GNV$GNU pointing at that. Installing any of the updated GNV tools like GNV Bash will install a command procedure into [vms$common.sys$startup].

The common directory shared by Jenkins and the OpenVMS systems either needs to be exported/served by OpenVMS or mounted by OpenVMS. I chose to mount an NFS export from Linux on OpenVMS.

In this case jazz_root is the /home directory from the Jenkins server. The directory structure of interest to the Jenkins and OpenVMS system is in the home directory of the Linux Jeeves account, of interest to us are three areas.

• Workspace directories which are job specific.
• Mirror directory - where repository mirrors are set up.
• OpenVMS script directory - common build scripts for OpenVMS.

Note my startup files are modularized and set up so that they can also be run after the system is booted if I make changes to them.

$if .not. f$getdvi("JAZZ_ROOT","exists")
$then
$   if arch_name .nes. "VAX"
$   then
$       nfs_opts="/transport=UDP/structure=5"
$   else
$       nfs_opts=""
$   endif
$   tcpip mount dnfs: jazz jazz_root -
    /host=jazz-san.xile.realm/path="/home"-
    /system'nfs_opts'/acp_params=(buffer_limit=819200)/timeout=00:00:05
$endif

I have two Perl versions on my system. The HP provided Perl 5.8.6 kit, and one of Craig Berry's updated VMS PERL Kits

Some program depend on the HP provided perl, so it is the default Perl on the system.

I set up a logical name so that the login.com of the builders can easily set up for the newer Perl with out having to track the version.

$! Perl - HP Distribution
$!------------------------
$perl_root_dir = "perl5_8_6"
$file = "sys$common:[000000]''perl_root_dir'.dir"
$if f$search(file) .nes. ""
$then
$   define/system/exec/trans=concealed perl_root -
        sys$sysdevice:[vms$common.'perl_root_dir'.]
$   file = "perl_root:[000000]perlshr.exe"
$   if f$search(file) .nes. ""
$   then
$       define/system/exec perlshr 'file'
$   endif
$   if f$file_attributes("perlshr","known")
$   then
$       install replace perlshr/open/share/header
$   endif
$endif
$!
$!
$ define/system/exec perl5_new disk$bigdisk:[vms$common.perl-5_18]

A local rooted directory was set aside for the use of the OpenVMS Build agent and the contents needs to be writable by the OpenVMS build agent account. Since I currently only have one build agent account, I have it own it.

The directory levels are set up to allow multiple build agent accounts, Jenkins build agents and OpenVMS build agents,

• JEEVES - The username for the OpenVMS build agent.
• JAZZ - The hostname for the Jenkins build agent.
• EAGLE_1 - The hostname for the OpenVMS build agent and the tag number in case there are multiple build agents on that host.
• WORKSPACE - Used pretty much the same as the Jenkins workspace directory.
• JENKINS_JOB_NAME - The jenkins job name. It is generally good practice to use Jenkins job names that can be used as directories with out special quoting or escape characters, since Jenkins uses them as directories.

$define_concealed jenkins_root disk$bigdisk:[jenkins_root.]

Directory DISK$BIGDISK:[000000]

JENKINS_ROOT.DIR;1         1   5-JUL-2016 20:45:52.14  [CI,JEEVES]
        (RWE,RWE,RE,RE)

Directory JENKINS_ROOT:[JEEVES.JAZZ.EAGLE_1.WORKSPACE]

AR_TOOLS_AXP_0840.DIR;1
                           1   5-JUL-2016 21:20:13.06
BASH_AXP_0840.DIR;1
                           1   5-JUL-2016 21:22:16.66

In the job directory a typical build job creates the following directories, this can be adjusted based on the needs of the build procedure. The release notes for the updated GNV components describe these in more detail.

In general the build job should clean out this directory before loading it up with files for a new build to make sure a project can build from the same source that is in change control.

• lcl_root - This is where all the build products are put.
• new_gnu - This is used for local changes to the GNV$GNU directory tree, for running the GNU install procedures, etc.
• src_root - This is typically the unmodified upstream source for most projects.
• vms_root - This is typically the OpenVMS specific files that were added to the project to get it to build.

Directory JENKINS_ROOT:[JEEVES.JAZZ.EAGLE_1.WORKSPACE.BASH_AXP_0840]

lcl_root.DIR;1             1   5-JUL-2016 21:22:16.93
new_gnu.DIR;1              1   5-JUL-2016 21:22:16.95
src_root.DIR;1             1   5-JUL-2016 21:22:16.90
vms_root.DIR;1             1   5-JUL-2016 21:22:16.91

The stage_root directory is a location that the OpenVMS build agent will place any PCSI kits or ZIP archives to potentially be uploaded for distribution.

The workspace.ini will be described later. It contains the build information needed by the OpenVMS build agent to do the build.

Other files may be placed there by the build like test_output.xml. This is a junit format file that contains the results of running self tests by the build procedure. The Jenkins jUnit plugin will display reports from this file and track the number of tests that have run, passed, failed, and are skipped between jobs.

Directory JAZZ_ROOT:[jeeves.jazz.eagle_1.workspace]

ar_tools_axp_0840.DIR;1
                           1  11-JUL-2016 08:07:51.45
bash_axp_0840.DIR;1
                           1  28-JUL-2016 07:17:33.45

Directory JAZZ_ROOT:[jeeves.jazz.eagle_1.workspace.gnu_awk_4_1_axp_0840]

stage_root.DIR;1           1  23-JUL-2016 18:54:28.77
test_output.xml;1         52  23-JUL-2016 19:14:15.57
workspace.ini;1            1  23-JUL-2016 18:54:17.08

The mirror directory lays has a directory tree for each branch of each repository. Each repository host has its own root directory to prevent repository name collisions. Each branch is also in its own subdirectory.

Directory JAZZ_ROOT:[jeeves.mirrors.gnv]

ar_tools.DIR;1             1  25-FEB-2016 20:08:40.67
bash.DIR;1                 1  25-FEB-2016 20:48:14.90

Directory JAZZ_ROOT:[jeeves.mirrors.gnv.bash]

V4^.3.DIR;1                1  28-JUL-2016 07:17:12.56

The OpenVMS script directory holds scripts that are common to building. These files are available in the jenkins_support HG repository in the vms_agent directory.

Directory JAZZ_ROOT:[jeeves.vms]

build_simple_vms_project.com;47
                          15   4-JUL-2016 17:47:43.46
get_workspace_info.com;39
                          15  10-JUL-2016 17:53:43.82
setup_logical_disks.com;15
                           8   7-MAR-2016 22:24:23.16

The OpenVMS builder sys$login:login.com helps set up local information for the build.

$!
$! GNV Make leaves these behind.
$ if f$search("sys$login:make*.;") .nes. "" then delete sys$login:make*.;*
$!
$@perl5_new:perl_setup.com
$!
$ file = "sys$login:tcpip$ssh_rcmd.log"
$ if f$search(file) .nes. ""
$ then
$   purge/keep=3 'file'
$ endif
$!
$ file = "sys$login:tcpip$rsh_run.log"
$ if f$search(file) .nes. ""
$ then
$   purge/keep=3 'file'
$ endif

There is a file in the OpenVMS builder default directory that is used to find where the Jenkins shared directory is and where the local Jenkins work directory is located.

This makes it so that none of the OpenVMS build scripts are dependent on a particular disk configuration.

EAGLE> type user_root:[jeeves]jenkins_defaults.ini
! sys$login:jeeves_defaults.ini
!
! Local JEEVES defaults for building
!
vms_work_root jazz_root
vms_disk_root jenkins_root

I tried using logical disks on an NFS mounted volume for the OpenVMS Builder work area. This seemed to work on OpenVMS 8.4/HP TCP/IP 5.7. It definitely did not work on older versions of TCP/IP, with the worst results on OpenVMS/VAX 7.3. OpenVMS Backup, which is currently used to create the source kits in the packages does not work well with directories on NFS served volumes.

OpenVMS Build Procedures

The OpenVMS Build procedure consists of the following files:

• sys$login:login.com - Local build symbols and logical name setups.
• sys$login:jeeves_defaults.ini - Information about the what disk volumes that OpenVMS build agent should use.
• workspace.ini in the Jenkins workspace root.
• build_simple_vms_project.com - Takes the workspace.ini information and runs the scripts listed in it.
• get_workspace_info.com - Parses workspace.ini into Job logical name for the build procedures.
• get
• setup_logical_disks.com - Set up concealed rooted directories for the build procedures.

The sys$login:login.com and jeeves_default.ini were described above.

The workspace.ini file

The workspace.ini is generated by the Jenkins job and the Unix path to it is passed to the build_simple_vms_project.com.

This is a sample from building GAWK 4.1 stable branch. Since this project does not have GNV changes that are not in the upstream, there is no VMS_SOURCE entry.

PROJECT_SHORT_NAME gawk
REFERENCE_SRC /home/jeeves/mirrors/gnu/gawk/gawk-4.1-stable
WORKSPACE /home/jeeves/jazz/eagle_1/workspace/gnu_awk_4_1_axp_0840
REPO_SIZE 287144
BUILD_DEFAULT PRJ_ROOT:[gawk]
BUILD_SCRIPT_1 [.vms]pcsi_product_gawk.com
DEFAULT_2 PRJ_ROOT:[gawk.test]
BUILD_SCRIPT_2 [-.vms]vmstest.com
PCSI_PRODUCER GNV
PCSI_PRODUCER_FULL_NAME John E. Malmberg

The build_simple_vms_project.com

The build_simple_vms_project.com is invoked by the Jenkins build agent, as copied here from the log:

+ ssh eagle.xile.realm '@jazz_root:[jeeves.vms]build_simple_vms_project.com' '"/home/jeeves/jazz/eagle_1/workspace/gnu_awk_4_1_axp_0840"'

The build_simple_vms_project.com then calls the get_workspace_info.com procedure that uses the Unix path for the workspace to find the workspace.ini file and parse it to be usable by command procedures, as detailed below.

After the workspace.ini file is parsed, the setup_logical_disks.com procedure is called. This procedure sets up and
populates the directories that will be used for the build.

Then the procedure sets the default directory to the BUILD_DEFAULT value passed and starts iterating through the BUILD_SCRIPT_n procedures changing the default directory based on the DEFAULT_n value.

This allows flexibility in which and how many command procedures are run for the build.

Once the build is done the build_simple_vms_project.com looks for any files named test*.xml that are in the lcl_root build directory and copies them into the Jenkins build agent workspace.

These XML files are expected to be in the JUnit XML format which Jenkins knows how to analyze and report on.

Currently I only have a handful of jobs that are producing JUnit format results. Hopefully that number will increase as the GNV and VMS-PORTS projects mature.

The get_workspace_info.com

The get_workspace_info.com procedure reads in the sys$login:jenkins_defaults.ini and for each line in it, it creates a job logical name that starts with "JENKINS_". Job logical names are used so that updated values can easily be passed to other command procedures.

For my current jenkins_defaults.ini, this results in two logical names. "JENKINS_VMS_WORK_ROOT" which is defined as "jazz_root" for where the build job gets its source files and Jenkins information from, and "JENKINS_VMS_DISK_ROOT" which is where the OpenVMS build agent actually does the build.

The get_workspace_info.com then reads in the workspace.ini file that was passed to it. Again it creates job logical names for each of the values in it. This time the logical names are prefixed with "JOB_".

The procedure does a quick and dirty conversion of the path from Unix format to VMS format for the WORKSPACE.INI, JOB_REFERENCE_ROOT, JOB_VMS_SOURCE (if present) values. This could probably be improved once the GNV project gets a robust filename conversion utility written. This conversion also handles some NFS ODS-2 encoding to support VAX or older Alpha versions.

The above paths need to be passed as Unix format paths because they are built from Jenkins environment variables that the Jenkins built agents use.

The setup_logical_disks.com

This procedure sets up and populates concealed rooted directories on the OpenVMS build agent workspace directory based on the values parsed by the get_workspace_info.com.

The lcl_root logical name is used for a directory tree to contain all build created files.

The vms_root logical name is used for a directory tree that contains VMS specific changes not in the upstream repostory, if there are any changes. This directory will normally not be updated by a build procedure.

The src_root logical name is used for a directory tree with the unmodified Unix code. While this directory is normally unmodified by the build procedure, occasionally files need to be "touched" to fix their modification dates after being restored from archives or copied from the actual archive location.

The new_gnu logical name is used to set up a directory tree for staging installs by non-privileged users. The gnv$gnu logical name is changed to be a search list starting with new_gnu, with the logical name old_gnu referencing the installed GNV programs.

The stage_root logical name is set up in the Jenkins build agent workspace for the results of the build procedure that could potentially be published.

Future Directions

I am just getting started with this project. My current focus is getting all the GNV components building under Jenkins control. Next will I will start on the VMS-PORTS projects that I am working on.

The capability is there to build against the development branches of the upstream. By doing that there will be less surprises when upgrading to a new release. This is something that I will probably start looking at once all the GNV components are building.

This build procedure is not set up yet for building more complex projects that would use multiple Jenkins jobs for artifacts. An example would be a project under development that needed libraries that are also under development. This is not that hard to do, I just need to have the library jobs produce artifacts that the other project knows how to pull in.

Currently I am building most packages with the release version names. Only some build procedures know how to detect and set pre-release or beta kit names. It would be useful to have Jenkins figure out from the repository information and notify the build procedure.

My ISP does not allow me to operate a public web server. So I am looking at other options for making build status available.

One option that may be used is a "Dashboard" Jenkins operated on a public web server. One problem with this approach aside from the work of maintaining a secure public server is that it could expose all the build artifacts, and as I noted above, not all projects differentiate between pre-release and release kits. So this could result in confusion of which build a kit is for. So there is a bit of testing that needs to be done for this.

Other options are sending e-mail to the GNV-Developers list when a build succeeds, or posting to a Public Google Calendar.

Having the mirror jobs watch for version releases and notify me or a relevant mailing list is another possibility.

I may be able to figure how to get Jenkins to upload data that can be used by or modify the SourceForge Wiki pages.

Related Link Below

It appears that sourceforge adds a reference link as below to the Wiki if code checked into a repository has a link to that wiki article. Since this is added as a link to the article instead of two the repository, it seems a bit useless.