The “RTC ClearCase workspace synchroniser“ (the synchroniser) is a Linux/UNIX based tactical solution that has been written to allow base IBM Rational ClearCase VOB sub-directory components/products that exist in a Linux/UNIX/windows interoperation (interopt) environment to be bi-directionally and dynamically synchronised with IBM Rational Team Concert (RTC) source control Components.
Notes:
i. The synchroniser runs as a daemon process that deals with the changes to data objects as and when they occur, obviously, if 1000’s of files are added to source control then there will be delays in synchronisation as this amount of data takes time synchronise.
ii. The synchroniser has been successfully stress-tested with many 1000’s of files of varying sizes up to an individual size of 600MB.
2. Glossary
CLI Command Line Interface
JTS Jazz Team Server
POC Point Of Contact
RTC Rational Team Concert
SCM Software Configuration Management
UCM Unified Change Management
URI Unique Resource Identifier
VOB Versioned Object Base
3. A brief description of the core concept of the synchroniser
The following two statements give a brief description of how the bi-directional synchroniser works in either direction, however, you first need to understand the key concept that the synchroniser uses an RTC Local (sandbox) workspace that is actually located in a ClearCase snapshot view and this is how the linkage between the two applications (RTC and ClearCase) is achieved.
ClearCase to RTC source control synchronisation:
The synchroniser runs a “cleartool update” against the VOB subdirectory Component in the ClearCase snapshot view and this identifies the changes that have been made to the objects in the VOBs sub-directories. These changes create “Unresolved” changes, in the RTC local (sandbox) workspace, which the synchroniser then “checks in” (scm checkin) to the Repository workspace and “deliverers” (scm deliver) to its “Flow Target” project integration Stream. From here the changes flow to the projects team members for them to “Accept” into their Repository and local (sandbox) workspaces.
RTC to ClearCase synchronisation:
The synchroniser Accepts (scm accept) the incoming changes (Change Sets) from the projects integration stream into its Repository workspace. This creates changes in the ClearCase snapshot view/local (sandbox) workspace, which the synchroniser deals with by running the appropriate ClearCase commands to update the underlying ClearCase VOB/VOBs.
Notes:
i. If it was only that easy, which is why the current version (v2.0.2) of the synchroniser is currently 29,000+ lines long. Most of this is made up of code comments, administration functionality and having the ability to run the synchroniser in a debugging mode. However, because of its dynamic nature it has to deal with things, such as, “blocking checkouts” in ClearCase, auto-resolve of conflict, etc...
ii. Running the synchroniser with the --CONFIG switch builds a new working daemon instance of the synchroniser that is immediately deployable.
iii. Running the synchroniser with the --TEST switch runs the synchroniser in a debug mode where the output is sent to the screen as well as the logs, and this has proved very useful during the setup and configuration of new synchroniser instances.
4. What the synchroniser currently does (v2.0.2 Sep 2011)
One of the best aspects of the synchroniser is the fact that it provides a very efficient way of getting the code from ClearCase into RTC source control without impacting on the development areas day-to-day business.
It does this by using existing ClearCase integration branches and sourcing its data from a ClearCase snapshot view copy of the VOB/VOBs data. Once a synchroniser instance is up and running the synchroniser currently provides the following major pieces of functionality:
a) Any Components that are added or removed from an RTC Repository Workspace or Stream, which is linked to a synchroniser instance, will be added or removed from the associated VOB location in ClearCase.
b) Any Components that are added or removed from ClearCase location, which is linked to a synchroniser instance, will be added or removed from the associated RTC Repository Workspace or Stream.
Notes:
i. The big advantage with the synchroniser is that during the setup RTC Components can be linked to VOB sub-directory Components that are at differing locations within the ClearCase VOB. You can also rerun the configuration and setup to add additional more complex configurations.
ii. The only constraint here are:
1. once a synchroniser instance is up and running and you add a new Component in RTC SCM the synchroniser needs to know where to create the equivalent sub-directory Component in ClearCase and you have to define this location, which is known to the synchroniser as the NEW_COMPONENT_LOCATION, when you configure the synchroniser instance.
2. The same applies in ClearCase once a synchroniser instance is up and running only Components added to the NEW_COMPONENTS_LOCATION will get added to RTC SCM.
3. If you want to add or remove a Component that is not in the NEW_COMPONENTS_LOCATION you would stop the synchroniser create the “scm share” snapshot view VOB sub-directory location manually and either rerun the synchroniser in - - CONFIG mode or manually edit the “components” file to reflect the addition.
iii. Various “scm share” methods are used by the synchroniser to ensure that file systems for Java and C++ projects get added as multiple Eclipse projects rather than one Eclipse project with a root folder.
iv. When RTC Components are removed from a synchroniser linked Repository Workspace or Steam, they are hidden from view in ClearCase using the “cleartool rmname” command, which means previous labelled baselines of RTC Components will still be accessible from ClearCase if required.
v. Offsite code which gets delivered on site via ClearCase MultiSite, automatically get delivered to the Project Integration Stream as part of the base ClearCase/synchroniser functionality.
Component level:
c) Once a synchroniser instance is up and running all Component updates (changes), additions and deletions are bi-directionally synchronised between RTC source control and ClearCase.
d) Refactoring (move and rename) of objects is bi-directional tracked in both directions without lose of history, which seen as one of the big advantages of using the synchroniser.
e) Deletions and Refactoring (move and rename) of objects that are part of the same “Change Set” in RTC are reflected as a single directory “Checkout” in ClearCase so that a single “cleartool rmver” of a directory can reverse the “Change set” in ClearCase.
Notes:
i. With reference to d) above, the only exception, which is due to an RTC SCM limitation, is that if you move a ClearCase object between Component VOB sub-directories, which is allowed in ClearCase, this will be seen as a deletion and then an addition in RTC, as refactoring between Component SCM is not supported in RTC.
ii. With reference to both c) and d), any changes bound from RTC for ClearCase can be blocked due the object or the object parent directory being checked out and the synchroniser handles this be emailing the ClearCase checkout owner and the admin team (once a day) until the checkout has been resolved. In the interim the pending changes are stored and will be applied once the blocking item has been removed.
f) The synchroniser attempts to auto-resolve any RTC SCM Component conflicts before emailing the Component Pont Of Contact (POC) and the synchroniser admin team.
g) RTC Component Baselines are automatically replicated into ClearCase as a labelled baseline of the linked sub-directory Component location and its parent directories and this “added-value” option has been well received by the client.
Project level:
h) By default RTC SCM sets the MIME type for text/plain files delimiters to “Platform”, which can cause “on the fly” changes to Carriage Return (CR) and line Feeds (LF) settings that can break the differencing of files in ClearCase if you are working in an interop (Window/UNIX) environment.
The synchroniser assures the ability to difference files in ClearCase when the changes have come from RTC SCM by stopping the initial MIME type delimiter changes from being sent back to ClearCase and changing MIME type delimiter for text/plain files to “None”.
Synchroniser/Miscellaneous:
i) By default the “lscm” daemonised version of the SCM CLI, which run considerably quicker than “scm”, stores daemon state configuration files in the user’s home directory and if this home directory is shared across the server farm (standard practice) then the “lscm” CLI can only be used from the first server it is started from. The synchroniser creates a local copy of the configuration files, which allows the “lscm” daemonised CLI to be used across multiple servers.
j) You can run more than one synchroniser instance per VOB as long as the sub-directory locations they are synchronising don’t cross.
k) A single synchroniser instance can synchronise multiple VOBs.
l) Permissions are key to the success of the synchroniser, when doing its “black-box” processing in the co-joined RTC local (sandbox) workspace/ClearCase snapshot view location. As such, the synchroniser has a permission function that preserves the permissions and consistency of the workspace at all times.
m) When RTC SCM reports that the local workspace is in an “inconsistent” state, the synchroniser automatically fixes this by running “scm repair”.
n) The synchroniser automatically re-synchronises Components that become out of sync in the workspace by running “scm load --force”.
Note: The synchroniser currently runs this twice as experience has shown that you often need to run this command twice.
o) The Synchroniser has a CONFIG mode, which allows administrators to quickly build and deploy working synchroniser instances.
p) The synchroniser has a - -TEST standalone mode, which allows you to analyse the full output of the synchroniser on screen.
q) The synchroniser has a logging facility, which can be set at different levels (Standard (Default) or Full), which automatically logs the output of meaningful RTC SCM and cleartool commands used.
r) The synchroniser produces a metrics file at both the synchroniser and Component levels, which allows you to monitor performance of the synchroniser and the underlying RTC SCM and ClearCase commands it runs.
s) Where it can the synchroniser runs the ”lscm” (daemonised SCM Command Line Interface (CLI), however, when synchronising multilevel VOB subdirectories this is not possible so it reverts to running the slower single command at a time “scm” CLI
t) The synchroniser uses the concept of ignores files to control what is and is not synchronised.
For instance:
1. The “.clearcase.ignore” file prevents any files that are named in the file from being synchronised with ClearCase.
2. The use of a “.classpath.ignore” file means different “.classpath” files can exist in both ClearCase and RTC source control environments and the same is true for .“build.xml.ignore” .
Note: The ignore file concept can be deployed at either the synchroniser or the individual Component level.
u) The synchroniser can be run in a single direction as well as a bi-directional (default) mode.
v) The synchroniser emails the defined administrators address when it experiences any problems it cannot fix, such as, un-resolvable conflicts.
w) The synchroniser has a Component Point Of Contact (POC) email capability, which means project members, as well as the synchroniser admin team, gets emailed if synchroniser has a Component conflict that it cannot resolve
x) The synchroniser uses a “start|stop|restart” concept to ensure it exits gracefully without compromising the consistency of the local data area.
5. What the synchroniser currently does not do and other issues
Like the IBM Rational synchroniser software the use of cleartool rmver does not get reflected fully into RTC source control. In particular if a version is removed that reintroduces a previously removed sub-directory Component this will not automatically get added back to RTC source control. You can however manually re-share the resource and edit the synchroniser snapshot view config spec to reintroduce the Component to RTC source control.
Because the synchroniser allows the sharing of VOB sub-directories as RTC SCM Components, there is the possibility that the VOB sub-directory Component can be renamed in ClearCase. This cannot be allowed to happen because, from an RTC SCM point-of-view, this is seen as renaming of a project folder and the underlying “scm mv” generated by the synchroniser to complete this activity will fail saying this operation is prohibited.
The synchronise will terminate if this type of activity occurs and an email is sent to the admin team tell them to reverse the “cleartool mv” command and restart the synchroniser. It also advises the user to conducting this refactoring activity in RTC source control, which is permitted adn will then be reflected in ClearCase as required.
Note: v2.0.3 of the synchroniser will contain a pre-operational trigger that will prevent the renaming of VOB sub-directory Components in ClearCase.
6. ClearCase information required before you setup the synchroniser
The information required from the project area before you setup the synchroniser is:
The name of the ClearCase VOB/VOBs where they store their code.
The sub-directory locations in the VOB/VOBS that contain the components/products that the project wants to bi-directionally synchronise with RTC source control
The ClearCase integration branches that the projects software developers deliver their code to. Note: This can be the main branch for small teams.
The location in the ClearCase VOB where any new Components will be created once the synchroniser is up and running.
7. Pre-synchronisation setup work required
Note: We hope to script this aspect of the synchronisation in the future.
In ClearCase
a) Create a snapshot view using the following naming convention:
<project name="">synchroniser_snapshot_view
The naming convention is not mandatory but it makes sense to use a common naming convention. You should also add “_pilot” , “live”, _maintenance”, dev” etc… after the project name if applicable to make the snapshot view easily recognisable.
Update the snapshot View “config spec” (Suggestion: Take the config spec the project uses to deliver code the integration branch/branches) adding the appropriate “load rules” to ensure the correct data gets loaded and subsequently bi-directionally synchronised between RTC source control and ClearCase.
Note: This is key to the success of the synchronisation activity.
b) On the ClearCase View server host that hosts the synchroniser snapshot View processes ensure that the appropriate version of the RTC Eclipse Client is installed and if necessary add the scmtools location to the PATH variable.
export PATH=$PATH:/opt/IBM/TeamConcert/scmtools
Note: Not mandatory to use the View server host but is a convention we are using.
In RTC Source Control
c) Ensure that the synchroniser processing user account (i.e. vobadm) is a team member of the Project/Team area.
d) Create a project integration Stream (if it does not already exist)
<project>_integration_stream
Note: Again naming not mandatory but it is nice to have a convention.
e) Create empty Components in the Stream to match the Components/Products identified in ClearCase under Section 4.
Notes:
i. The names can be and often are different, it is just easier if they match. To date they have tended to differ as projects have used the move to RTC SCM as an opportunity to refactor.
ii. Please ensure that the ownership of the Stream and Component are given to the project and that the Components access control are correctly set according to the projects requirements (i.e. private, scoped, public)
f) Create a synchroniser processing user workspace using the following convention:
( i.e. vobadm_samecs_development_synchroniser_workspace )
Notes:
i. Again naming not mandatory but good to have a convention
ii. We are using the "vobadm" process account because on the test system it owns all of the ClearCase VOBs; therefore, there will be no issues when writing data to the ClearCase VOBs via the snapshot view.
iii. Creating a Repository Workspace - In RTC Eclipse client Project logged on as vobadm right-click the Stream and go New -> Repository Workspace
g) Link each of the empty RTC Repository Workspace Component with the sandbox VOB subdirectory Component location using the “scm share” command:
scm share <workspace_UUID> <Component_UUID> <extended snapshot view/VOB pathname to the Component> -r https:// <hostname>:9443/jazz -P <password>
Notes:
i. You can get the workspace_UUID and Component_UUID by running the following command:
scm list workspaces -v -r https://<hostname>:9443/jazz
ii. Java and C++ hierarchies should be added with an <extended snapshot="" view="" VOB="" pathname=""> that ends …/
i.e. …/webservices/src/java/
The will ensure the sub-directories in the Component get loaded as separate Eclipse project rather than as a single Eclipse project under a root folder in Eclipse.
iii. If there is a directory hierarchy structure in the extended snapshot view/VOB path then add those in the synchronisation root last. (i.e…/vobs/samecs/src/webservices/src/java before ../vobs/samecs/src/webservices/scripts and ../vobs/samecs/src/webservices/utils)
iv. If you do get this wrong temporarily rename the lower .jazz5/ directory will allow the child sharing to go ahead (i.e. mv ../vobs/samecs/src/webservices/scripts/.jazz5/ ../vobs/samecs/src/webservices/scripts/.jazz5_tmp/)
v. Deliver the share you have just created using the “scm deliver” command:
vi. Link each RTC Repository Workspace Component with the sandbox VOB subdirectory Component location using the “scm share” command:
scm deliver
Alternative RTC Source Control into ClearCase
Everything under the section above assumes that the data is present in ClearCase and will be loaded into empty RTC source control Compoents, which will be the case if you are doing the first import of data from ClearCase into RTC before starting the synchroniser. However, it is perfectly acceptable to start with the data in RTC source control and load this into ClearCase using the following steps:
a) Add the RTC Stream Component to the synchroniser Repository Workspace with:
scm workspace add-comp ….
b) Accept the new Component into the Repository Workspace with:
scm accept –C <Component_UUID>
c) Load the data into the workspace/snapshot view ready to be added to ClearCase:
The following additional steps will also be required in ClearCase
f) Add the view private data in the ClearCase snapshot view to ClearCase source control.
Note: As part of the synchroniser development we create a recursive add to ClearCase source control function and we have a local standalone version of this function which could be used to do this.
8. Creating a synchroniser instance
You create synchroniser instance by running:
./rtc_clearcase_workspace_sync --CONFIG
from the location where you have stored your copy of the synchroniser. In --CONFIG mode the synchroniser will prompt you for the 8 arguments, which are required to support the synchroniser at “runtime” and they are described in the section below:
Initialisation:
After checking that the server you are running on is capable of supporting the synchroniser by validating the RTC SCM CLI, the synchroniser will prompt you for the following 8 arguments.
Note: Argument variable name used by the synchroniser is given in brackets i.e. (LOGPATH):
1. (a) A "unique" none source control UNIX/Linux directory (LOGPATH)
Note: This should be file share with both Network File System (NFS) and, if you are using ClearCase in a windows environment as well, a Common Internet File System (CIFS) access, as it is used by the synchroniser to create and maintain access control files.
(i.e. /snapshot_views/sync/<project name> )
(b) You will then be asked for the CIFS domain name and offered a default (the synchroniser has a defaults file where you can detail the default values offered by the synchroniser), which you can accept by hitting “Return”.
From this entry the synchroniser will attempt to calculate the CIFS share equivalent of the UNXI/Linux LOGPATH you entered under 1. (a)
Note: CIFS share accress is required for the synchronisr postop lnname trigger, which, in Linux/UNIX runs a shell script and in windows runs a Perl script to track ClearCase refactoring using the “cleartool mv <target>” command.
2. The synchronisation root directory (SYNCROOT)
This will be the directory location in the extended ClearCase snapshot view/VOB pathname and should be the highest common point in the directory structure from where all synchronised Component objects in ClearCase can be accessed.
Note: If the synchroniser is synchronising multiple VOBs this will be the ../vobs/.. directory.
An example of an extended ClearCase snapshot View/VOB/sub-directory location is given below:
Note: The synchroniser then validates your entry to see it exists before continuing.
3. The Jazz Repository URI of the Jazz Team Server (REPHOST)
You will be offered a default that you can accept by hitting “Return”
(i.e. https://samecs.dev.com:9443/jazz)
4. The admin support email address for the sychroniser (EMAIL)
You will be offered a default that you can accept by hitting “Return”
(i.e. support@samecs.dev.com)
and the synchroniser uses this address to highlight any unresolvable conflicts or problem the synchroniser encounters.
5. Synchroniser processing user password file (PASSWDFILE)
You will need to create this file as part of the set-up process and is the password required by the synchroniser to logon to the Repository host so that it can run scm commands.
You will be offered a default that you can accept by hitting “Return”
(i.e. /home/vobadm/vobadm_passwd.txt)
6. Creation location for new Components(NEW_COMPONENTS_LOCATION)
This is the extended ClearCase snapshot View/VOB pathname location where any new Components created in RTC should be created in ClearCase and vice versa.
Note: This is one of the only constraints when using the synchroniser but it does need to know where to create and source new Components that need synchronising.
This location will be the same as the SYNCROOT if you are using a single VOB with Components contained in a single directory and this will be offered as a default that you can accept by hitting “Return”
If you are synchronising multiple VOBs under one synchroniser instance then a choice has to be made where to store new Components in just one of the VOBs. If this does not meet with the projects ClearCase requirements then we would suggest creating seperate synchroniser instances for each VOB.
At this point the synchroniser will validate this entry to ensure that it is a valid RTC SCM share location by logging on to the Jazz Team Server (JTS) with the Jazz Repository URI and password you have supplied in arguments 3 and 5.
7. The directory location housing the synchroniser(SCRIPT_LOCATION)
You will be offered a default that you can accept by hitting “Return”
8. The primary group of the ClearCase VOBs being synchronised (GROUP)
You will be offered a default that you can accept by hitting “Return”
Synchroniser instance creation and configuration
++++++++++++++++++++++++++++++++++++++++++++++++
Once you have entered the 8th argument the synchroniser will continue the setup and configuration process by:
a) Attempting to calculate he VOBSROOT location (The pathname in the extended snapshot view/VOB pathname up to and including where ../vobs/… occurs.
( i.e. /net/samecs.dev.com/snapshot_views /synchroniser_evaluation_snapshot_view/vobs )
which it uses to check the existing triggers on the VOB to see if they could potentially conflict with the running of the synchroniser
For example, if it finds any triggers that use a “clearprompt”, which may attempt to interact with the synchroniser, it will display the trigger information and ask if you want to proceed or terminate to allow you to take remedial action (i.e. remove the trigger or set ClearCase environmental variables to stop the trigger interacting with the synchroniser).
Another option would be to add –nuser <synchroniser processing="" account=""> (i.e. vobadm) so that the trigger does not fire for the synchroniser when it is being run by that user.
b) Continuing its setup process and creating the log (.synclog), control and metrics files and applying the ClearCase triggers, etc… that it will need at runtime.
c) Attempting to populate the “components” file, which it does by checking the “scm status” of the extended ClearCase snapshot view/VOB Repository Workspaces for Component entries. At this point you will be prompted to either edit of accept what it has found. It will be presented to you in the format:
<$WORKSPACE_UUID>*<$COMPONENT_UUID>*<Location relative to $SYNCROOT>
and this format is fully explained in section “10. Control Files and synclog”
d) Finally, it soources the “Flow Target” Stream (STREAM_UUID) for the synchroniser Repository Workspace. The Stream UUID is used inconjction with the synchroniser Repoistory Workspace UUID to determine when Components have been added and removed from synchronisation.
This completes the synchroniser setup and configuration process and you will be given the option to either start the synchroniser instance or Exit.
Do you want to start the synchroniser now? Y/N
Notes:
i. The synchroniser stores the valuses of the 8 arguments and other information in a “config” file that is stored in the logs/control files (LOGPATH) location.
This file is sourced at runtime and is reused if you rerun the synchroniser setup with the --CONFIG option:
./rtc_clearcase_workspace_sync --CONFIG
ii. An additional 9th argument of --TEST will run the synchroniser in standalone test mode and this is set a runtime as describe in the “Running a synchroniser instance” section below.
9. Running a synchroniser instance
Daemonised script
Once you have completed the “rtc_clearcase_workspace_sync --CONFIG” setup process, the synchroniser will use the argument information to build a working synchroniser instance, which can be run as daemon process that is controlled from the logs/control files (LOGPATH) location with:
./sync start
./sync restart
./sync stop
Non-Daemonised script
If you run the synchroniser with the following options it runs in a standalone mode and sends its output to the screen as well as the log files. This is a useful option for debugging and use during the infancy of a new synchroniser:
./sync start --TEST
./sync restart –TEST
Updating an existing synchroniser instance
You can also rerun the initial:
./rtc_clearcase_workspace_sync --CONFIG
for the synchroniser instance by running:
./sync –CONFIG
This will take your initial responses for each of the 8 arguments and prompt you for acceptance or editing, so this is how you would update a synchroniser instance.
10. Control Files and synclog
The Control Files location is home to a large number of configuration and transient files used by the synchroniser. The following is a list of the most important control files:
components
The most important control file is the “components” file and in --CONFIG mode the synchroniser attempts to build this file, however, I would advise you review and where necessary edit this file before starting the synchroniser for the first time. The format for the components file is:
i. The <Component pathname=""> is the pathname relative to the RTC source control/ClearCase VOB synchronisation root (SYNCROOT)
ii. An additional *- at the end of the line allows the sub-directories of the parent directory to be loaded as Eclipse Java and C++ projects.
iii. The synchroniser creates a copy of the “components” file called “components_copy”, which can be useful investigating synchroniser issue if components stop being synchronised.
config
This file, amongst other things, contains the 8 argument inputs used when starting the synchroniser and if it exists the arguments are sourced and passed back to the synchroniser for reuse when running the commands:
metrics.txt (One at Component level one at the synchroniser level)
These, comma delimited, files store metrics information of the synchroniser’s activities in the following format:
Date&Time,SynchroniserInstance,Component,Direction,Command,Action,NumberOfFiles,Data,TimeTaken23_06_11@06_53_00:,samecsdev,samecsproductsjava,ClearCase->RTC,"scm -u y -a n checkin",---c,28,0,10...
Breakdown of the Headings:
Date&Time: Date (DD_MM_YY) and time (HH_MM_SS) of the activity.
Synchroniser Instance: name of the synchroniser instance
Component: The name of the Component being processed
Direction: Either “RTC to ClearCase” or “ClearCase to RTC”
Command: Short description of the command (i.e. scm checkin)
Action: A 4 character string highlighting what the command did:
Renamed/Moved = m---
Deleted = -d--
Added = --a-
Changed (updated) = ---c
NumberOfFiles: The number of files being processed.
Data: The amount of data involved in bytes
Time Taken: The time taken for the activity in seconds.
refactor.sh & refactor.pl
The synchroniser applies a post operational ClearCase trigger on the hidden “lnname” command to track refactoring activity in ClearCase so that it can be mirrored into RTC SCM and maintain the history of such activity.
To do this it creates two control file refactor.sh and refactor.pl, which are accessed by the trigger from UNIX/Linux and windows respectively.
.synclog
Any meaningful transactions (i.e. cleartool and scm commands that create change) are logged in the .synclog
Note: You can also turn on a higher level of logging by placing a file named “.full_logging” in the synchroniser instances control file location.
An example of a log output is give below:
03_08_11@10_33_24: Running "scm -u y -a n accept -C rc5 test"
03_08_11@10_33_33: (See output below:)
Repository: https://samecs.dev.com:9443/jazz/
Workspace: (_8IojcKe8EeCbbt4zvI36Cg) "vobadm_synchroniser_development_workspace"
Component: (_Sv9Y0L26EeCYhu87w_ih8w) "rc5 test"
Change sets:
(_4fY9Ib27EeCYhu87w_ih8w) ---$ Geoff Bush <No comment="">
Changes:
---c- /rc5 test/level 1 test.sh
---c- /rc5 test/Level 1/test.sh ************
03_08_11@10_33_33: The following files are hijacked (See output below):
level1 test.sh
./Level 1/test.sh*********
Running: cleartool co -usehijack | cleartool ci against the above*********
03_08_11@10_33_33: synchroniser metrics generated
03_08_11@10_33_33:,synchroniser_development,rc5 test,RTC->ClearCase,"cleartool co -usehijack & ci",---c,2,13652,1************
Other control files of interest with a brief description:
.script_run_stop
Contains either the words “run” or “stop” and is populated by the ./sync start|stop|restart script and allows the synchroniser to either continuing running or terminate (gracefully).
.hostname
contains the hostname of the server that the instance on the synchroniser is being run on. Ensures you do not start a second instance on the same server and that you issue the command to stop the synchroniser on the correct server.
.lastrun
A continually updating time stamped output from the “scm status” command. Useful for ensuring that the synchroniser is still running.
.rtc_to_clearcase_sync.ignore
When created, stops RTC SCM events being sent to ClearCase. This can be set at the synchroniser and individual Component levels
Note: This means that the RTC SCM Change Set “backup” in the synchroniser Repository Workspace; therefore, you should be very careful when you remove this control file as it would “open the floodgates” with changes bound for ClearCase, which could cause the snapshot view/local (sandbox) workspace to become corrupt.
.clearcase_to_rtc_sync.ignore
When created, stops ClearCase events being sent to RTC source control. This can be set at the synchroniser and individual Component levels.
.clearcase.ignore
When created and populated, the “.clearcase.ignore” file prevents any files that are named in the file from being updated or added to source control in ClearCase, when they have been added or updated under RTC SCM. This can be set at the synchroniser and individual Component levels.
.classpath.ignore & .build.xml.ignore
When created in the control files area it means the different “.classpath” files can exist in both ClearCase and RTC source control environments and no updates are sent between the two systems for .classpath files. The same is true for “build.xml” files when the .”build.xml.ignore” file is created. This can be set at the synchroniser and individual Component levels.
.get_symlinks
RTC Source Control does not support symbolic linking; therefore, when created, the .get_symlinks control file lists the ClearCase symbolic links in the VOB sub-directory Component that will not be reflected fully in RTC Source Control. For each Component the output is sent to the file:
<$LOGPATH>/$COMPONENT_UUID/.symlinks
Note: You should not leave the .get_symlink flag turned on permanently as this will affect the performance on the synchroniser.
.poc
When created, which you need to do to activate this feature, it should contain the email address of any Point Of Contact (POC) that you want to be emailed if the synchroniser experiences issues, such as, conflict between updates moving between ClearCase and RTC SCM. You can set a .poc file at the synchroniser and individual Component level to allow for flexibility.
.full_logging
When this file is created in the control files area, it will, for example, turn on logging of individual checkout and checkin transactions. This can be set at the synchroniser and individual Component level.
.clearcase_master
By default the synchroniser will attempt to resolve any workspace conflict in favour of RTC SCM, however, if you create a .clearcase_master file in the control file area, it will change this to resolve the conflict in favour of ClearCase. This can be set at the synchroniser and individual Component level.
.lscm_overide
----If the Clear Case VOB sub-directory Components exist at different levels then the synchroniser defaults to using “scm”, however, sometimes you can run “lscm” if the synchronisation paths don’t cross; therefore, if a “.lscm_overide” is created in the $LOGPATH/control files directory it will override the default behaviour and run the “lscm” command line.
Transient files in the control file area
The sychroniser also creates a large number of transient files generated from “cleartool” and “scm” command outputs, which it uses for logging and error trapping purposes. Note: These files will appear in the root of the synchronisers control files and under the component_UUID directory location.
11. WARNING: Permissions are key so don’t manually change things in the synchroniser workspace
Once you have created the jazz scm share you are not advised to manually edit anything in that is shown as the black-box synchroniser in the diagram on the next page. For instance if you manually check out and then check in a file into the VOB from the synchroniser ClearCase snapshot view it will change the files permissions in such a way that the local workspace (sandbox) see this as a change and will create a circular update.
To reiterate the synchroniser controls this behaviour with the permissions function, where basically the permissions of a file are captured before it is updated and reset back to exactly the same permissions after the update.
Permissions are key throughout and the synchroniser ensures files can be written to when they appear/are created in the local workspace (sandbox)/snapshot view location.
12. Using the RTC Eclipse client GUI to resolve issues in the synchroniser local (sandbox) workspace
It is perfectly acceptable to stop the synchroniser and start the RTC Eclipse client GUI as vobadm to resolve any conflict issues:
/opt/IBM/TeamConcert/Eclipse
Note: If the synchroniser has been running the “lscm” daemonised CLI it will continue to run once you have stopped the synchroniser. To rectify this run:
RTC ClearCase workspace synchroniser v2.0.2 Sep 2011
1. The “RTC ClearCase workspace synchroniser”
The “RTC ClearCase workspace synchroniser“ (the synchroniser) is a Linux/UNIX based tactical solution that has been written to allow base IBM Rational ClearCase VOB sub-directory components/products that exist in a Linux/UNIX/windows interoperation (interopt) environment to be bi-directionally and dynamically synchronised with IBM Rational Team Concert (RTC) source control Components.
Notes:
i. The synchroniser runs as a daemon process that deals with the changes to data objects as and when they occur, obviously, if 1000’s of files are added to source control then there will be delays in synchronisation as this amount of data takes time synchronise.
ii. The synchroniser has been successfully stress-tested with many 1000’s of files of varying sizes up to an individual size of 600MB.
2. Glossary
CLI Command Line Interface
JTS Jazz Team Server
POC Point Of Contact
RTC Rational Team Concert
SCM Software Configuration Management
UCM Unified Change Management
URI Unique Resource Identifier
VOB Versioned Object Base
3. A brief description of the core concept of the synchroniser
The following two statements give a brief description of how the bi-directional synchroniser works in either direction, however, you first need to understand the key concept that the synchroniser uses an RTC Local (sandbox) workspace that is actually located in a ClearCase snapshot view and this is how the linkage between the two applications (RTC and ClearCase) is achieved.
ClearCase to RTC source control synchronisation:
The synchroniser runs a “cleartool update” against the VOB subdirectory Component in the ClearCase snapshot view and this identifies the changes that have been made to the objects in the VOBs sub-directories. These changes create “Unresolved” changes, in the RTC local (sandbox) workspace, which the synchroniser then “checks in” (scm checkin) to the Repository workspace and “deliverers” (scm deliver) to its “Flow Target” project integration Stream. From here the changes flow to the projects team members for them to “Accept” into their Repository and local (sandbox) workspaces.
RTC to ClearCase synchronisation:
The synchroniser Accepts (scm accept) the incoming changes (Change Sets) from the projects integration stream into its Repository workspace. This creates changes in the ClearCase snapshot view/local (sandbox) workspace, which the synchroniser deals with by running the appropriate ClearCase commands to update the underlying ClearCase VOB/VOBs.
Notes:
i. If it was only that easy, which is why the current version (v2.0.2) of the synchroniser is currently 29,000+ lines long. Most of this is made up of code comments, administration functionality and having the ability to run the synchroniser in a debugging mode. However, because of its dynamic nature it has to deal with things, such as, “blocking checkouts” in ClearCase, auto-resolve of conflict, etc...
ii. Running the synchroniser with the --CONFIG switch builds a new working daemon instance of the synchroniser that is immediately deployable.
iii. Running the synchroniser with the --TEST switch runs the synchroniser in a debug mode where the output is sent to the screen as well as the logs, and this has proved very useful during the setup and configuration of new synchroniser instances.
4. What the synchroniser currently does (v2.0.2 Sep 2011)
One of the best aspects of the synchroniser is the fact that it provides a very efficient way of getting the code from ClearCase into RTC source control without impacting on the development areas day-to-day business.
It does this by using existing ClearCase integration branches and sourcing its data from a ClearCase snapshot view copy of the VOB/VOBs data. Once a synchroniser instance is up and running the synchroniser currently provides the following major pieces of functionality:
RTC Repository Workspace/Stream – ClearCase View level
a) Any Components that are added or removed from an RTC Repository Workspace or Stream, which is linked to a synchroniser instance, will be added or removed from the associated VOB location in ClearCase.
b) Any Components that are added or removed from ClearCase location, which is linked to a synchroniser instance, will be added or removed from the associated RTC Repository Workspace or Stream.
Notes:
i. The big advantage with the synchroniser is that during the setup RTC Components can be linked to VOB sub-directory Components that are at differing locations within the ClearCase VOB. You can also rerun the configuration and setup to add additional more complex configurations.
ii. The only constraint here are:
iii. Various “scm share” methods are used by the synchroniser to ensure that file systems for Java and C++ projects get added as multiple Eclipse projects rather than one Eclipse project with a root folder.
iv. When RTC Components are removed from a synchroniser linked Repository Workspace or Steam, they are hidden from view in ClearCase using the “cleartool rmname” command, which means previous labelled baselines of RTC Components will still be accessible from ClearCase if required.
v. Offsite code which gets delivered on site via ClearCase MultiSite, automatically get delivered to the Project Integration Stream as part of the base ClearCase/synchroniser functionality.
Component level:
c) Once a synchroniser instance is up and running all Component updates (changes), additions and deletions are bi-directionally synchronised between RTC source control and ClearCase.
d) Refactoring (move and rename) of objects is bi-directional tracked in both directions without lose of history, which seen as one of the big advantages of using the synchroniser.
e) Deletions and Refactoring (move and rename) of objects that are part of the same “Change Set” in RTC are reflected as a single directory “Checkout” in ClearCase so that a single “cleartool rmver” of a directory can reverse the “Change set” in ClearCase.
Notes:
i. With reference to d) above, the only exception, which is due to an RTC SCM limitation, is that if you move a ClearCase object between Component VOB sub-directories, which is allowed in ClearCase, this will be seen as a deletion and then an addition in RTC, as refactoring between Component SCM is not supported in RTC.
ii. With reference to both c) and d), any changes bound from RTC for ClearCase can be blocked due the object or the object parent directory being checked out and the synchroniser handles this be emailing the ClearCase checkout owner and the admin team (once a day) until the checkout has been resolved. In the interim the pending changes are stored and will be applied once the blocking item has been removed.
f) The synchroniser attempts to auto-resolve any RTC SCM Component conflicts before emailing the Component Pont Of Contact (POC) and the synchroniser admin team.
g) RTC Component Baselines are automatically replicated into ClearCase as a labelled baseline of the linked sub-directory Component location and its parent directories and this “added-value” option has been well received by the client.
Project level:
h) By default RTC SCM sets the MIME type for text/plain files delimiters to “Platform”, which can cause “on the fly” changes to Carriage Return (CR) and line Feeds (LF) settings that can break the differencing of files in ClearCase if you are working in an interop (Window/UNIX) environment.
The synchroniser assures the ability to difference files in ClearCase when the changes have come from RTC SCM by stopping the initial MIME type delimiter changes from being sent back to ClearCase and changing MIME type delimiter for text/plain files to “None”.
Synchroniser/Miscellaneous:
i) By default the “lscm” daemonised version of the SCM CLI, which run considerably quicker than “scm”, stores daemon state configuration files in the user’s home directory and if this home directory is shared across the server farm (standard practice) then the “lscm” CLI can only be used from the first server it is started from. The synchroniser creates a local copy of the configuration files, which allows the “lscm” daemonised CLI to be used across multiple servers.
j) You can run more than one synchroniser instance per VOB as long as the sub-directory locations they are synchronising don’t cross.
k) A single synchroniser instance can synchronise multiple VOBs.
l) Permissions are key to the success of the synchroniser, when doing its “black-box” processing in the co-joined RTC local (sandbox) workspace/ClearCase snapshot view location. As such, the synchroniser has a permission function that preserves the permissions and consistency of the workspace at all times.
m) When RTC SCM reports that the local workspace is in an “inconsistent” state, the synchroniser automatically fixes this by running “scm repair”.
n) The synchroniser automatically re-synchronises Components that become out of sync in the workspace by running “scm load --force”.
Note: The synchroniser currently runs this twice as experience has shown that you often need to run this command twice.
o) The Synchroniser has a CONFIG mode, which allows administrators to quickly build and deploy working synchroniser instances.
p) The synchroniser has a - -TEST standalone mode, which allows you to analyse the full output of the synchroniser on screen.
q) The synchroniser has a logging facility, which can be set at different levels (Standard (Default) or Full), which automatically logs the output of meaningful RTC SCM and cleartool commands used.
r) The synchroniser produces a metrics file at both the synchroniser and Component levels, which allows you to monitor performance of the synchroniser and the underlying RTC SCM and ClearCase commands it runs.
s) Where it can the synchroniser runs the ”lscm” (daemonised SCM Command Line Interface (CLI), however, when synchronising multilevel VOB subdirectories this is not possible so it reverts to running the slower single command at a time “scm” CLI
t) The synchroniser uses the concept of ignores files to control what is and is not synchronised.
For instance:
1. The “.clearcase.ignore” file prevents any files that are named in the file from being synchronised with ClearCase.
2. The use of a “.classpath.ignore” file means different “.classpath” files can exist in both ClearCase and RTC source control environments and the same is true for .“build.xml.ignore” .
Note: The ignore file concept can be deployed at either the synchroniser or the individual Component level.
u) The synchroniser can be run in a single direction as well as a bi-directional (default) mode.
v) The synchroniser emails the defined administrators address when it experiences any problems it cannot fix, such as, un-resolvable conflicts.
w) The synchroniser has a Component Point Of Contact (POC) email capability, which means project members, as well as the synchroniser admin team, gets emailed if synchroniser has a Component conflict that it cannot resolve
x) The synchroniser uses a “start|stop|restart” concept to ensure it exits gracefully without compromising the consistency of the local data area.
5. What the synchroniser currently does not do and other issues
Like the IBM Rational synchroniser software the use of cleartool rmver does not get reflected fully into RTC source control. In particular if a version is removed that reintroduces a previously removed sub-directory Component this will not automatically get added back to RTC source control. You can however manually re-share the resource and edit the synchroniser snapshot view config spec to reintroduce the Component to RTC source control.
Because the synchroniser allows the sharing of VOB sub-directories as RTC SCM Components, there is the possibility that the VOB sub-directory Component can be renamed in ClearCase. This cannot be allowed to happen because, from an RTC SCM point-of-view, this is seen as renaming of a project folder and the underlying “scm mv” generated by the synchroniser to complete this activity will fail saying this operation is prohibited.
The synchronise will terminate if this type of activity occurs and an email is sent to the admin team tell them to reverse the “cleartool mv” command and restart the synchroniser. It also advises the user to conducting this refactoring activity in RTC source control, which is permitted adn will then be reflected in ClearCase as required.
Note: v2.0.3 of the synchroniser will contain a pre-operational trigger that will prevent the renaming of VOB sub-directory Components in ClearCase.
6. ClearCase information required before you setup the synchroniser
The information required from the project area before you setup the synchroniser is:
The name of the ClearCase VOB/VOBs where they store their code.
The sub-directory locations in the VOB/VOBS that contain the components/products that the project wants to bi-directionally synchronise with RTC source control
The ClearCase integration branches that the projects software developers deliver their code to. Note: This can be the main branch for small teams.
The location in the ClearCase VOB where any new Components will be created once the synchroniser is up and running.
7. Pre-synchronisation setup work required
Note: We hope to script this aspect of the synchronisation in the future.
In ClearCase
a) Create a snapshot view using the following naming convention:
<project name="">synchroniser_snapshot_view
The naming convention is not mandatory but it makes sense to use a common naming convention. You should also add “_pilot” , “live”, _maintenance”, dev” etc… after the project name if applicable to make the snapshot view easily recognisable.
Update the snapshot View “config spec” (Suggestion: Take the config spec the project uses to deliver code the integration branch/branches) adding the appropriate “load rules” to ensure the correct data gets loaded and subsequently bi-directionally synchronised between RTC source control and ClearCase.
Note: This is key to the success of the synchronisation activity.
b) On the ClearCase View server host that hosts the synchroniser snapshot View processes ensure that the appropriate version of the RTC Eclipse Client is installed and if necessary add the scmtools location to the PATH variable.
export PATH=$PATH:/opt/IBM/TeamConcert/scmtools
Note: Not mandatory to use the View server host but is a convention we are using.
In RTC Source Control
c) Ensure that the synchroniser processing user account (i.e. vobadm) is a team member of the Project/Team area.
d) Create a project integration Stream (if it does not already exist)
<project>_integration_stream
Note: Again naming not mandatory but it is nice to have a convention.
e) Create empty Components in the Stream to match the Components/Products identified in ClearCase under Section 4.
Notes:
i. The names can be and often are different, it is just easier if they match. To date they have tended to differ as projects have used the move to RTC SCM as an opportunity to refactor.
ii. Please ensure that the ownership of the Stream and Component are given to the project and that the Components access control are correctly set according to the projects requirements (i.e. private, scoped, public)
f) Create a synchroniser processing user workspace using the following convention:
<synchroniser process="" user="">_<project>_synchroniser_workspace
( i.e. vobadm_samecs_development_synchroniser_workspace )
Notes:
i. Again naming not mandatory but good to have a convention
ii. We are using the "vobadm" process account because on the test system it owns all of the ClearCase VOBs; therefore, there will be no issues when writing data to the ClearCase VOBs via the snapshot view.
iii. Creating a Repository Workspace - In RTC Eclipse client Project logged on as vobadm right-click the Stream and go New -> Repository Workspace
g) Link each of the empty RTC Repository Workspace Component with the sandbox VOB subdirectory Component location using the “scm share” command:
Notes:
i. You can get the workspace_UUID and Component_UUID by running the following command:
ii. Java and C++ hierarchies should be added with an <extended snapshot="" view="" VOB="" pathname=""> that ends …/
i.e. …/webservices/src/java/
The will ensure the sub-directories in the Component get loaded as separate Eclipse project rather than as a single Eclipse project under a root folder in Eclipse.
iii. If there is a directory hierarchy structure in the extended snapshot view/VOB path then add those in the synchronisation root last. (i.e…/vobs/samecs/src/webservices/src/java before ../vobs/samecs/src/webservices/scripts and ../vobs/samecs/src/webservices/utils)
iv. If you do get this wrong temporarily rename the lower .jazz5/ directory will allow the child sharing to go ahead (i.e. mv ../vobs/samecs/src/webservices/scripts/.jazz5/ ../vobs/samecs/src/webservices/scripts/.jazz5_tmp/)
v. Deliver the share you have just created using the “scm deliver” command:
vi. Link each RTC Repository Workspace Component with the sandbox VOB subdirectory Component location using the “scm share” command:
Alternative RTC Source Control into ClearCase
Everything under the section above assumes that the data is present in ClearCase and will be loaded into empty RTC source control Compoents, which will be the case if you are doing the first import of data from ClearCase into RTC before starting the synchroniser. However, it is perfectly acceptable to start with the data in RTC source control and load this into ClearCase using the following steps:
a) Add the RTC Stream Component to the synchroniser Repository Workspace with:
b) Accept the new Component into the Repository Workspace with:
c) Load the data into the workspace/snapshot view ready to be added to ClearCase:
d) Share the Component with everyone else in the team with:
e) Deliver the Share with:
The following additional steps will also be required in ClearCase
f) Add the view private data in the ClearCase snapshot view to ClearCase source control.
Note: As part of the synchroniser development we create a recursive add to ClearCase source control function and we have a local standalone version of this function which could be used to do this.
8. Creating a synchroniser instance
You create synchroniser instance by running:
from the location where you have stored your copy of the synchroniser. In --CONFIG mode the synchroniser will prompt you for the 8 arguments, which are required to support the synchroniser at “runtime” and they are described in the section below:
Initialisation:
After checking that the server you are running on is capable of supporting the synchroniser by validating the RTC SCM CLI, the synchroniser will prompt you for the following 8 arguments.
Note: Argument variable name used by the synchroniser is given in brackets i.e. (LOGPATH):
1. (a) A "unique" none source control UNIX/Linux directory (LOGPATH)
Note: This should be file share with both Network File System (NFS) and, if you are using ClearCase in a windows environment as well, a Common Internet File System (CIFS) access, as it is used by the synchroniser to create and maintain access control files.
From this entry the synchroniser will attempt to calculate the CIFS share equivalent of the UNXI/Linux LOGPATH you entered under 1. (a)
which you can accept by hitting “Return”.
Note: CIFS share accress is required for the synchronisr postop lnname trigger, which, in Linux/UNIX runs a shell script and in windows runs a Perl script to track ClearCase refactoring using the “cleartool mv
2. The synchronisation root directory (SYNCROOT)
This will be the directory location in the extended ClearCase snapshot view/VOB pathname and should be the highest common point in the directory structure from where all synchronised Component objects in ClearCase can be accessed.
Note: If the synchroniser is synchronising multiple VOBs this will be the ../vobs/.. directory.
An example of an extended ClearCase snapshot View/VOB/sub-directory location is given below:
Note: The synchroniser then validates your entry to see it exists before continuing.
3. The Jazz Repository URI of the Jazz Team Server (REPHOST)
You will be offered a default that you can accept by hitting “Return”
4. The admin support email address for the sychroniser (EMAIL)
You will be offered a default that you can accept by hitting “Return”
and the synchroniser uses this address to highlight any unresolvable conflicts or problem the synchroniser encounters.
5. Synchroniser processing user password file (PASSWDFILE)
You will need to create this file as part of the set-up process and is the password required by the synchroniser to logon to the Repository host so that it can run scm commands.
You will be offered a default that you can accept by hitting “Return”
6. Creation location for new Components(NEW_COMPONENTS_LOCATION)
This is the extended ClearCase snapshot View/VOB pathname location where any new Components created in RTC should be created in ClearCase and vice versa.
Note: This is one of the only constraints when using the synchroniser but it does need to know where to create and source new Components that need synchronising.
This location will be the same as the SYNCROOT if you are using a single VOB with Components contained in a single directory and this will be offered as a default that you can accept by hitting “Return”
If you are synchronising multiple VOBs under one synchroniser instance then a choice has to be made where to store new Components in just one of the VOBs. If this does not meet with the projects ClearCase requirements then we would suggest creating seperate synchroniser instances for each VOB.
At this point the synchroniser will validate this entry to ensure that it is a valid RTC SCM share location by logging on to the Jazz Team Server (JTS) with the Jazz Repository URI and password you have supplied in arguments 3 and 5.
7. The directory location housing the synchroniser(SCRIPT_LOCATION)
You will be offered a default that you can accept by hitting “Return”
8. The primary group of the ClearCase VOBs being synchronised (GROUP)
You will be offered a default that you can accept by hitting “Return”
Synchroniser instance creation and configuration
++++++++++++++++++++++++++++++++++++++++++++++++
Once you have entered the 8th argument the synchroniser will continue the setup and configuration process by:
a) Attempting to calculate he VOBSROOT location (The pathname in the extended snapshot view/VOB pathname up to and including where ../vobs/… occurs.
( i.e. /net/samecs.dev.com/snapshot_views /synchroniser_evaluation_snapshot_view/vobs )
which it uses to check the existing triggers on the VOB to see if they could potentially conflict with the running of the synchroniser
For example, if it finds any triggers that use a “clearprompt”, which may attempt to interact with the synchroniser, it will display the trigger information and ask if you want to proceed or terminate to allow you to take remedial action (i.e. remove the trigger or set ClearCase environmental variables to stop the trigger interacting with the synchroniser).
Another option would be to add –nuser <synchroniser processing="" account=""> (i.e. vobadm) so that the trigger does not fire for the synchroniser when it is being run by that user.
b) Continuing its setup process and creating the log (.synclog), control and metrics files and applying the ClearCase triggers, etc… that it will need at runtime.
c) Attempting to populate the “components” file, which it does by checking the “scm status” of the extended ClearCase snapshot view/VOB Repository Workspaces for Component entries. At this point you will be prompted to either edit of accept what it has found. It will be presented to you in the format:
and this format is fully explained in section “10. Control Files and synclog”
d) Finally, it soources the “Flow Target” Stream (STREAM_UUID) for the synchroniser Repository Workspace. The Stream UUID is used inconjction with the synchroniser Repoistory Workspace UUID to determine when Components have been added and removed from synchronisation.
This completes the synchroniser setup and configuration process and you will be given the option to either start the synchroniser instance or Exit.
Notes:
i. The synchroniser stores the valuses of the 8 arguments and other information in a “config” file that is stored in the logs/control files (LOGPATH) location.
This file is sourced at runtime and is reused if you rerun the synchroniser setup with the --CONFIG option:
ii. An additional 9th argument of --TEST will run the synchroniser in standalone test mode and this is set a runtime as describe in the “Running a synchroniser instance” section below.
9. Running a synchroniser instance
Daemonised script
Once you have completed the “rtc_clearcase_workspace_sync --CONFIG” setup process, the synchroniser will use the argument information to build a working synchroniser instance, which can be run as daemon process that is controlled from the logs/control files (LOGPATH) location with:
Non-Daemonised script
If you run the synchroniser with the following options it runs in a standalone mode and sends its output to the screen as well as the log files. This is a useful option for debugging and use during the infancy of a new synchroniser:
Updating an existing synchroniser instance
You can also rerun the initial:
for the synchroniser instance by running:
This will take your initial responses for each of the 8 arguments and prompt you for acceptance or editing, so this is how you would update a synchroniser instance.
10. Control Files and synclog
The Control Files location is home to a large number of configuration and transient files used by the synchroniser. The following is a list of the most important control files:
components
The most important control file is the “components” file and in --CONFIG mode the synchroniser attempts to build this file, however, I would advise you review and where necessary edit this file before starting the synchroniser for the first time. The format for the components file is:
Notes:
i. The <Component pathname=""> is the pathname relative to the RTC source control/ClearCase VOB synchronisation root (SYNCROOT)
ii. An additional *- at the end of the line allows the sub-directories of the parent directory to be loaded as Eclipse Java and C++ projects.
iii. The synchroniser creates a copy of the “components” file called “components_copy”, which can be useful investigating synchroniser issue if components stop being synchronised.
config
This file, amongst other things, contains the 8 argument inputs used when starting the synchroniser and if it exists the arguments are sourced and passed back to the synchroniser for reuse when running the commands:
metrics.txt (One at Component level one at the synchroniser level)
These, comma delimited, files store metrics information of the synchroniser’s activities in the following format:
Breakdown of the Headings:
Date&Time: Date (DD_MM_YY) and time (HH_MM_SS) of the activity. Synchroniser Instance: name of the synchroniser instance Component: The name of the Component being processed Direction: Either “RTC to ClearCase” or “ClearCase to RTC” Command: Short description of the command (i.e. scm checkin) Action: A 4 character string highlighting what the command did: Renamed/Moved = m--- Deleted = -d-- Added = --a- Changed (updated) = ---c NumberOfFiles: The number of files being processed. Data: The amount of data involved in bytes Time Taken: The time taken for the activity in seconds.refactor.sh & refactor.pl
The synchroniser applies a post operational ClearCase trigger on the hidden “lnname” command to track refactoring activity in ClearCase so that it can be mirrored into RTC SCM and maintain the history of such activity.
To do this it creates two control file refactor.sh and refactor.pl, which are accessed by the trigger from UNIX/Linux and windows respectively.
.synclog
Any meaningful transactions (i.e. cleartool and scm commands that create change) are logged in the .synclog
Note: You can also turn on a higher level of logging by placing a file named “.full_logging” in the synchroniser instances control file location.
An example of a log output is give below:
03_08_11@10_33_24: Running "scm -u y -a n accept -C rc5 test"
03_08_11@10_33_33: (See output below:)
Repository: https://samecs.dev.com:9443/jazz/
Workspace: (_8IojcKe8EeCbbt4zvI36Cg) "vobadm_synchroniser_development_workspace"
Component: (_Sv9Y0L26EeCYhu87w_ih8w) "rc5 test"
Change sets:
(_4fY9Ib27EeCYhu87w_ih8w) ---$ Geoff Bush <No comment="">
Changes:
---c- /rc5 test/level 1 test.sh
---c- /rc5 test/Level 1/test.sh
************
03_08_11@10_33_33: The following files are hijacked (See output below):
level1 test.sh
./Level 1/test.sh*********
Running: cleartool co -usehijack | cleartool ci against the above*********
03_08_11@10_33_33: synchroniser metrics generated
03_08_11@10_33_33:,synchroniser_development,rc5 test,RTC->ClearCase,"cleartool co -usehijack & ci",---c,2,13652,1************
Other control files of interest with a brief description:
.script_run_stop
Contains either the words “run” or “stop” and is populated by the ./sync start|stop|restart script and allows the synchroniser to either continuing running or terminate (gracefully).
.hostname
contains the hostname of the server that the instance on the synchroniser is being run on. Ensures you do not start a second instance on the same server and that you issue the command to stop the synchroniser on the correct server.
.lastrun
A continually updating time stamped output from the “scm status” command. Useful for ensuring that the synchroniser is still running.
.rtc_to_clearcase_sync.ignore
When created, stops RTC SCM events being sent to ClearCase. This can be set at the synchroniser and individual Component levels
Note: This means that the RTC SCM Change Set “backup” in the synchroniser Repository Workspace; therefore, you should be very careful when you remove this control file as it would “open the floodgates” with changes bound for ClearCase, which could cause the snapshot view/local (sandbox) workspace to become corrupt.
.clearcase_to_rtc_sync.ignore
When created, stops ClearCase events being sent to RTC source control. This can be set at the synchroniser and individual Component levels.
.clearcase.ignore
When created and populated, the “.clearcase.ignore” file prevents any files that are named in the file from being updated or added to source control in ClearCase, when they have been added or updated under RTC SCM. This can be set at the synchroniser and individual Component levels.
.classpath.ignore & .build.xml.ignore
When created in the control files area it means the different “.classpath” files can exist in both ClearCase and RTC source control environments and no updates are sent between the two systems for .classpath files. The same is true for “build.xml” files when the .”build.xml.ignore” file is created. This can be set at the synchroniser and individual Component levels.
.get_symlinks
RTC Source Control does not support symbolic linking; therefore, when created, the .get_symlinks control file lists the ClearCase symbolic links in the VOB sub-directory Component that will not be reflected fully in RTC Source Control. For each Component the output is sent to the file:
<$LOGPATH>/$COMPONENT_UUID/.symlinks
Note: You should not leave the .get_symlink flag turned on permanently as this will affect the performance on the synchroniser.
.poc
When created, which you need to do to activate this feature, it should contain the email address of any Point Of Contact (POC) that you want to be emailed if the synchroniser experiences issues, such as, conflict between updates moving between ClearCase and RTC SCM. You can set a .poc file at the synchroniser and individual Component level to allow for flexibility.
.full_logging
When this file is created in the control files area, it will, for example, turn on logging of individual checkout and checkin transactions. This can be set at the synchroniser and individual Component level.
.clearcase_master
By default the synchroniser will attempt to resolve any workspace conflict in favour of RTC SCM, however, if you create a .clearcase_master file in the control file area, it will change this to resolve the conflict in favour of ClearCase. This can be set at the synchroniser and individual Component level.
.lscm_overide
----If the Clear Case VOB sub-directory Components exist at different levels then the synchroniser defaults to using “scm”, however, sometimes you can run “lscm” if the synchronisation paths don’t cross; therefore, if a “.lscm_overide” is created in the $LOGPATH/control files directory it will override the default behaviour and run the “lscm” command line.
Transient files in the control file area
The sychroniser also creates a large number of transient files generated from “cleartool” and “scm” command outputs, which it uses for logging and error trapping purposes. Note: These files will appear in the root of the synchronisers control files and under the component_UUID directory location.
11. WARNING: Permissions are key so don’t manually change things in the synchroniser workspace
Once you have created the jazz scm share you are not advised to manually edit anything in that is shown as the black-box synchroniser in the diagram on the next page. For instance if you manually check out and then check in a file into the VOB from the synchroniser ClearCase snapshot view it will change the files permissions in such a way that the local workspace (sandbox) see this as a change and will create a circular update.
To reiterate the synchroniser controls this behaviour with the permissions function, where basically the permissions of a file are captured before it is updated and reset back to exactly the same permissions after the update.
Permissions are key throughout and the synchroniser ensures files can be written to when they appear/are created in the local workspace (sandbox)/snapshot view location.
12. Using the RTC Eclipse client GUI to resolve issues in the synchroniser local (sandbox) workspace
It is perfectly acceptable to stop the synchroniser and start the RTC Eclipse client GUI as vobadm to resolve any conflict issues:
Note: If the synchroniser has been running the “lscm” daemonised CLI it will continue to run once you have stopped the synchroniser. To rectify this run:
ps –ef | grep scm
vobadm 418 1 0 07:11 pts/1 00:00:08 /opt/IBM/TeamConcert/scmtools/eclipse/scm --config /home/vobadm/.jazz-scm daemon start --connection-timeout 120000 --inactive-timeout 7200000
and kill the process (i.e. kill 418)