INT-PF-3

Controls for the INT PFC:
Configuration management

Issue 3.1; 11th January 1995

Royal Greenwich Observatory,
Madingley Road,
Cambridge CB3 0EZ

Telephone (0223) 374000
Fax (0223) 374700
Internet gtr@mail.ast.cam.ac.uk

1 Introduction

Experience on the WHT tells us some attributes of a good version-control system:

z All versions of each basic component are retained in a library. On the WHT, this is done through DEC's Code Management System (CMS). Each version of a component has a generation number and the generation/file-name pair uniquely identifies the state of a component at the time it was returned to the library.

z Where a program is built from a selection of components, the program itself has a version number (a `class' in CMS) which translates to a unique set of component versions.

z When a program is built for operational use, a specific version is built by an automatic system that supports the version-control standards.

z Any component fetched from the library for building includes its version number in the data. The WHT system does not provide this.

z A built program (object code) can be decomposed to recover the versions of its components. The WHT system doesn't provide this either.

z The version-control libraries are flexible and components can easily be moved between libraries.

z Safeguards are provided against simultaneous changes by two developers. Normally, only one person has write access to each component, but two components from the same library can be changed in parallel.

z Parallel libraries can be maintained at RGO and ING. (Duplicating the libraries is undesireable, but without superlative network performance between Cambridge and La Palma, it's essential.) The libraries can be regularly merged to provide the most recent generations of all files in each copy. This process is not compromised if changes are made to both libraries, provided that no one component is changed in both libraries.

z Once a version of the system is built, it is stored in such a way that a user can easily find the correct programs and there is no chance of executing the wrong version of a program. Ideally, all executable files for a system version are in the same directory.

z Running versions of the system can be patched for emergency maintenance and for testing. Patches never remove files from an operational system, but occlude certain programs and data-files with altered copies. It is easy for an engineer to add and remove patches. Any user can easily determine which patches are in use.

This document descibes first how versions of components are to be managed for the INT-PFC control-system. I then define a method by which versions of programs can be associated with groups of component versions. Finally, the management of complete systems is discussed.

2 Versions of components

For the PFC-controls project, a component is a file in the Unix

file-system. Files become system components if they are delivered with the system or generated during installation. For example, each file of source code is a component, and each object module; however, a list of user preferences generated during observations is not a component.

Each component has one version number. Where several subroutines or functions are contained in one file, they are one component and share the same version number.

2.2 Storage of components

Text and source-code components are stored in SCCS libraries, which are simply directories containing an SCCS history-file for each component. When a component is first entered, SCCS writes its text verbatim to the history file. When updates to the component are entered, SCCS records only the differences from the previous generation. Hence, a history file contains all generations of a component. History files are independent of each other and can be moved freely between libraries (but see the notes on `checked-out' files below).

Binary components, such as object code compiled from source components, are not stored in SCCS. It is assumed that any such file can either by recovered from source or is outside the version-control system.

The contents of history files should only be processed by SCCS or by trusted, SCCS-compatible software. Operating-system commands should only be used on history files if they preserve the contents of the file.

SCCS has an upper limit on line-length, which may vary between implementations. The last line of a component must end in the new-line character.

In its base state, a component only exists as its SCCS history-file. A component may extracted for reference or for system building; in this case, SCCS creates a read-only copy outside the library and makes no change to the history file. The component may also be `checked out' for changes in which case SCCS creates a writeable copy outside the library and a lock-file inside the library. A checked-out component cannot be retrieved for system building or checked out by a second programmer. Removing the lock file with rm, or moving the history file without the lock-file defeats the lock. Moving the history file and the lock file together preserves the lock.

Ideally, all the components are kept in one library of which there is only one copy. If the need arises, components can be moved to subsidiary libraries by using the mv command on the appropriate history files, but these transfers should be kept to a minimum. If parallel copies of a library are required at ING and RGO, the copies can be synchronized by exchange of history files.

In each library there should exist a component called Inventory that lists and defines the other components.

2.3 Version-numbers and tags

SCCS assigns version numbers to components using the Dewey decimal system. Each version number (the SCCS manual calls them SIDs) has at least a major part and a minor part separated by a decimal point. The manual pages for SCCS call the major part the `release' and the minor part the `level'. By default, the first generation of a component is numbered 1.1 and SCCS adds one to the level for each new generation entered. When creating a new generation, you can choose to increase the release number, but cannot control the level number; SCCS will not allow a level of zero in any release.

Occasionally, branches in the line of descent are created by entering changes to a generation that is not the most recent. In this case, generations in the branch have version-numbers with four numeric parts separated by decimal points. Because of this, version numbers must be processed as strings in the most general case and cannot be handled as real numbers.

When a component is retrieved for a build, the developer chooses which generation to retrieve. SCCS provides a way of marking the component's text with the chosen generation such that generations cannot be confused. A `tag'

containing `ID keywords' can be embedded in the text which SCCS expands to include current information. The value of the tag can be retrieved from the component by the SCCS command what, which searches for the escape sequence @(#) and then reports all following characters up to end-of-line.

The standard for the PFC-controls project is to use the keywords %M% (module name) and %I% (version number), %P% (history-file), %E% (date of last modication) and %U% (time of last modification) on one line in a specific syntax:

@(#) Component %M% %I% %P% %E% %U%

For a data-file, the tag can go in any line. For a C source-file, the tag should be put in the initializer of a static array of characters:

const char BOM[] = "@(#) Component %M% %I% %P% %E% %U%";

For documents, the issue number should be made the same as the SCCS version number wherever possible (hence, most documents start at issue 1.1). Since all well-written documents include the issue number in the visible text, the automatic tagging may be omitted.

2.4 Version control procedures

2.4.1 Accessing the right library

Before using SCCS commands, check that you will operate on the correct library and that you are working from a suitable directory. If a component does get into the wrong library you can move it later, but if you apply changes to the wrong component you may lose information.

Either set the environment variable PROJECTDIR to be the pathname of the library or unset PROJECTDIR and cd to the parent directory of the library. In general, you need write access to the SCCS library, to the history files for the components you are working with and to the directory in which you are working.

It is best not to cd into the SCCS library itself.

2.4.2 Creating a new component

When the text of a component has been typed in and checked, ensure that there are no lines longer than SCCS' upper limit and that the last line ends with a new-line character. If the component is a document, check that it displays issue number 1.1.

Add the version-control tags. Consider whether the component might be exempt from tagging (e.g., it is not compilable and already includes the version/issue number in visible text). For source-code, consider whether the component may need a unique variable name for the tag (e.g. it will be included by another component); check that the variable used is large enough for the expanded for of the tag.

Move or copy the candidate component

to the current working directory; note that this file will be destroyed in the process of creating the component, but that you can later recover the data from the library. Use the SCCS command enter to create the history file:

% sccs enter component

Instead of sccs enter you can use sccs create. The two commands are the same except that sccs create automatically retrieves a read-only copy of version 1.1.

2.4.3 Retrieving a component for editing

Consider the name of the component: any file with the same name in the current working directory will be over-written when your retrieve the new copy. Consider also that retrieving the component locks it until you check in your changes. (In necessary, you can later abandon your changes and remove the lock.)

Use the SCCS command edit to retrieve the component:

% sccs edit [-rrelease] component

Checking out a component for editing reserves a version for the altered component; SCCS reports this as the component is checked out. The -r switch specifies a release number for the version. If this switch is not used, the new version has the same release number as the previous version and has a level number one higher. When a new release number is started, the level number goes to one. You cannot change the version of the edited component while it is checked out, but you can check it in without changes (c.f. CMS unreserve) and then check it out again as a different release. See `unlocking a checked-out', below.

2.4.4 Retrieving a component for building

Consider the name of the component: any file with the same name in the current working directory will be over-written when your retrieve the new copy.

Use the SCCS command get to retrieve the component:

% sccs get component

2.4.5 Recording a new version of a component

Check the line-length of the component and that the last line ends with the new-line character. Pause and consider whether it's really time to record a new version. Note that SCCS will delete the external copy of the component that you are entering and that you need write access to this file.

Use the SCCS command delta to enter the new version in the library:

% sccs delta [-r version] component

SCCS deletes the external copy of the new version and unlocks the history file. You cannot use sccs delta on a component that has not been checked out for editing.

2.4.6 Unlocking a checked-out component

If you have a component checked out, and hence locked, you can release the lock with the SCCS command unget:

% cd sccs-library
% /usr/sccs/unget s.component

This operation does not alter or delete the copy that you checked out, but does prevent you from checking in any changes as a new version. To get round this, you can later check the component out again into a scratch directory, discard the copy that SCCS then gives you and go back to the copy that you were editing previously, which can now be checked in. Beware of parallel edits by other programmers!

2.4.7 Changes to parallel libraries

When a component exists in two libraries, extra procedures apply. First, check the component out of all libraries which include it. Choose one library as the working library and save intermediate versions there until the changes are ready for release; these versions should not be used in operational systems. When the changes are complete, check the component into the working library and then copy the component's history file from the working library to all other libraries. Destroy the lock files for the component in all the libraries except the working library (in the latter, the lock file was destroyed when thecomponent was replaced). Note that this is not a merging of versions between libraries; the record from the working library over-writes and destoys the record in all the other libraries. Any changes not in the workin library would be lost.

The process of synchronizing the libraries is delicate. It is dangerous to do it manually using mv, cp and rm. The version-control tool sccs-sync

does the process automatically. The command

% sccs-sync xyz.c 2.5

3 Version control for assemblies

3.1 Definition: bill of materials

Any assembly can be defined entirely in terms of the version numbers its components;

if an assembly arises for which this is not true, something is badly wrong with the system. Hence, a `Bill of Materials' can exist which lists the component versions and allows the assembly to be recreated from the SCCS libraries. A bill of materials is a file within the system, and hence a controlled component; the version number of the assembly is the version number of the bill. An assembly for which no bill has been filed is a rogue element and does not belong in an operational system.

For the PFC-controls project, a bill is a text file in which each line names a component, assembly tag-file. or makefile; the meaning of the entries is outlined in section 3.3 and described in more detail in appendix B. The usual effect of processing a bill is to fetch all the component of the target assembly, to build the assembly by running its makefile, and to include in the assembly a tag stating its version.

The filename of the bill is generally the name of the assembly with the suffix .bom. For example, the executable program obslog would be specified by the bill obslog.bom, while the bill for the object library fits.a would be fits.a.bom.

3.2 Creating a bill of materials

Bills of materials are intended to reproduce versions of assemblies that have been previously built manually. It is possible to write a bill, using a text editor, before any of its components exist; however, one may get the bill more easily, and with less risk of error, by analyzing a copy of the assembly that has been shown to build and run correctly.

The SCCS program what extracts version tags from any file (binary or text) and writes then to standard output. When used on an assembly built from tagged components, what writes out a list of component versions required to recreate the assembly. Typically, the version of the makefile, and possible of a few other components, must be added in a separate run of what. For example:

% what autolog     >  autolog.bom
% what autolog.dat >> autolog.bom
% what autolog.mk  >> autolog.bom

For the bill to be complete and valid, all components referenced must be checked in to the SCCS library. If what is run on an assembly that includes uncontrolled components, these will not appear in the bill. Any components that were built into the assembly while checked out will appear with their ID keywords unexpanded, e.g.

Component %M% %I% %P% %E% %U%

3.3 Building an assembly from a bill of materials; the bom utility

The utility-program bom is provided to process bills of materials. It read a bill line by line and reacts to the keywords `Component', `Assembly', `Makefile' and `Bill' when they are followed by the proper version information. The entries have the syntax

Component component-name  version  [location]
Assembly assembly-name  version  [location]
Tag assembly-name  version  language
Makefile makefile-name  make-target
Bill bill-name  [bom-flags]

Assembly entries provide cross checks on the software environment: they suggest that particular versions of assemblies on which the current assembly depends are installed in specific locations. For assembly entries, bom looks for the named assembly in the place given by location and checks its version number using what. A warning is given if the correct version is not found. Assembly entries do not create or alter files in the build directory and have no effect on the construction of the assembly.

Tag entries allow make to record the version of the assmbly that it builds. For tag entries, bom writes a version tag for the assembly into the file TAG.x where x is a filename suffix appropriate to the specified language. Make can then compile and link the tag file into the assembly.

Makefile entries allow bom to build the assembly by calling make. For makefile entries, bom executes the specified target of the named makefile.

Bill entries invite bom to become recursive, which is useful when building an entrue system. For bill entries, bom tries to run a copy of itself on the named bill.

Bom and make can be used together in various ways. The suggested way is to put the tag entry at the top of the bill followed by the assembly entries (if any are required), then the component entries and finally the makefile entry. When such a file, xyx.bom say, is processed by the command

% bom xyz.bom

The bom utility is described in more detail in Appendix B. Sample bills of materials are shown and described in Appendix C.

3.4 Cleaning up a build directory

After running make, it may be useful to clear away the residual source-files. The command sccs clean deletes from the current working directory any file that can obviously be recovered from the current SCCS library. Use this command with caution: it will delete components that were not used in the last build but are present in the library.

3.5 Installation after building.

In some cases, many assemblies will be installed in the same directory for operational use. In this case, it is undesirable to build the assemblies in the target directory since the files left over from building may cause side-effects and sccs clean is not effective when several libraries have contributed to the system. Ideally, each releaseable assembly should come with an installation script that copies the built files into the directory for operation. A particularly clever script might parse the original bill to find out which assemblies to install.

4 Defining the System

The version number of the entire system is important; it is the only version number that the end-user is likely to know about and so tends to feature in feed-back and bug reports. It helps if the system's version corresponds to meaningful changes in behaviour. In particular, it sours relations between users and support staff if `system a.b' changes behaviour because of an undisclosed change in some assembly. Hence, any planned change to the system should increment at least the minor part of the version number.

The overall system is a meta-assembly containing assemblies built using bills of materials and makefiles. Hence, the version number of the system is the version number of a single file, itself a bill, that lists the versions of all the assemblies. This bill will contain one `Assembly' entry for each assembly and thus running the bill through bom checks the completeness of the system. Additionally, the master bill contains a `Component' entry for each subsidiary bill and thus can be used as the entry point for a make of the entire system.

For the PFC system, the major part of the system version tracks changes in specification (typically, the addition of new functions), while the minor part tracks bug-fixes and changes that are not apparent to the end user.

4.2 Patches

In a system with more than a few major assemblies, defining a system version may be a lot of work. Certainly, one does not want to define and install a complete new system just to try out a new feature in engineering time. Some mechanism is needed whereby certain parts of the system can be temporarily patched.

A mechanism for patching the the PFC system has not yet been chosen. However, some rules for patches are already clear.

z A properly installed system is complete when no patches are present, even if it is too buggy to use. Patches are outside any version of the system.

z Patches do not alter any part of an installed system. Instead, they occlude certain assemblies or components, possibly by appearing earlier in search paths.

z A patch may be transparent, especially to end-users, but it must also be detectable. That is, there must be some straightforward way for a non-technical user to verify that, say, `we are running system 3.2 with patches 11, 12, 14 and 15, but not patch 13 which breaks the FITS-writer; this is what we were told to expect.'

z Patches must be cleanly removable. Removing a patch must have no lasting side-effects.

z Whenever the system version is logged, the current set of patches should also be logged.

In general, I see patches being used by engineers and only very occasionally by observers. Any operational system that runs at night with a patch is broken by definition and someone should be fixing it.

4.3 System storage on disk

The Unix software for the PFC is stored in the directory tree based on the directory /int. In this tree there is one branch (directory) for the main SCCS library, one branch for each built system and one branch for each patch. Other branches may be added to store utility programs that apply to multiple systems and any large installations of software (IRAF?) that are required by but not part of the systems. In the branch for a system or patch there are the directories bin (executable programs) lib (object libraries), src (any source-code retained after build) and etc (small data-files used by the system but not the bulk-data-files used for storing observations). The tree looks like this.

In this example, three system versions are in the tree and three patches 11, 13 and 25 are extant; the substructure has been shown for one of each, but is the same for all systems and patches. A tools directory is present to hold the version-control tools and IRAF has been given its own directory tree to avoid keeping a copy with each system version. A general SCCS library is present, with a scratch directory above it to hold temporary files created while maintaining the library (this sort of thing should not be done in the /int directory). The tools branch has its own SCCS library so that tools maintenance and observing-system maintenance can be kept separate.

The system versions are independent: no system version makes reference to assemblies in any other version, although all versions may refer to the tools or iraf branches of the tree and all derive components from /int/src/SCCS.

4.4 Selecting a system

To use an observing system, a user must put that system in the relevant search-paths. Patchs may be added to a system by putting them earlier in the paths. For example, if

PATH = /int/s1.1/bin:/bin:/usr/bin:/ucb/bin:.

PATH = 
.:/int/p13/bin:/int/p25/bin:/int/s1.1/bin:/bin:/usr/bin:/ucb/bin:

LD_LIBRARY_PATH may also need to be set if the system includes dynamically-linked programs. MANPATH is not needed since the system doesn't include Unix manual-pages.

4.5 Building a version of the system

Building a system is very similar to building one assembly. A master bill of materials (the file whose version defines the system version) is used, and the system may also have a makefile.

The system or `master' bill must contain a component entry for each of the bills describing the constituent assemblies, and one bill entry to activate each subordinate bill. Component and makefile entries may also be present for the system's makefile if one is used. It may be useful to put in assembly entries for the constituent assemblies to verify the build: these should naturally come after the bill entries.

The master bill is run through bom in the normal manner. Standard output should be saved to a file as a record of the build:

%! Fetch the system bill (v3.2) from SCCS.
%   sccs get -r3.2 system.bom
%!
%! Last, careful check on the state of the build directories. 
%   <whatever>
%!
%! Deep breath, cross fingers...
%   bom system.bom > system_3.2_build.log

Any change to a component

of a system that is not part of a patch as defined above generates a new version of the system; there should be no exceptions. To ensure system integrity, the new system version should be rebuilt from sources. Copying the previous version and upgrading a few assemlies is possible but unsafe and may also be mor ework than a complete, automated build.

To define a new system version, it is necessary to update the master bill of materials. This must be done manually: there is no equivalent of what that analyzes an entire system. When updating the bill, ensure that you

1. update the version number in the assembly entry for each changed assembly;

2. update the version number in the component entries for the corressponding bills;

3. add and assembly entry for each new assembly;

4. add a component entry for the bill relating to each new assembly.

Appendix A. Document History

Issue 1.1 01/12/94 Original issue, produced for review by DRM. Nothing in this issue was actually implemented.

Issue 2.1 06/01/95 Revised and expanded: new format for bills of materials; new implementation for bom. The section on system structure and building made more definite. Appendicies B and C added.

Issue 3.1 11/01/95 The Bill entry introduced; description of system building changed. Better examples in appendix C.

Appendix B. The bom utility

This appendix describes v3.1 of bom.

The utility program bom performs version-checking and SCCS operations under control of bills of materials. The utility is invoked from the shell with the command

% bom [-i item] [-m make-target] bill1 [bill2...]

% some-program | bom [-i item] [-m make-target]

An entry in a bill is any line containing one of three pre-set patterns. The patterns

Component filename  version  location

Component filename  version

Assembly filename  version  location

Assembly filename  version

Makefile filename  make-target

Tag filename  version  language

Bill filename  flags
Bill filename

Assembly entries cause bom to check the version of the assembly at location. If location is not given, bom assumes that the assembly is a file with name filename in the current directory. A warning is output if the assembly is not found or if it does not have the right version tag. Assembly entries do not use or affect SCCS libraries.

Makefile entries cause bom to run make on the makefile filename, which is assumed to be exist in the current working directory. One target only is made: that named onthe command line for bom. Makefile entries do not use or affect SCCS linbraries unless the makefile itself uses SCCS. In particular, a makefile entry will not fetch the makefile from SCCS and will fail if the makefile is not present. It is normal to have a component entry for the makefile before the makefile entry.

Tag entries cause bom to generate a file containing a version tag for the assembly named by filename. The tag is written in the specified language, which currently must be C or PERL (upper case is mandatory for this parameter). C-language tags are written in TAG.c and PERl-language tags in TAG.perl; note the case of these filenames.

Bill entries cause bom to process a subordinate bill of materials. A copy of bom is run on the stated bill in a sub-process. The child bom is passed bom-flags on its command line if any were specified in the bill entry that invokes it; otherwise, the child bom is passed the same command-line switches as its parent. Setting bom-flags to just a hypen in the parent bill causes the child bom to run with no command-line switches.

Errors in processing entries do not abort bom.

To be valid, each entry must satisfy the following points of grammer and syntax.

z The captialization of the keywords Component Assembly and Makefile must be exact.

z In each entry, the fields indicated above must appear in the order shown.

z Fields must be separated by one or more white-space characters. The fields themselves must not contain white-space.

z Each entry must be contained on one line. No line may contain more than one entry.

z Entries must contain only ASCII text. Other parts of the bill may contain non-ASCII data, but there is a small danger that such data might by mistaken for an entry.

z Entries may contain other text (commentary). The commentary may come after the fields of the entry or before them or both, but not between the fields.

z Lines not containing commentary can be freely mixed with lines containing entries.

If an entry is syntactically invalid, it will usually be passed over as commentary. Bom does not count the entries processes and will not report an error if it processes a bill with no valid entries.

Appendix C. Examples of bills of materials

The first example defines version 8.3 of the hypothetical program logger for version 10.2 of the parent system.

Tag       logger     8.3  C
Component logger.c   2.4  /int/src/SCCS/s.logger.c
Assembly  fits.a    10.1  /int/s10.2/fits.a
Component log1.c     2.1  /int/src/SCCS/s.log1.c
Component log2.c     2.3  /int/src/SCCS/s.log2.c
Component parse.c   20.4  /rgo/generic/SCCS/s.parse.c
Component logger.mk  1.2  /int/src/SCCS/s.logger.mk
Makefile  logger.mk logger

This bill names all the required components and assemblies. logger.mk can assumes that all the files required for the build will be present when make is run. However, if bom finds problems with any component, or with the assembly fits.a, it will not abort. The makefile will still be run and the make will probably fail. Because of this, it is safer (a) to run the build in a scratch directory that can be wiped clean after a failed build; (b) to copy the built assembly to its operation directory only if the build succeeds - this may rule out automatic installation by make.

The second example shows an assembly entry without a location, used to verify the build.

Component ccd1.c     2.4  /int/src/SCCS/s.ccd1.c
Component ccdsub.c   3.4  /int/src/SCCS/s.ccdsub.c
Comonent  ccd1.mk    1.2  /int/src/SCCS/s.ccd.mk
Tag       ccd1       4.1  C
Makefile  ccd1.mk ccd1
<Many other lines concerned with related assemblies.>
Assembly  ccd1 4.1

The third example shows a version of the logger bill that is less dependent on location in the file system.

Component logger.c   2.4
Assembly  fits.a    10.1  $LIBDIR/fits.a
Component log1.c     2.1
Component log2.c     2.3  
Component parse.c   20.4  $RGOSCCS/s.parse.c
Component logger.mk  1.2  logger.mk
Tag       logger     8.3  C
Makefile  logger.mk logger

A system bill (for a trivially small system) might look like this.

Component prog1.bom       2.5
Component prog2.bom      12.5
Component prog3.bom       1.3
Component config.sh.bom  28.9
Component system.mk       3.4
Bill prog1.bom
Bill prog2.bom
Bill prog3.bom
Assembly prog1      2.5
Assembly prog2     12.5
Assembly prog3      1.3
Assembly config.sh 28.9
Makefile system.mk INSTALL

The final example shows a bill rigged for a different use of make.

Component  ccd.mk  4.5
Makefile   ccd.mk  ccd
Component  ccd1.c  2.6
Component  ccd2.c  8.2
Component  ccd3.c  1.4

% bom -i ccd.mk ccd.bom

ccd2.c :
   bom -i ccd2.c ccd.bom