Particle Physics and Astronomy
Research Council
Royal Greenwich Observatory
Configuration management
standards for
ING Observing Systems
Guy Rixon
Issue 4. 1; 16th May 1996
Royal Greenwich Observatory,
Madingley Road,
Cambridge CB3 OEZ
Telephone (0223)374000
Fax (0223)374700
Internet gtr@mail.ast.cam.ac.uk
CONTENTS
1 Introduction
1.1 Purpose
of this document
1.2
Scope of the software
1.3 Glossary
1.4 References
2 Goals
3 Version control
3.1 Definition
of 'component'
3.2 Storage
of components
3.3 Version
numbers
3.3.1 Version-number
tags
3.4 Version
control procedures
3.4.1 Accessing
the right library
3.4.2 Creating a new component
3.4.3 Retrieving
('checking out') a component for editing
3.4.4 Retrieving a component
for building
3.4.5
Recording ('checking in') a new version of a component
3.4.6 Unlocking a checked-out
component
3.4.7 Installing
components and synchronizing libraries
4 Build control
4.1 Definition
of 'assembly'
4.2 Definition
of 'bill of materials'
4.3 Creating
a bill of materials
4.4 Building
sub-systems with the bom utility
4.5
Building systems
4.6
Standard targets for makefiles
5 Installation
in an observing system
5.1 Assigning
the version number
5.2 The
release tree
5.3 Environment
variables
5.4 Patches
Appendix A. Document History
Procedures are described for version control of source-files, programs, sub-systems and systems involved in ING observing systems. The principles described here have been evolved during production of the INT DAS and were later accepted by ING computing staff as the basis of an ING standard. All programmers intending to deliver code to ING should read this document.
This document supersedes document INT-PF-3.
These procedures are intended to apply ING observing systems; that is to all software written specifically for ING in direct support of observing. Data-reduction packages such as IRAF are not controlled in this way and neither are infrastructure packages such as DRAMA.
The current scheme can only be applied to software that is stored in a Unix file-system. Hence, FORTH systems, VAX-ADAM systems and the TCS do not conform.
[1] BOM: a configuration-control language
ING/RGO document SOF-BOM-1 by Guy Rixon.
Observing-systems for ING telescopes are noted for large-scale reuse of code (consider the D-tasks in the WHT ADAM-system), frequent, minor changes, and the critical interdependencies between sub-systems. These systems rapidly decay and fail if proper version-control is not applied.
Experience on the WHT tells us some attributes of a good version-control system:
Any non-trivial software-systems is built from a number of source-code components. Configuration management requires that system versions be capable of analysis into versions of components and that all versions of the components remain available for inspection and re-use.
For the Unix-based observing systems, a component is a file in the Unix file-system. Files become system components if they are delivered with the system or generated during installation. For example, each file of source code is a component, and each object module; however, a list of user preferences generated during observations is not a component.
Each component has one version number. Where several subroutines or functions are contained in one file, they are one component and share the same version number.
Text and source-code components are stored in SCCS libraries, which are simply directories containing an SCCS history-file for each component. When a component is first entered, SCCS writes its text verbatim to the history file. When updates to the component are entered, SCCS records only the differences from the previous generation. Hence, a history file contains all generations of a component. History files are independent of each other and can be moved freely between libraries (but see the notes on 'checked-out' files below).
Binary components, such as object code compiled from source components, are not normally stored in SCCS. It is assumed that any such file can either by recovered from source or is outside the version-control system. Where a component only exits in binary form, it can be stored by uuencoding the data.
The contents of history files should only be processed by SCCS or by trusted, SCCS-compatible software. Operating-system commands should only be used on history files if they preserve the contents of the file.
SCCS has an upper limit on line-length, usually 132 characters, which may vary between implementations. The last line of a component must end in the new-line character.
In its base state, a component only exists as its SCCS history-file. A component may extracted for reference or for system building; in this case, SCCS creates a read-only copy outside the library and makes no change to the history file. The component may also be 'checked out' for changes in which case SCCS creates a writeable copy outside the library and a lock-file inside the library. A checked-out component cannot be retrieved for system building or checked out by a second programmer. Removing, the lock file with rm, or moving the history file without the lock-file defeats the lock. Moving the history file and the lock file together preserves the lock.
The primary SCCS library is /ing/src/SCCS. All source-components for ING sub-systems are kept here. Test materials are stored separately in /ing/test/SCCS. and some general tools (those that fall outside the scope of the observing system) are kept in /ing/tools/SCCS.
Parallel copies of the libraries exist at RGO and ING. Sets of history files may be copied between the two sites by tar When using tar it is necessary to check carefully that generations are not lost when a history file is over-written. A tool to check this is planned. History files should never be copied or updated while the elements concerned are checked out for editing from either library.
In each library there should exist a component called Inventory that lists and defines the other components.
SCCS assigns version numbers to components using the Dewy decimal system. Each version number (the SCCS manual calls them SIDs) has at least a major part and a minor part separated by a decimal point. The manual pages for SCCS call the major part the 'release' and the minor part the 'level'. By default, the first generation of a component is numbered 1.1 and SCCS adds one to the level for each new generation entered. When creating a new generation, you can choose to increase the release number, but cannot control1 the level number; SCCS will not allow a level of zero in any release.
Occasionally, branches in the line of descent are created by entering changes to a generation that is not the most recent. In this case, generations in the branch have version-numbers with four numeric parts separated by decimal points. Because of this, version numbers must be processed as strings in the most general case and cannot be handled as real numbers.
When a component is retrieved for a build, the developer chooses which generation to retrieve. SCCS provides a way of marking the component's text with the chosen generation such that generations cannot be confused. A 'tag'2containing 'ID keywords' can be embedded in the text which SCCS expands to include current information. The value of the tag can be retrieved from the component by the SCCS command what, which searches for the escape sequence @(#) and then reports all following characters up to end-of-line.
The standard is to use the keywords %M% (module
name) and %I% (version number), %P% (history-file), %E%
(date of last modification) and %U% (time of last modification)
on one line in a specific syntax:
@(#) Component %M% %I% %P% %E% %U%Tags of this form3are recognized by the bom utility described in section 4.4 [1]. A tag is said to be 'unexpanded' when it has the literal form above and 'expanded' if the values of the keywords have been filled in.
For a data-file, the tag, can go in any line. For a C source-file, the tag should be put in the initializer of a static array of characters:
For documents, the issue number should be made the same as the SCCS version number wherever possible (hence, most documents start at issue 1.1). Since all well-written documents include the issue number in the visible text, the automatic tagging may be omitted.
In interpreted languages such as PERL, the tag can either be assigned to a variable or can be included in a comment.
3.4 Version control procedures
3.4.1 Accessing the right library
Before using SCCS commands, check that you will operate on the correct library and that you are working from a suitable directory. If a component does get into the wrong library you can move it later, but if you apply changes to the wrong component you may lose information.
Either set the environment variable PROJECTDIR to be the path-name of the library or unset PROJECTDIR and cd to the parent directory of the library. In general, you need write access to the SCCS library, to the history files for the components you are working with and to the directory in which you are working.
It is best not to cd into the SCCS library itself.
3.4.2 Creating a new component
When the text of a component has been typed in and checked, ensure that there are no lines longer than SCCS' upper limit andthat the last line ends with a new-line character. If the component is a document, check that it displays issue number 1.1.
Add the version-control tag. Consider whether the component might be exempt from tagging (e.g., it is a document).
Move or copy the candidate component4 to the current working directory; note that this file will be destroyed in the process of creating the component, but that you can later recover the data from the library. Use the SCCS command enter to create the history file:
csh> sccs enter component
where component is the path-name of the file containing the new component. SCCS will create, in the library, the history file s.component and will rename the file from which it read the component's text to, component. The latter file can be deleted if the SCCS command completed without errors5.The component is now recorded at version 1.1.
Instead of sccs enter you can use sccs create. The two commands are the same except that sccs create automatically retrieves a read-only copy of version 1.1.
3.4.3 Retrieving ('checking out') a component for editing
Consider the name of the component: any file with the same name in the current working directory will be over-written when your retrieve the new copy. Consider also that retrieving the component locks it until you check in your chances. (If necessary, you can later abandon your changes and remove the lock.)
Use the command sccs edit to retrieve the component:
csh> sccs edit [-rrelease] componentIf you intend to create a new minor version - e.g. to upgrade from v2.1 to v2.2 - omit the –r version switch. SCCS will assign the new version-number automatically. Use the -r switch if you intend to create a new major version. For example, to go from v2.2 to v3. 1, give the command:
csh> sccs edit -r3 componentThe minor part of the version-number is not given in the command: SCCS sets this to one automatically. No space should be typed between -r and the number as SCCS does not accept that syntax. Users of CMS should note that SCCS assigns the new version when the component is checked out, not when it is checked back in.
SCCS creates component in the current working directory and allows you write access to it. The component in the library is now locked. If the component is tagged, its ID keywords are not expanded.
You cannot change the version of the edited component while it is checked out, but you can check it in without changes (c.f. CMS unreserve) and then check it out again as a different release. See ‘unlocking a checked-out component’, below.
3.4.4 Retrieving a component for building
Consider the name of the component: any file with the same name in the current working directory will be over-written when your retrieve the new copy.
Use the SCCS command get to retrieve the component:
csh> sccs get componentSCCS creates component in the current working directory and denies you write access to it (you can use chmod to get write access later if necessary). The component in the library is not locked, If the component is tagged, its ID keywords are expanded.
Most builds are done under the control of the bom utility as described below. In this case there is no need to retrieve the components manually.
3.4.5 Recording ('checking in') a new version of a component
Check the line-length of the component and that the last line ends with the new-line character. Pause and consider whether it's really time to record a new version. Note that SCCS will delete the external copy of the component that you are entering and that you need write access to this file.
Use the command sccs delta to enter the new version in the library:
3.4.6 Unlocking a checked-out component
If you have a component checked out, and hence locked, you can release the lock with the command sccs unedit:
3.4.7 Installing components and synchronizing libraries
Components should never be edited at ING and RGO at the same time as merging the changes is difficult and unsafe. With current working practices, parallel changes are not likely.
Libraries should be brought into line by exchanging individual components. It is usually not possible to export an entire library between sites as some components are typically maintained at each site: neither library contains the most recent versions of all the components.
The components to be exchanged can be determined by comparing listings of the two libraries. The command
Components to be exchanged should be checked in at both sites.
To transfer components, make a tar file of the history files and copy it to the other site. The filenames in the tar file should be relative paths. That is:
To install the transferred files, unpack the tar file into the target library e.g.
Sub-systems within the observing system need configuration management to the same degree as software components. At this level, versions and revisions start to become visible to managers and end-users.
The ING standard for build control extends the version numbering scheme for components to 'assemblies' (as defined below) by means of the Bill-Of-Materials Language (BOM).
An 'assembly' is a special case of a configuration-management baseline. It may be any group of inter-related components, or something made from those components. Examples are:
4.2 Definition of 'bill of materials'
A bill of materials is an executable script in the BOM language, as described in [1]. Bills are readable by engineers (the language is interpreted and the bills are text files), and can be used as instructions for manual building of a sub-system. Normally, however, a bill is executed by the utility bom to build an assembly from source components. The execution involves fetching components from SCCS, executing the makefile and applying version-numbering to the assembly.
Execution of bills is hierarchical: a statement in one bill can invoke the processing of subordinate bills for lower-level assemblies. In this way, an observing-system may be defined in terms of versions of sub-systems and the sub-systems may be progressively decomposed into lower-level assemblies and finally into source components: the version-number of the system defines uniquely the building of the entire system.
Each bill is a version-controlled component. The version-number of an assembly is the version number of the bill that defines it
A bill may define more than one assembly8. In this case, each assembly has the same version number.
The file-name of the bill is generally the name of the assembly with the suffix .bom. For example, the executable program obslog would be specified by the bill obslog.bom, while the bill for the object library fits.a would be fits.a.bom.
4.3 Creating a bill of materials
A bill should always contain the following statements:
It is trivial to write a bill of materials with a text
editor if you already have a listing of the version of each component.
An easy way to generate such a listing is to run the SCCS program what
on
a previously-built copy of each assembly and to concatenate the results.
The SCCS program what extracts version tags from any file (binary or text)
and writes then to standard output. When used on an assembly built
from tagged components, what writes out a list of
component versions required to recreate the assembly. Typically, the version of the makefile, and possibly a few other components, must be added in a separate run of what. For example:
For the bill to be complete and valid, all components referenced must be checked in to the SCCS library. If what is run on an assembly that includes uncontrolled components, these will not appear in the bill. Any components that were built into the assembly while checked out will appear with their ID keywords unexpanded, e.g.
4.4 Building sub-systems with the bom utility
Normally, a system will be rebuilt by sub-systems. For example, the das sub-system (meaning the set of programs and utilities concerned specifically with the Data-Cell interface) is defined by das.bom which is itself defined in terms of dasn.bom, dasm.bom and das_clients.bom. Building this sub-system generates five executable programs from the three lower-level bills.
The recommended procedure is as follows.
2. Set the environment variable OBSSYS to define the version of the observing system (see section 5). Unset OBSSYSPATH.
3. Set ING to define the root of the source tree.
4. Unset PROJECTDIR. This prevents make
from
fetching files from SCCS and reduces the chance of building in a
rogue component.
5. Fetch from SCCS the bill of materials, e.g.
8. Wipe the build directory, but keep the log of results
for quality assurance.
The master bill of materials for the observing system is obssys.bom. In principle, executing this bill will build and install the entire system. In practice, some sub-systems require others to be pre-installed before their makefiles can operate (e.g. das requires libcia to be installed) and the bom utility cannot currently achieve this. Currently, executing obssys.bom fetches bills for versions of subsystems to be built manually. This situation will be improved in due course.
4.6 Standard targets for makefiles
The principal make-targets that build the assemblies are stated in their respective bills and so need not have standard names. It is conventional to provide the standard targets install to move assemblies into the release tree, deinstall to strip assemblies out of the release tree and clean to wipe the build directory.
5 Installation in an observing system
5.1 Assigning the version number
The version number of the entire system is important; it is the only version number that the end-user is likely to know about and so tends to feature in feed-back and bug reports. It helps if the system's version corresponds to meaningful changes in behaviour. In particular, it sours relations between users and support staff if 'system a.b' changes behavior because of an undisclosed change in some assembly. Hence, any planned change to the system should increment at least the minor part of the version number.
As discussed in the previous section, the system is the highest-level assembly in a hierarchy of assemblies, and its version number is the version number of its bill of materials. Typically, the major part of the system version tracks changes in specification (e.g. the addition of new functions), while the minor part tracks bug,-fixes, or changes that are not apparent to the end user.
Each version of the system is installed independently and depends from a particular directory. For example, the first system, version s1.1, depends from /ing/s1.1. It is anticipated that all systems will be installed in the /ing tree, but this need not be the case; in general, software should not rely on this installation.
Below the system's directory, e.g. s1.1, are directories bin (for executable programs and scripts), lib (for object libraries and loadable Tcl modules), inc (for include files), and etc (for data-files needed at run-time). This arrangement is standard and applications (and their makefiles) can rely on it.
The release tree also contains directory structures for patches, as defined below. Patch directories have the standard naming pn, where n is the number of the patch. They contain the same sub-directories as systems.
The tree looks like this:
As well as the system and patch directories, /ing also contains the src directory where the components live and a tools branch containing tools needed at build-time, but not at run-time. (For example, /ing/tool/bin is where bom lives.) /ing/bin and /ing/etc contain files needed at run-time that are not part of anyobserving system.
The environment variables OBSSYS and/or OBSSYSPATH must be set when running an observing system: they allow pro-rams within the system to traverse the system tree without being aware of which system they are in or where that system tree is mounted.
OBSSYS should always be set to indicate the root of the system tree, e.g.
There are standard components available to programs in the observing system that search for a file along OBSSYSPATH or in OBSSYS if OBSSYSPATH is not set. A program might ask for the location of "etc/prog.settings" and be given "/ing/p2/etc/prog.settings" or "/ing/s1.1/etc/prog.settings " according to the values of the environment variables.
When setting OBSSYS and OBSSYSPATH, PATH should also be set to include the bin sub-directory of the base system and every patch in the list.
The script obssys is routinely used by observers to select a system and a set of patches. It is kept in /ing/bin (i.e. outside any observing system) and needs to be source'd by the user. The version of the base system is hard-coded into obssys and any arguments supplied are taken as the numbers of patches. Obssys always sets OBSSYS and PATH and sets OBSSYSPATH if any patches were requested. The entries in PATH for the system and patches are put at the front.
The environment variable ING is not used at run-time.
In a system with more than a few major assemblies, defining and building a system version may be a lot of work. Certainly, one does not want to define and install a complete system just to try out a new feature in engineering time. Some mechanism is needed whereby certain parts of the system can be temporarily patched.
A patch is a partial observing system and is assumed to replace and occlude certain files in a base system. Typically, new versions are supplied for a handful of programs or data-files. Since each patch lives in its own tree, as described above, creating a patch does not permanently alter the observing system to which it applies. Patching at run-time is explicit: the user must request patches by number.
Each patch is identified by a single number. It is convenient, for purposes of recording, to count from one and never to re-use patch numbers. In any case, it is impossible to have two patches with the same number in the release tree at once.
Patches are mainly intended as a suitable environment for trying out new versions of sub-systems. They may also be used as quick fixes: it is reasonable to say to an observer 'You are still using system s2.3 but you'll need to load patch number 42 to fix the bug we saw last night'. It is less reasonable to run the same set of patches night after night; patches that are fixes should be consolidated in a new version of the system as soon as they have been tested. Requiring the observer to load dozens of patches (all in the right order of precedence) and then varying the set each night is evil. Varying the contents of a patch without telling the observer is doubly evil.
1. In fact, you can force SCCS to skip
to a higher level-number by entering a series of Generations without changes.
2. 'Tag' is my term of choice and does
not appear in the SCCS manual.
3. In fact, only the part @(#) Component
%M% %I% is parsed by bom; the rest is ignored.
4. There's a semantic problem here.
Before SCCS first operates on a file, it isn't under version control, so
is it worthy of the name 'component'?
5. SCCS may report 'no id keywords found'.
This an error if the component was meant to be tagged, but is OK otherwise.
6. Did you pack your tar file
yourself? Has it been left unattended at any time? Does it contain any
hazardous code or firmware likely to interfere with the operating system?
7. By convention, the build is defined
by the makefile and the makefile is a version-controlled component
of the assembly.
8. Suppose that progs.bom contains
the building instructions for executable programs prog1 and prog2.
Both
prog1
and prog2 are assemblies because they are built programs. The union
of prog1 and prog2 is also an assembly because it is created at
the same time from the same bill: this assembly is a package or
possibly a sub-system. The package and the two programs each take the version
number of progs.bom.
9. The SCCS keyword %1% gets expanded
into the current version number when the bill is extracted from the library
for execution. This reduces the risk of the wrong version number being
written.
Issue 1.1 01/12/94
Original issue as INT-PF-3, produced for review by DRM. Nothing in this issue was actually implemented
Issue 2.1 06/01/95
Revised and expanded: new format for bills of materials; new implementation for bom. The section on system structure and building made more definite. Appendices B and C added.
Issue 3.1 11/01/95
The Bill entry introduced; description of system building changed. Better examples in appendix C.
Issue 4.1 16/05/96
Rewritten as a general standard (previous issues addressed
only the INT WFC project). Glossary added; section on patches made formal;
section on build control extended, but details of BOM and bom devolved
to [I].
Appendices C and D deleted. Document reclassified SOF-STD-3.