Issue 1.1; 8th February 1996
This manual is provided as a guide to ING staff in installing, maintaining and configuring the Unix-based Data-Acquisition System (DAS). It is most definitely not an instruction to visiting observers on how to drive the system during normal observing; misuse of the techniques described below can subvert the integrity of the system and lose observations.
1.2 Scope of the software
The DAS is the computer sub-system that records observations on disk. The detailed, technical requirements for the DAS are listed in [1] and the design of the software is described in [2]. Image data are input to the DAS via fibre-optic links from the CCD controllers (CCDCs) and textual data describing each observation are read from files on disk. The DAS outputs disk-FITS files each of which contain one run with an explanatory FITS header.
The DAS does not include the software to drive the CCD controller (CCDC), to compile the observation log or to archive observations on tape.
1.3 Glossary
Archive run: a science observation destined for the log, FITS tapes and La Palma archive.
bill of materials: a script in the language BOM (see [3]) defining the components that make up a program or group of programs.
CCDC: CCD controller.
channel: the data path to the DAS from one CCD. Channels have numbers 0 to 7.
client program: a program that uses a service provided by dasm or dasn.
cmd: a client program for general engineering commands.
csh> : represents a C-shell prompt in examples of commands. You do not type these characters when issuing the command.
DAS: Data-Acquisition System.
dasformat: a client program that sets the DAS' readout format.
dasm, DASM: the DAS' master server; dasm is the executable file and DASM is its name in the DRAMA message-net.
dasn, DASn: the server for one DAS channel.
dasreset: a client program that resets or reconfigures the entire DAS.
glance run: a run that persists on disk only until the next acquisition and is then over-written. All runs are glance runs if they aren't archive or scratch runs.
headcode: the byte attached to a pixel which identifies the CCDC that produced it.
header packet: a disk-file containing a contiguous fragment of the FITS header for a run.
libcia: a library of support subroutines needed to build the DAS client-programs.
obsdata: the client program that changes data directories.
run: the quantum of recorded observation. Each run is stored in a separate FITS file.
run number: runs are numbered, counting from one, and the number sequence for a given telescope is continuous for the entire lifetime of the telescope.
scratch run: a run preserved on disk for examination but not archived.
1.4 References
[1] Software Requirements for the Data Acquisition Servers
ING/RGO document INS-DAS-8.
[2] Implementation of the Data Acquisition Server
ING/RGO document INS-DAS-9
[3] BOM: a Configuration-Control Language
ING/RGO document SOF-BOM-1
[4] FITS headers for INT observations
ING/RGO document INT-DAS-2
[5] Dialogues with the user: general principles
ING/RGO document OBS-TALK-1.
2.1 Environment
The build process uses two environment variables: OBSSYS and ING. ING should be set to show the root of the source tree from which components can be retrieved: e.g. /ing or /lpss10/ingsw. OBSSYS should name the root of the observing system into which the DAS will be built: e.g. /ing/s1.0 or /ing/p1. ING and OBSSYS must always be absolute path-names and may not include shell constructs (wild-cards; references to environment variables; the tilde character).
To build the DAS completely the major assembly libcia must first be built and installed. Check the release notes for compatible versions of libcia. Certain source-components associated with libcia are required. The DAS also requires source components from the srv series. All source components are drawn from the library ${ING}/src/SCCS.
Many of the configuration techniques require the command tool cmd.
The DAS can be built (but cannot run) on a computer that does not have the FOX interface described in [1].
2.2 Installation of components and building of assemblies
The DAS source-components must be installed in the SCCS library ${ING}/src/SCCS. This is not an option: the bills of materials for the DAS will not operate correctly if the sub-system is installed elsewhere.
There are three bills of materials at the lowest level: dasn.bom, dasm.bom and das_clients.bom. These can be run directly as described in [3] and each build one assembly (the das_clients assembly consists of several executable programs). The bill das.bom builds all the assemblies in the sub-system (it executes the other three bills). Hence, the DAS as a whole has a version number and the three assemblies each have a version number.
After building the sub-system there should be five executable programs:
dasn is the server for one DAS channel; one copy of it is run per active channel;
dasm is the DAS' master server which manages start-up and shut-down;
obsdata is a client that changes data directories;
dasformat is a client that sets and reports the readout format;
dasreset is a client that resets and reconfigures the DAS.
2.3 Carrying over information from previous versions
When installing a version of the DAS in a new observing system, the files that maintains continuity of run numbers are re-used in the new system. See section 4.7 for details. It is absolutely vital that these files contain the correct numbers when observing starts; if the numbers are wrong observations may be mis-filed and lost.
Settings files (see section 4.1) from previous versions of the DAS may still be compatible: check the release notes to find out. If the settings files are compatible, it is legitimate to copy the files ${OBSSYS}/etc/DAS*.settings from the previous system to the new system.
The running DAS contains the task DASM (executable file dasm) and one task DASn (n = 0..7; executable file dasn for all copies) for each active channel. In this context, `active' means `likely to be used tonight' rather than `currently receiving data'; an inactive channel cannot be used without restarting the DAS. DASM is the parent of DASn and starts them according to the memory allocated to each channel.
The image-acquisition interface reserves a certain amount of memory for each channel. The amount of memory is set when DASM loads DASn and, for each channel, must be large enough to hold the largest image to be acquired on that channel. Normally, the memory allocation is enough to hold one full frame of the CCD attached to the channel. Allocating zero bytes of of buffer-memory to a channel deactivates it; allocating a positive number of bytes actives the channel.
To start the DAS, invoke dasm from the shell:
csh> dasm telescope [m0 m1 m2 m3 m4 m5 m6 m7] &
The task DASM then runs in the background until killed. The arguments m0..m7, which are optional, are the memory allocations for channels 0..7 respectively and should be given in bytes. Any memory allocations not given will default to the numbers retrieved from the settings file (see section 4.1) for DASM. The system records the allocation in the settings file. Once started, DASM starts DASn for each channel which has a non-zero allocation. The argument telescope should be set to INT, JKT or WHT. DASM passes this to DASn which use it to select the correct series of run numbers.
To stop the DAS, invoke the EXIT action on DASM:
csh> cmd DASM EXIT
As it dies, DASM kills off all copies of DASn. Any runs in progress are stopped immediately and their data are lost.
To restart the DAS with the same memory allocations (e.g. to clear an error), use the das_reset client:
csh> das_reset
To alter the memory allocations, use das_reset with command-line arguments:
csh> das_reset m0 m1 m2 m3 m4 m5 m6 m7
where m0..m7 are the same arguments as for the dasm command.
You can start DASn directly for engineering purposes:
cash> dasn telescope channel memory &
where telescope is the telescope name as for dasm, channel is the channel number and memory is the memory allocation in bytes. Do not do this if you intend to run DASM at the same time. To shut down your DASn, use its EXIT action:
csh> cmd DASn EXIT
The DAS has control settings which it remembers from night to night: the settings are recorded in the files DASn.settings (n = 0..7) and DASM.settings. The former files specify the data directory, pixel headcode, header-packet list and readout format for channels 0..7; the latter specifies the memory allocations for each of the eight channels. The settings files live in ${OBSSYS}/etc.
When a version of the DAS is installed, it has no settings files.1 The system creates default files as it starts up. The memory allocations are defaulted to zero bytes (i.e. an uninitialized DAS has no active channels). The other settings files have a validity flag for each item which is set false when the files are created.
The validity flags may be read from the DAS' parameter set; to check validity on channel n use:
csh> cmd -g DASn READY
which returns a boolean flag (1 = true, 0 = false) for the validity of each control setting. The flag READY.RUN does not refer to a control setting but indicates whether the channel is armed to receive image data from the CCDC. If any ready flags, except READY.RUN, are false, the channel is not able to record observations. making a valid control-setting, as described below, changes the associated flag to true.
Control settings can and should be set up by ING staff: this would logically be part of an instrument change. The settings remain in force until they are over-ruled or the settings files are removed. Normally, observers will only need to change the readout-format settings and, very occasionally, the data directory. Normal practice should be to set up the system for the observers such that all ready flags show true.
The settings files are binary. DASM.settings is in SDS format and can be read by the utilty sdslist from the directory $SDS_LIB. DASn.settings cannot be read except by dasn. It is sometimes possible for observers to preserve their settings files between observing runs: this allows them to recreate formats exactly. However, the settings files are not necessarily compatible between versions of the DAS. Using an incompatible version of a settings file may cause very odd failures.
The settings files must be ritual by the programs dasn and dasm. This generally means that the files must be writeable by any user.
4.2 Activating channels
See section 3 for instructions on activating channels.
4.3 Setting the data directory
The directory in which observations are stored may be set separately for each channel. In all cases, the absolute path-name of the directory must be given and the name may not contain shell constructs such as references to environment variables. Observers should use the client program obsdata to change directories:
csh> obsdata channel directory
where channel is the channel number and directory is the path-name. Obsdata uses interlocking and will not act when there is a danger of losing data.
Engineers may note that, interlocking apart, obsdata is equivalent to
csh> cmd DASn OBSDATA directory
A successful change to a data directory generates a broadcast alarm advising that the directory change should be passed on to the logging and archiving systems.
The current data directory may be checked using the command
csh> cmd -g DASn OBSDATA
4.4 Setting the pixel headcode
A pixel headcode is a byte appended to each pixel input to the DAS that indicates which CCD the pixel came from. A DAS channel is attached to a CCD by setting the headcode for that channel. Set the headcode thus:
csh> cmd DASn HEADCODE headcode
where headcode is the decimal value of the headcode. Note that headcodes are often specified as hexidecimal values: headcode 20, meaning CCDC number 2, head number 0, implies a hexidecimal value.
Changing the headcode during an acquisition will lose the pixels from that run; the command is not interlocked to prevent this. Headcodes should only be changed by ING staff.
The current headcode may be checked using the command
csh> cmd -g DASn HEADCODE
4.5 Setting the packet list
The list of header packets defines the data that will be copied into the FITS header of each run on the channel in question. There is a standard set of packets for particular instruments: see [4]. The packet list can be set with the command
csh> cmd DASn PACKETS list count
where list is a space-separated list of packet names enclosed in double quotes (e.g. "observation telescope autoguider pfc ccd" and count is the total number of FITS descriptors that will appear in all the packets. Changing the packet list has no effect on runs in progress but applies to all subsequent runs. The packet list should only be changed by ING staff.
The current packet-list may be checked using the command
csh> cmd -g DASn PACKETS
4.6 Setting the readout format
The readout format describes how the DAS should reconstruct the image on a channel, taking account of CCD size, windows and binning. The format also allows for rotation and reflection of the image relative to the geometry of the CCD.
In general, observers will set the format by running client programs that are outside the scope of the DAS: it is necessary to set equivalent formats in the DAS and the CCDC. Engineers can set the DAS format independently using the dasformat command:
csh> dasformat [-c ccd-size -b binning -w enabled -1 window -2 window \
-3 window -4 window -t transformation -p -r -d] channel
where channel is the channel number. If none of the options are invoked the effect is to set the format to default values:
CCD size defaults to 100:100;
binning factors default to 1:1;
windowing defaults to disabled;
transformation defaults to no transformation;
the windows default to 0:0:0:0.
The -p (`read parameter') option avoids the defaults and initializes the new format to the values currently set on the DAS channel.2
Most of the options alter an aspect of the format:
-c sets a new CCD size. The size of the unwindowed, unbinned image is given in pixels in the format x:y, e.g 1180:1280.
-b sets binning factors. The x and y factors are given in the format x:y, e.g. 2:4.
-w enables or disables windowing. Its argument is 1 to enable windowing and 0 to disable.
-1 sets window 1. The argument is in the format x-size:y-size:x-origin:y-origin, e.g. 100:1280:500:0 and the sizes and origins are measured in unbinned pixels. The leftmost column and bottom row are numbered 0: in the example above, the window spans the full height of the CCD.
-2 sets window 2 with the same formalism as -1.
-3 sets window 3 with the same formalism as -1.
-4 sets window 4 with the same formalism as -1.
-t sets the geometric transformation. The argument is a quoted string in which each character specifies a transformation: `x' is a reflection in the x axis, `y' a reflection in the y axis, `+' a rotation by +90 degrees (i.e. anti-clockwise) and `-' a rotation by -90 degrees (i.e. clockwise). The transformations are applied in order from left to right.
The -r option causes dasformat to read back and report to the terminal the format that it has set. Using -p and -r with no other options reprocesses and reports the format that was set at the start of the command.
The -d option instructs the DAS to dump to the terminal its array of `readout strips'; see [2]. This array is an internal representation of the format and can be inspected to check the format processing. It is no use whatsoever during observing.
The current format can also be listed to the terminal using the command
csh> cmd -g DASn FORMAT
which will show the positions of the windows in the output image as well as their position on the original CCD frame (dasformat -r doesn't show this). It is not possible to set a format using cmd.
4.7 Setting the run number
A contiguous series of run numbers is used for each of the three telescopes: the numbers do not start from one for each new group of observers and are not set back to one when a new version of the DAS is brought into use. The record of the last run-number used at each telescope is therefore recorded outside any observing system.
There are three files, INT.run, JKT.run and WHT.run, recording the last run-number to be allocated at the respective telescopes. They are kept in a directory etc that is in the same directory tree as the observing systems but not a part of any system. For example, if the observing systems /ing/s1.1 and /ing/s2.4 are on disk, the run-number files would be in /ing/etc. This arrangement is critical to the DAS: the tasks DASn look for the run-number files in ${OBSSYS}/../etc when they start up. The files must be writeable by the program dasn. This generally means that the files must be writeable by any user.
Each file is one line of ASCII code containing the number followed by a new-line character. If the run-number file is lost for any reason, it can be recreated using a text editor. The last-used number can be recovered from the observation logs.
4.8 Supporting software
The DAS is designed to be run in conjunction with a Talker program (see [5]) such that messages to the user and alarms are handled correctly. The DAS servers connect to a default Talker when they are started: this is the task named by the environment variable TALKER in the job from which DASM is started. Other talkers can connect to the DAS later and certain messages will be broadcast to all connected Talkers. The DAS will operate satisfactorily with no Talkers attached; in this case any `broadcast' messages are written to the shell from which the DAS was started.
The operation of DASM is enhanced and made more reliable if the IMP master-task is running. However, DASM can operate without the master.
The DAS is normally run with an image display to view the data as they are put onto disk. The coupling between the DAS and the display is made as weak as possible. The display sub-system reads the images from the FITS files on disk. The display can find out when new data is available by monitoring the parameter RUN in DASn. The DAS works perfectly well when the image display is not running.
Logging of observations and archiving of data on FITS tapes are done by separate facilities. At the time of writing, these programs interact with the DAS only by reading the disk-FITS files. Thus, each time a data directory is changed in the DAS, an equivalent change should be made in the logging and archiving programs.
The parameter RUN in DASn reports the progress of a run and is updated as data are acquired. You can get the current value of this parameter with the command
csh> cmd -g DASn RUN
or monitor the progress continually with
csh> cmd -p DASn START RUN &
(the latter command is a poor man's mimic-system). Here is a typical view of the parameter:
ArgStructure Struct
RUN Struct
run Uint 2030
readout Ushort 12
header Ushort 100
validFits Int 1
fileReleased Int 0
disposal Char [16] "Archive"
foxFinished Int 0
imageFinished Int 0
headerFinished Int 1
foxStatus Uint 0
imageStatus Uint 0
headerStatus Uint 0
In this snap-shot, the channel is caught part-way through an acquisition. The run-number is seen
to be 2030 and the file will be disposed of in the archive at the end of the run. The readout from the
CCDC is 12% complete and the FITS header is 100% complete. The file is already valid FITS
(because the header is complete) but is not yet released to the logging and archiving programs. The
last six entries show the state of three threads of control within DASn: one (`Fox') is reading pixels
from the image-acquisition interface; one is reconstructing and filing the image and the third was
assembling the header. The header thread has finished with good status and the other two are still
working.
Watching the parameter READY in DASn indicates when the channel is ready to receive pixels: pixels will be recorded while READY.RUN shows true and will be thrown away at all other times.
5.2 Archive, scratch and glance runs
The DAS distinguishes three classes of acquisitions. Archive runs are destined for the FITS tapes, observation log and, ultimately, the La Palma archive. Scratch runs are temporary data, recorded to allow off-line checks of the observing system. Glance runs are temporary data that survive only long enough to be examined in the image display.
All runs start off as glance runs; the DAS saves all incoming data to disk in the files DASn.fit. At the end of archive runs, DASn.fit is renamed to rn.fit where n is the run number. Only Archive runs are given run numbers. At the end of scratch runs, dasn.fit is renamed to whatever name the observer chose for the scratch file. At the end of glance runs there is no renaming; the next run over-writes the previous glance.
5.3 Checking disk-space
Monitoring disk-space is considered to be a function of the logging system and the DAS does not do it. If there is insufficient space for a run in a data directory, the run will fail and will be lost. This will become clear at the beginning of the run (probably before the shutter has a chance to open) and very little observing time need be lost.
You can check the space yourself using the command
csh> cmd -g DASn OBSDATA
to extract the name of the directory and
csh> df -k directory
to report the available space in kilobytes. Note that this report refers to the disk partition holding the directory. It is quite possible to run the DAS with eight different data-directories depleting space from one partition. In the general case it is not possible to predict the number of runs that can safely be made on a particular channel.3
`There's too little buffer-memory to read this format' - message from DASn. You have allocated too few bytes to the image-acquisition buffer for the readout format. If you get this message when starting or resetting the DAS you have probably allocated memory for the wrong type of CCD or forgotten to multiply by two bytes-per-pixel. If you get this message when changing format you have probably allocated memory only for a binned and/or windowed image. Wait for all current runs to finish and then reset the DAS with more memory on the affected channel. See section 3.
CCDC completes its readout but the run doesn't finish; RUN.HEADER is < 100%. The channel is probably waiting for a header packet to become available. Check the packet list (see section 4.5) for spurious names. If you see a name that shouldn't be there you can salvage the run by creating an empty packet:
csh> touch ${OBSDATA}/etc/packet.xyz
where packet is the packet name and xyz is the number of the current run (0 for scratch and glance runs). Then fix the packet list (see section 4.5).
CCDC completes its readout but the run doesn't finish; RUN.READOUT remains at zero. The pixels have arrived but have been ignored; your observation is lost. Check the parameter READY.RUN: if it's false the pixel-stream was sent when the DAS wasn't ready, which is some kind of fault in the client program executing the run. Otherwise, go to the computer in which the FOX interface is mounted and check for messages on its console window. If you see `Unexpected data on unmapped channel x' you can assume that someone has set the wrong headcode for the channel. See section 4.4 for the solution.
`Cannot register DAS0: task name already in use' - message while loading or resetting the DAS. (This message can refer to any DASn or to DASM.) Another program has allocated the name by which your program wants to log into the DRAMA message net: the names are exclusive. Either the DAS was already running or one of the programs failed to close down last time it was stopped. You can usually kill the rogue program with
csh> cmd program EXIT
where program is the name that you need to reuse in the message net. If this doesn't work the rogue program may be hung; get its process number from ps and kill it as you would any other Unix command. (NB: don't use kill -9 for this; the program must be given a chance to run its exit handlers or it won't log out of the message net.) If the program isn't running you can clear its mortal remains with the command cleanup. Be warned: this command will remove every DRAMA task in the observing system.
`Window n will not fit on the CCD' - message from dasformat or other format-setting command. Well, should it fit? Remember that there is a row zero and a column zero. A window set to 50:1124:100:1 on a 1124-square CCD would start at the second row and would extend one row beyond the top of the CCD. Window origins equal to one are usually mistakes.
Data in the output images do not not match the known orientation of the CCD. Check the transformation in your image format (see section 4.6). Particular care is needed with reflections in spectroscopic images. Perhaps you should take and check an arc frame at hand-over time?
`I can't open /dev/rgofoxnn: device busy' - message received while starting or resetting the DAS. This file is the Unix access-point to the image-acquisition device; the nn in the file-name is the channel number affected. These devices only allow one program to use them at a time and someone else got there first. Presumably, the interfering program isn't an instance of DASn or you'd have seen the `can't register; name already in use message' (see above) instead. Someone is running an engineering program on your channel; take a blunt instrument and go reason with them.
Appendix A. Document history
Issue 1.1 08/02/96 Original document.
2
. That is, it reads the value of the parameter FORMAT and starts with that.
3
. You're wondering `Did it do six runs or only five?'. In all this confusion the DAS can't remember. You
have to ask yourself `Do I feel lucky?'