Data storage for observations

This page is part of the ING document INS-DAS-29: Operations manual for UltraDAS

Raw data of observations are first stored by UltraDAS in memory, then saved to magnetic disc as FITS files. Somewhat later, the files on disc are archived on CD-ROMs. Observers make data tapes to take away starting with the FITS files on the magnetic discs.
 

Which raw-data discs are available?

The DAS computer "owns" a set of magnetic discs on which UltraDAS records raw observations.  The discs are set up to provide the shortest-possible data-saving times.  UltraDAS will only use this prescribed set of fast discs; it cannot use an arbitrary disc elsewhere in ING.

Separate raw-data discs are used for each telescope. The discs are consistently named <tel><x> where <tel> is the telescope name, and x is a letter of the alphabet: e.g. whta, whtb, intf, etc..

The observing-log display lists all the data discs for the telescope, showing the amount of space used and the amount of space free. However, some of the discs shown may be allocated to reserve computers, or computers of alternate systems, and these will not be available to you. Ask your support astronomer or telescope operation which discs are live on your system.
 

How do I see the raw-data discs?  Where are my data?

You can see into the raw-data discs from the system computer where you control the observations, and from the data-reduction computer assigned to the telescope where you made the observations. You may also be able to see the data discs from other computers at ING. In all cases, the discs appear in the file system below the /obsdata directory. For example, UltraDAS at the INT, at the time of writing, has available two discs called intf and intg, and these are visible as the directories 
    /obsdata/intf
    /obsdata/intg
Inside each of these directories, you will find one directory for each night on which observations were written on that disc. The directory is named for the date of observation in the format yyyymmdd, e.g. 20000717 for 17th July 2000. The "night" is considered to start and end at 1200hrs UT, so that all observations from one night are stored in the same directory. The data in the directory name is the date on which the night started. For example, 
    /obsdata/intf/20000717
would store all the observations from 1200hrs on 17th July until 1159hrs on 18th July.
 

How do I select a raw-data disc?

It doesn't matter to UltraDAS which disc you use; they all tend to have the same performance. Clearly, however, it is more convenient to group all the data from one observing night, or, where possible, from an entire observing programme, on one disc.

Left to itself, UltraDAS will use the same data disc until it forced to change (due to that disc filling up or going off-line), or until you explicitly tell it to change. UltraDAS remembers which disc it was using for a particular detector from night to night, and across restarts of the control system; it will not change arbitrarily.

We recommend that review the choice of disc:

You can either pick a disc yourself, or ask UltraDAS to choose the emptiest. To let UltraDAS make the choice, give the obsdata command at the SYS> prompt thus: 
    SYS> obsdata
To pick a specific disc yourself, add the name of the disc to the obsdata command, e.g.: 
    SYS> obsdata /obsdata/intg

Can I write other files to the data discs?

When logged in to the observing account, you can write freely to the raw-data directories but probably should not do so. If you fill up the disc there is a small chance of losing an observation.
 

What happens if the disc fills up?

UltraDAS checks the free space on the disc at the start of each observation. If the disc is full, UltraDAS selects the emptiest disc from the others in its list of data-discs, and writes the observation there; the observation is not lost and no time is wasted. Subsequently, UltraDAS uses the same disc for all subsequent observations on the same camera.

If the disc is filled up during an observation by non-UltraDAS software, there is a chance that UltraDAS will not be able to save the observation and will lose it. For this reason, you should write reduced data and similar files to a scratch disc, not to the raw-data disc.
 

What happens if all the data discs fill up?

If all the raw-data discs become totally full, you will not be able to make any more observations; UltraDAS will refuse to start runs.

This should never happen. The total disc space is always much grater than the data volume that can be acquired in one night, and old observations are erased from disc between nights as necessary. If space is short, ING staff may erase observation files older than 24 hours without warning.  For this reason, you should always commit your data to tape (see below) within 24 hours of observation.

You should not yourself remove files form the raw-data discs without checking with ING staff.
 

How are the observations stored on disc?

For each numbered run, one FITS file is written. The file is named r<n>.fit, where <n> is the run number.

Inside the FITS file, there is a primary header describing the run as a whole, and one FITS image-extension per image. That is, simple, single-CCD cameras produce files containing a single image-extension, but CCD mosaics tend to produce files with one image-extension per CCD in the mosaic. If you window your detector, you will get one image-extension per readout window. When naming FITS-files to IRAF, you need to state the number of the extension as well: e.g. 

    CL> display /obsdata/intf/20000717/r1234567.fit[1]
to display the first image in run 1234567.
 

When and where are the data archived?

The observations are copied from the raw-data discs automatically by the ING data-manager, which is a computer system shared between the three ING telescopes. The data manager copies the FITS files to CD-ROM, and generates an archive catalogue in an on-line database.

In normal operation, the data-manager will copy the observation files within 24 hours of the observation. You can check the on-line archive catalogue from the web pages of the ING intranet.

In due course, the CD-ROMs from the data manager are sent to the Astronomical Data Centre at Cambridge, UK. The catalogue of observations is there made available on the WWW. There is typically a delay of several months between the observations being recorded and going on-line at Cambridge.
 
 

What data-tapes are made? Who makes them?

It is normal practice for the observer to make two tape copies of the data. One copy - the "C-tape" - is for you to take away. The other - the "D-tape" - is kept at ING as a back-up copy. (In times past, the D-tape was sent to cambridge for inclusion in the archive. The CD-ROMs from the data manager have now taken over this function.)

It is your responsibility as observer to make the tapes.