6.
Notes on design choices6.
Notes on design choicesThe fibre link from an SDSU detector-controller to the DAS runs at 40 Mbits/s, and each pixel, with framing bits, is sent as 18 bits. That is, the fibre link can transmit a pixel every 450 ns.
Each controller can drive from one to eight channels (i.e. separately-clocked areas). In general, mosaics of EEV42 CCDs should have a detector controller for each four channels. A group of four EEV42 transmits a pixel every 550 ns which is a good match to the fibre capacity.
Each controller needs an interface in the PCI bus of the DAS computer. Where one camera has several controllers, the controllers come under control of one server programme and must be interfaced to only one computer. That means that on an Ultra 10, which has two PCIbus slots for camera interfaces, up to 8 channels can be served without derating the CCDs, and up to 16 if the CCDs are runs at half their best speed. In the future, PCI-interface boards may be available that drive two controllers each.
Currently, the largest cameras at ING (INT WFC, INGRID) each have four channels. Two sizes of larger mosaic have been proposed: a 12Kpixel-square camera (18 EEV42) and a 16 Kpixel-square camera (32 EEV42). The WFC and INGRID will be equiped as single detector-groups of four channels on one controller. The 12K-square camera could be built with three controllers and six channels per controller: this would degrade the throughput. Alternatively, it could be made with six controllers, three channels per controller and two controllers per PCI interface and would then run at full speed. The 16K-square mosaics cannot be built on an Ultra 10 at all until the dual interface-boards are available. Then, the big mosaic has three interfaces, six controllers and (up to) five channels per controller; it degrades the CCD performance slightly.
The maximum pixel-output of one controller is 4.5 MB/s. The capacity of PCI bus is 132 MB/s. Each pixel crosses the bus twice, one coming from the camera and once going to the disk server (or to the local disks; Ultra 10 computers have their SCSI interface on the PCIbus). Hence, the theoretical capacity of the DAS computer is 14 controllers reading out in parallel and planned cameras use no more than half this capacity.
Writing to disk could be a bottleneck. Current disks of good quality can write at between 9 and 20 MBs; this the speed at which data can be put onto the media, and the interface between the computer and the disk may be slower. In tests, a Fujitsu MAB3045 with an UltraSCSI interface wrote consistently at 12.6 MB/s. The storage-area network associated with UltraDAS is planned to use a fibre-channel arbitrated-loop interface which runs at up to 100 MB/s. The intended disks have published throughputs of 10 to 17 MB/s.
UltraDAS is required to run successive observations of negligable exposure time at no more than 30-second intervals. If five seconds of the cycle are reserved for clearing CCDs, working the shutter, collecting the FITS header and other overheads, then the data saving in each cycle can take up to 25 seconds. For the canonical EEV42, the readout time is 19.5 seconds.
Three arrangements for data-saving were considered:
Options 2 and 3 each require a write speed of 22 MB/s for a 32-channel mosaic on a 30-second cycle. Option 3 requires this for any observation, but option 2 could use a lower speed if the cycle time was allowed to extend for very short exposures. In each case, if the speed was raised to 28 MB/s then the data saving would take no longer than the readout. Option 2 requires twice as much RAM in the DAS computer.
Option 2 was chosen for UltraDAS when it became clear that the fast disc-server would not be available when UltraDAS was first released and might never be available. The cost in porgramming complexity was extreme, but there was no other way to recover the performance.
To support UltraDAS, the disc server needs to accept data at the following speeds:
If it were a stand-alone system, UltraDAS could be built as a set of monolithic control-programmes on the dedicated DAS computer. Each programme might control one camera (or perhaps one programme would control all cameras) and the programmes would include a specialist command-line and GUI. However, the DAS has to co-operate with the ICS and TCS. All three aspects of the observing system must be controllable from a single point.
The TCS, ICS and the preceeding DAS share a client-server architecture.
UltraDAS uses the same architecture for compatibility.
The old, Sbus SPARCstations were considered and discarded. Sun is ceasing production of these machines and they will become untenable within the planned lifetime of UltraDAS.
The VME solution was proposed in RGO's study before the development of the PCI interface for SDSU controllers was announced. A sophisticated hardware-architecture was conceived that could handle the required dataflows in a reliable manner. This solution had low technical risk when applied to CCD cameras (the deterministic nature of VxWorks allows the I/O performance to be guaranteed early in the design) but is hard to extend to IR and esoteric cameras: there is no cheap way to put sufficient CPU power in the VME computer to do the necessary preprocessing. The overall cost of the solution was thought to be relatively high.
The solution based on an Ultra computer was first proposed as a cheap alternative to the VME system. The cost of the computer hardware per telescope is much lower (but those hardware costs are small compared with the cost of the detector controllers and interfaces). The development cost on VME and Ultra were initially estimated to be similar, but the Ultra-based system was thought to allow large-scale reuse of software from the Data-Cell DAS, making it much cheaper. (Later analysis of the new requirements showed that this reuse was not possible.) The Ultra platform was thus chosen on grounds of cost.
The Sun Ultra 10 was chosen as the cheapest UltraSPARC with a suitable PCI bus. To satisfy the user requirements, the DAS computer must have 3 PCI slots for camera interfaces and one for the disk interface. The Ultra 10 has the necessary four slots; the Ultra 5, which is cheaper, has only three slots.
An Ultra-10 clone from Hamcom was also considered. That machine has six PCI slots (of which one might need to be given up to a 100baseT interface) and does not sacrifice a slot for a SCSI interface. Currently, Sun's original machine is cheaper and meets the requirements well enough. It may be worth looking again at the clone if a larger number of peripherals are needed.
In general, Ultra 10 computers are sufficient for current cameras and future mosaics of up to eight channels. If larger mosaics are to be built then the builders should supply Solaris-compatible DAS computers with enough bus slots to run the camera at full speed.
The PC platform was not seriously considered. ING has no experience
with Solaris on Intel processors and has no wish to introduce such systems.
There is no EPICS support for the SDSU hardware and EPICS definitely cannot run on the processors of the detector-controllers themselves. It would be hard, or even impossible, to build an EPICS database on the Ultra computer; EPICS databases usually run on VME computers. Once the VME-based DAS was rejected, EPICS was not considered any further.
Thus, DRAMA communications were required by default. In fact, the use
of DRAMA was implied by the plans to reuse code from the Data-Cell DAS.
The old Perkin-Elmer DAS on the INT did not support multi-channel cameras. The WHT DAS does not support them either.
The Data-Cell DAS supports only one multi-channel camera: the INT wide-field camera. For this camera, it puts each of the four channels into a spearate FITS file. This approach makes the DAS' design simpler but complicates logging and archiving of observations. The seperation into four files actually makes CCD data easier to reduce and for this reason the four-files arrangement was made a requirement by the project scientist for the wide-field camera. The plans to reuse code from the Data-cell DAS assumed the "one-file-per-channel" convention.
There is a default assumption that multiple channels are treated in the same way whether they come from separate amplifiers on one chip (e.g. INGRID) or from separate chips (e.g. WFC).
The teams building INGRID and LIRIS stated at an UltraDAS requirements-review (in October 1998) that they could not work with groups of separate files for each observation. At this point, plans to use code and conventions from the Data-Cell DAS were dropped and it was provisionally agreed that the pixels from each channel be assmbled into one raster in one FITS file per observation.
After the meeting, weaknesses in the new solution were pointed out: