Return to ING home page6.2  Real time arrangements in support of the unbuffered interface

This page is part of the ING document INS-DAS-31 Design notes for UltraDAS

As noted above, the S-bus interface has very little buffering for pixels of a readout. At maximum readout-rate on four readout channels, the buffer is expected to overflow in roughly 0.25ms. Once started, the readout may not be interrupted in the controller. The only way to prevent overflow and loss of the readout is for the device to continually drain the buffer into the DAS computer by executing DMA to memory on the far side of the S-bus. This is clearly a problem in hard-real-time I/O, with a moderately-challenging deadline.

Two basic techniques allow the DAS to perform hard-real-time I/O.

  1. The application udas_camera must read every pixel of the readout in one entry to the read system-call. If it tries to capture the readout in chunks, in order to allow processing of one chunk while the next is being acquired, the time taken to swap from kernel mode to user mode and back  - i.e. to get out of one call to read and back into the next one is sometimes too long for the available buffering.
  2. In Solaris, I/O in the kernel runs in the same thread and light-weight process (LWP) that invoked the I/O; the scheduling attributes of the LWP are retained. This means that the user-land thread that calls read to get the pixels must already have made all the real-time arrangements that the kernel need to keep to the correct timing. If the thread makes the call with normal, time-sharing scheduling, the readout has a small but significant chance of failing.
The second technique may seem puzzling. Is the real-time I/O not controlled from interrupts, and therefore immune to scheduling problems? In the S-bus driver, no. The lower half of the driver is interrupt-driven, true, but the upper half - the part executing in the context of the user-land thread - has to provide the interrupts with buffer space for the chain of DVMA operations. If the buffers are not ready on time, the DVMA engine stalls and the readout is lost.

To get the readout thread to work in real time requires three measures.

  1. The thread must be bound to a LWP and must be the only thread on the LWP. If the Solaris is juggling two threads on the crucial LWP, there is no guarantee that the right one will get the CPU.
  2. The LWP with the readout thread must be scheduled in the RT (real-time) class; by default, it will be in the TS (time-sharing) class.
  3. All the memory used in the readout must be locked into physical RAM to avoid long delays while the system pages.
In practice, only the main acquisition buffer (inside the Camera-i/f) object needs to be locked into physical RAM. The code for the read operation is a module in the kernel and this is permanently in RAM. The acquisition buffer is the only part of the application that the kernel needs to see during readout; it is locked into RAM using the POSIX mlock call.

Getting the LWP into the RT class is difficult to do in a portable way. It is unclear whether the programme as a whole would work if every thread was bound to a RT LWP, so the POSIX sched_xxx calls cannot be used to change the scheduling class of the whole application. Instead, the priocntl call is used; this is a Solaris-specific calls and understands how to operate on just one LWP.

To lock memory, and to send the LWP into the RT class, the thread running the readout must run as root - i.e. with a UID of zero. This means that udas_camera has to be a setuid-root programme. However, it is not appropriate to run all the threads as root: the ownership of created files would be wrong. Instead, the programme abdicates its root privileges by setting its effective UID to the real UID of the user running it. When it needs to be root for a readout, the programme sets the effective UID to the stored UID, which is the UID of zero - root - that it originally started with.