6.2
Real time arrangements in support of the unbuffered interface
This page is part of the ING document INS-DAS-31
Design
notes for UltraDAS
As noted above, the S-bus interface has very little buffering for pixels
of a readout. At maximum readout-rate on four readout channels, the buffer
is expected to overflow in roughly 0.25ms. Once started, the readout may
not be interrupted in the controller. The only way to prevent overflow
and loss of the readout is for the device to continually drain the buffer
into the DAS computer by executing DMA to memory on the far side of the
S-bus. This is clearly a problem in hard-real-time I/O, with a moderately-challenging
deadline.
Two basic techniques allow the DAS to perform hard-real-time I/O.
-
The application udas_camera must read every pixel of the readout
in one entry to the read system-call. If it tries to capture the
readout in chunks, in order to allow processing of one chunk while the
next is being acquired, the time taken to swap from kernel mode to user
mode and back - i.e. to get out of one call to read and back
into the next one is sometimes too long for the available buffering.
-
In Solaris, I/O in the kernel runs in the same thread and light-weight
process (LWP) that invoked the I/O; the scheduling attributes of the LWP
are retained. This means that the user-land thread that calls read to get
the pixels must already have made all the real-time arrangements that the
kernel need to keep to the correct timing. If the thread makes the call
with normal, time-sharing scheduling, the readout has a small but significant
chance of failing.
The second technique may seem puzzling. Is the real-time I/O not controlled
from interrupts, and therefore immune to scheduling problems? In the S-bus
driver, no. The lower half of the driver is interrupt-driven, true, but
the upper half - the part executing in the context of the user-land thread
- has to provide the interrupts with buffer space for the chain of DVMA
operations. If the buffers are not ready on time, the DVMA engine stalls
and the readout is lost.
To get the readout thread to work in real time requires three measures.
-
The thread must be bound to a LWP and must be the only thread on the LWP.
If the Solaris is juggling two threads on the crucial LWP, there is no
guarantee that the right one will get the CPU.
-
The LWP with the readout thread must be scheduled in the RT (real-time)
class; by default, it will be in the TS (time-sharing) class.
-
All the memory used in the readout must be locked into physical RAM to
avoid long delays while the system pages.
In practice, only the main acquisition buffer (inside the Camera-i/f) object
needs to be locked into physical RAM. The code for the read operation is
a module in the kernel and this is permanently in RAM. The acquisition
buffer is the only part of the application that the kernel needs to see
during readout; it is locked into RAM using the POSIX mlock call.
Getting the LWP into the RT class is difficult to do in a portable way.
It is unclear whether the programme as a whole would work if every thread
was bound to a RT LWP, so the POSIX sched_xxx calls cannot be used
to change the scheduling class of the whole application. Instead, the priocntl
call is used; this is a Solaris-specific calls and understands how to operate
on just one LWP.
To lock memory, and to send the LWP into the RT class, the thread running
the readout must run as root - i.e. with a UID of zero. This means that
udas_camera has to be a setuid-root programme. However, it is not
appropriate to run all the threads as root: the ownership of created files
would be wrong. Instead, the programme abdicates its root privileges by
setting its effective UID to the real UID of the user running it. When
it needs to be root for a readout, the programme sets the effective UID
to the stored UID, which is the UID of zero - root - that it originally
started with.