Occassionally, the data-acquisition sub-system in UltraDAS (i.e. the
software on the DAS computer)misbehaves and needs to be reset. Sometimes
it's an internal problem, sometimes the trouble starts in the the detector
controller, which the DAS software talks to. These notes describe a set of
procedures for recovering the fault with minimal disruption to the rest
of the observing system.
Resetting the DAS
What not to do
It is usually a bad idea to stop and restart the whole observing system,
at least until you have tried the other procedures listed below. Why is
this bad? Firstly, because it takes much longer than the procedures below
(it is a superset of those procedures). Secondly, because it increases
the chance of hitting other problems elsewhere in the system. In particular,
the shutter and filter controls on the INT WFC will be disrupted by a full
restart and these do not always restart properly on the first try.
If you are using the INT WFC and you get problems with the shutter or
filter wheel (or with the task MCA that manages them), please note that
this is nothing to do with the DAS. The shutter of the INT WFC is controlled
by the ICS and resetting the DAS won't solve any ICS problems.
What to do
There are four procedures; try them in order until you get satisfaction.
Use the dasreset command. This is quick - it takes less than
a minute - and non-disruptive. No need to call the duty engineer
if this works.
Power-cycle the SDSU detector controller, then use dasreset. This
takes longer because you need to slew the telescope to get at the controller.
Stop and restart the DAS, but not the other sub-systems.
This takes longer still and disrupts software on other computers.
This may be a good time to call the duty engineer.
If all else fails, but not before, stop and restart all Unix sub-systems.
Type dasreset <camera-name> at the SYS> prompt: e.g.
dasreset WFC or dasreset WHTWFC. Case is significant: the command
must be in lower case and the name of the camera in upper case.
dasreset returns the prompt as soon as the command is accepted.
That is, it does not wait for the command to take effect.
Watch the talker: you should see the camera first announce that its
going off-line and later announce that it is back on-line.
If the camera fails to come back on-line, then this procedure hasn't worked,
so go on to the next procedure.
If the camera comes on-line you are ready to go again except that
you will need first to reselect your binning, readout speed and readout
Controller reset followed by dasreset
Slew the telescope so that you can reach the SDSU detector-controller:
see the notes below. If you don't know for certain what or where
this device is, call the duty engineer now.
Power-cycle the SDSU controller.
Do the dasreset thing from the previous procedure.
If dasreset works, you are ready to observe. If not, go on to the
Stop and restart the DAS
Log on to the DAS computer.
Run the obssys command on the DAS computer.
Give the command shutdownobssys to the DAS computer, at the DAS>
prompt.. (Be careful not to give the command at the SYS> prompt
by mistake: the idea here is not to shut down the ICS.)
Wait for the DAS software to die.
Give the command startobssys to the DAS computer, at the DAS>
prompt (see notes below).
Wait for the camera to come back on-line: it will anounce this in
Connect to the DAS by giving the command startobssys at the SYS>
prompt (see notes below).
Wait for the mimics to appear and connect.
If you have no outstanding errors, you are ready to observe. Otherwise,
despair and go on to the next procedure.
Stop and restart both the DAS and the software on the system computer
At the SYS> prompt on the system computer, give the command shutdownobssys.
Wait for the system-computer software to go away.
At the DAS> prompt on the DAS computer, give the command shutdownobssys.
Wait for the DAS software to go away.
Log off the DAS computer.
Log off the system computer.
Slew the telescope to get at the SDSU detector-controller (see notes below).
Power-cycle the SDSU detector controller (see notes below).
Log on to the system computer as you would at the start of the night.
From the system computer log on to the DAS computer. (The system may log
in automatically on your behalf. Look for a terminal window with the DAS>
Run the obssys command on the DAS computer at the DAS> prompt.
Run the startobssys command on the DAS computer at the DAS> prompt.
Run the obssys command on the system computer at the SYS> prompt.
Run the startobssys command on the system computer at the SYS> prompt.
You should now be ready to observe.
These instructions apply to observing-system s8.2 at the INT and observing-system
s8. 3 at the WHT, those being the two systems currently using UltraDAS
components. The procedures do not apply to other DAS variants. The
restart procedures are likely to change radically with the forthcoming
systems s9. Hopefully, this will simplify matters.
The telescope position for access to the SDSU detector-controller varies
from instrument to instrument.
The detector controller is in two parts: An aluminum box for the main
electronics and a grey box for the power-supply. The on-off switch is on
the power-supply box.
For INT prime, slew to access park.
For WHT prime, slew to access park.
For IDS, slew to zenith.
For UES, don't slew: the controller is on the Nasmyth platform and is accessible
from all positions of the telescope.
The system typically gives you an terminal-window to the DAS computer
when you log on to the system computer. This window may go away if not used
(Unix logs out the terminal session), but this does not mean that the
DAS has crashed. The DAS doesn't need the terminal window to be there.
Don't reset just for this.
The obssys command selects and enables certain Unix commands
that you need to work on the system; if you miss out obssys, the
computer will not know about commands such as startobssys and
shutdownobssys.Obssys is an idempotent command: it doesn't
matter if you do it twice in the same terminal session.
When startobssys completes successfully on the DAS computer,
the DAS is on-line, but is not yet connected to the system computer because
the latter has to make the connection. At this stage, observing commands
like run will not work. When you then do startobssys on the
system computer, you make the connections to the DAS computer that allow
the observing commands to work properly.
There is no reason that I can see to stop and restart either the TCS or
the autoguider as part of these procedures.
Guy Rixon (firstname.lastname@example.org) 2000-05-02.