Troubleshooting

Known problems

1. 'WARNING: Header collection failed' message appears in the system window when performing any form of run.
This indicates that UltraDAS cannot connect to the Header Collection Task (HCT), the TCS, or one of the associated FITS header packet colletor tasks. The HCT is restarted by issuing the startobssys command in the system window.

2. The camera mimic disappears or stops updating and all runs fail with the message 'The camera could not be contacted - please ensure that the camera is running on the DAS machine' .
This indicates that the system computer has lost its connection to the DAS camera server. This is due either to a communication failure to the DAS computer, or the camera server running on the DAS computer has crashed. If this is the case, restart the DAS as shown below in the section below "Stop and restart the DAS".
 
 

Resetting the DAS

Occasionally, the data-acquisition sub-system in UltraDAS (i.e. the software on the DAS computers) misbehaves and needs to be reset. Sometimes it's an internal problem, sometimes the trouble starts in the the detector controller, which the DAS software talks to. These notes describe a set of procedures for recovering the fault with minimal disruption to the rest of the observing system.

What not to do

It is usually a bad idea to stop and restart the whole observing system, at least until you have tried the other procedures listed below. Why is this bad? Firstly, because it takes much longer than the procedures below (it is a superset of those procedures). Secondly, because it increases the chance of hitting other problems elsewhere in the system. In particular, the shutter and filter controls on the INT WFC will be disrupted by a full restart and these do not always restart properly on the first try.

If you are using the INT WFC and you get problems with the shutter or filter wheel (or with the task MCA that manages them), please note that this is nothing to do with the DAS. The shutter of the INT WFC is controlled by the ICS and resetting the DAS won't solve any ICS problems.

What to do

There are four procedures; try them in order until you get satisfaction.
  1. Use the dasreset command.  This is quick - it takes less than a minute - and non-disruptive.  No need to call the duty engineer if this works.
  2. Stop and restart the DAS, but not the other sub-systems. This takes longer still and disrupts software on other computers.
  3. Power-cycle the SDSU detector controller, then use dasreset. This takes longer because you need to slew the telescope to get at the controller.
  4. If all else fails, but not before,  stop and restart all Unix sub-systems.
  • Always remember to restore your binning, readout speed and readout window settings after resetting the DAS.
  • Detailed procedures

    dasreset alone

    1. Type dasreset <camera-name> at the SYS> prompt: e.g. dasreset WFC or dasreset RED. Case is significant: the command must be in lower case and the name of the camera in upper case.
    2. dasreset returns the prompt as soon as the command is accepted. That is, it does not wait for the command to take effect.
    3. Watch the talker: you should see the camera first announce that it is going off-line and later announce that it is back on-line.
    4. If the camera fails to come back on-line, then this procedure hasn't worked, so go on to the next procedure.
    5. If the camera comes on-line you are ready to go again except that you will need first to reselect your binning, readout speed and readout windows.

    Stop and restart the DAS

    1. Log on to the DAS computer. An orange login window should be available on the system computer. If more than one orange window is visible (ie. with ISIS at the WHT), check the CCD whiteboard to determine the relevant DAS computer for the camera needing to be restarted.
    2. Run the obssys command on the DAS computer.
    3. Give the command shutdownobssys to the DAS computer, at the DAS> prompt.. (Be careful not to give the command at the SYS> prompt by mistake: the idea here is not to shut down the ICS.)
    4. Wait for the DAS software to die.
    5. Give the command startobssys to the DAS computer, at the DAS> prompt (see notes below).
    6. Wait for the camera to come back on-line: it will anounce this in the talker and in the DAS window.
    7. Wait for the camera mimics to re-connect.
    8. If you have no outstanding errors, you are ready to observe. Otherwise go on to the next procedure.

    Controller reset followed by dasreset

    1. Slew the telescope so that you can reach the SDSU detector-controller: see the notes below. If you don't know for certain what or where this device is, call the duty engineer now.
    2. Power-cycle the SDSU controller.
    3. Do the dasreset thing from the previous procedure.
    4. If dasreset works, you are ready to observe. If not, go on to the next procedure.

    Stop and restart both the DAS and the software on the system computer

    1. At the SYS> prompt on the system computer, give the command shutdownobssys.
    2. Wait for the system-computer software to go away.
    3. At the DAS> prompt on the DAS computer(s), give the command shutdownobssys.
    4. Wait for the DAS software to go away.
    5. Log off the DAS computer(s).
    6. Log off the system computer.
    7. Slew the telescope to get at the SDSU detector-controller (see notes below).
    8. Power-cycle the SDSU detector controller (see notes below).
    9. Log on to the system computer as you would at the start of the night.
    10. From the system computer log on to the DAS computer. (The system should automatically log on your behalf. Look for orange terminal windows with the DAS> prompt.
    11. Run the obssys command on the DAS computer(s) at the DAS> prompt.
    12. Run the startobssys command on the DAS computer(s) at the DAS> prompt.
    13. Run the obssys command on the system computer at the SYS> prompt.
    14. Run the startobssys command on the system computer at the SYS> prompt.
    15. You should now be ready to observe.

    Notes

    These instructions apply to observing-systems at the INT and at the WHT, those being the systems currently using UltraDAS components. The procedures do not apply to other DAS variants.

    The telescope position for access to the SDSU detector-controller varies from instrument to instrument.

    The detector controller is in two parts: An aluminum box for the main electronics and a grey box for the power-supply. The on-off switch is on the power-supply box.

    The system typically gives you an terminal-window to the DAS computer when you log on to the system computer. This window may go away if not used (Unix logs out the terminal session), but this does not mean that the DAS has crashed. The DAS doesn't need the terminal window to be there. Don't reset just for this.

    The obssys command selects and enables certain Unix commands that you need to work on the system; if you miss out obssys, the computer will not know about commands such as startobssys and shutdownobssys.Obssys is an idempotent command: it doesn't matter if you do it twice in the same terminal session.

    When startobssys completes successfully on the DAS computer, the DAS is on-line, but is not yet connected to the system computer because the latter has to make the connection. At this stage, observing commands like run will not work. When you then do startobssys on the system computer, you make the connections to the DAS computer that allow the observing commands to work properly.

    At the WHT, startobssys at the system computer also starts the Header Collection Task (HCT) on the system computer.

    There is no reason to stop and restart either the TCS or the autoguider as part of these procedures.


    Guy Rixon (gtr@ast.cam.ac.uk) 2000-05-02, Richard Bassom (rab@ing.iac.es) 2011-05-11