WFC Troubleshooter

Problems with subsystems
General procedure for solving issues with the instrument
Problems with the telescope control system (TCS)
- 3.1 Log out of the TCS
- 3.2 Starting up the TCS
Data reduction PC
- 4.1 Log out
- 4.2 Log in
Acknowledgements

1. Problems with subsystems

1.1 Dome

If the dome does not rotate when the telescope is preset to a target, then the DOME SHTRS button may still be pressed on the Engineering console. Depress it to set the dome back to computing mode. Alternatively, the dome might be erroneously left in local (engineering) mode. Turn the key in the dome control panel located inside the dome to set it back to remote (computing mode).

1.2 WFC shutter

If, when you start an exposure, you get the message The shutter failed to open at the start of the exposure accompanied by red/blinking/beeping, quit the alarm and repeat the last command.

Elapsed time is counted in a blue box on the WFC Graphical User Interface (GUI) while the shutter is open. After read-out has completed, the box turns to green. If the box turns orange with time elapsed = 0, then a fault has occurred. This exposure will then either be logged with a 0 second exposure time, or not quoted at all, in the observations log and in the headers. The fault is a result of a problem with the MCA controller (the electronics rack mounted on the WFC which drives the shutter and filter wheel). To recover type:

 SYS> cmd MCA RESET

With the next exposure, the exposure time should be recorded correctly again. If this does not fix the issue, bring the telescope down to access park and power cycle the MCA controller (this is the controller on the bottom of the WFC assembly [rotator at 180 deg.]), followed by:

 SYS> cmd MCA RESET

1.3 Filter wheel

On WFC start-up, it can be the case that the MCA controller has problems identifying which filters are loaded in the wheel. In this case, it may be necessary to send the command:

 SYS> cmd MCA RESET

When changing between filters in the wheel, you might get the message The filter wheel detent pin is malfunctioning accompanied by red/blinking/beeping, or Filter wheel movement timed out. If this happens, quit the alarm and try to change filters again. If this does not help, bring the telescope down to access park and power cycle the MCA controller, followed by:

 SYS> cmd MCA RESET

1.4 Data acquisition

Data acquisition term refers to the management of an exposed image, reading of data from the CCDs and saving to disk. This may fail in one of several stages, for example:

Run, Finish, Abort or Re-Time commands are not successful
Run process stops after CCD clearing
Run spontaneously aborts during readout or transfer to disk
Transfer to disk does not complete to 100%
Data on disk is corrupted (e.g. torn frames)

These may or may not be accompanied by a pop-up error box/alarm. If the error pop-up mentions the (ultra) DAS or reset the camera and you cannot continue observing, then the following procedure is recommended:

 SYS> dasreset WFC

When this completes, try a new run and see if the above fault reoccurs. If it does, then a lower-level reset of the CCD controller may be necessary. In this case, move the telescope down to access park and power cycle the unit labeled SDSU CCD controller (located to the left of the cryostat), followed by:

 SYS> dasreset WFC

Note: The above procedures deal with CCD controller problems. Procedures on how to carry out a full restart of the DAS (Data Acquisition System) can be found in section 2.

1.6 Meteorological Data System (MDS)

Occasionally the measurements from one, two, or all three of the meteorological stations will fail to reach the MDS, leaving all the windows red. If this continues for more than a few minutes the MDS can be restarted. To close it type Alt+F4 on the attached keyboard, and reopen by double clicking on the MDS icon near the middle of the screen. After a minute this should update and work as normal. If this does not not help, restart the computer with username: skyview and entering the password found on the MDS monitor.

1.6 Bad Pointing

First, check target positions and epoch are correct. Second, check focal station.

If these are OK, next repeat ZEROSET at zenith (see section 6 of the INT/WFC Operation Manual). Note that in HA there are two positions separated by only 13 arc minutes. Make sure to take the one closest to the indicating arrow. Continue with:

 USER> CAL LAST

1.6.1 Checking pointing with the autoguider

Firstly, check the pointing with a pointing grid star and the autoguider:

 USER> gocat [object name]

 USER> aper 5

Mark the autoguider window centre (SETUP/TOGGLE CENTER CROSS)

 SYS> agwin acq pos 1032 512

 SYS> autotv on

If the star comes up near the centre, the pointing is fine.

If the star does not appear near the centre of the autoguider, then repeat the ZEROSET at zenith (see section 6 of the INT/WFC Operation Manual) and check that the apertures are defined correctly (see section 1.7.4).

 USER> cal default

1.6.2 Repeat single star calibration

If pointing does not improve after re-doing the zeroset, perform another single star calibration (see section 14 of the INT/WFC Operation Manual).

1.6.3 Seven star calibration

For the ING SA supporting the first S/D night: the seven star calibration should be done always during the first WFC D-night. Please write down the resulting rms and send the value to the WFC instrument specialist.

For the observer: In case the single star calibration does not improve the pointing, then perform a manual 7 star calibration, here is the procedure:

 SYS> agwin acq pos 1032 512

 SYS> autotv on

 USER> enter aperture 0 1065 -42

(redefines aperture 0, i.e. the default pointing position, to the centre of the autoguider chip)

 USER> cal faint

(this runs the 7-star calibration script) The method to follow is exactly the same as that of a single star calibration (centering the star on the crosshairs, etc). However, instead of breaking out of the script using ctrl+z once the first star has been found, simply position it accordingly and press the HANDSET button to automatically slew the telescope to the next star. The script will then end after 7 stars have been centered.
You will then be asked to accept the result. Follow with:

 USER> next

in order to re-centre the star using the updated pointing model.

 USER> enter aperture 0 -251 326

(back to normal aperture 0, centre of chip 4)

 SYS> autotv off

1.6.4 Check focus and aperture

Check that the focus is set to the correct plane (WFC = prime focus):

 USER> show focal

Check if apertures have been defined and have been defined correctly:

 USER> show aper

ap 0 -251 326 nominal aperture, object centred on CCD#4
ap 1 704 5 centre object on CCD#1
ap 2 357 -1043 centre object on CCD#2
ap 3 -701 -10 centre object on CCD#3
ap 4 0 0 nominal aperture, object in centre of mosaic camera
(shifted from centre of CCD#4)
ap 5 1316 -368 centre object on AG-CCD (X=1032, Y=512)
ap 6 251 -326 centre object on RC (rotator centre)

If the apertures are not defined, the observing system (ICS) must be restarted. See Section 2.4 for instructions on how to do this.

After the system returns, re-issue the following commands:

 USER> show aper

 USER> show cal

 USER> cal def

and redo a single star calibration.

1.7 TCS disconnects

Check that the TCS USER window is responsive. If it is not, it is possible that the TCS TELD task has died. To restart, go to the x-terminal where the TCS user interface and info display are running and type:

 intobs@inttv> telnet lpas2

Then, log onto the computer named lpas2 (username: int_login, password can be found on the lpas2 monitor), and select the option > TELD This will re-establish contact between the observing system and the TCS. When contact is re-established, the TCS icon at the top of the WFC GUI will turn green.

1.8 TCS communication hangs

In the case of loss of communication between the TCS and the user interface, i.e. if either the TCS USER window stops responding, or the TCS DISPLAY window stops updating, clearing the communications may recover the system:

 In the blue band at top of the window, click on
 Commands --> Clear Communications

1.9 WFC disconnects

If WFC disconnects, as shown by the orange warning-label on the mimic, the DAS has most likely crashed. Check the orange telnet window that is logged onto the DAS. If it is not available, you have to start a new telnet connection from the background.

First run obssys with the option shown on the whiteboard first in order to get the DAS> prompt (there is also an option to start up a new DAS window found by right-clicking on the desktop). Press the return key to see if there is a termination message for udas_server (the WFC task is an instance of the udas_server program). If udas_server has stopped running, do the following:

Type startobssys at the DAS> prompt.
Watch the talker. Wait until you have seen WFC come on-line.
Type startobssys at the SYS> prompt.
Wait for the mimic to reconnect to WFC.

2. General procedure for solving problems with the instrument

For any issues which appear to affect the instrument, including control mechanism errors, communication errors, or observing system problems, try the following actions in order until the problem is resolved

2.1 Reset the Data Acquisition System (DAS)

This can also be important when solving issues with high read-out noise.

 SYS> dasreset WFC

2.2 Problems with the instrument control system (ICS)

This takes about 5 minutes, and will clear most communication errors.

In the pink window type:

 SYS> shutdownobssys

 SYS> exit

On the orange DAS window for WFC (triton), type:

 DAS4@triton> shutdownobssys

Are you sure you want to shut down the DAS? Y/N? > Y

 DAS4@triton> exit

Right click on the desktop, and choose exit

Once logged out, log back in again using the username and password on the ICS monitors.

Locate the orange DAS window for the WFC (triton), and type:

 intobs@triton> obssys

followed by the number of the observing station you need (4).

 Choose your station> 4

Then, to begin, type:

 DAS4@triton> startobssys

Once the DAS is running, repeat the same steps in the pink Observing System window.

 intobs@lpss7> obssys

followed by the number of the observing station you need (4).

 Choose your station> 4

Then, to begin, type:

 SYS> startobssys

Then, wait for the following windows to appear.

Talker
Observation Log
TCS Infodisplay (Check UTC and LST are updating)
Autoguider
Who goes there?
WFC GUI
WFC Autoguider Sequencer

Wait for the GUI will read in all the filters. Remember to check the read-out mode and binning.
Check the filter wheel can be rotated and if the shutter works. If one or the other does not work, type:

 SYS> cmd MCA RESET

In case your observations do not need guiding, check and eventually shutdown the wfc_ag_sequencer, according to the "Not Guiding" section of the WFC Operation Manual.

2.3 Restart of individual system controllers

If you are not 100% sure how to power cycle any system, do not attempt to do this without first contacting your support astronomer or the WHT Telescope Operator! The next stage requires the telescope to be parked and brought down to access park.

CCD, read-out, or communications

CCD SDSU controller

filter wheel or camera shutter

MCA controller

autoguiding system

AG controller

If problems persist, you should now contact the WHT Telescope Operator, and clearly explain the problem and the actions you have taken so far.

3. Restart of the Telescope Control System (TCS)

If you encounter problems with the telescope system, focussing, pointing or tracking, you may need to restart the TCS. Shut down the observing system and log out of the ICS as described above. Be sure to follow this order:

Park the telescope at Zenith using park zen command. If the TCS is lost, manually put the telescope at Zenith using engineering mode;
Log out from the ICS and then log out from the DAS;
Log out from the TCS;
Start up the TCS;
Zeroset the telescope;
Start up the DAS and then the ICS.

3.1 Log out of the TCS

To log out type:

 USER> tcsexit

Then log out of the machine by using the Fedora menu.

3.2 Starting up the TCS

Log onto the left computer (inttv) using: username: intobs, password: can be found on the TCS monitor
Use the xterm window (start a new one if there is not one already available) and establish a telnet session to the machine:
```
 USER> telnet lpas2
```
username:int_login, password: can be found on the TCS monitor
At next menu choose start from the menu options. This will bring up both the telescope status display and the USER> prompt by default.

Remember to do a ZEROSET whenever the TCS is restarted! Then you can restart the ICS following the instructions in the section above.

4. Data reduction PC

4.1 Log out

In the IRAF window, at the prompt type:

 ecl> logout

then close any remaining windows and exit the system as previously shown.

4.2 Log in

To log in type: username: intguest and the password can be found on the intdrpc monitor (below). Then click on the xterm icon to open up a xterm window. Here you should type:

 > iraf

Followed by:

 cl> wfc_ql

Check what directory you are in by typing:

 cl> pwd

The correct directory is displayed on the WFC GUI. If you are not in the correct directory, type:

 cl> cd /obsdata/inta/yyyymmdd

where yyyymmdd is your observing date, e.g., 20110321 for the night started in 21 of March 2011.

6. Acknowledgements

Credits should be given to the following ING students who contributed recently (May 2012) to improve this manual: Ilona Busenbender, Lindsay Magill, Annemieke Janssen, Amy Tyndall, Hugo Ledo and Liam Hardy and also to Andrew Cardwell for his support with past student training at the INT.