ING Banner
Home > Astronomy > Wide Field Camera > Observing Guide > Troubleshooter

WFC Troubleshooter

Contents

  1. Problems with Subsystems
  2. General Procedure for solving issues with the instrument
  3. Problems with the telescope control system (TCS)
  4. Data Reduction PC
  5. Special Observations
  6. Acknowledgements

Problems with Subsystems

Dome

If the dome does not rotate when the telescope is pre-set to a target, then the DOME SHTRS button may still be pressed on the Engineering console. Depress it to set the dome back to computing mode. Alternatively, the dome might be erroneously left in local mode. Turn the key in the dome control panel located inside the dome to set it back to remote.

WFC shutter

If, when you start an exposure, you get the message The shutter failed to open at the start of the exposure accompanied by red/blinking/beeping, quit the alarm and repeat the last command.

Elapsed time is counted in a blue box on the WFC Graphical User Interface (GUI) while the shutter is open. After read-out has completed, the box turns to green. If the box turns orange with time elapsed = 0, then a fault has occurred. This exposure will then either be logged with a 0 second exposure time, or not quoted at all, in the observations log and in the headers. The fault is a result of a problem with the MCA controller (the electronics rack mounted on the WFC which drives the shutter and filter wheel). To recover type: 
 SYS> cmd MCA RESET
With the next exposure, the exposure time should be recorded correctly again. If this does not fix the issue, bring the telescope down to access park and power cycle the MCA controller (this is the controller on the bottom of the WFC assembly [rotator at 180 deg.]), followed by: 
 SYS> cmd MCA RESET

Filter wheel

On WFC start-up, it can be the case that the MCA controller has problems identifying which filters are loaded in the wheel. In this case, it may be necessary to send the command: 
 SYS> cmd MCA RESET
When changing between filters in the wheel, you might get the message The filter wheel detent pin is malfunctioning accompanied by red/blinking/beeping, or Filter wheel movement timed out. If this happens, quit the alarm and try to change filters again. If this does not help, bring the telescope down to access park and power cycle the MCA controller, followed by: 
 SYS> cmd MCA RESET

Data acquisition

Data acquisition term refers to the management of an exposed image, reading of data from the CCDs and saving to disk. This may fail in one of several stages, for example:
  • Run, Finish, Abort or Re-Time commands are not successful
  • Run process stops after CCD clearing
  • Run spontaneously aborts during readout or transfer to disk
  • Transfer to disk does not complete to 100%
  • Data on disk is corrupted (e.g. torn frames)
These may or may not be accompanied by a pop-up error box/alarm. If the error pop-up mentions the (ultra) DAS or reset the camera and you cannot continue observing, then the following procedure is recommended: 
 SYS> dasreset WFC
When this completes, try a new run and see if the above fault reoccurs. If it does, then a lower-level reset of the CCD controller may be necessary. In this case, move the telescope down to access park and power cycle the unit labeled SDSU CCD controller (located to the left of the cryostat), followed by: 
 SYS> dasreset WFC
Note: The above procedures deal with CCD controller problems. Procedures on how to carry out a full restart of the DAS (Data Acquisition System) can be found in section 2.

Autoguider

If the guide on or guide off commands do not return, press Ctrl+c to kill the command. If this has no effect, type: 
 SYS> fixGuideTaskProblem
(NB: This is a case sensitive command!)

Meteorological Data System (MDS)

Occasionally the measurements from one, two, or all three of the meteorological stations will fail to reach the MDS, leaving all the windows red. If this continues for more than a few minutes the MDS can be restarted. To close it type Alt+F4 on the attached keyboard, and reopen by double clicking on the MDS icon near the middle of the screen. After a minute this should update and work as normal. If this does not not help, restart the computer with username: skyview and entering the password found on the MDS monitor.

Bad Pointing

First repeat ZEROSET at zenith (see section 6 of the INT/WFC Operation Manual). Note that in HA there are two positions separated by only 13 arc minutes. Make sure to take the one closest to the indicating arrow. Continue with: 
 USER> CAL LAST

Check pointing with Autoguider

Firstly, check the pointing with a pointing grid star and the autoguider: 
 USER> gocat [object name]
 USER> aper 5
Mark the autoguider window centre (SETUP/TOGGLE CENTER CROSS) 
 SYS> agwin acq pos 1032 512
 SYS> autotv on
If the star comes up near the centre, the pointing is fine.

If the star does not appear near the centre of the autoguider, then repeat the ZEROSET at zenith (see section 6 of the INT/WFC Operation Manual) and check that the apertures are defined correctly (see section 1.7.4). 
 USER> cal default

Repeat single star calibration

If pointing does not improve after re-doing the zeroset, perform another single star calibration (see section 14 of the INT/WFC Operation Manual).

Seven star calibration

If the single star calibration does not improve the pointing, perform a manual 7 star calibration: 
 SYS> agwin acq pos 1032 512
 SYS> autotv on
 USER> enter aperture 0 1065 -42
(redefines aperture 0, i.e. the default pointing position, to the centre of the autoguider chip) 
 USER> cal faint
(this runs the 7-star calibration script) The method to follow is exactly the same as that of a single star calibration (centering the star on the crosshairs, etc). However, instead of breaking out of the script using ctrl+z once the first star has been found, simply position it accordingly and press the HANDSET button to automatically slew the telescope to the next star. The script will then end after 7 stars have been centered.
You will then be asked to accept the result. Follow with: 
 USER> next
in order to re-centre the star using the updated pointing model. 
 USER> enter aperture 0 -251 326
(back to normal aperture 0, centre of chip 4) 
 SYS> autotv off

Check focus and aperture

Check that the focus is set to the correct plane (WFC = prime focus): 
 USER> show focal
Check if apertures have been defined and have been defined correctly: 
 USER> show aper
ap 0 -251 326 nominal aperture, object centred on CCD#4
ap 1 704 5 centre object on CCD#1
ap 2 357 -1043 centre object on CCD#2
ap 3 -701 -10 centre object on CCD#3
ap 4 0 0 nominal aperture, object centred on CCD#4
ap 5 1316 -368 centre object on AG-CCD (X=1032, Y=512)
ap 6 251 -326 centre object on RC (rotator centre)

If the apertures are not defined, the observing system (ICS) must be restarted. See Section 2.4 for instructions on how to do this.

After the system returns, re-issue the following commands: 
 USER> show aper
 USER> show cal
 USER> cal def
and redo a single star calibration.

TCS disconnects

Check that the TCS USER window is responsive. If it is not, it is possible that the TCS TELD task has died. To restart, go to the x-terminal where the TCS user interface and info display are running and type: 
 intobs@inttv> telnet lpas2
Then, log onto the computer named lpas2 (username: int_login, password can be found on the lpas2 monitor), and select the option > TELD This will re-establish contact between the observing system and the TCS. When contact is re-established, the TCS icon at the top of the WFC GUI will turn green.

TCS communication hangs

In the case of loss of communication between the TCS and the user interface, i.e. if either the TCS USER window stops responding, or the TCS DISPLAY window stops updating, clearing the communications may recover the system:   
 In the blue band at top of the window, click on
 Commands --> Clear Communications

WFC disconnects

If WFC disconnects, as shown by the orange warning-label on the mimic, the DAS has most likely crashed. Check the orange telnet window that is logged onto the DAS. If it is not available, you have to start a new telnet connection from the background.

First run obssys with the option shown on the whiteboard first in order to get the DAS> prompt (there is also an option to start up a new DAS window found by right-clicking on the desktop). Press the return key to see if there is a termination message for udas_server (the WFC task is an instance of the udas_server program). If udas_server has stopped running, do the following:
  1. Type startobssys at the DAS> prompt.
  2. Watch the talker. Wait until you have seen WFC come on-line.
  3. Type startobssys at the SYS> prompt.
  4. Wait for the mimic to reconnect to WFC.

General Procedure for solving problems with the instrument

For any issues which appear to affect the instrument, including control mechanism errors, communication errors, or observing system problems, try the following actions in order until the problem is resolved

Reset the Data Acquisition System (DAS)

This can also be important when solving issues with high read-out noise. 
 SYS> dasreset WFC

Restart of the observing control system (ICS)

This takes about 5 minutes, and will clear most communication errors.
  • In the pink window type: 
     SYS> shutdownobssys
     SYS> exit
  • On the orange DAS window for WFC (triton), type: 
     DAS> shutdownobssys
    Are you sure you want to shut down the DAS? Y/N? > Y 
     SYS> exit
  • Right click on the desktop, and choose exit
  • Once logged out, log back in again using the username and password on the ICS monitors.
  • Locate the orange DAS window for the WFC (triton), and type: 
     SYS> obssys
    followed by the number of the observing station you need (4). 
     Choose your station> 4
    Then, to begin, type: 
     SYS> startobssys
  • Once the DAS is running, repeat the same steps in the pink Observing System window. 
     SYS> obssys
    followed by the number of the observing station you need (4). 
     Choose your station> 4
    Then, to begin, type: 
     SYS> startobssys
  • Then, wait for the following windows to appear.
    1. Talker
    2. Observation Log
    3. TCS Infodisplay (Check UTC and LST are updating)
    4. Autoguider
    5. Who goes there?
    6. WFC GUI
    7. WFC Autoguider Sequencer
    Wait for the GUI will read in all the filters. Remember to check the read-out mode and binning.
    Check the filter wheel can be rotated and if the shutter works. If one or the other does not work, type: 
     SYS> cmd MCA RESET

    Restart of individual system controllers

    If you are not 100% sure how to power cycle any system, do not attempt to do this without first contacting your support astronomer or the WHT Telescope Operator. The next stage requires the telescope to be parked and brought down to access park.

      If you have problems with the CCD, read-out, or communications, you may need to power cycle the CCD SDSU controller, which is located to the left of the cryostat, as seen in this photo. Turn off the switch at the back of the power supply, wait for one minute, then turn back on. After this you may need to complete a DAS reset or ICS restart as described above.
      If you have problems with the filter wheel or camera shutter, you may need to power cycle the MCA controller. This is located below the cryostat, as seen in this photo. Turn off the controller, wait for one minute, then restart. After this you may need to complete a DAS reset or ICS restart as described above.
      If you have problems with the autoguiding system, you may need to power cycle the AG controller. This is located to the right of the cryostat, as seen in this photo. Turn off the controller, wait for one minute, then restart. After this you may need to complete an ICS restart as described above.
    If problems persist, you should now contact the WHT Telescope Operator, and clearly explain the problem and the actions you have taken so far.

    Restart of the Telescope Control System (TCS)

    If you encounter problems with the telescope system, focussing, pointing or tracking, you may need to restart the TCS. First, park the telescope at zenith, as you will have to complete a zeroset.
    Shut down the observing system and log out of the ICS as described above. Be sure to follow this order:
    1. Log out from the ICS and then the DAS
    2. Log out from the TCS
    3. Start up the TCS
    4. Start up the DAS and then the ICS

    Log out of the TCS

    To log out type: 
     USER> tcsexit
    Then log out of the machine with a right click and then choose exit from the options.

    Starting up the TCS

    1. Log onto the left computer (inttv) using: username: intobs, password: can be found on the TCS monitor
    2. Use the xterm window (start a new one if there is not one already available) and establish a telnet session to the machine: 
       USER> telnet lpas2
      username:int_login, password: can be found on the TCS monitor
    3. At next menu choose start from the menu options. This will bring up both the telescope status display and the USER> prompt by default.
    Remember to do a ZEROSET whenever the TCS is restarted! Then you can restart the ICS following the instructions in the section above.


    Data Reduction PC

    Log out

    In the IRAF window, at the prompt type: 
     ecl> logout
    then close any remaining windows and exit the system as previously shown.

    Log in

    To log in type: username: intguest and the password can be found on the intdrpc monitor (below). Then click on the xterm icon to open up a xterm window. Here you should type: 
     > iraf
    Followed by: 
     cl> wfc_ql
    Check what directory you are in by typing: 
     cl> pwd
    The correct directory is displayed on the WFC GUI. If you are not in the correct directory, type: 
     cl> cd /obsdata/inta/yyyymmdd
    where yyyymmdd is your observing date, e.g., 20110321 for the night sterted in 21 of March 2011.


    Special Observations

    Observations below 33 degrees

    If you wish to observe at a zenith distance between 57 deg and 70 deg (33 deg - 20 deg elevation), then you need to raise the lower shutter (see section 12 of the INT/WFC Operation Manual).

    There will not be any warning given from the TCS if you are observing the inside of the dome, so you will have to check the zenith distance (ZD) on the TCS monitor from time to time.

    Please consider if opening the lower shutter is essential. Opening up takes around 10 minutes and the image quality will decrease due to observing through higher air masses.

    Remember to bring the lower shutter down afterwards, as you will not be able to observe above 45 degrees elevation if it is left in the raised position.

    Observations of moving targets

    If you wish to observe a moving target, then you need to compute its motion rates in the equatorial coordinate system. The TCS requires the differential tracking rates to be expressed as:
      da = d(alpha)/dt in SECONDS/sec
      dd = d(delta)/dt in ARCSECONDS/sec
    Often (e.g. JPL), ephemerides are given as
      d(alpha)/dt x cos(delta) in ARCSECONDS/hour (1)
      d(delta)/dt in ARCSECONDS/hour (2)
    Thus, to transform these for TCS input, compute
      da = (1) / (3600 x 15 x cos(delta))
      dd = (2) / 3600
    The differential tracking rates are then input to the TCS using the following sequence of commands: 
     USER> gocat {object name}
     USER> diff_rates da dd
     USER> next
    To return to sidereal tracking (e.g. for standards) issue the TCS command: 
     USER> diff_rates 0 0

    Acknowledgements

    Credits should be given to the following ING students who contributed recently (May 2012) to improve this manual: Ilona Busenbender, Lindsay Magill, Annemieke Janssen, Amy Tyndall, Hugo Ledo and Liam Hardy and also to Andrew Cardwell for his support with past student training at the INT.


    • Top | Back

    WFC Instrument Specialist
    Last modified: 10 April 2012