ING Banner
Home > Astronomy > Wide Field Camera > Observing Guide > Troubleshooter

WFC Troubleshooter

Contents

  1. Problems with subsystems
  2. Problems with the instrument control system (ICS)
  3. Problems with the telescope control system (TCS)
  4. Data reduction PC
  5. Reporting/Searching faults at the ING Fault Management System
  6. INT usual alarms
  7. Useful ING web addresses
  8. Acknowledgements

1. Problems with subsystems

1.1 Dome

If the dome does not rotate when the telescope is preset to a target, then the DOME SHTRS button may still be pressed on the Engineering console. Depress it to set the dome back to computing mode. Alternatively, the dome might be erroneously left in local (engineering) mode. Turn the key in the dome control panel located inside the dome to set it back to remote (computing mode).

1.2 WFC shutter

If, when you start an exposure, you get the message The shutter failed to open at the start of the exposure accompanied by red/blinking/beeping, quit the alarm and repeat the last command.

Elapsed time is counted in a blue box on the WFC Graphical User Interface (GUI) while the shutter is open. After read-out has completed, the box turns to green. If the box turns orange with time elapsed = 0, then a fault has occurred. This exposure will then either be logged with a 0 second exposure time, or not quoted at all, in the observations log and in the headers. The fault is a result of a problem with the MCA controller (the electronics rack mounted on the WFC which drives the shutter and filter wheel). To recover type:
 SYS> cmd MCA RESET
With the next exposure, the exposure time should be recorded correctly again. If this does not fix the issue, bring the telescope down to access park and power cycle the MCA controller (this is the controller on the bottom of the WFC assembly [rotator at 180 deg.]), followed by:
 SYS> cmd MCA RESET 

1.3 Filter wheel

On WFC start-up, it can be the case that the MCA controller has problems identifying which filters are loaded in the wheel. In this case, it may be necessary to send the command:
 SYS> cmd MCA RESET
When changing between filters in the wheel, you might get the message The filter wheel detent pin is malfunctioning accompanied by red/blinking/beeping, or Filter wheel movement timed out. If this happens, quit the alarm and try to change filters again. If this does not help, bring the telescope down to access park and power cycle the MCA controller, followed by:
 SYS> cmd MCA RESET

1.4 Data acquisition

Data acquisition term refers to the management of an exposed image, reading of data from the CCDs and saving to disk. This may fail in one of several stages, for example:
  • Run, Finish, Abort or Re-Time commands are not successful
  • Run process stops after CCD clearing
  • Run spontaneously aborts during readout or transfer to disk
  • Transfer to disk does not complete to 100%
  • Data on disk is corrupted (e.g. torn frames)
These may or may not be accompanied by a pop-up error box/alarm. If the error pop-up mentions the (ultra) DAS or reset the camera and you cannot continue observing, then the following procedure is recommended:
 SYS> dasreset WFC
When this completes, try a new run and see if the above fault reoccurs. If it does, then a lower-level reset of the CCD controller may be necessary. In this case, move the telescope down to access park and power cycle the unit labeled SDSU CCD controller (located to the left of the cryostat), followed by:
 SYS> dasreset WFC
Note: The above procedures deal with CCD controller problems. Procedures on how to carry out a full restart of the DAS (Data Acquisition System) can be found in section 2.

1.5 Meteorological Data System (MDS)

Occasionally the measurements from one, two, or all three of the meteorological stations will fail to reach the MDS, leaving all the windows red. If this continues for more than a few minutes the MDS can be restarted. To close it type Alt+F4 on the attached keyboard, and reopen by double clicking on the MDS icon near the middle of the screen. After a minute this should update and work as normal. If this does not not help, restart the computer with username: skyview and entering the password found on the MDS monitor.

1.6 Bad Pointing

First, check target positions and epoch are correct. Second, check focal station.

If these are OK, next repeat ZEROSET at zenith (see section 6 of the INT/WFC Operation Manual). Note that in HA there are two positions separated by only 13 arc minutes. Make sure to take the one closest to the indicating arrow. Continue with:
 USER> cal last

1.6.1 Checking pointing with the autoguider

Firstly, check the pointing with a pointing grid star and the autoguider:
 USER> gocat [object name]
 USER> aper 5
Mark the autoguider window centre (SETUP/TOGGLE CENTER CROSS)
 SYS> agwin acq pos 1032 512
 SYS> autotv on
If the star comes up near the centre, the pointing is fine.

If the star does not appear near the centre of the autoguider, then repeat the ZEROSET at zenith (see section 6 of the INT/WFC Operation Manual) and check that the apertures are defined correctly (see section 1.7.4).
 USER> cal default

1.6.2 Repeat single star calibration

If pointing does not improve after re-doing the zeroset, perform another single star calibration (see section 14 of the INT/WFC Operation Manual).

1.6.3 Seven star calibration

For the ING SA supporting the first S/D night: the seven star calibration should be done always during the first WFC D-night. Please write down the resulting rms and send the value to the WFC instrument specialist.

For the observer: In case the single star calibration does not improve the pointing, then perform a manual 7 star calibration, here is the procedure:
 SYS> agwin acq pos 1032 512
 SYS> autotv on
 USER> enter aperture 0 1065 -42
(redefines aperture 0, i.e. the default pointing position, to the centre of the autoguider chip)
 USER> cal faint
(this runs the 7-star calibration script) The method to follow is exactly the same as that of a single star calibration (centering the star on the crosshairs, etc). However, instead of breaking out of the script using ctrl+z once the first star has been found, simply position it accordingly and press the HANDSET button to automatically slew the telescope to the next star. In case one star does not show up in the field (possible closer to the North pole where the pointing could be worse), then it could be ignored by typing "I". The script will then end after 7 stars have been centered. The resulted RMS should be smaller than 10 arcsec for a normal pointing model.

You will then be asked to accept the result. Follow with:
 USER> next
in order to re-centre the star using the updated pointing model.

Prepare the telescope pointing back to the centre of chip 4:
 USER> enter aperture 0 -251 326
 SYS> autotv off

1.6.4 Check focus and aperture

Check that the focus is set to the correct plane (WFC = prime focus):
 USER> show focal
Check if apertures have been defined and have been defined correctly:
 USER> show aper
ap 0 -251 326 nominal aperture, object centred on CCD#4
ap 1 704 5 centre object on CCD#1
ap 2 357 -1043 centre object on CCD#2
ap 3 -701 -10 centre object on CCD#3
ap 4 0 0 nominal aperture, object in centre of mosaic camera
      (shifted from centre of CCD#4)
ap 5 1316 -368 centre object on AG-CCD (X=1032, Y=512)
ap 6 251 -326 centre object on RC (rotator centre)

If the apertures are not defined, the observing system (ICS) must be restarted. See Section 2.2 for instructions on how to do this.

After the system returns, re-issue the following commands:
 USER> show aper
 USER> show cal
 USER> cal def
and redo a single star calibration.

Regarding other possible focus problems, the SA or OSA can check the focus encoders using following engineering "INT Sony incremental focus transducer" document related to the absolute and incremental focus encoders.

1.6.5 No stars in the autoguider window

Check if the automatic procedure for finding guide stars is activated, using the command:
       SYS> cmd INFODISP PREDICT ON
This enables the automatic star acquisition. It predicts the x,y pixel coordinates at which suitable guide stars are expected to appear in the autoguider CCD at a given telescope pointing. The prediction is activated with every gocat command.

If no stars are visible in the autoguider window, although the focus and pointing have been checked or/and they are assumed to be fine, then check the camera shutter.

First, select a broadband filter (preferably G or R), otherwise a narrow band filter that is centered in the range 500-700 nm.

If you still don't see a bright star in the autoguider, then check on the GUI whether the camera shutter is open. This image shows the two possible states of the shutter. Make sure that it is in the OPEN position. If the shutter is closed, then click on the image on the WFC mimic window to open it.

NOTE (FR 24392): When the connection to the TCS is lost, then the INFODISP task that does the prediction exits.  The recommended solution is to start the ICS, which then starts the INFODISP task and turns the predictions on. Nevertheless, it is possible to run the task without starting the whole ICS: 
  • Check that the TCS is shown as connected (shown in green at top of INT Wide field Camera Mimic).
  • In the main INTICS pink window start the infodisp then guide star predictor, by typing:
  •        SYS> online TCS && ( infodisp >/dev/nul & )
           SYS> cmd INFODISP PREDICT ON
In case of a further connection problem, the INFODISP task will exit. Once the problem is resolved then repeat last two instructions.

1.6.6 Check RA/DEC encoders

If you experience possible tracking problems while observing (e.g. the stars appear elongated in the images) it is important to save the logs of the encoders that describe the position and movement of the telescope. This information allows to do a better diagnostic of the problems and helps to find a possible solution. To save the encoders' logs there are the two TCS commands (so they have to be typed in the TCS computer) described below.

  • To log the encoders' information for the next minutes you should type in the TCS

    USER> log enc on <time>

    where <time> is the number of minutes in the future you want to be logging the encoders' information. The <time> duration parameter defaults to 15 minutes, if omitted it will be prompted for. It is recommended to accept the default and in case is necessary, repeat the command after the previous log was saved. When the <time> have passed a file with the encoders' information is saved and in the TCS the file name will be displayed. Please take note of this file name and include it in the Fault Report.


  • To save the encoders' information of the past minutes you should type in the TCS

    USER> log enc keep <time>

    where <time> is the number of minutes in the past for which you want to save the encoders' information. The maximum duration is 30 minutes. After you type this command, a file with the encoders' information is saved and in the TCS the file name will be displayed. Please take note of this file name and include it in the Fault Report.

Example:

USER> log enc keep 30

Then in the TCS it will be displayed the name of the encoder log file just created, which will look as:

USER> SYS$SYSDEVICE:[INT.DATA.ENCODER]ENC151027.DAT;1

Sometimes a work around can be, if possible, to switch to observe another object in a different sky position and come back to the previous object later in the night, being in another sky position as the time has passed may not be a problem anymore.

1.7 TCS disconnects

Check that the TCS USER window is responsive. If it is not, it is possible that the TCS TELD task has died. To restart, go to the x-terminal where the TCS user interface and info display are running and type:
 intobs@inttv> telnet lpas2
Then, log onto the computer named lpas2 (username: int_login, password can be found on the lpas2 monitor), and select the option > TELD This will re-establish contact between the observing system and the TCS. When contact is re-established, the TCS icon at the top of the WFC GUI will turn green.

1.8 TCS communication hangs

In the case of loss of communication between the TCS and the user interface, i.e. if either the TCS USER window stops responding, or the TCS DISPLAY window stops updating, try one of the following solutions:

S1) Try to restart the TCS by opening the INT TCS terminal on the TCS computer and logging in with the INT TCS username and password.

S2) If this does not work, then clearing the communications may recover the system:

 In the blue band at top of the window, click on
Commands --> Clear Communications

S3) In case this still does not work, please contact the WHT OSA asking him/her to restart the TCS display using the following steps:

Log in again to lpas2 using the usual userid and password. Select the option REOPEN from menu as shown below:

 This is the INT TELESCOPE CONTROL SYSTEM
 
 The TCS is running, using version I30-0-2
 
 Please enter an option -
 
 STOP     - Stop the TCS
 
 TELD     -  Start the TELD task
 
 REOPEN   - Open the TCS user and display windows
            Info display on inticsdisplay.ing.iac.es
            USER window  on inticsdisplay.ing.iac.es
 
 DISPLAY  - Select different display devices
 
 NFSMOUNT -  Mount the ICS Unix partition
 
 EXIT    - Exit this menu and log out

NOTE: It is NOT necessary to do a Zeroset of the telescope after reopening the windows using the REOPEN.

1.9 WFC disconnects

If WFC disconnects, as shown by the orange warning-label on the mimic, the DAS has most likely crashed. Check the orange telnet window that is logged onto the DAS. If it is not available, you have to start a new telnet connection from the background.

First run obssys with the option shown on the whiteboard first in order to get the DAS> prompt (there is also an option to start up a new DAS window found by right-clicking on the desktop). Press the return key to see if there is a termination message for udas_server (the WFC task is an instance of the udas_server program). If udas_server has stopped running, do the following:
  1. Type startobssys at the DAS> prompt.
  2. Watch the talker. Wait until you have seen WFC come on-line.
  3. Type startobssys at the SYS> prompt.
  4. Wait for the mimic to reconnect to WFC.

1.10 The wfc_ag_sequencer window missing

In case the small wfc_ag_sequencer windows is closed, it can be activated using the command:
       SYS> wfc_ag_sequencer& 

1.11 The autoguider window missing

In case the WFC autoguider window is missing, first you should check in the bottom programs bar, maybe it was minimised by another user. Search for the "ING Autoguider" button showing a small telescope icon (usually the first from the left in the bottom screen programs bar) and click on it to restore the window.

In case the window was indeed closed, then you can start it again by typing in the pink window:
       SYS> agstart prime 
To establish a connection, choose "File" botton, then "Connect", and confirm a message which appears in a separate window.

2. Problems with the instrument control system (ICS)

For any issues which appear to affect the instrument, including control mechanism errors, communication errors, or observing system problems, try the following actions in order until the problem is resolved

2.1 Reset the Data Acquisition System (DAS)

This can also be important when solving issues with high read-out noise.
 SYS> dasreset WFC

2.2 Problems with the instrument control system (ICS)

This takes about 5 minutes, and will clear most communication errors.

2.2.1 Logout from the ICS

In the pink window type:
 SYS> shutdownobssys
 SYS> exit

2.2.2 Logout from the DAS

On the orange DAS window for WFC (triton), type:
 DAS4@triton> shutdownobssys
Are you sure you want to shut down the DAS? y/n? > y
 DAS4@triton> exit
 intobs@proteo> exit

2.2.3 Start up the DAS

Single left-click on the DAS term icon, then type:
 intobs@triton> obssys
followed by the number of the observing station you need (4).
 Choose your station> 4
Then, to begin, type:
 DAS4@triton> startobssys

2.2.4 Start up the ICS

Once the DAS is running, repeat the same steps in the pink INTICS window. If INTICS window is not displayed already, just single left-click on the ICS icon. Then type:
 intobs@lpss7> obssys
followed by the number of the observing station you need (4).
 Choose your station> 4
Then, to begin, type:
 SYS> startobssys
Then, wait for the following windows to appear.
  1. Talker
  2. Observation Log
  3. Who goes there?
  4. WFC GUI
  5. Autoguider
  6. WFC Autoguider Sequencer
Wait for the GUI will read in all the filters. Remember to check the read-out mode and binning.
Check the filter wheel can be rotated and if the shutter works. If one or the other does not work, type:
 SYS> cmd MCA RESET
In case your observations do not need guiding, check and eventually shutdown the wfc_ag_sequencer, according to the "Not Guiding" section of the WFC Operation Manual.

In case you need guiding, in the autoguider window, overlay the cross in the centre by selecting TOGGLE CENTRE CROSS from the SETUP menu.

2.3 Restart of individual system controllers

If you are not 100% sure how to power cycle any system, do not attempt to do this without first contacting your support astronomer or the WHT Telescope Operator (WHT OSA)! The next stage requires the telescope to be parked and brought down to access park.
    * If you have problems with the CCD, read-out, or communications, you may need to power cycle the CCD SDSU controller, which is located to the left of the cryostat, as seen in this photo. Turn off the switch at the back of the power supply, wait for one minute, then turn back on. After this you may need to complete a DAS reset or ICS restart as described above.

    * If you have problems with the filter wheel or camera shutter, you may need to power cycle the MCA controller. This is located below the cryostat, as seen in this photo. Turn off the controller, wait for one minute, then restart. After this you may need to complete a DAS reset or ICS restart as described above.

    * If you have problems with the autoguiding system, you may need to power cycle the AG controller. This is located to the right of the cryostat, as seen in this photo. Turn off the controller, wait for one minute, then restart. After this you may need to complete an ICS restart as described above.
If problems persist, you should now contact the WHT Telescope Operator, and clearly explain the problem and the actions you have taken so far.

3. Problems with the telescope control system (TCS)

If you encounter problems with the telescope system, focussing, pointing or tracking, you may need to restart the TCS. Shut down the observing system and log out of the ICS as described above. Be sure to follow this order:
  1. Park the telescope at Zenith using park zen command.
    If the TCS is lost, manually put the telescope at Zenith using engineering mode;
  2. Log out from the ICS and then log out from the DAS;
  3. Switch to engineering mode;
  4. Log out from the TCS;
  5. Start up the TCS;
  6. Zeroset the telescope;
  7. Start up the DAS and then the ICS.

3.1 Log out of the TCS

To log out type:
 USER> tcsexit
Then log out of the machine by right-clicking on Desktop, then choose Leave, then Log out.

3.2 Starting up the TCS

    On the INTICS machine (lpss7):

  1. Check the desktop environment, which should be KDE rather than GNOME. You can find the KDE option on the bottom right of the screen.
  2. Start the detector and instrument control systems by clicking the "DAS term" icon
  3. Log in with the intobs account (within the OpenWindows desktop; Session=Plasma)
  4. Single left-click on TCS icon, then use: username: int_login and password which can be found on the left monitor.
  5. At next menu choose START from the menu options. This will bring up both the telescope status display and the USER> prompt by default. If it doesn't work, type "REOPEN"

  6. Remember to do a ZEROSET whenever the TCS is restarted!
    After the TCS is restarted, check that the telescope focus value is what you need. If not, check the focal station.


    Then you can restart the ICS following the instructions in the Section 2.2.

3.3 DEC synchro arrows jumping

  1. Effective 10 December 2015: Quite often, the DEC arrows in the engineering rack are not moving correctly, jumping back and forth sometimes (instead of settling) after the telescope has reached its demanded position (during the zeroset or going to a new sky field).

    These arrows synchro units are 50 years old and can't be repaired reliably anymore. Fortunately they are not essential for the operation of the INT, being only useful as a reference especially when you do zeroset the telescope.

    The way to work-around if this DEC arrow start jumping while doing a zeroset is to check the black tape marks on the right hand DEC bearing of the telescope (photo here) in this order:

    1. Move in DEC to zenith. If the two marks coincide, then you are close to zenith;
    2. Then, from the engineering rack move DEC and find the point where the DEC LED lights up;
    3. Do zeroset drive DEC in positive direction over the point where the DEC LED lights up.

    At the moment the two tape marks are not visible from the control room window when the telescope is at Zenith, being hidden by the balance weigh control box. We are trying soon to provide a solution to be able to check DEC zenith reference marks when looking through the control room window.

    If the DEC arrows jumping happens while going to a sky field, then you can either simply disregard them (but check the first sky image) or work-around by going to a nearby field (for example using the command offset arc 0 1000) then gocat back to the required sky position, and the arrows should settle in this second intend.

4. Data reduction PC

4.1 Log out

In the IRAF window, at the prompt type:
 ecl> logout
then close any remaining windows and exit the system as previously shown.

4.2 Log in

To log in type: username: intguest and the password can be found on the intdrpc monitor (below). Then click on the xterm icon to open up a xterm window. Here you should type:
 > iraf
Followed by:
 ecl> wfc_ql
Check what directory you are in by typing:
 ecl> pwd
The correct directory is displayed on the WFC GUI. If you are not in the correct directory, type:
 ecl> cd /obsdata/inta/yyyymmdd
where yyyymmdd is your observing date, e.g., 20110321 for the night started in 21 of March 2011.

Reporting/Searching faults at ING Fault Management System

The ING has a Fault Management System to be used by ING staff and visiting observers mainly to record faults that occur during the operation of the telescopes and to review previous faults and their possible workaround. The INT-WFC observers may use the system by "Login as guest" being able to create and review fault reports.

Reporting a fault in the ING Fault Management System is a way of communication between visitor observers and the ING staff. Therefore, the faults reported can be of diverse nature: instrument problems, telescope problems, INT car issues, etc. The ING Fault Management System is reviewed daily in the morning by Telescope and Instrument Engineers, Operations Team and Instrument Specialists, who will address the reported situations and update the Fault Report with the 'Workaround' or any other relevant information related to the report.

5.1 Searching for previous fault reports

Searching for fault reports can be useful to identify similar problems that occurred in the past and help the WHT operator to find a possible solution. INT observers should not try actions described in the fault reports without the explicit authorization and supervision of the WHT operator, unless they have done it before and they fully understand what are they doing.

To look for a fault report in the database proceed as follow: Go to the ING Fault Management System and chick on "Login as guest". It will open a window with three tabs on the top.

Click on 'Search' and there, the best option is to select the 'Advanced' tab. A window with a space to write 'keywords' and a series of options for several searching parameters will appear.

Write down the searching keywords and open the menu in the relevant parameters for your search to choose the correct option then click on the left side box to select the parameter. Note that it is not necessary to select all the parameters available (but check for example 'INT' option under 'Site' parameter to constrain the search).

After you have made your choices go to the 'Search' button. This will produce a list of previous fault reports that match your requested keywords and parameters. The list contains basic information about the report. To access to the complete Fault Report (which includes the 'Fault Description' and the 'Workaround' or an updated status of the report) click on the 'ID' number in the first column (it will open the complete report in a new tab in your browser) or click on the Fault Report title (it will open the complete report in the same tab).

5.2 Reporting a fault

Reporting a fault in the ING Fault Management System it is the way to communicate to the ING staff the problems you had during the night. It is recommended to report most of the faults whether solved or not (to keep track of the system failures in the former case and to allow ING staff handle the problem during the day in the later case). Minor issues as a single "read time out" message after a command in the ICS do not need to be reported unless it keeps on repeating several times.

It is important that the Fault Report has all the relevant information related to the problem described, written in a concise and ordered way. Find below a guideline for writing a Fault Report:

Go to the ING Fault Management System and chick on "Login as guest". It will open a window with three tabs on the top. You will already be in the tab for writing the fault report.

The 'Title' should shortly describe the problem. It will appear in the Fault Report list when doing a search for previous fault reports. In the main window below the 'Title' you have to describe the problem. The basic information there should be:
  • What happened.
  • When did it happen (shortly describe the situation).
  • If applicable, describe the actions, in chronological order, you have followed trying to solve it.
  • Was the fault finally solved?.
  • At the end of the fault report window there is a series of buttons with options to choose or fill in. Please, try to specify them as much as possible, leaving as 'Unknown' only those items that you are not sure about (normally Duty Eng. and System).
  • Once you have finished take a few minutes to read the Fault Report you have written, to be sure that what is written is clear and understandable (for someone that was not in the INT control room when it happened), also check that you have not forgotten any essential information. Finally, press 'Submit Fault'.
If you were working together with the WHT Operator to solve the fault, and you are not sure about what and how to write the Fault Report you can ask them for help with this as well.

6. INT usual alarms

There are 3-4 alarms at the INT that are triggered sometimes, but the observers should not worry about them:

  • Oil pump: Sounds about a minute after switching off the oil pumps. Just need to be "accepted" from the engineering console.

  • Horizon limit: It may sound when slewing to access park (AP) for filling the WFC cryostat or parking the telescope with the WFC.

  • Mirror support: It might sound when slewing to a new sky position. If its character is not repetitive, then it can be accepted.

  • Point beyond INT pointing limits: It may occur a beeping if you try to point the telescope beyond the pointing limits. By simply pointing the telescope to another allowed position, the beeping should stop.

If there is an alarm different from the ones described above, or you are doubting or the alarm is persisting please call the WHT OSA to inform the situation and look for advice.

7. Some useful ING web addresses

It may occur while you are observing that you cannot access to common user INT web pages (even when Internet is working properly). Sometimes (depending on the origin of this problem), an easy work around that allows access to the pages is just copy and paste in the browser the web address of the page (instead of using the page's link from other web pages). Use this list of common use web addresses for observers at INT to copy-and-paste the address in the browser in case links from other pages do not work (and whenever you have time make a Fault Report of the problem so it can be solved as soon as possible).

There is a printed copy of this page in the INT control room, in the same folder as the IDS Operational Manual and Troubleshooter.

8. Acknowledgements

Credits should be given to past ING students who contributed to improve this manual and also to former staff members Andrew Cardwell and James McCormac for their support with past student training at the INT.


Top | Back


WFC Instrument Specialist
Last modified: 1 June 2022 (Section 3.2)