Trouble Shooting

  It is possible as with all complex instruments and electronic systems, that a time will come when the user encounters a problem, a hangup or a crash. This section details known problems and how to recover from them cleanly. Please seek the help of your SA or TO when a problem occurs! It is likely they have `seen it all before'.

WHIRCAM REFUSES TO STARTUP

• The PROBLEM: After typing whircam_startup, at some point you get an obscure error message and the startup task exits before all windows are created.
• The SOLUTION: Check to ensure that the ADAMNET task is running on both the WHIRCAM Vaxstation and on the VAX cluster by typing SH SYS on the WHIRCAM terminal, and on LPVS3 or LPVS2, and make sure the ADAMNET task is running on both. If it isn't, then type @UKIRT_DIR:ADAMNET
  If ADAMNET was running, or if one of the error messages is ``nonsense SP request,'' then shutting down WHIRCAM and logging out, then logging in and re-starting should solve the problem. If it still gets stuck, reboot IRCMVX (the WHIRCAM Vaxstation) (ask your SA or Duty Engineer).
  

THE ALICE OCCAM/TRANSPUTER SYSTEM HANGS/DIES

• PROBLEM: At some point, the Occam/Transputer system will crash/die with one of the following error messages appearing on the IRCAM3_DEV SMS menu window.

  BDS_ERRCHK> Error while getting ackn from CORE of GO completion
  - BDS_CORE_SERVER> ERROR: Channel comms timed out on tag receive
  WFG_OBEYW> Error completing action COMPLETE in 4^^1 task.
  BDS_CORE_SERVER> ERROR: Channel comms timed out on tag receive
  !! AIM> After INTEGRATE TASK_DONE SAI-I-ERROR, Error
  %SAI-I-ERROR, Error
  148013867

  or

  BDS_ERRCHK> Error while sending parameter block
  - BDS_CORE_SERVER> ERROR: Channel comms timed out on tag send
  WFG_CONFIGURE> Action SEND_PARAMETERS in BDS task failed.
  - BDS_CORE_SERVER> ERROR: Channel comms timed out on tag send
  !! ALICE_DONE> Action failed to complete in WFG : STATUS = 
  SAI-I-ERROR, Error
  !  ALICE_DONE> Returned OUTVAL =BDS_CORE_SERVER> ERROR: Channel 
  comms timed out 
  ! on tag send
  !! AIM> After CONFIGURE TASK_DONE SAI-I-ERROR, Error
  %SAI-I-ERROR, Error
  148013867

  What is happening is that the BDS gets a command (to say, GO) and sends this onto the CORE transputer system. This should be, and is, acknowledged. When the observation completes, the CORE has to send an acknowledgement to the BDS that the (GO) completed successfully - it is this acknowledgement that the BDS never receives (e.g. ackn from CORE of GO completion). This is a sure sign that the Occam is hanging for some reason - the subsequent error:
  (starting BDS_ERRCHK;SPMgt; Error while sending parameter block ...)
  is a further indication that the Occam side cannot communicate with anything else.
  

• The SOLUTION: Try the following first:
  1. Click on small ALICE parameter display window to re-activate it.
  2. Do cntrl C three times. The result of this will be to re-gain the DCL $ prompt.
  3. Open a new DECterm window under the Applications menu of the Session Manager.
  4. In the new DECterm type:

     ALICE_CONFIG <CR>
     ALICE_OCCAM <CR>

     This will re-load the Occam/Transputer side.
  5. Click on IRCAM3_DEV window to re-activate it again.
  6. Go up to the top level IRCAM3 menu.
  7. Go into the `Engineering' menu.
  8. Go into the `ALICE_engineering' menu.
  9. Go into the `Reinitialise_occam' item. This will re-establish communication with the Occam/Transputer system.
  Unfortunately, this appears not to be successful ALL the time. If this procedure does not work then run down the whole WHIRCAM system, logout and start again! This process will take about 5 minutes.
  

THE ALICE/TELESCOPE LINK HANGS/DIES DURING A SLIDE

• The PROBLEM: If your EXEC calls for a series of SLIDES (eg for a dithered pattern), and the telescope is also autoguiding, then occasionally the link between WHIRCAM and the telescope control software (TCS) dies and the EXEC is suspended. This is because the TCS was waiting for a handshake from the autoguider, telling it that the guide star has been re-acquired after the SLIDE, but it didn't get it within the default response time.
• The SOLUTION: On the ICL terminal, type `SEND AUTO CANCEL GUIWIND', which should cancel the hung state of the TCS. Then get your TO to re-aquire the guide star with the autoguider, then on the IRCAM3 Instrument Control screen select `Resume_EXEC' and the EXEC should resume where it left off.
  

ETHERNET FAILS

• The PROBLEM: Communications fail.
• The SOLUTION: First, check for a flashing light on the ethernet box in the WHT computer room. WHIRCAM is connected to port 4. If its flashing, reset the box.

  If no flashing light, then type SET PROC/PRIV=ALL in WHIRCAM's SMS window, then type UCX. At the UCX prompt, type `ping cgs' which checks for a response from ALICE. If no response, then something is likely wrong with either the ethernet, the B300, or IRCMVX ethernet. Call a duty tech.
  

THE WHIRCAM MOTOR DRIVES FAIL

• The PROBLEM: Being a mechanical part, the motor drives that position WHIRCAM's filter wheels are the one part that may fail at some time (usually about 3 am). As the filter wheels are positioned according to indents on the wheels tripping a microswitch, ALICE should tell you if it can't move the filter wheels, such as when a drive motor fails.
• The SOLUTION: First, check that it is a problem, and select the `datum_motors' menu in `ircam_engineering' (via the `engineering' and `set_instrument' menus). If this fails, then reload the motor drive software, as described in the WHIRCAM DUTY ENGINEER CHANGE MANUAL (in the WHT control room or the WHT manager's office). If this fails, then the motor(s) is/are probably stuffed.

  Replacing the failed motor drive is the obvious solution, but so as to avoid losing a large fraction of the night, there is an option of running WHIRCAM without motor control, in which all the other functions will continue as normal, but filter changes will have to be done by hand. In this case, close down WHIRCAM, and restart with `whircam_manual' instead of `whircam_start'. The motor drives sit on the back of the WHIRCAM cryostat, and when not powered (switch off the drives in the GHRIL console area), can be rotated by hand. The datum position (position 1) is marked on the drive wheel, and each new position (in the direction of the arrow) can be felt by hand as you rotate the wheel. While this may be time consuming and laborious, real astronomers prefer this than to going to bed early.
  

HIGH DARK CURRENT

• The PROBLEM: The results from an array_tests (see §3.3) gave non-nominal results for the dark current.
• The SOLUTION: Repeat array_tests to make sure it wasn't a fluke. If it still persists, check that the cold head temperatures on the temperature displays in the GHRIL room are nominal (Table 9).
  

   

  If any temperatures aren't nominal, then the He in the closed cycle cooler may be low, and needs to be topped up. Call the Duty Tech.
  

HIGH READOUT NOISE

• The PROBLEM: The results from an array_tests (see §3.3) gave non-nominal results for the readout noise.
• The SOLUTION: If your ALICE display shows a lot of diagonal streaks, then ALICE probably isn't switched on. The enable switch is a key in the upper middle part of the ALICE rack up in the GHRIL control room, and should be fully turned CLOCKWISE (a green LED to the right of the key will be on if enabled).
  If the display looks flat (apart from hot/dead pixels), then repeat array_tests to make sure it wasn't a fluke. If it still persists, then check if the chip is being properly read out. To achieve fast readout times, ALICE uses 4 ADCs, each reading out every 4th column. Check to make sure all readout channels are reading out similarly, by plotting a histogram of each readout output. To do this, open another DECterm window, type `set def DISK$IRCAM:[WHIRCAM.DATA.IRCAM_DATA.19960229.RODIR]', then `ircam_setup', then type `cgs4dr' (the Data Redustion System for UKIRT's CGS4 IR spectrograph). This will create another SMS window session. Within this, select the `engineer' menu, then `graphics', then `plot_histogram'. Within this menu set Data to the 4th image in the array_tests sequence (this is a DARK), and the following:

  Display_port = 1
  Data_area = DATA
  Number_of_bins = 50
  X_step = 4
  Y_step = 1
  Smooth_size = 0
  Soft_or_hard_copy = SOFT
  Display_whole_area = FALSE
  _xstart_if_not_whole = 1
  _ystart_if_not_whole = 1

  This should plot a histogram of the first transputer readout in the upper left quadrant of the image display. To display the second transputer readout, reset Display_port to 2 (shifts to the next display quadrant in upper right) and _xstart_if_not_whole to 2. Repeat for the third and fourth transputer readout. You should then end up with 4 histograms, one per display quadrant. They should be all about the same mean, with widths of typically a few counts. If one is noticeably wider, then check the cable for that transputer (in the ALICE rack in the GHRIL control room) to make sure it isn't loose. If the SA doesn't know which cable belongs to which transputer (eg the brown cable is channel 4), you'll have to disconnect each in turn, repeat array_tests and re-plot the histograms to see which one changes.

  If all cables seem to be OK, then it may be that one of the ADC cards is broken. Spare cards exist, so call your Duty Engineer.
  

MISALIGNMENT OF WHIRCAM AND/OR DICHROIC

• The PROBLEM: Normally, WHIRCAM should be properly aligned with respect to the rotation and optical axes by the SA at the start of a run. However, if some clod happens to later bump into either WHIRCAM or the dichroic, then you'll need to call your SA (or the site SA) to re-align the optics.
• The SOLUTION: This is the method your SA should use to realign. If only the optical breadboard was bumped, then use the alignment jigs to re-position it back to its nominal position and orientation. Stand next to the optical table, facing WHIRCAM's snout (ie looking over the dichroic box towards WHIRCAM, with the telescope and de-rotator to your left). Get someone else to face you, on the other side of the table (the breadboard is heavy and won't slide very easily, so you'll need someone else to push/pull while you pull/push). There are two metal alignment tabs attached to your side of the breadboard, and one tab on your left side of the breadboard, near the re-rotator. Slide the breadboard such that all 3 tabs touch the sides of the optical table. Make sure that the various location clamps on the breadboard are still tight.

  If the dichroic was moved, then you must tweak the micrometers to re-align it:

  • In the GHRIL control room, disconnect the graphics monitor cable, and connect up a PC monitor (which if not in the GHRIL control room will have to be `borrowed' from someones desk), and place it on the optical table.
  • Run MOVIE with 0.5 sec integration, in ND_STARE mode, the nbL filter, and 0.68V BIAS. Put the statistics window on a dead-pixel-clear part of the chip (select set_graphics, statistics_window menus, then set centre_column to 145, centre_row to 120, width to 10, height to 10). Take a DARK and set graphics to subtract it from subsequent exposures.
  • Point telescope to near zenith, make sure dome and mirror covers are open, and that you are pointing at blank sky and not a bright star, and make sure the telescope is tracking.
  • Adjust first one micrometer so as to minimise the total counts from sky (the night sky at L should always be less emissive than the secondary baffle!). Make sure stray fingers don't touch the dichroic, and are well away from WHIRCAM's FOV when you take the measurements. Now do the same for the other micrometer (ie the other axis). Finally, repeat for the first micrometer.
  • You should now be re-aligned. Take note of the micrometer readings. (In January 1996 these were twist (azimuth) = 5.810 mm, shout (altitude) = 5.135 mm.)
  • To be sure, you can repeat the above, but first set up on a bright star, set the filter to J, and now try to maximise the total counts (check to make sure the peak pixel isn't saturated, and keep the whole of the stellar image within the statistics box on the monitor).
  • The final micrometer readings should be the same as those obtained on sky in NBL. If not, repeat on sky with the NBL filter.
  You may need to re-determine the rotation centre (your TO should know how), but make sure it is for SKY PA 0. (In Jan 1996 the rotation centre was at WHIRCAM pixel position 103, 195).