Taming the JAG

A diagnostic guide for the JKT A&G box  (ver 2.4)


Occasionally the JAG fails due to power cuts and other (unknown) phenomena and this guide should hopefully enable the engineer to get the instrument back on line as quickly as possible. It cannot cover EVERY problem that may happen, but now being the last person still around to have worked in the team at RGO, Herstmonceux that built the JAG back in 1986... There ain't much I don't know about this instrument :-)

In the many years that I have been involved with the JAG, experience has shown that most of the problems can be solved by following the instructions below. The first version of  "Taming the JAG"  I wrote back in 1990, but the JKT ICS has changed considerably over the years and this document reflects these changes.

I have attempted to make this version of the document an 'expert' procedure?  Just click on the links below and follow the steps in a logical order. Clicking on the links within the sections will take you to the to the next stage and you will have the JAG back on the sky within a short time. Any problems with this document or the JAG in general then drop an email to: ejm@ing.iac.es

For more information refer to manuals :

ER0 45  The Local Control System for the 1m CCD A& G Box
ER0 46  4MS Local Controller Software Manual

These documents and a full set of circuit diagrams (Document folder No 23) can be found in the JKT control room .

I have broken down this document into six main parts:

JAG can you hear me?
Analysis
Diagnosis and cure
Lost movement in autoguider probes
Problems with filter wheels
Final checkout with the Sparc based ICS

Appendix

The status word explained
Important notes


JAG can you hear me?

You have arrived at the JKT and the JAG has problems with one or more mechanisms.

Rule 1...  As "The Hitch Hiker's Guide to the Galaxy" states:    DONíT PANIC!!

All these checks and recovery procedures below will be done from the JAG ENGINEERING VT220 TERMINAL which sits on the table beside the met station PC. This must be connected AT ALL TIMES. If the Engineer's terminal is not connected to the 4MS controller, it has been found that 4MS can 'lock up' due to pick-up on the RS232 cable running from the telescope to the control room.

n.b.  You will be entering instructions to the FORTH interpreter thus ALL commands must be typed in: UPPER CASE.

  1. If the ICS is running, shut down the observing system  by typing:  shutdownobssys at the SYS> prompt in the pink window. If the JAG has problems and the ICS is running you CANNOT do sensible work from the engineering terminal due to the ICS constantly polling the JAG 4MS controller for status messages.
  2. Push the RETURN button a few times on the engineering terminal and make sure that you get the 'ok' prompt. Type: HELP and a screen of engineering commands will be displayed. If this is ok, move on to: Analysis
  3. If there's no response, it could be that the JAG 4MS controller has crashed. Go out to the telescope and power cycle (wait a few seconds) the 4MS controller. The VT220 monitor should respond with :
  4. RGO 4MS.FORTH V.2
    hi
  5. Push return a few times and check for the ok  prompt. If this is good, move on to:  Analysis
  6. If there is still no communication with the 4MS crate, this will need to be investigated.
  7. Is the VT220 terminal configured correctly? Push the setup key (F3) and go to comms. The engineering terminal's RS232 parameters are:

  8. 9600 bds 7 data bits, even parity and 1 stop bit.
  9. If still no luck, check :
  10. The ribbon cables from the ACIA boards in the 4MS crate are seated correctly
  11. The cards themselves in the 4MS crate are seated correctly.
If still no luck you're in for a LONG night! The problem lies within the 4MS crate modules and is beyond the scope of this document.


Analysis

A block of data (shown in bold font) will appear showing the status and step counts of the various mechanisms. You can also use the mimic where the status is shown below the step count value. A healthy example is shown below.
 
Mechanism Status Step count Comments
%X 1004 005740 Autoguider X probe (at pos 5740)
%N 1104 1 Autoguider ND filter (pos 1)
%G ******** ****** Red and blue grisms  (removed)
%Y 1104 010000 Autoguider Y probe (at pos 10000)
%C 1000 3 Autoguider Colour filter (pos 3)
%F 1104 O TV flat mirror (out)
%S 1004 C TV shutter (closed)
%T **** ****** Drift scan table  (not used now)
%D 1104 1 Main CCD filter wheel (pos 1)
%B 0002 Overall box status added by 4MS

The thing to look for is that the most significant bit in the status line (bit 12 of the status word) is set to 1. This means that the mechanism has been (or still is) initialised. The overall box status %B never has the high order bits set so ignore this.

n.b. This does NOT apply now to mechanisms %G (grisms) or %T (table) as these have been physically removed or de-commissioned.

THE BOTTOM LINE!


Diagnosis and Cure

  1. If the msb of any mechanism shows a value of zero '0' it will need to be re-initialised.

  2. Solution goto :  RESTORING A MECHANISM
     
  3. If the msb is a  8, 6, 4, or A (in the X, Y, F or S fields), the problem is that the mechanism has moved into either an upper or lower limit switch.

  4. Solution goto :  MOVING OUT OF A LIMIT SWITCH
     
  5. If the status or step count values show random ASCII characters (e.g. 4?f* $6#ba!), the SMDM battery backed ram has corrupted.

  6. Solution goto :  RESTORING A CORRUPTED SMDM
     
  7. If a group of mechanisms show zero's  0000 000000  an SMDM has either lost power or is not communicating with the 4MS controller.

  8. Solution goto :  NO SMDM COMMUNICATION

Restoring a mechanism

Use the ALL? NOW command to determine which mechanisms have 0 msb's. Restore the mechanism using the command format below.
 
Command Mechanism names Comment
SETUP mech_name XPROBE,  ND,  YPROBE,  COLOUR,  FLAT,  SHUTTER, CCD Reload mechanism parameters
ZEROSET mech_name XPROBE,  ND,  YPROBE,  COLOUR,  FLAT,  SHUTTER, CCD Re-initialise by moving the mechanism to datum switch at engineering velocity
SETUP mech_name XPROBE,  ND,  YPROBE,  COLOUR,  FLAT,  SHUTTER, CCD Restore normal velocity by reloading parameters

e.g.  To re-initialise the autoguider Y probe:

SETUP YPROBE
ZEROSET YPROBE
SETUP YPROBE

See Note 5

Now enter the ALL? NOW command.

The data block below shows what the status should be for any mechanism after a successfull initialisation.  In practice, only the re-initialised mechanism will show the status as shown in this table.
 
%X 1304 000005
%N 1104 U
%G ******** ******
%Y 1304 000005
%C 1104 U
%F 1104 U
%S 1104 U
%T **** ******
%D 1104 U
%B 0002

Undefined Mechanisms
Some mechanisms will show a U value. This means although the mechanism was initialised successfully, the actual position of a particular mechanism is Undefined.  Driving a filter wheel or the TV mirror or shutter to a defined position will clear this. Moving a mechanism from ICS will also clear these U flags. In engineering mode, the commands and example values to do this are :
 
Command Valid positions 
15.000  XPROBE 5 to 29200 steps
3.  ND Valid position are 1 to 8
15.000 YPROBE 5 to 29200 steps
4.  COLOUR Valid positions are 1 to 8
IN FLAT IN or OUT
CLOSE SHUTTER OPEN or CLOSE
3.  CCD Valid positions are 1 to 6

IMPORTANT
An integer value as given on an X, Y, ND, COLOUR or CCD move must contain a period '.' This can be placed anywhere in the number  i.e. 15.000 is the same as 1.5000, 150.00, 15000. etc.

If the mechanism(s) work ok in engineering mode, proceed to:  Final checkout


Moving out of limit switch

Use the ALL? NOW command to determine which mechanism has moved into a limit switch.

n.b. The CCD, ND and COLOUR  filter wheels do NOT have associated limit switches

See Notes 3 and 4

If the X PROBE is in a limit then enter the commands :
 
LIMITOVERRIDE XPROBE override limit switch
ZEROSET XPROBE move mechanism to datum switches at engineering speed
SETUP XPROBE restore normal speed by re-loading mechansim parameters

Check the mechanism has been initialised correctly by typing:  X?   Status returned must be %1304

If the Y PROBE is in a limit then enter the commands :
 
LIMITOVERRIDE YPROBE override limit switch
ZEROSET YPROBE move mechanism to datum switches at engineering speed
SETUP YPROBE restore normal speed by re-loading mechansim parameters

Check the mechanism has been initialised correctly by typing:  Y?   Status returned must be %1304

If the TV FLAT is in a limit then type the commands :
 
LIMITOVERRIDE FLAT override limit switch
ZEROSET FLAT move mechanism to datum switch at engineering speed
SETUP FLAT restore normal speed by re-loading mechanism parameters

Check the mechanism has been initialised correctly by typing:  F?   Status returned must be %1104

If the TV  SHUTTER is in a limit then type the commands :
 
LIMITOVERRIDE SHUTTER override limit switch
ZEROSET SHUTTER move mechanism to datum switch at engineering speed
SETUP SHUTTER restore normal speed by re-loading mechanism parameters

Check the mechanism has been initialised correctly by typing:  S?   Status returned must be %1104

Verify that all mechanisms have been successfully initialised by typing the ALL? NOW command.

Some mechanisms will show a U status after initialisation. Follow the steps for an Undefined mechanism to clear this flag.

If the mechanism(s) work ok in engineering mode, proceed to:  Final checkout


Restoring the battery backed RAM in the SMDM

The JAG uses three Stepper Motor Drive Modules (labelled SMDMís 1, 2 and 3) and each box has 4 channels. You will also notice that the data blocks displayed with the ALL? NOW command displays mechanism status in box order.  The boxes and associated mechanisms are:
 
Box number Channel 1 Channel 2 Channel 3 Channel 4
SMDM 1 AG   X probe AG   ND  filter Red Grism (n.a) Blue Grism (na.)
SMDM 2 AG   Y probe AG   COLOUR  filter TV flat mirror TV shutter
SMDM 3 Table (n.a) CCD filter wheel Not used Not used

n.a.  Not Applicable. These mechanisms have been removed (Grisms) or are de-commissioned (Drift scan table)

See Note 2   for more information.

If any of the values in the data block display random ascii values then the battery backed ram is corrupted and will need to be cleared and restored.  Use the ALL? NOW command to determine which SMDM is corrupt. Follow the instructions below to restore the corrupted SMDM.

e.g. If SMDM 1 is corrupted, you will see rubbish like this !

%X 5tY!J8*3?dh
%N 3g$w4DT8>e
%G 1[/r %3hvR6s

If  SMDM 1 is corrupt then  :
 
RESTORERAM XPROBE clear out contents of SMDM 1 RAM
SETUP XPROBE download X probe parameters
SETUP ND download AGND  parameters)
ZEROSET XPROBE move X probe to datum switches
ZEROSET ND move AGND filter to datum switch
SETUP XPROBE restore normal mechanism velocity
SETUP ND  restore normal mechanism velocity

If  SMDM 2 is corrupted then :
 
RESTORERAM YPROBE clear out contents of SMDM 2 RAM
SETUP YPROBE download Y probe parameters
SETUP COLOUR download AGCOL  parameters
SETUP FLAT  download TV flat parameters
SETUP SHUTTER download TV shutter parameters
ZEROSET YPROBE move Y probe to datum switches
ZEROSET COLOUR move AGCOL to datum switch
ZEROSET FLAT move TV flat to datum switch 
ZEROSET SHUTTER move TV shutter to datum switch
SETUP YPROBE restore normal mechanism velocity
SETUP COLOUR restore normal mechanism velocity
SETUP FLAT restore normal mechanism velocity
SETUP SHUTTER restore normal mechanism velocity

If  SMDM 3 is corrupted then :
 
RESTORERAM CCD clear out contents of SMDM 3 RAM
SETUP  CCD download CCD filter wheel parameters
ZEROSET CCD move filter wheel to datum switch
SETUP CCD restore normal mechanism velocity

Verify that all mechanisms have successfully initialised by typing the ALL? NOW command.

Some mechanisms will show a U status after initialisation. Follow the steps for an Undefined mechanism to clear this flag.

If the mechanism(s) work ok in engineering mode, proceed to:  Final checkout


No SMDM communications

If an SMDM returns zeros when sending the ALL? NOW command to check status. It can be either one of two things: LOSS OF POWER

The 24V supply for the SMDM's is derived from two 12V Vero PSU working in series. It has been known to happen that one of these PSU's can crowbar or fail. Drop the front cover of the  JAG 4MS controller and check the health of these PSU's. Measuring and seeing 24V across the pins on the SMDM power connector is useful, but is not the ultimate test as a PSU could be breaking down under load.

LOSS OF COMMUNICATIONS

Generally this is due to a loose connection with the DIL ribbon cable connectors going from the pair of  triple ACIA boards in the 4MS crate to the 4 pin connectors on the back panel. Drop the front panel of the 4MS controller and make sure that these connectors are seated firmly down on the ACIA boards. The other possibility of course is a broken lead between an SMDM and the 4MS controller.

A more serious problem has been the RS422 line driver and receiver chips blowing up! These are located on both the ACIA boards in the 4MS crate and on the CPU card within the SMDM. If the 24V power to the SMDM's is ok, replace (or swop in the first instance) the RS422 chips on the ACIA board. These are marked as such.

n.b. Not all the chips on the ACIA boards use RS422 tx/rx chips. Some are for RS232 usage so BEWARE!

If the problem still persists, then change the RS422 tx/rx chips on the SMDM cpu board. Usually its best to change the entire board. There are spares available. It is quite simple to remove the 4 front plate screws with the SMDM box still in-situ and remove the CPU board. If this is done, you will need to go through the steps of restoring the on board ram and mechanism setup as described above.


Checking for lost movement in the autoguider X and Y probes

The only position when the autoguider probes show a %1304 status (both linear and rotary datum flags active) is when the X and Y probes are in their PARKED position of 5 steps. This is a good test of checking for lost movement. i.e. Moving the probes using the ICS commands: autocentre and autopark a few times then checking for a status of %1304 when parked will show that the probes are NOT losing position.

The ABSOLUTE position for the autoguider probes can only be read from the engineering mimic which shows the status bits. From the ENGINEERING TERMINAL enter the following:

15.000  XPROBE  15.000  YPROBE

This will CENTRE the probes. Check that the engineering mimic status is:  %1004 15000  for both probes. Now enter:

5.  XPROBE  5.  YPROBE

This will PARK the probes. Check the mimic that the status returned is:  %1304 000005 for both probes

See Notes 3 and 4

n.b. If only the mechanism step count values are shown and NOT the status information on the engineering mimic,  type: ON DETAIL
There is no need to go into IC mode for this operation.

If the X or Y probe does not return a status of %1304 at a step count of 5 then re-initialise it as described in:  Restoring a mechanism

If the mechanism(s) work ok in engineering mode, proceed to:  Final checkout


Problems with filter wheels

There are times when an observer may complain that he/she can't find guide stars or that the CCD frame has a strange shadow or effect.
This has often been caused by a filter wheel losing position and vignetting the beam.

You will notice checking the engineering mimic that whenever the filter wheel moves to position 1, the datum switch flag bit is set. This is a good test for lost movement. i.e. If a filter is suspected of vignetting the beam, move it to position 1 and check for the %1104 status.

n.b. If only the mechanism step count values are shown and NOT the status information on the engineering mimic,  type: ON DETAIL
There is no need to go into IC mode for this operation.

If a wheel shows a %1004 status at position 1 it is misaligned and will need to be re-initialised as described in: Restoring a mechanism

It has also been found that the locking screw that holds the main CCD filter wheel can work loose causing the wheel to slip. If the wheel is suspected of losing position, check that it is inserted correctly, is in mesh with the sprocket and that the locking screw is firmly (but not over) tightened before re-initialising.

It should also be noted that a microswitch operates when the door to the filter wheel compartment is opened. Although the flag set (bit 11) by this switch is not read by 4MS, its status is detected at ICS level. Thus a status of %1504 means that although the wheel has been zeroset the door is open. This will prevent a move of the filter wheel  from the ICS.

If the mechanism(s) work ok in engineering mode, proceed to:  Final checkout


Final checkout under ICS

The Sparc ICS (Instrument Control System) communicates with the JAG via an RS232 link (protocol:  9600 7E1). This connection is made between a  serial interface connector at the rear of the Sparc and the INSTRUMENT COMPUTER port connector on the JAG 4MS controller. It should should be noted that for diagnostic purposes, sometimes this serial link cable is plugged into an RS232 switch box which is located in the 'Instrument Electronics' area adjacent to the control room.

The switch on this break-out box  MUST be set to the 'SPARC' position. The other position allows for example; a laptop PC to be connected for line diagnosis or monitoring.

When you are satisfied that the JAG works ok under engineering control proceed as follows:

  1. Type ENG to restore normal engineering commands and divert status messages back to the ICS.
  2. Type ON DETAIL (if needed) to bring up status information on the engineering mimic display.
  3. Restart the ICS by typing startobssys at the SYS> prompt in the pink window.

  4. n.b. When the ICL starts up, there may be a RED ALARM WINDOW flashing and bleeping. Ignore this. If this persists it may be best to take the ICS down completely, log out and start afresh from JKTOBS.
  5. Check the ICS mimic display for a GREEN STATUS on all mechanisms. If any mechanism shows BLUE STATUS  this is probably caused by a U flag being set during a mechanism restoration in engineering mode which still hasn't been cleared. Moving any mechanism to a valid position under ICS should cure this.
Test the JAG as follows by entering these commands at the SYS> prompt:
 
ICS Command Operations Engineering mimic status
SYS>  tv on Moves the TV flat mirror into the beam and opens the TV shutter F=%1004     S=%1104  (datum active)   Note
SYS>  tv off Removes the TV flat mirror from the beam and closes the TV shutter F=%1104 (datum active)    S=%1004 
SYS>  autocentre Moves the autoguider X and Y probes to centre of field (15000 steps) X=%1004    Y=%1004
SYS>  autopark Parks the autoguider probes (pos  X=5   Y=5) X=%1304  Y=%1304  (both datums active)
SYS>  autocol  4 Moves the autoguider colour filter wheel to position 4 %1004
SYS>  autocol  1 Moves the autoguider colour filter wheel to position 1 %1104  (datum active)
SYS>  autond  4 Moves the autoguider ND filter wheel to position 4 %1004
SYS>  autond  1 Moves the autoguider ND filter wheel to position 1 %1104  (datum active)
SYS>  filter 3 Moves the main CCD filter wheel to position 3 %1004
SYS>  filter 1 Moves the main CCD filter wheel to position 1 %1104 (datum active)

Note
After the TV shutter opens and closes a few times, the datum switch is not always activated when the shutter is open? This is due to a small amount of hysterisis with this mechanism. This is not important. If however no light is entering the TV camera (and the camera appears to be working ok) it would be advisible to re-initialise the TV shutter.

THAT'S   IT !

If the JAG still doesn't work, there is a REAL hardware problem that will need to be investigated.


APPENDIX

Description of the STATUS WORD

The status block consists of 4 bytes designated VWXY (15=msb of V, 0=lsb of Y). Bytes VWX come directly from the SMDM. Byte Y is generated from 4MS.

The example shown below would be valid status for mechanisms Xprobe or Yprobe after initialisation.

i.e.  %1304
 
Bits generated from SMDM Bits added by 4MS
Bit  15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
Value 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0
Function lower
limit
switch 
active
upper
limit 
switch
active
stepper
motor 
moving
mech
init'sed
ok
datum
switch 
4
active
datum
switch 
3
active
datum
switch 
2
active
datum
switch 
1
active
not
used
not
used
datum
failure
engr's
control 
on
monitor
mode
invalid
bit
last
comm
compl'd
bit
not
used
mech
busy
Notes 1 1 2 3 4

1.   Datum switch 3 is not used. Datum switch 4 is used only for the filterwheel door  (open or closed status).
2.   A LIMITOVERRIDE forces this to Bit=1
3.   Bit=1 Last command completed      Bit=0  Command in progress or failed to complete
4.   Bit=1 Mechanism busy                    Bit=0  Mechanism not busy


Important notes

    Note 1
    When using the engineering terminal in the IC (Instrument Computer) mode, the following responses will be seen: FYI:  Why does just hitting the return [cr] key produce an @N I response? A VALID command MUST be entered first FOLLOWED by a [cr]

    Note 2
    Parameter values for each mechanism are stored in EPROM in the 4MS controller. These include things like:

    Unlike the SMDM's used on ISIS or the WHT A&G box, the parameter tables must be loaded individually after the battery backed ram is cleared  i.e.  You MUST always enter a SETUP command for each mechanism after a RESTORERAM command.

    Note 3
    When doing a ZEROSET, a mechanism moves at engineering velocity. This is very slow and can take sometime to complete especially in the case of the autoguider X and Y probes if they happen to be a long way off from the datums. n.b. If a mechanism is ALREADY at the datum and a ZEROSET is entered, it will move out of the switch a small distance at normal velocity then re-seek the datum switch at engineering speed.

    For reasons unknown, sometimes the command does not always complete and the mechanism stops short of the datum(s). If this is the case, type ZEROSET again. If you are using the JAG A&G box engineering terminal in the control room, listening to mechanisms moving over the loudspeaker gives confidence that at least something is happening!

    Note 4
    You may see a B (busy) status returned if you enter a command more than once. This is sometimes the case if the autoguider probes have not initialised at the first attempt and you re-enter the command. If this happens, type RESET which will reboot the 4MS and try again. You will have to re-type IC so that the ALL? NOW command shows all mechanism status. If the terminal happens to lock up when typing RESET, you will have to power cycle the 4MS crate in the telescope rack.

    Note 5
    Once a mechanism is successfully initialised the normal velocity must be restored. If this is not done, the ICS software will generate a timeout message as it expects an operation to be completed within a defined time period.
    i.e. You MUST enter a further SETUP command AFTER doing a ZEROSET to restore the mechanism's correct operating speed.


Document history

Ver 2.0 This re-written document first issued : 25th May 2001 ejm
Ver 2.1 Filterwheel door switch info added : 12th Sept 2001 ejm
Ver 2.2 The autoguider probe positions using the AUTOPARK and AUTOCENTRE commands from the ICS are now in agreement on both the observers and engineering mimic displays. Removed temporary notes : 13th Dec 2001 ejm
Ver 2.3 4MS PSU info added : 29th July 2002 ejm
Ver 2.4 ON DETAIL command added in Analysis section : 19th Sept 2002 ejm

This file: http://www.ing.iac.es/~eng/ops/jkt/taming_the_jag.html