• Communications/ top gui
• Optical alignment
• No light / not enough light
• Mechanisms
• DM + FSM
• Fisba + laser-flattening
• WFS/SDSU
• White-light flattening
• Grabs from science camera
• Simplex
• INGRID
• Miscellaneous
• Diagnostic procedures

TopGui freezes while guiding.

Don't panic, it's probably still correcting (but the autoguide signal is not being sent to the TCS). First type c40CommunicationsRestart. TopGui will disappear. Type TopGui & to recover it, and the loop should still be closed. Open the loop immediately, stop the autoguider (ctrl/c to get rid of window if necessary), start autoguider, close the loop.

Suspect communications

ping scorpio, aocontrol1, fisba, naomisdsu, naomiomc, then c40Ping 1, c40Ping 2 etc. c40Ping 0 checks contact with the communications process running on aocontrol1. WFS Status is useful for checking the status of all the C40s. Check network connections (usually ports 5 6 7 8 10 in wall behind control cabinet) by plugging in a portable PC. fisba is only pingable when plugged into its own port (reason unknown). naomiomc and naomisdsu are VMEs (no disks) so are not pingable unless aocontrol1 is. C40s 0-8 are WFS, 9-16 are SG If naomiomc and naomisdsu won't ping from navis, after aocontrol1 reboot, try powering them off and on.

aocontrol1 not responding

May happen if not been properly shut down. To check state, plug VT220 into RS232 (need to check parity), look at messages on screen. If looks bad, try pressing the reset button on the aocontrol1 card. To power cycle, rsh -l root aocontrol 1 (password in the back of the NAOMI logbook), type shutdown now (i.e. 2 words). Wait a few sec for shutdown to complete. Power off aocontrol1 using switch top right on front of aocontrol1 cabinet. aocontrol1 is the left-hand card in top crate. Power on again. Ideally, have a VT220 plugged into the socket on the back of the aocontrol1 box while this happens, to watch the messages as it reboots. To simply reboot, type reboot on the VT220 keyboard. 3 NaomiRestarts (on navis) are needed after an aocontrol1 reboot (c40ResetSystem in an aocontrol1 session may also work). After a power cycle, the C40s may not correctly boot (NAD email 15/7/02), in which case you need to c40Run with an old configuration, by typing, at aocontrol1?:
c40Run NaomiInterleaveZeiss
which should do a naomirestart as normal (still needed after cold start 11/02).
If this doesn't work, worth checking all cable connections (especially after a NAOMI move). (When aocontrol1 taken down, no longer necessary to rlogin aocontrol1 then c40Run NaomiDev.)

Alternative to using vt220 plugged in back of navis: tip session then use the TIP name as a command-line option to start a tip session to access the rs232 engineering ports for NAOMI on Portserver 11 (PCM email 9/7/02).

NAVIS dead

E.g. after entering password into screen lock, screen goes blank and won't respond to keyboard or mouse. May be disk full of fisba logs in /var/adm/wtmpx, call CFG.

Alternatively, the AOcontrol1 network connection switch (at the back of the top crate) may not be switched to BNSC. In general, if communications problems are suspected, first ping aocontrol1 from the navis prompt to see if navis can talk to aocontrol1.

NAVIS very slow on startup

If it takes ~ 10 mins to login and then another ~ 10 mins to start TopGui, which then freezes, a reboot may be necessary. Hit the stop/a keys, typoe boot at the go prompt. If there's a disk problem, navis asks for root password. To bypass this (and fix from another machine), ctrl/d.

Can't telnet to navis from SLO

Telnet not permitted, try rlogin instead

TopGui comes up as empty square

Check that logged in as naomi, not root.

TopGui crashes or freezes or no centroids

After TopGui freezes, may still be correcting closed-loop, try: c40CommunicationsRestart at the aocontrol1 prompt, TopGui window disappears, run it up again, open the loop, stop the autoguider, ctrl/c autoguider window if still open, start autoguider again (IKS 9/03).
Or... ps -ef | grep TopGui, then kill nnnn, then at navis prompt, TopGui &. To restart TopGui without touching C40s, navis promt, NaomiRestart -slave -noc40Boot. The -slave parameter ensures that only the machine logged into is affected. See Nigel Dipper's list of restart procedures.

No communication with OMC

Press reset button on 167 card.

C40s problem (no response to c40ping 0)

At navis prompt, WFS Status to see if still framing (frame numbers will be incrementing). If C40 comms lost, try c40CommunicationsRestart. If this doesn't work, NaomiRestart. Don't add '&'! If this doesn't work, try c40Run NaomiInterleaveZeiss (NAD email 15/7/02).

TopGui, IngridAlign no longer recognised

E.g. on typing TopGui, error message is: `libspinbox.so.1 open failed, no such file', or on typing IngridAlign, error message is: `Python not configured for Tk'. LD_LIBRARY_PATH should contain /usr/local/lib (for e.g. libSpinbox.so.1) so change to path suspected. Last edit to .postcshrc removed (the simplextex line), logged out, restarted, but not clear whether change to file, or restart, was critical factor.

Top Gui won't come up

Check that the following processes have all started up on navis by NaomiRestart (list provided by Nigel Dipper 6/02):

> ----------------------------------------------------------------
>      UID   PID  PPID  C    STIME TTY      TIME CMD
>  eladmin 22934     1  0 04:03:33 ?        0:00 BillDoors -v -l /software/Electra/save/log/BillDoors.log
>  eladmin 22945     1  0 04:03:34 ?        0:01 python /software/Electra/bin/Sequencer2.py
>  eladmin 22949     1  0 04:03:34 ?        0:00 /bin/sh /software/Electra/bin/TemperatureCompensation.scheduler
>  eladmin 22880     1  0 04:02:31 ?        0:00 NameServer -addr navis:2000 -p /software/Electra/save/log/ports.log -V -r /soft
>  eladmin 22942     1  0 04:03:34 ?        0:03 python /software/Electra/bin/EPM2.py
>  eladmin 22953     1  1 04:03:34 ?        0:32 python /software/Electra/bin/EPMDataServer

NaomiRestart also starts the following processes on aocontrol1:

/software/Electra/bin/BillDoors
/software/Electra/bin/c40Comms
(/software/Electra/bin/BillDoors.py - Different to above. Not much used)

Note that all of the services started by NaomiRestart belong to a user called 'eladmin'.

Reconstructor apparently not running

The button on the front page isn't activated until you click on the realtime page. It should say 'running', when it is. The start and stop buttons on the realtime page should work. WFS GetReconstructorMatrix returns the matrix. If it contains only zeros, the reconstructor is not running.

Pupil-view mode on WFS

Check pupil-view on WFS. Set camera stage to 'pupil' or 'near_pupil' (usually 'focussed'). Should give superimposed images of lenslets and DM, parallel light impinging on each lenslet.

DM not aligned with secondary mirror.

Check table is not floated.

WFS not aligned with DM

10/02 back end of WFS needed swinging round ~ 4 mm to realign, to fix half-cell shifts of WFS spots.

Not enough light on WFS from pinhole

Check lamp on, beamsplitter in, attenuator set to minimum, NCU mask out (especially after power-cycle), correct pinhole in, covers off, filter and dichroic have sufficient bandwidth.

Light on WFS but not on dectector

Check above, plus INGRID filter and pupil wheels datumed, instrument focus, sensitivity to ambient light.

Not enough light from star

Check above, plus clouds, dome position, target coordinates and mag in relevant band, telescope focus, mirror petals open, Nas flat positioned, table down, pinhole out, beamsplitter out. If any light at all from star, check out-of-focus image (e.g. if half of pupil missing, dome is probably in wrong position).

Too noisy

Check ambient light (moonlight, dome lights, GHRIL room lights, monitors, fisba, LEDs), detector temperature (WFS Peltier and chiller both have to be on), etector read-speed.

INIT does not move a mechanism, INDEX does (search for datum switch).

All mechanisms have status unknown

The fix is to go into the old eng mechs gui, open e.g. the WFS pickoff page, then close it. The mechs status on the TopGui pages updates.

Mechanisms don't move on startup, or stop moving

Jure advises 5/12/05 that if the WFS mechanisms fail, check the status of the EPICS control by with the following:

1. Open Naomi mechanism control GUI
2. Press Naomi systems and WFS components
3. Right click on Device displays (right side of Assembly control GUI) and choose "CCD stage device"
4. On the window that appears press "more Status".
5. On the new window check for error messages, especially the "Action state" status.
6. Report the findings.

The camera stage now has the encoder enabled but the velocity is reduced to 5.

Recipe for recovery: INIT then INDEX from top gui (needed after power-cycling mechs crate). If this fails, go to low-level epics gui via the pickoff.dl gui, INIT, START, INDEX, START. If INIT, START fails, try INIT, STOP first (NAD 9/02). For the pickoff x,y and WFS focus, which are combined under one page, it may be necessary to go another page down to e.g. move the pickoff mirror in x without the camera stage also being asked to move. WFS foreoptics stage has stopped moving in past, because of dirt on the track. NB the devices index from each other. Check NAOMI PSU (below mechs crate) is powered on at back. If this doesn't work, telnet to naomiomc (no login) and type reboot 1 , or power-cycle the crate and initialise the mechanisms.
The WFS carriage is particularly troublesome, summer 2003. If there are problems, record probe and carriage positions, from epics screens, and commands/scripts being used, file a fault report.

Problems with USP mechs.

Go into lower-level epics pages with start_epics from SYS prompt.

Runaway pickoff mirror.

Stop top gui, restart.

Lamp too faint or too bright

From *old* MECHS gui, NCU, calibration lamp, devices, set value between 0 (faint) and 230 (brightest?). See plot in AJL's binder for calibration of intensity vs setting.

Bench not floating properly

Pump up basketballs, but *not* so hard that they prevent table setting onto the mounts when it is lowered.

Most elements can't be moved from gui

Check `secret switch' at bottom RH corner of back of strain-gauge power-amplifier rack, which controls power (e.g. left off after move 12/02). If this doesn't work, try SG ChanPrint with a few values and check last line (strain-gauge ADC sample). If these are floating to max (0xffff) or min (0x0) values, there is a problem with the SG electronics (RM 4/1/03).

Mirror element oscillating (especially no. 18), appears as streak instead of spot

Open and close the SG loop.

DM appearance doesn't change when load flat or mid-range, but does change on opening or closing loop.

naomirestart (synonym for NaomiRestart).

DM appearance peculiar after naomi restart.

At navis prompt, TemperatureCompensation.

Sudden mirror degradation.

TemperatureCompenstion needed if full NaomiRestart not executed.

Use AutoRectifyFlat

DM almost mid-ranges on using AddZernike

Seen 13/6/03 - DM moved successively closer to mid-range with each AddZernike. Reloaded flat, then OK.

FISBA fringes fuzzy or won't stabilise, mirror won't flatten

Check gain ~ 0.8, not e.g. 0.000001. There must be almost no fringe motion(< 1 fringe per 5 sec) for the laser-flattening to work. If there is more than ~ 1 fringe wander per 5 sec, laser-flattening will probably make things dramatically worse. Unstable fringes can be due to:

• Table not up, one of legs making contact with table, telescope oil pumps on, INGRID compressor on in GHRIL control room, people working on telescope e.g. filling cryostats.
• Convection (in the test focal station, check the plastic sheet is protecting the bench from air currents). If at GHRIL, try rotating the WHT in azimuth to reduce prevailing draughts.
• Slow oscillation of the table (the beach balls need pumping up).
• Laser still warming up (intensity rises and falls), usually takes about < 30 minutes, but can take > 6 hours. At the end of 2001, the fringes drifted continuously for days before finally settling down. Seems OK now (6/02).

If laser-flattening still giving poor results, try reducing the x,y,z gains. Or put in pinhole, bright lamps, zero WFS offsets, AutoCentre, close WFS loop, open loop WITHOUT reloading flat to leave last closed-loop demand, SG GrabMirror.

FISBA ftp dies, or display won't come up from eng gui

In GHRIL, restart the FISBA PC from the button bottom left of screen, then zoom display *3 when it comes up (menu button top-left of display). Takes ~ 5 minutes. If it won't reoot satisfactorily, delete all windows, remove sessions on bar at the bottom of the display, double click on microshape ikon. If it won't shut down, close from file button. If still no response, power-cycle the cpu.

FISBA image degraded

The FISBA image may be degraded by dirt in the optics e.g. on the beamsplitter or CCD (or, 6/01, by a lens falling out and leaving a lump of glue close to the focal plane). The FISBA optics can be removed from the back of the lens unit and blown clean. (NB hatching on the displayed image indicates saturation, turn down the intensity.)

FISBA fails with memory error

Check autoguider not running.

FISBA misaligned

The fring-analysis software can cope with x,y shifts and changes of scale as long as the red crosses are realigned (set display to contiguous), but probably not with rotations of more than ~ 2 deg (which might be a problem > 11/05, after the installation of 2 fold flats in the light path).

General WFS problems

At the navis prompt, type WFS Stop, WFS Start, WFS Status. Always hit the WFS Start button on TopGui as well (Nigel's email 17/7/01).

Vignetting, e.g. spots towards top right are fainter than the rest.

Check that the filter wheel isn't in an error state, by attempting to change to another filter. If top gui displays an error state, initialise the filter wheel (bring up the mechs epics gui, then from that select (right click) the filter-wheel gui).

Bright stripes (1000s counts) left and right, high background several 10s counts, bright pixels

Hydraulic cooling has probably failed, check the flow meter. May simply need switching off and on again. 5/06 - looks as though it can't cool both the CCD39 and L3 heads?

WFS display comes up in bizarre colours.

Reset the simplex display from rainbow to grey, and/or kill any ds9 (and maybe netscape) sessions.

Top spot (starlight) in each column on WFS display vignetted.

Unfloat the table!

WFS spots elongated

ADC may be in wrong position, try moving one ADC to see if value of the other changes. May need init and index.

Spots missing along one radial direction.

Malfunctioning mirror petal.

Spots for segments 70, 75 are invisible on WFS display, even though centred on fisba display, but can be made to appear just below their nominal positions if offset in y.

CCD39/L3 switch not correctly switched to CCD39 position. It may feel in the right position, but moving it out and in again may cure the problem.

WFS blank even though it is framing

Hit stop then start on the WFS page.

WFS quadrants going blank at random

This happens especially with 1875-microsec readout, if decimate is set too low (try 20). With 1250 microsec, it always happens, i.e. this mode can't be used.

WFS spots are too faint

Is the bar sticking out of the NAOMI side of the NCU fully out?

WFS illumination asymmetrical (e.g. left half bright, right half faint, spots show large rms on average x, y in WFS align

Power cycle the SDSUs and camera controllers. Camera controllers on first, then SDSU?

WFS centroids jumping all over the place in some quadrants

Could be background light in corner of quadrant (which affects subtaction for that quadrant). Investigate by switching off background subtraction with WFS Testmode?? But sometimes this is not the cause, SDSU reset or power cycle probably needed.

Few spots on the WFS, distributed at random

Check that pickoff mirror is at sensible position, e.g. not 0,0.

SDSU not responding e.g. won't update exposure time

Hard reset by pressing reset on VME card. Wait for fault light to clear. Press reset on BOTH fibre interface cards. In software, either click on INIT on SDSU page of TopGui, OR bring up CamVxWorks terminal, gives 'minimal' prompt, type < startup_cam_no_epics, gives 'naomiCam' prompt, type < readoutC40_cam1, check framing (with WFS Status), close camera terminal.
Don't click on 'continuous' before the C40s are framing.
NaomiRestart does NOT start the SDSUs.
This advice is from Nigel Dipper's email 19/6/01.

< cam1 hangs

Sometimes takes ~ 2 mins

There are patches on WFS where the spots are greatly attenuated.

May be condensation, check the air hose is plugged into the back of the WFS.

White-light flat giving rms ~ 1.e-13 instead of ~ 10s, or fails with divide-by-zero error

In the interferogram .peakH or .peakV files, check for lines starting with #. For these element pairs the interferometry failed. If most start with #, lamp may simply not have been bright enough, OR, was executed through a dichroic with insufficient spectral diversity (use the transparent one). Probably the wlflat has done nothing at all. The last column in the .peak* files gives 1000/S:N. If this number is > 200, data not useful. Also, the wl flat procedure will yield rms = 0 if the SG loop is left open.

White-light flat yielding << 180 measurements.

Check for vignetting on the WFS, see possible causes in section above.

To plot white light flat, type mplot, and follow prompts for interactive or non-interactive use.

For the latter, select the required option:

• DAC values a b c printed at each mirror-element position
• a-b, b-c
• a-b, b-c plotted as vector
• mean of a, b and c = piston
• colour representation of tilt, red = a-b, green = c-b
• colour representation of piston, red = +ve, blue = -ve

Set NORM = 1 in the parameter file to plot difference mirror(i) - mirror(1). The names of the mirror files are given at the end of the parameter file. The file mplot.bad lists the elements on pass-through (rendered hatched on the colour plots). There are example mplot.par and mplot.bad files in ~/crb.

To subtract one flat from another (or add), and display

difflat = ~/bin/DiffFlat or addflat = AddFlat (the ones in ...appsrc don't work)

To plot wl flat in 3D

cdflat, then plotflat3d filename = ~bin/FlatPlotter.py filename

To plot wl flat such that each element is represented by a column of asterisks

plotflathist filename = ~/bin/PlotFlat filename (NB master copy is in /software/Electra/bin).

AddZernike apparently doing nothing

Check correct flat is loaded.

Plot interferometry from white-light flat

E.g. if one suspects the white-light flat is not doing anything useful, try:
wlplot filename.scanH (or .scanV) = WhiteLightPlot filename.scanH (or .scanV) n1 n2
to look at the intensity vs separation plot for adjacent mirror elements n1 and n2. If there's no clear peak, something's wrong. If n1 and n2 are omitted, the procedure plots for all adjacent element pairs. On each plot the 7 curves correspond to individual pixels in the WFS cell. The plot is made twice, first the data then the fit.
If the .residual and .solution files are empty, something's wrong. If you need to recreate the output, try:
WhiteLightProcedure --existingScan (RM 20/7/01)
*** wlplot is not working at 8/02 - library missing. Substitute 5/04: ./scan2txt filename.scanH > junk (RM email 27/5/04). Then run crb's scanpl programme (edit scanpl.par to give junk as name of data file) to plot the fringes for all segment pairs. The .scanH and .scanV files are now in directories e.g. ~naomi/WhiteLight/20050101

Problems with udas detectors

Try dasreset in the DAS window.

INGRID grab stalls

Often during a simplex, the grab stalls for ~ 1 min, then restarts of its own accord. If it stalls for longer, try a dasreset at the INGRID das window (RO 8/02).

http fails

dasreset INGRID at SYS prompt.

Grab won't start - socket error

Probably another IngridAlign running, kill it.

INGRID grab fails (this needs rewriting for Craige's new version)

1. ctrl/c out of navis window which has start-Drama command in it.
2. stop-DramaServer.sh (ctrl/c after a few sec if it hasn't returned the prompt).
3. drama-cleanup
4. Dserver.csh (return probably needed to get prompt back)
5. start-DramaServer.sh (doesn't return a prompt)

IngridAlign --camera SBIG crashes with dtm error or timeout.

Check no other SBIG processes running. Alternatively, the C40s may be down, see above.

INGRID align gui crashes

Happens if set zeros in start X or Y at top left of the gui. Kill window (see below), restart gui.

INGRID grab slowing down intermittently

If the INGRID das has been up several days, restart it, the memory may be getting fragmented. If this doesn't solve problem, rec 9/02 is to power cycle the INGRID controller (240V rocker switch off, wait 10 sec, then switch on). restart -slave 25/9/02 boosted speed from 1.2 to 3 Hz.

Simplex slows to 1 Hz

Reboot navis. The IngridAlign.py interface also has a memory leak, best to exit it after each simples, before starting another.

Simplex yields 2 or 4 symmetrically-disposed spots instead of 1.

Mirror may be split in piston, e.g. could happen after simplexing through narrow-band IR filter. Make sure using broad-band filter, or rerun simplex with more aggressive piston search.

Simplex good in H or J but not the other

If some piston errors have crept in, e.g. after several days without a white-light flat, it's possible for the H simplex to generate a diff-limited pinhole image, but with some mirror elements lambda/2 out. On switching to J, the image is badly degraded. The same will probably happen after simplexing in J and switching to H. The cure is to white-light flatten, or to simplex with an aggressive piston limit. On one occasion (8/02) it appears that this problem may have arisen due to poor laser-flattening (because of slow fringe drift, beach balls needed pumping up), followed by ineffective white-light flats (gradients ~ 1.e-14 in the x direction) for reasons unknown.

Unable to grab.

Swap power and camera-head cables of two black boxes on top of right-hand electronics cabinet in GRACE.

INGRID controller power cycled

dasreset ingrid

Pupil stop needs realigning

1. login to crater as whtmimic, or as ingrid
2. cd /bin
3. start_ingrid, brings up gui
4. select pupil stop wheel, CAD record
5. action = move, move mode = absolute
6. abspos = 197.4, start
7. the wheel moves to a position marked invalid (which is OK)
8. exit
   To get eng control, take out *left* ribbon cable, yields 'PSU drive on' message on the mimic.

Response very slow, e.g. when doing hysteresis.

Check WFS exposure time not set to 50000 (SDSU page).

ds9 won't run up from inside iraf.

Try /opt/astro/bin/ds9.

Autoguider drives straight to limit, e.g. if accidentally close loop with gains set = 0.

restart = NaomiRestart. Autoguider also does this if the AO loop is closed before the TCS loop, usually on next occasion of opening and closing loop.

Navis window won't close

Background menu. right mouse button, select windows, select destroy, position cursor on window, left mouse button.

FSM not moving when loop closed

Check that gain (REALTIME gui) is both set *and* applied.

Peculiar image on the WFS

1. Too faint? Check beamsplitter out, pinhole mask out, WFS exposure time correct, telescope not pulled off by autoguider, correct filter in WFS.
2. Scale of spots has changed? Probably due to telescope focus error (incorrect compensation for flexure with elevation).
3. Spot array expanding and contracting irregularly? Probably a strong focus term in the seeing.

Image split on science camera

may be a reconstructor problem, try turning it off.

DM temperature log

PrintTemperatures 20030715 20030716, where any pair of dates can be used. Log starts at 00:00 on first date. Only works for 48 hours at the moment?
Alternatively, use SJG's /software/Electra/log/Temperature//TempLog (see his email 2/12/03)

Status of WFS

At navis prompt, type WFS Status, for status of the 8 cpus.

Strain-gauge pass-thru

At the navis prompt, SG PrintPass to see which strain gauges are on pass-thru mode. The result is labelled 'cpu11', but the table is the same for all cpus.

What's using the cpu?

At a navis prompt, type top.

Help on commands

At the navis prompt, type WFS or SG to see lists of very-long-name commands.

Using JOSE as an imager

Lenslet array can be unscrewed from reimaging lens. If removed, provides reasonably in-focus images at 0.25 "/pixel.

Inject tip-tilt noise into FSM

Unit in GHRIL control room?

Temperature logging

SGlogData /software/Electra/save/TemperatureLog

Power spectrum of FSM trace

Note that this takes several sec to appear, and is not autoscaled, you may need to change y scale. ou need to login to aocontrol1 and issue the command

DM temperature logging

See RM's email 24/6/02 for location and format of DM temperature logging.

To grab WFS image

WFStoFITS at command line. Directory name given at front of this document.

To GRAB WFS centroid data

login to aocontrol1 and issue the command
diagDowload -dma

To get true exposure time on WFS

EstimateFrameInterval

---

Last Updated: 2006 May 16 Chris Benn crb@ing.iac.es