This manual is not intended as an introduction to IRAF, even though most of the processing routines are written for IRAF. There is copious documentation at the IRAF home page to get you started (see especially A Beginner's Guide to IRAF .) This manual is also not intended as an introduction to CCD data reduction. Again, see some of the documentaion at the IRAF home page (especially A User's Guide to CCD Reductions with IRAF).
This manual starts first with a set of installation instructions. Next is a description of the data files that WFCRED will handle. These are known as "Multi-Extension Fits" files (MEFs). These are a bit more complex than the simple FITS images you may be used to from single chip observations. (The very nature of observing with a mosaic camera argues very strongly for using a MEF style format). The next section gives an overview of WFCRED and the reduction steps that it can perform. The final section describes all of the processing routines available in the WFCRED package.
% cd <destpath>where <destpath> is the destination directory for the package% mkdir wfcred
% cd wfcred
% gunzip -c <tarfilepath>wfcred.tar.gz | tar -xvf -where <tarfilepath> is the location of the tar file you downloaded.
% perl < configAmongst other things, this will create a file called wfc_startup.csh which you'll need later.
% setenv wfcred <destpath>/wfcred/iraf/wfcred/N.B. If you are installing this as IRAF manager, then you can put:
% cl
cl> set wfcred = "<destpath>/wfcred/iraf/wfcred/"
cl> cd wfcred$
cl> mkpkg -p wfcred <platform>where <platform> is the platform alias. Here are a few of the more common ones:cl> mkpkg -p wfcred update
solaris -- ssun
redhat linux -- redhat
dec alpha -- alpha
sunos -- sparcThis last command should build and install the IRAF binaries.
set wfcred = "<destpath>/wfcred/iraf/wfcred/"in the usual location in hlib$extern.pkg and then build the package as normal. Otherwise users will have to put the above lines in their personal loginuser.cl file.
task wfcred.pkg = "wfcred$wfcred.cl"
The important thing to remember about MEFs when using IRAF is that you (in general) must be careful to specifiy explicitly which extension you want to work on. If you have a MEF from the INT Wide Field Camera (1 primary header unit + 4 image extensions) called wfcfile.fit and you wanted to display it like so:
cl> display wfcfile.fit 1You'd get an error like:
cl> ERROR: FXF: must specify which FITS extension (wfcfile.fit)
The trick it to specify which extension you want. One of the ways to do this (and they way we'll always do it in this package) is the give an extension number along with the container file name. So if what we really want is to display the first extension in this file we'd do the following:
cl> display wfcfile.fit[1] 1
where the number in square brackets gives the extension number.
This is by no means a complete description of MEFs as that sort of thing is well beyond the scope of this manual. A more complete (although by no means exhaustive) treatment of the subject can be found in the users guide for mscred by typing:
ms> help mscguide
An even better source is the IRAF FITS Kernel User's Guide which is available at the IRAF home site.
Another key point about WFCRED is that it is designed to allow for minimal user interaction. Although it is highly desirable that data should be processed with no user interaction, in practice this is often not possible, especially at the stage of mean calibration frame construction. Problems with light leaks, overexposed or underexposed flat field images or computer hardware faults can cause problems with the final mean calibration frames if they are not detected right from the beginning. Some of these problems could possibly be detected automatically, but not all. So it's important that a small amount of user interaction is allowed at this stage. This is a small price to pay as once the mean frames are done, then the target frames can be reduced automatically.
WFCRED is also designed so that frames can be reduced in a piecemeal way if that is what you want. That is you can stop after a certain processing stage and look at the data. Then resume the reduction and WFCRED will not attempt any of the steps that have already been done. This is also useful in the event that WFCRED fails due to lack of disc space. Once more disc space has been created you can just resume the processing and it will pick up where it left off.
The data coming from the INT Wide Field Camera was discovered early on to be non-linear atsome level in each of the chips. The source of the non-linearity seems to be in the ADC and it can vary with time. For this reason a non-linearity correction must be done. The values that need correcting are those over and above the mean bias level in the frame. So a mean bias level <B> is calculated from the overscan region. The corrected value, N(x,y) is found from the raw value, R(x,y) via:where L() denotes a lookup table value for that which is in parenthesis.N(x,y) = L(R(x,y) - <B>) + <B>
This refers to the standard practice of debiassing and flat fielding images. Debiassing can be done either by subtracting a mean bias frame or subtracting a constant bias value as found onthe overscan strip of the individual target frames. Flat fielding is done using a previously defined mean flat field frame. The buildingof mean bias and mean flat field frames is discussed later.
Images taken in some red passbands suffer greatly from fringing. To minimise the problem mean fringe frame in scaled and subtracted from each image taken in a passband where this might be a problem. We defer the formation of mean fringe frames to later on in this document.
Mosaic camera images are large to start with (raw Wide Field Camera imagesare 72 Mb each). Once a bit of processing is applied the files essentially double in size as they change from integer to real type data. Squishing refers to a lossy compression whereby the data range is clipped at useful lower and upper bounds (say -1000 and 80000 counts) and then compressed to scaled 16-bit integers.WCS definitionFuture releases of WFCRED will include a catalogue generation routine, a WCS refinement routine and an object classifier.
This is the path to the WFCRED configuration files. Don't ever change this unless you really know what you're doing.
WFCRED can reduce data from several instruments. This is the instrument whose data you want to reduce. The current options available are:More will become available soon.
- wfc -- INT Wide Field Camera
- pfc -- WHT Prime Focus Camera (PFIP)
This defines the source of the astrometric information used for fitting the WCS for the target frames. It can take one of these values:
- local -- A local FITS table will be used. If this is the case the you must also set the value of WFC_ASTCAT
- apmcat -- The astrometric information will be taken from one of the APM surveys. This option requires an Internet connection. It also requires that WFC_ASTSUR be set too.
- any -- A combination of the above. If chosen, then the APM POSS and then UKST surveys are tried. If these fail, then the local catalogue defined in WFC_ASTCAT is tried.
If you choose the apmcat or any option for WFC_ASTSRC, then this parameter needs to be set. It can take on one of two values. Both of these options exclude regions near or in the galactic plane.
- poss1 -- APM catalogue of POSS1 plates.
- ukst -- APM catalogue of UKST plates
If you choose the local or any option in WFC_ASTSRC then this needs to be set to the full path to a fits file with the required astrometric information. This must be a FITS table with four columns:
- id -- (integer) An ID number
- ra -- (real) The R.A. of the object in decimal degrees.
- dec -- (real) The Dec of the object in decimal degrees.
- mag -- (real) The magnitude of the object (any colour).
This is defined so that in case you have any special definitions in your IRAF environment that you want to be used in WFCRED, then you specify a full path to a file called loginuser.cl with these definitions inside.
This is a colon delimited set of full paths to all directories where mean calibration images can either be found or where they can be stored once they've been created.
The algorithm that matches mean calibration frames to target exposures takes the value of this environment variable as a guide to which one to choose. This is a colon delimited menu of preferences. The values can be:A typical value might be "within:nearest:before:after". That is, if there are no frames where valid dates include the observation date, then the chronologically nearest mean frame is chosen.
- within -- Choose a frame that is valid for a range of dates including the observation date of the current frame.
- nearest -- Choose a frame that is valid for a range of dates that is as close as you can get to the date of the current frame.
- before -- Choose a frame that is valid for a range of dates that is just before the date of the current frame
- after -- Choose a frame that is valid for a range of dates that is just after the date of the current frame.
When creating a mean fringe frame there are certain areas of sky with large extended objects which should be avoided (e.g. centre of M31). The file defined in this variable is
an exclusion file. It has the central coordinates of a region to avoid. If the central RA and Dec of the target frame are within a degree of any of these fields, the frame in ignored for the purposes of forming a mean fringe frame. The file has three columns. The first two are the RA and Dec of the fields to be exluded (both in decimal degrees). The third is a label.
When creating a mean fringe frame, images with exposure times less than the value of this variable will be ignored. That will cause all short exposures to be ignored for the purposes of mean fringe frame combinations.
This is a colon delimited list of filter bands where defringing will be necessary.
This is the full path to the IRAF cl startup script. Don't fiddle with this -- it isn't worth it.
This is the full path to the IRAF mkiraf application. This shouldn't ever be changed.
There are currently two preprocessors available for the INT Wide Field Camera. The first is called preproc and is valid for data spanning roughly September 1999 to September 2000. Data from after that time should be preprocessed using pp2. Data before September 1999 needs to be packed into MEFs. Contact the author for instructions on how to do this. The processors can be invoked from the command line:
% pp2 r\*.fitor from the IRAF cl:
wf> pp2 r*.fitNote that on the command line characters like '*' need to be escaped with a backslash to avoid them being interpreted by the shell.
If preprocessors for other instruments are needed for other instruments
or the current ones need updating, these will be added to the package and
this manual will be updated accordingly. It is strongly recommended
that all files be preprocessed no matter how much or how little
processing you plan to do with them.
There are currently two ways of creating mean calibration frames
using WFCRED. Before attempting either you should make sure that
you preprocess all your calibration data.
wf> hselect r*.fit[4][1000:1100,1000:1100] $I '(obstype == "FLAT")' > yukThis should very quickly show you which mean flats are overexposed. Those where the number of counts are say > 30000 and < 10000 can be deleted.
wf> imstat @yuk
wf> delete yuk
A current and persistent problem with the INT Wide Field Camera is that of light leaks. These show up on bias frames and can be a real nuisance. It is worthwhile displaying one image from each bias exposure you intend to use (you only need 10 or so) to make sure this hasn't been a problem (it should show up as slowly rising ramp with no corresponding rise in the overscan). Another problem is interference from the INT autoguider during readout. This quite often shows up on bias frames (as well as others) and manifests itself as a ripple effect in Y. It is by far most prevalent in chip 1 and any bias exposure that shows this should be rejected.
Once you have pruned out all of the offending files you should be ready to invoke calcombine. One final thing to keep in mind is that if you wish to subtract off a bias frame rather than a constant bias then the bias has to be done first. If your raw bias frames are in the same directory as your raw flat frames then calcombine will process the biases first so that the mean bias frame is there and ready for the processing of the flat frames. If you want to keep them separate, make sure you calcombine the biases first. If you don't have any raw bias frames and you still want to use a mean bias frame for processing your raw flats, then calcombine will look for a suitable mean bias frame in the directories defined in WFC_SAVEPATH. calcombine currently has the following options (default in parenthesis):
% calcombineTo change a command line parameter:
% calcombine -biasframe noThis means that calcombine will not be using a bias frame during the CCD processing of flat frames.
If you set the rejmethod command line parameter, you have to
also specify the rest of the rejection parameters that are relevant for
the method you've chosen. In the table below is listed each of the
rejection methods available and the parameters they need. Default
values are in parenthesis.
Method Name | Associated Parameters |
---|---|
none | |
minmax | nlow (1), nhigh(1) |
ccdclip | nkeep(1),mclip(yes),lsigma(3),hsigma(3),rdnoise(0),gain(1),snoise(0) |
crreject | nkeep(1),mclip(yes),lsigma(3),hsigma(3),rdnoise(0),gain(1),snoise(0) |
sigclip | nkeep(1),mclip(yes),lsigma(3),hsigma(3) |
avsigclip | nkeep(1),mclip(yes),lsigma(3),hsigma(3) |
pclip | nkeep(1),lsigma(3),hsigma(3),pclip(0.5) |
Thus to change the rejection method on the command line you'd type (for example):
% calcombine -rejmethod minmax -nlow 2 -nhigh 3
% caltool <directory> [-dtoi yes|no] [-biasframe yes|no]where <directory> is the name of the directory where your raw flats and biases are. The
dtoi
If set, the flat data will be linearisedbiasframe
If set, then the bias will be subtracted from flat frames using a mean bias frame. Otherwise a constant bias will be determined from the overscan region of each flat frame.
This is the directory where the raw frames reside. Changing this at any time will cause caltool to search that directory for raw calibration files. If this directory has many files, this could take a bit of time, so be patient.savepath
This is a colon-separated list of directories where mean frames can either be found or stored. At startup time the default value is taken from the user's value of WFC_SAVEPATH.filter
The first thing that caltool does when it has all the files it needs from the current directory (directory) is to group them by filter. All of the available groupings are listed in this box with the number of frames found for each noted in parenthesis after the group designiation. As there may be more than one filter used with the same passband the grouping is by both passband name and filter identification number, so the group designation will be the passband name and filter ID, separated by a colon. Bias frames will have the designation "BIAS:0". Clicking on one of the filter sets will cause it to be highlighted and become the current group.exposure sets
When one of the filter sets is highlighted, this lists all of the run numbers that comprise the current group. By default all of the runs are 'selected' -- that is if you don't deselect them, they will form part of the next mean frame you make for the current group. Click on the run numbers to either select or deselect. There is one key binding in the exposure sets window. Pointing at a particular run number and typing the letter i brings up the information menu. This allows the user to examine the file header and display the images. To display the images you must be sure to have a display server running ahead of time, such as ximtool or ds9.mean calibration frames
If a filter group has been selected, then caltool searchs through the directories specified in savepath to see if there are any mean frames that correspond to the current filter group. If so, it displays them in this window. When frames are found in the savepath directories, their names are followed by an asterisk. When a new mean frame is created, it will be displayed in this window, but without an asterisk until it is explicitly saved. To save a new mean frame, just point at the file name in this window and type s. The save menu requires that you give two dates (format yyyymmdd) which correspond to the dates for which this frame will be valid. You can also delete a frame from the save menu. The information menu is also available in this window by pointing at a frame and typing i.dialogue
This is just telling you how things are going during processing.quit
Quits out of a session of caltool. If there are mean frames you have created, but not yet saved, you'll be asked what you want to do with them.abort
If you are in the middle of a processing cycle and you want out, then this button kills the processing without killing the whole session.help
Push this to get the help window.proc
Once a filter group has been highlighted and frames have been chosen from the exposure set window, pressing this button causes the processing cycle to start. The abort button will kill this if you've made a mistake. Messages in the dialogue window should keep you informed of the progress of the processing cycle.files
Allows the user to specify what type of files will be used in forming the mean frame. Note that this is ignored for bias frames. For flat frames you can form a mean frame from either frames that are designated as flats in their headers, or you can use all frames of the appropriate filter group, be they flat or target in the header.reject
Allows the user to select which rejection algorithm is wanted. Depending upon which algorithm is chosen, a second menu will pop up with parameters for that method. These are all the same as described above for caltool.options
There are currently two additional options:As before there is one final thing to remember. If you are subtracting a bias frame from the flat frames during processing, you must make sure to have a mean bias frame available in one of the directories in savepath. This means either having one there before you fire up caltool or making sure you process your bias frames in caltool before doing any flat fields.Do D to I conversion (yes)If selected then the raw flat field data will be linearised.Subtract a bias frame (yes)If switched on then the bias will be removed from flat frames with a mean bias frame. Otherwise it will be estimated from the overscan region.
A quick word is needed regarding the names of the output files. WFCRED separates all files by their observation type and by their filter. The filter is specified by both a band name and an id number (if this is available for the particular instrument). Each mean frame is also assigned a time stamp to say when it was created. These are combined to form the output file name. (Much more about the way in which the file was formed is included in the header). For example a recently created flat field frame for the Wide Field Camera was called: i:215_983787471.fit. The 'i:215' refers to the filter band and the id number of the piece of glass. This distinction is important since it is known to happen that a person will observe, for example, with both KPNO and Harris B filters on the same night. Bias frames will be called something like BIAS:0_983787381.fit.
There is a third type of calibration frame that is relevant to WFCRED and that is the mean fringe frame. These have to be formed much further down the processing line and hence are discussed in the next section.
apm (no)
If set, then an image detection will be done on the current frames. This will be automatically set if the wcs switch is set. The difference is that if you specify that you want the WCS to be fit, but don't switch on apm then the object tables will be deleted.biasframe (yes)
If set then the bias will be correct by subtracting a mean bias frame. If not, then a constant bias will be subtracted.defringe (yes)
If set, then the frames in the wavebands listed in WFC_FRFILT will be defringed.dtoi (yes)
If set, then the data will be linearised.realout (yes)
Frames are processed using real arithmetic. If this is set then the resulting output will be a 32 bit real frame. If this switch is unset, then the resulting output will be unsigned short data which is also 16 bit.squish (yes)
If real data are chosen via realout then it is still possible to save some space by using squish. This process is described earlier in this document.wcs (yes)
knit
This takes a MEF that has had a WCS applied and writes it to a single image. Although this may not be a very accurate rebinning, the orientation and spacing of the chips is quite close to the truth. This can be quite useful if you want to judge the quality of a reduction. e.g.makegif% knit <input MEF> <output file>
This takes all the MEFs in the current working directory and creates a GIF image for each FITS image. Can be useful for advertising.
This routine does an image detection analysis similar to that done by the APM machine. This is not intended as a definitive image analysis of a processed frame, but rather a good guide to be used in defining the WCS. The output is a set of STSDAS tables (.tab) files, one for each image in the input MEF. The information in these can be accessed using any of the applications in the tables package in IRAF. The columns in the output files are as follows:
column name description x X coordinate of object (pixels) y Y coordinate of object (pixels) total Total intensity of object (adu) area Total number of pixels covered by the object (pixel**2) peak Peak intensity of the object (adu) xxterm XX cross-term xyterm XY cross-term yyterm YY cross-term eccentricity Eccentricity of the object p0 - p7 First 8 areal profiles (pixel**2) The parameters for apm are:
input
List of input MEFs.(pnop)Minimum number of pixels an object must occupy before it can be considered an object.(thresh)This relates to the detection threshold. If thresh is positive, then the detection threshold is thresh ADUs above sky. If thresh is negative, then the detection threshold is the absolute value of thresh times the sky noise above the local sky.(grid)The first thing apm does is to divide the input image into a set of subrasters with a maximum dimension of grid pixels on a side. This is used to determine the broad structure of the background. Later a two point interpolation of this background map is performed in to estimate a sky value for each pixel in the image raster.(verbose)This is set if you want diagnostic information.(keepback)If you want to keep the background map, then set this parameter. The background map will be stored in a file called <MEFname>_<ext#>.bg.fit where <MEFname> is the name of the original MEF and <ext#> is the extension number.
This program takes a mean fringe frame and scales it such that a maximal amount of fringing will be removed from each image. Both object and fringe frames are set to zero mean and large scale variations across the field are taken out. A scale factor based on the exposure times and the background means of each frame is calculated as well as a window. The fringe subtraction is done at both extremes of the window as well as at the calculated scale factor. The quality of the fringe removal is judged by the mean absolute deviation of the subtracted frame. Given these three 'fits' a better scale factor and smaller window is calculated. The iterations continue until the window size is small enough so that it isn't worth continuing. The final scale factor is used to subtract the fringes. The mean background and large scale structure is added back in. The parameters are:inlist
List of MEF files that need to have fringes removed.frframeA mean fringe frame. No checking is done to make sure the filters match, so that's up to you to get right.(opmode)This can either be auto or manual. If auto then the final scale factor is calculated using the algorithm outlined above. If manual then no fitting is done and the final scale factor is given by the user in the parameter scfac.(scfac)If operating in manual mode, then the scale factor to be used should be entered here. Otherwise this parameter is ignored.(verbose)Switch on for verbose output. Otherwise it operates silently.
This routine does the linearisation of the raw data as described in Section 3.2.
The parameters are:input
List of input MEF files requiring linearisation.lookupsThis is the name of the container file with the lookup tables for each chip in the MEF. There are a number of these available in the standard directory wfcred$stds/lintabs. No checking is done that the linearisation information is appropriate for the input MEFs -- that's up to the user. Each table only has one column called y and 65536 rows. The value of y for the ith row is the replacement value for an input data value of i.
As described in Section 3.2 it is sometimes desirable to save space by converting 32-bit real images to scaled 16-bit integer images. The program squish windows the data to minimum and maximum values and then scales them so that they can be written as 16-bit values. This is slightly lossy, but a reasonable compromise.
The parameters are:input
List of input MEFs to be squished.datamin, datamaxThe minimum and maximum data values. Any data above the maximum or below the minimum is set to datamax or datamin respectively.maxbscaleThe maximum value of bscale allowed.logfileA log file for information on the scaling employed for each image. If left blank, no logging is done.
For a list of MEFs, object x,y coordinates are matched up with astrometric standard positions. These are then used to map the (x,y) -> (RA,Dec) for the image. The WCS information is written to the header of the image. Before this routine can be used, each MEF must have had apm and wfcwcs run. These do the image detection and set a rough WCS. The parameters areinput
List of MEFs that need WCS information(interact)This is set if the actual fitting is to be done interactively. Otherwise it will be done silently.(catsrc)This defines the source of the astrometric information used for fitting the WCS for the target frames. It can take one of these values
(survey)
- local -- A local FITS table will be used. If this is the case the you must also set the value of catname and catpath.
- apmcat -- The astrometric information will be taken from one of the APM surveys. This option requires an Internet connection. It also requires that survey be set too.
- any -- A combination of the above. If chosen, then the APM POSS and then UKST surveys are tried. If these fail, then the local catalogue defined in catname and catpath is tried.
If catsrc is set to apmcat then this specifies which of the APM surveys will be used. This can either be:(catname)
- poss1 -- Palomar Sky Survey (Northern)
- ukst -- UK Schmidt Survey (Southern)
If catsrc is set to local then this has to be set to the FITS table with the astrometric standard information. The format for this is described in section 4.3(catpath)If catsrc is set to local then this has to be set to the full path to the catalogue defined in catname.
A dead reckoning rough WCS is given to each image extension based on header information. This WCS will be good to 10-15 arcseconds. The parameters are:inlist
List of input MEFs needing a rough WCS.