Basic CCD Reduction, TO training, part 1
In this exercise we explain the basic steps in preparing the CCD images for
spectal extraction.
If IRAF is used for the first time on an account run 'mkiraf'. Now start
'xgterm -sb &' (-sb to get a scroll bar) and 'ximtool &', then start
IRAF in the xgterm and cd to were your data is.
cl> imred
im> ccdred
Getting started
First of all we have to read the fits files and convert them to IRAF
format. We use 'list_fit' and 'list_imh" to define the fits data source
and the IRAF filename to be given. They are simple text files containing
the names of the files with one name per line. The fits files you find
in the 'fit' directory. First you have to copy them into your working
directory. You may use 'list_fit' as a template when creating 'list_imh'
useing EMACS, selecting edit, query replace.
cc> lparam rfits
fits_file = "@list_fit" FITS data source
file_list = File list
iraf_file = "@list_imh" IRAF filename
(make_image = yes) Create an IRAF image?
(long_header = no) Print FITS header cards?
(short_header = yes) Print short header?
(datatype = "") IRAF data type
(blank = 0.) Blank value
(scale = yes) Scale the data?
(oldirafname = no) Use old IRAF name in place of iraf_file?
(offset = 0) Tape file offset
(mode = "ql")
cc> rfits
Now you can save disk space by discarding the redundant fits files.
cc> !rm *.fit
Combining the bias frames: zerocombine
In principal the first stage of the process is to prepare the bias frames
for use.
Display a bias and check for structure, if there is none you don't need
zerocombine! You will see there is no structure, hence zerocombine isn't
needed.
Also, bias frames can tell you if there are any offsets between overscan
and the 'real' bias level. However, the bias level can change during
night so one has to be careful intepretating these values.
Observers should check for structure taking a bias. If there is, they will
take many (at least 25, so they are not dominated by readout noise)
'zero second' exposures which need to be combined into a single image,
hence the current task zerocombine.
In the example below, 'list_bias' is a text file containing the names of the
bias frames to be combined with one name per line of the file. If there are
very few such frames, you can give zerocombine the image names directly via
eparam. The output is a single image called 'biascomb'. Most parameters can
safely be left at default values; however ccdtype needs to be set to ' ',
since in general, the image type is not specified correctly in the image
header. The image type needs to be correct for the next stage, so correct it
if necessary (use hedit to set the 'imagetyp' of biascomb to 'zero').
cc> lparam zerocombine
input = @list_bias List of zero level images to combine
(output = "biascomb") Output zero level name
(combine = "average") Type of combine operation
(reject = "minmax") Type of rejection
(ccdtype = "") CCD image type to combine
(process = no) Process images before combining?
(delete = no) Delete input images after combining?
(clobber = no) Clobber existing output image?
(scale = "none") Image scaling
(statsec = "") Image section for computing statistics
(nlow = 0) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject (neg
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(pclip = -0.5) pclip: Percentile clipping parameter
(blank = 0.) Value if there are no pixels
(mode = "ql")
cc> zerocombine
cc> hedit biascomb imagetype "zero"
As mentioned above zerocombine isn't needed for our data reduction
exercise, we continue therefore without using the result 'biascomb'.
Subtracting bias level and trimming
We use the task 'ccdproc', this task incorporates a veritable host of minor
functions, all combined into the one easy-to-use task. We will use 'ccdproc'
later again. Now we use it for subtracting the average over the columns in
the overscan region (to remove any frame-to-frame variations in the average
zero level). A fit (generally a constant) is performed on the overscan region
of each image as a function of line number and is then subtracted from all
columns of the data part of the image.
After subtraction of the overscan region the image is trimmed to leave just
the part containing useful data.
It is necessary to specify the portions of the image containing data and the
overscan with the parameters biassec and trimsec; these may be determined by
inspection with a task such as implot. The format for biassec and trimsec
is: [start column:end column,start line:end line].
To find out how to use 'implot' examine the help page. Then determine the
values for biassec and trimsec.
cc> help implot
cc> implot r62506 (press 'r' for redraw, 2 times 'e' to define window,
:c 500 to plot column 500, :l 50 to plot line 50)
Some example values for biassec and trimsec are included in the parameter
listing below. These serve as an approximation, but you should check (with
implot) to be sure.
The ccdproc task can also correct bad pixels by interpolation (if
fixpix=yes) in this case a file listing known bad pixels for the CCD used
is required.
There are quite a few other things ccdproc can do, we will see more later on.
cc> lparam ccdproc
images = "@list_imh" List of CCD images to correct
(ccdtype = " ") CCD image type to correct
(max_cache = 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?\n
(fixpix = no) Fix bad CCD lines and columns?
(overscan = yes) Apply overscan strip correction?
(trim = yes) Trim the image?
(zerocor = no) Apply zero level correction?
(darkcor = no) Apply dark count correction?
(flatcor = no) Apply flat field correction?
(illumcor = no) Apply illumination correction?
(fringecor = no) Apply fringe correction?
(readcor = no) Convert zero level image to readout correctio?
(scancor = no) Convert flat field image to scan correction?\n
(readaxis = "line") Read out axis (column|line)
(fixfile = "") File describing the bad lines and columns
(biassec = "[3:46,*]") Overscan strip image section
(trimsec = "[52:1074,*]") Trim data section
(zero = "") Zero level calibration image
(dark = "") Dark count calibration image
(flat = "") Flat field images
(illum = "") Illumination correction images
(fringe = "") Fringe correction images
(minreplace = 1.) Minimum flat field value
(scantype = "shortscan") Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines\n
(interactive = no) Fit overscan interactively?
(function = "legendre") Fitting function
(order = 1) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
cc> ccdproc
Combining the flat field frames: imcombine
A second type of calibration frame is the flat field frame. A flat field is
an image of a uniform intensity distribution, e.g. the dome flat-field
screen or the twilight sky, (the latter is to be preferred). This is used to
determine and correct for the relative spectrograph and detector response.
Imcombine combines any number of frames in similar fashion to zerocombine,
the ouput file in this example being called 'flatcomb'. There
is one important difference though. Since the exposure time is non-zero,
there will inevitably be cosmic ray events or tracks (CREs) on the image.
Imcombine uses an algorithm to weed these out. Note that median uses the
average of the two central values when the number of pixels is even.
cc> lparam imcombine
input = "@list_flat" List of images to combine
output = "flatcomb" List of output images
(plfile = "") List of output pixel list files (optional)
(sigma = "") List of sigma images (optional)
(logfile = "flatcomb.log") Log file\n
(combine = "median") Type of combine operation
(reject = "none") Type of rejection
(project = no) Project highest dimension of input images?
(outtype = "real") Output image pixel datatype
(offsets = "none") Input image offsets
(masktype = "none") Mask type
(maskvalue = 0.) Mask value
(blank = 0.) Value if there are no pixels\n
(scale = "none") Image scaling
(zero = "none") Image zero point offset
(weight = "none") Image weights
(statsec = "") Image section for computing statistics
(expname = "") Image header exposure time keyword\n
(lthreshold = INDEF) Lower threshold
(hthreshold = INDEF) Upper threshold
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject (neg (mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(sigscale = 0.1) Tolerance for sigma clipping scaling correction (pclip = -0.5) pclip: Percentile clipping parameter
(grow = 0) Radius (pixels) for 1D neighbor rejection
(mode = "ql")
cc> imcombine
Normalizing the combined flat frame
The IRAF package 'kpnoslit' must be loaded first, the task 'response' will
be used.
The normalization spectrum is obtained by averaging 'flatcomb' across the
dispersion to form a one dimensional spectrum and smoothing the spectrum
by fitting a function.
The image header keyword DISPAXIS must be present with a value of 1 for
dispersion parallel to the lines. This parameter must be added using hedit,
as it isn't existing yet.
cc> kpnoslit
kp> hedit flatcomb DISPAXIS 1
The fitted function may be examined and modified interactively when
the parameter interactive is set. In this case the normalization
spectrum and the fitted function or the residuals of the fit are graphed.
kp> lparam response
calibration = "flatcomb" Longslit calibration images
normalizatio = "flatcomb" Normalization spectrum images
response = "flatnorm" Response function images
(interactive = yes) Fit normalization spectrum interactively?
(threshold = INDEF) Response threshold
(sample = "*") Sample of points to use in fit
(naverage = 1) Number of points in sample averaging
(function = "spline3") Fitting function
(order = 1) Order of fitting function
(low_reject = 0.) Low rejection in sigma of fit
(high_reject = 0.) High rejection in sigma of fit
(niterate = 1) Number of rejection iterations
(grow = 0.) Rejection growing radius
(graphics = "stdgraph") Graphics output device
(cursor = "") Graphics cursor input
(mode = "ql")
kp> response (press 'w''e''e' to define window, 'w''a' to get rid of window,
: order 10 to change fitting function, 'k' to get ratio,
'h' to return)
We leave the kpnoslit package now and go back to the ccdred package.
kp> bye
Flat field correction
We now come to use 'ccdproc' again to apply the flat field correction.
After this the data will be ready for spectral extraction.
A few parameters have to be changed.
cc> eparam ccdproc (only parameters listed which have to be changed)
images = "@list_div" List of CCD images to correct
(overscan = no) Apply overscan strip correction?
(trim = no) Trim the image?
(flatcor = yes) Apply flat field correction?
(flat = "flatnorm") Flat field images
cc> ccdproc
If you also want to subtract the zero frame, you should set zerocor=yes
and zero=biascomb.
You might ask yourself why we used the task 'ccdproc' twice, we could have
done it all in one go!
In theory, the effect of not subtracting bias level from flat field before
normalizing has got only a marginal effect if the bias level is much less
than the flat field level and the bias frame is nice and smooth.
Not trimming (exculding overscan strip) the flat field before normalizing has
however a practical impact. 'response' and other tasks uses a fitting routine
that runs across the whole frame. If the overscan is NOT removed before
running 'response', you will have problems when you reach the border between
the flat field signal (typically counts ~40000 ADU) and overscan (few
100's ADU) since the fitting rountine will average these values. This means,
that your fit will 'tip over' near the borders. A way to avoid this, would be
to exclude the overscan regions interactively when you use response (see
online help on how to exclude regions).
It is recommended, however, to split ccdproc up in two parts. The time
overhead in running the task twice is not very big.