Quick Start Maunal for CRUSH. (Updated for 1.60)



Table of Contents
=================

	1.  Installing CRUSH
	2.  Running CRUSH
	3.  The Tools of the CRUSH Suite
	4.  Reducing Multiple scans
	5.  Pipeline Configuration
	     Pipeline elements
	     Source Model Placement in the Pipeline
	6.  Time Constants
	7.  SHARC2 In-band Opacity
	8.  Pointing
	9.  Calibration
	10. High-Pass Filtering
	11. Extended Sources
	12. Deep Field and Faint Compact Sources (-deep)


1. Installing CRUSH
===================
Read INSTALL.


2. Running CRUSH
================

Run crush from within the directory. Most scans shold reduce properly 
without specifying any options. Sometimes you may wish to override some of 
the scan specific options, like pointing (-fazo=, -fzao=), or coordinate 
epoch (-epoch=) or rotator zero angle (-rot0=) which were recorded 
erratically during the January 2003 run. To see what options you have 
available run 'crush [options] -help', or simply 'crush' without any 
arguments. For available configuration keys, see crush.cfg.


3. The Tools of the CRUSH Suite
===============================

CRUSH really is a collection of useful tools. Here's a short description
of what is there and what they are used for. Each tool, when run without
arguments (or with the -help option) will show you a list of available
command-line options.


crush		The principal reduction tool. 'crush' is meant to deal
		with raw SHARC-2 data files mainly to produce source maps
		out of them. (But there are a few other functions, like
		making ascii logs).

coadd		Add FITS images together. Use as a last resort tool as it is
		always better to reduced scans together with crush.

show		A simple display program for the FITS files, with various
		useful functions for simple analysis and image manipulation
		(think of it as a simple ds9 specifically for SHARC-2 images)

imagetool	A tool for manipulating SHARC-2 images. As of 1.60 it can
		also read and manipulate BoA images, and starting from 1.61
		the capability has been added to read single HDU (flux only)
		images as well, making it a generic purpose tool for all
		sorts of astronomical images

calibrate	A tool for deriving calibration information into an ascii 
		file.

difference      Difference two images. The result is either displayed or
		writen into a new image file.

histogram	Provide a histogram of signal-to-noise distribution of
		a SHARC-2 image, BoA image or a generic FITS image.

jiggle		Align multiple SHARC-2 images w.r.t. one-another.

covarsee	A visualization tool for SHARC-2 pixel-to-pixel covariance
		files (prdiced via the crush -covar= option).

detect		A source extraction tool for SHARC-2 maps. You should make
		sure that the noise is close enuogh to Gaussian (e.g. with
		'histogram') before fully relying on this. Nevertheless,
		the algorithm tries to adjust confidence levels based on
		observed deviations from Gaussian distribution. 


4. Reducing Multiple Scans
==========================

CRUSH is designed to be able to reduce multiple scans together. Indeed,
this is the prefered way of dealing with large data sets, rather than
reducing scans separately and co-adding them later.

Clearly, scans can be taken over several days, under different settings
and conditions. This is why CRUSH has options that affect a set of scans
only. These 'Scan Specific' options can be used on the command line
preceding the set of scan numbers they affect. 

The most important scan specific options are those relating to pointing
(-fazo= and -fzao=) and calibration (-scale= or -Jy=). 

E.g. say you have two sets of scans 8287-8291 and 8294-8298. You determined
that the pointing for the first set is (-101.3, -26.0) and for the second
set it is (-103.4, -29.2), while your calibration scans suggest that you
want to apply the calibration factor 1.42 and 1.53 respectively to the sets.
In this case,

crush [options...] -fazo=-101.3 -fzao=-26.0 -scale=1.42 8287-8291 \
  -fazo=-103.4 -fzao=-29.2 -scale=1.53 8294-8298

will apply the proper corrections to each set of scans.

Read sections 8-9 for more on pointing/calibration.


5. Pipeline Configuration
=========================

The CRUSH reduction is based on sequentially estimating solutions to the
various phyisical signals that are present the time-stream. To do this
successfully, the signal that is estimated at each stage must be 
dominant at that given stage (i.e. after brighter signals have been 
extracted from the data). For the most part, the default pipeline is
set up this way based on the knowledge of relative signal strengths. 
However, this does not include source strength, which may vary 7 orders
of magnitude (from Jupiter to the faint high-z galaxy), and therefore
special attention has to be given to where the source model is inserted
into the pipeline.

The CRUSH default is to try obtain a solution with the default pipeline
order, and then to reconfigure that pipeline for optimal order, based on
extracted signal levels. This works well for the most part, but you can
retain more control if you wish to. Read on to see how.


Pipeline elements
-----------------

The following pipeline elements are available (as of 1.50), in their
default ordering. 

	PixelOffsets
	CorrelatedNoise
	PixelWeighting
	ResidualSpikeRemoval
	PixelGains
	PixelNonlinearities
	TemporalPixelTest
	ChopResiduals
	SkyPolynomials
	TimeWeighting
	RowBiasDrifts *or* RowPolynomials
	MUXOffsets
	SourceMap
	PixelDrifts
	SpectralPixelAnalysis
	RegionalCorrelations *or* BlockCorrelations
	AccelerationResponse

You can set up your own pipeline, if you will, via the 
'pipeline.new = true' and 'pipeline.add' configuration keys.


Source Model Placement in the Pipeline
--------------------------------------

One way to explicitly specify how bright your source is, is to do this 
by loading one of the supplied configurations with the appropriate 
command line flag. To use the configurations, the following flags are 
defined:
	
	-default	Any (default).
	-bright		> 1,000 Jy
	-faint		< 1 Jy (can be extended)
	-deep		< 100 mJy (point like)

Your other option is to set the brightness more directly either by the
pipeline.map key in a config file (e.g. crush.cfg), or by the
-map= option in the command line. The argument specifies where to
insert the source model in the pipeline relative to other pipeline
elements. The following arguments are understood:

	first		Source Model should be the first model in the 
			pipeline.

	last		Solve for source structure last.

	before:<model>	Solve for source just before solving for
			<model>.

	after:<model>	Solve for source after having solved for
			<model>.		

E.g. '-map=after:PixelGains' in the command line will put the source
model after the "PixelGains" model in the pipeline.


6. Time Constants
=================

Some models have characteristic time constants associated to them, which
determine how often is the model approximated on the data. These time
constants are given by an integer number that is a multiple of the
recorded frame rate (36ms frames are default). A value of 30 is equivalent
to ca. 1 second.

Most of these should be set optimally in crush.cfg, but you can choose to 
change them and see what happens. 

Values of 'auto' are interpreted for -driftT=, -skyT=, -rowT=, -muxT=  and 
-regionT= flags (PIXEL_DRIFT_T, SKY_POLYNOMIAL_T, ROW_POLYNOMIAL_T, MUX_T, 
REGION_T keys), which when defined, will attempt to determine the optimal 
setting based on the first scan supplied. This is still somewhat 
experimental for the time being, so use it with care if really want to.

Another option '-TCScale=<value>' scales all time constants by a given
factor. This may be useful when facing especially unstable conditions. 
E.g. try '-TCScale=0.3' to deal with data taken in very variable weather.


7. SHARC2 In-band opacity
=========================
A linear tau dependence can be described to a parametric tau value. 
This is done by the DEFINE_TAU key or -tauScale= flag. Both take an 
argument of the form:
                                                                                
	id(A,B) where tau(id) = A * t + B
                                         where t is the tau parameter.
                                                                                
By default t is equivalent to the 225GHz tau at zenith. If you want to 
change this, and make the parameter to which other opacities are related 
become some other value, say the 350um tau, you can do this by setting the 
argument to:
                                                                                
	350um(1.0,0.0)
                                                                                
This will set tau(350um) = 1.0 * t + 0.0, that is t = tau(350um).
                                                                                
The reserved id's used by crush are:
                                                                                
	'225GHz'    to specify the 225 GHz radiometer tau.
	'350um'     to specify the 350 um dipper tau.
	'Sharc2'    to specify the tau in the Sharc2 pass-band.
                                                                                
By thefault 350um filter operation is assumed. If the data was taken by 
another filter setting, you must adjust the tau(Sharc2) value accordingly. 
You may do this by uncommenting the appropriate definition in crush.cfg 
(these are rather approximate for the time being), or by specifying it 
directly in the above described fashion.
                                                                                
In concert, the '-tau=id:value' flag or the corresponding TAU_SOURCE
configuration key can be used to determine tau. E.g. to explicitly specify 
a 225GHz tau value of 0.042, use the flag: 

	-tau=225GHz:0.042

To use the Mai-Tau daily tau fits for the 350um tipper data, use:

	-tau=MaiTau:350um

Notice, that the TAU key in the output FITS file is now carrying the 
in-band tau value that is parametrized by the id 'Sharc2'.


8. Pointing
===========

Reducing the data with the correct pointing can be crucial (esp. when 
attempting to detect/calibrate faint point sources). The CSO pointing 
model is not always perfect, and most of the time it requires fine
tuning. Luckily, since SHARC2 is a 2-D array, getting the exact pointing
offset wrong at the time of the observation has no seriuos consequence
as long as the source still falls on the array, and as long as the exact
pointing can be determined later. If you, at the time of the data 
reduction, have a better guess of what the pointing was at the time the
data was taken (e.g. by fitting a model to the night's pointing data), you 
can specify the pointing offsets that you believe were more characteristic
to the data, by using the -fazo= and -fzao= options *before* the scan 
numbers that should be reduced with the updated pointing. E.g.,

	-fazo=-103.5 -fzao=24.3

You can also make pointing changes after the reduction (alas, now in RA/DEC).
You can read off the apparent source position from each separately reduced
scan (e.g. by using 'show' and placing the cursor above the apparent source
center/peak). Then you can edit the individual FITS image headers (e.g.
using the '-changeKey' option of 'imagetool' avaiable from the SHARC-2 web site),
by putting the apparent RA/DEC pointing offsets into the RAP and DECP fields.
E.g.,

	> imagetool [...] -changeKey=RAP:3.0 -changeKey=DECP:-4.5 [...]
 
Then, 'coadd' will add these images together with the proper alignment. (Also, 
all other crush utilities, like 'show' or 'detect' will include this in their 
astrometry.)
Clearly, this method of adjusting the pointign is only practical if your source
is clearly detected in a single scan.

Thirdly, the tool 'jiggle' can be used to determine the above pointing offsets
through an interactive display in which the individual maps can be moved
w.r.t one-another. The cursor arrows will move the selected map in
the desired direction, while '+', and '-' can be used to change the map that
is selected for movement. Currently, 'jiggle' does not itself update the
RAP, DECP keys with the determined values. It only suggests what values
you will have to enter yourself in the FITS images.


9. Calibration
==============

A lot of effort has been spent to provide you with the best possible
calibration in CRUSH. At present the built in calibration is excellent
and only beam quality variations prompt an order of unity correction.

The built-in calibration accounts for line-of-sight opacity in the SHARC2
passband and the detector non-linearities. 

MAP_VOLTS_PER_JY configuration key or -mapJy= option flag determine the
calibration for the output map. The Calibration is defined as the Voltage 
(V) at the detector stage for a uniform gain pixel for a 1Jy per SHARC2 
pixel area (4.85 arcsec^2) source (NOT 1 Jy/beam!!!). The supplied 
calibration is a mean calibration for past SHARC-2 runs, and is only 
approximately correct (30-50%) for your purposes given variations in beam
size and shape (focus quality and DSOS changes).

Once the beam quality correction is taken into account, the residual 
calibration of CRUSH-1.40 is expected be ~10% or better.

As of v1.32, CRUSH comes with improved 850um calibration, thanks to Darren.

Please read the relevant documents in the doc/ sub-directory of the
distribution for detailed calibration procedures and corrections. While these
documents are fluid, once their contents are finalized, they will be made
available via the crush website www.submm.caltech.edu/~sharc/crush.

10. High-Pass Filtering
======================

As of 1.2b6, there is a possibility of high-pass filtering the time 
stream. This may have benefits when the source is relatively compact, 
and/or the scanning speed is large. For s source with angular size D, and 
scanning speed v, you want to set the filter time scale >> D/v. The 
setting 'auto' will attempt to do just that by setting the time scale to 
10 D/v. Beware thuogh, that the high pass filtering may affect your 
calibration if as some components of the source will inevitably filter out 
as well (thuogh their contribution may be kept reasonably small by 
carefully selecting filtering parameters). However, high-pass filtering 
may present an option for reasonably looking maps under poor 
stability, especially for large box scans with faint structures.
The filter is set by the -hipass= flag or "HIGH_PASS_FILTER" key. The 
value 'off' diables the filter (default).


11. Extended Sources
====================

Extended sources provide the most challenge, both because the redundancy 
of a larger scan pattern is lower -- making it harder to disentangle the 
various terms from the data, -- and often because some instrument specific 
terms (row drifts) and sky instabilities (sky gradients & turbulent 
artefacts) can interfere with faint extended source structures. 

As of 1.32b1, the default is to assume extended structures. This feature
can be toggled by the -extended/-compact flags, or with the 
EXTENDED_PRESERVE_TURNS configuration key. When reducing in the extended mode,
models that have sensitivity to extended structures are turned off until
a specified number of source model generations have been obtained (controlled 
by -Iextended= flag or EXTENDED_PRESERVE_TURNS key -- default is 2).

Warning! The trade-off is simple: to keep extended structures, you will likely
have to put up with more map artefacts or, you can clean the maps better by
risking the removal of larger scale flux (on the scale >= array size). 
When attempting to map faint extended structures, you are trying to do 
something very difficult, if not impossible. You'll have to rely on better
weather stability and even then will have to put up with higher levels of
map noise on all spacial scales.  

Here's a little FAQ to help dealing with difficulties:

	Q: Baselines are not flat, what can I do?

	A: Your best bet is to change the position of the source map in the
	   pipeline. (via -bright, -faint flags, or via the -map= flag)

 	   If the source precedes the correlated noise model, then the 
	   variation in the sky emission will be responsible for the baseline. 
	   Move the source after the correlated noise model..

	   More likely, though, the baselines are due to gradients, row 
	   drifts and/or turbulence, which happen on a faster scale than 
	   what it takes for the scan pattern to cycle. To avoid this problem, 
	   you can either change the source map position in the pipeline such
	   that the source follows the suspect term(s), or increase the 
	   time constant of the suspect models to high enough values to
	   get rid of the unwanted interference. The first method will
	   likely be more effective, but can also filter large scale 
	   structures.

	   An alternative or complementary solution is to use the high 
	   pass filtering capability in the time stream (with the -hipass=
	   and -diameter= flags). See more above.
	   It may also help to add in scans with considerable parallactic 
	   angle rotation, if they exist, which will increase redundancy.	


	Q: There are negative ridges or regions around my compact sources. 
	   How can they be made to disappear?

	A: Try more iterations, or delay/disable gain fitting (-Igain option),
	   or enable clipping below some S/N level (-cutoff=) for a while 
	   (-Ifidel=).
	

	Q: Extended structure seems missing, where did it go?

	A: Try one/some/all of the remedies below:

	     a. Make sure to use extended mode (-extended).
                Also controlled by EXTENDED_PRESERVE_TURNS key.
	     b. Increase extended source generations (-Iextended=).
                This will have the effect described in c-g below, if
                the starting iteration of those models is set to 'auto'.
		Also controlled by EXTENDED_PRESERVE_TURNS key. 
	     c. Delay gain modelling till later (-Igain=).
	     d. Delay sky gradients (-Isky=).
	     e. Delay row levels and gradients (-Irow=).
	     f. Delay multiplexer offsets (-Imux=);
	     g. Delay regional correlation modelling (-Iregion=). 
	     h. Use longer drift time constants (-driftT=).
	     i. Use longer time constants for the models in (c-g).
	   
	   Models like row drifts, bolometer drifts, regional 
	   correlations, gradient will, unfortunately interfere with 
           extended sources. Moving the source map ahead in the pipeline
	   (via -map= flag) so that the source extraction precedes the suspect 
	   model will do away with the unwanted interference, but you may pay 
	   the price of heavier baselines in the map. Otherwise, you can try 
	   increasing the model time scales, for those that precede source 
	   extraction, to values that compare to the time necessary for the 
	   sweep to cross the expected structure.
	
	   Another approach is to activate the sensitive models only
	   later on in the iterative process, once a more robust solution is 
           obtained by the other terms. To do this, use the various -Ixxx= 
           flags or an appropriate configuration file.
	   Also, if you are high-pass filtering, make sure it is done
	   on a time scale >> source crossing time.


12. Deep Fields and Faint Compact Sources (-deep)
=================================================

As should be clear from the above section, very faint and extended sources
are the most challenging to recover. What about point-like or compact sources
that are very faint? The -deep option is designed to deal with these, by
reducing maps with very aggressive settings. While these setting will tend
to filter extended structures away, the smaller scales will remain relatively 
unaffected (See the calibration memo for appropriate flux corrections). However,
while the default -deep setting will likely work in most cases, there are
exceptions. Depending whether you find exceess noise to be rid of, or you'd 
rather trade noise for larger source extents, the following are recommended:

Q. There is still excess noise in the maps. How can I get rid of it?

A. The excess noise is most probably sky noise. Sky noise has a power-law
   spectrum with larger scales having more power. The knee where this power-law
   behaviour blends into the white noise level of the background will determine
   how heavy a sky filter you need. The default is 30.0 arcsec FWHM. However, 
   there are days when the sky is noisier, and you need heavier filtering. Try
   setting the regional correlation filter size appropriately smaller (e.g.
   20.0 arcsec) via the -regionSize= flag or REGIONAL_CORRELATION_SIZE key.

   If you don't get the desired effect, you can try activating the Multiplexer
   Offset Model (which is disabled by default) via the -Imux=auto flag or
   MUX_TURN key.

   If you are still not satisfied, you have little room left for improvement. 
   Try shorter time constants for the Row Polynomial model (-rowT= option or
   ROW_POLYNOMIAL_T key), MUX model (-muxT= option or MUX_T key) or the
   Regional Correlation Model (-regionT option or REGIONAL_CORRELATION_T key).

Q. I'm not worried for sky noise, but want to see more extended structures
   in my map. What do I do?

A. Increase the Regional Correlation scale (-regionSize flag or
   REGIONAL_CORRELATION_SIZE key) from the default (30.0) to something
   appropriately larger. Alternatively, you may want to try to disable this
   model altogether (-Iregion=off or REGIONAL_CORRELATION_TURN=off).



