A Guide to using MiniCRUSH. 
(A minimalistic version of CRUSH for LABOCA & ASZCA)

Last updated 23 July 2008 -- Attila Kovacs <akovacs@mpifr-bonn.mpg.de>



Table of Contents
=================

1. Getting Started

  1.1 Installation
  1.2 Quick start
  1.3 The Tools of the CRUSH Suite
  1.4 A Brief Description of What CRUSH Does and How
  1.5 Command-Line Options and Scripting Support

2. Basic Configuration

  2.1 Global configuration options
  2.2 Pipeline configuration
  2.3 Recovery of Extended Structures
  2.4 Configuring the Source Map
  2.5 Scan-specific configuration

3. Instrument Specific Configuration

  3.1 Instrument Definition Files
  3.2 Laboca specific options
  3.3 ASZCA Specific options

4. Advanced Configuration

  4.1 Advanced Pipeline Options
  4.2 Advanced Source Map Configuration
  4.3 Advanced Scan-Specific Options
  4.4 Pipeline elements (Models) and their options
  4.5 Further output options

5. Understanding the Console Output

6. Pointing Corrections

7. Examples

8. Further information







#####################################################################
1. Getting Started
#####################################################################


1.1. Installation
=================

   After unpacking the gzipped tarball, enter the directory minicrush.

   If setting it up on a project account, edit 'minicrush.cfg' to 
   specify the location of your data files, and the project id. You can also 
   set the output directory. Example contents of minicrush.cfg:
	
	datapath ../data
	outpath ../maps
	project T-79.F-0002-2006

   Check to see if your files are compressed. If your data files end with
   '.fits.gz' extension then you will want to also include a line:

	compressed

   in the configuration file. (Or you can specify this individually for
   scans on the command-line, if you do not want a global setting).

   Alternatively, to create a user configuration somewhere outside of the
   minicrush/ directory, copy minicrush.cfg into .minirush/ in your home:

	> mkdir ~/.minicrush
	> cp minicrush.cfg ~/.minicrush
	
   This way every user can edit their local preferences. You can create any
   number of user specific configuration files in ~/.minicrush. You can invoke
   these via either the 'config <filename>' configuration entry or via the
   equivalent '-config=<filename>' runtime option.

   Java Configuration
   ------------------
   Edit the 'setup.sh' script, and adjust the -Xmx option of JAVAOPTS to the 
   maximum amount of memory you will allow the reduction to use. E.g. for 1GB,
   use:
	
	-Xmx1000M

   Note, that for 32-bit machines, the value of Xmx has to be below 2000.
   On 64-bit machines (with the appropriate 64-bit java VM installed) 
   up to 4G or more (depending on the OS configuration) can be specified (the
   latter may require the use of the '-d64' option as well). 

   You may also want to change the JAVA variable in 'setup.sh' to point to 
   the version of java that you want to use, if this is not the default one. 
   (The default is to use the latest version fron SUN in /usr/java/latest/bin) 
   The minicrush software needs at least version 1.5.0, but 1.6 is recommended.

   The GNU java (default on RedHat and Fedora systems) is rather buggy and 
   tends to cause trouble. Avoid using it if you can.

   To see what is your default java version, type:

	> java -version




1.2. Quick start
================


   To reduce data, simply specify the instrument (laboca or aszca) and the 
   scan numbers to minicrush. (You may have to add './' before minicrush if 
   the current directory '.' is not in your path.) E.g.,

	> ./minicrush laboca 10059

   will reduce LABOCA scan 10059 with the default reduction parameters. You can
   specify additional options. Some of these apply to the reduction as a whole
   while others will affect the scan processing for those scan numbers that are
   listed *after* the option flag. 

   If you are new to MiniCRUSH it is recommended that you read this document 
   through Sections 1--2 (Getting Started & Basic Configuration). This should 
   give you enough background to become a well-versed user already. :-)

   Once you get the hang of it, and feel like you need more tweaking ability,
   feel free to read on further to see what other fine tuning capabilities
   exist...

   Here're some quick tips:

 	* Reduce as many scans together as you can. E.g.
		
	  > ./minicrush 10657-10663 10781 11233-11235 [...]

 	* You can specify opacities, pointings, scaling etc, for each
	  of the set of scans listed (See section for Scan-specific options). 
	  E.g.,

	  > ./minicrush -tau=0.32 10657-10663 10781 -tau=0.18 11233-11235 [...]

	  will use a zenith tau value of 0.32 for 10657-10663 and 10781, and
	  0.18 for the last set of scans.

	* If you suspect that you are missing extended structure (see section
	  on the Recovery of Extended Structures), then you can specify the
	  'extended' option, which will better preserve large scale structures
	  albeit at the price of more noise. E.g.

	   > ./minicrush -extended 10657-10663

	* If your source is faint (meaning S/N < 10 in a single scan), then
	  you may try using the 'faint' option. E.g.

	  > ./minicrush -extended 10657-10663

	  or,	  
	 
	  > ./minicrush -faint -extended 10657-10663

	  to try preserve extended structures (see above).

	* For finer control of how much large scale structures are retained
	  you can use the 'sourcesize' option (instead of 'extended'), giving 
	  a typical source scale (in arc-seconds) that you'd like to see 
	  preserved. Then CRUSH will optimize its settings to do the best it
	  can to get clean maps while keeping structures up to the specified
	  scales more or less intact. E.g.

	  > ./minicrush [...] -sourcesize=45.0 10657-10663
  
	  Will optimize reduction for <~ 45 arcsec sources.

	* Finally, if your sources are not only faint, but also point-like, 
	  you should try the 'deep' option. This will filter large scale
	  structures heavily to yield the cleanest looking maps possible.
	  E.g.,

	  > ./minicrush -deep 10657-10663


   With just these few tips you should be able to achieve a decent job in
   getting the results you seek. There are a lot more fine-tuning possibilities
   for the curious mind. If interested, you can find all the documentation 
   further below...	



1.3. The Tools of the CRUSH Suite
=================================

CRUSH provides a collection of useful tools. Here's a short description
of what is there and what they are used for. Each tool, when run without
options (or with the -help option) will provide you a list of available
options on the console.

coadd		Add FITS images together. Use as a last resort tool as it is
		always better to reduced scans together.

show		A simple display program for the FITS files, with various
		useful functions for simple analysis and image manipulation
		After starting press 'h' for help.

imagetool	A tool for manipulating images. Can also deal with images
		produced by BoA (and to some degree other images also).

calibrate	A tool for deriving calibration information into an ascii 
		file.

histogram	Provide a histogram of signal-to-noise distribution of
		an image.

jiggle		Align multiple images w.r.t. one-another.

detect		A source extraction tool for maps. You should make
		sure that the noise is close enuogh to Gaussian (e.g. with
		'histogram') before relying on this. 

difference	Allows to look at the difference between two supplied
		images.





1.4. A Brief Description of What CRUSH Does and How...
======================================================

   CRUSH is a pipeline reduction that is principally meant to remove
   correlated signals (correlated noise) in the time-streams to arrive
   at clean & independent bolometer signals, which are used to make
   a source map.

   As such it is not an interactive reduction software (e.g. as opposed to 
   BoA). The term 'scripting' in CRUSH mainly means defining configuration 
   options (in the command line or through configuration files) which 
   are parsed in the order they are read.

   During the reduction CRUSH aims to arrive at a consistent set of
   solutions for various correlated signals, the corresponding gains and
   the pixel weights, as well as tries to identify and flag problematic data.

   This means a series of reduction steps, which are iterated a few times
   until the required self-consistent solutions are arrived at. 
   
   


1.5. Command-Line Options and Scripting Support
===============================================

   Configuration of minicrush is available either through command line 
   options or via scripting. You have seen scripting already in the form
   of 'minicrush.cfg', which stores some user-defined values. But there is 
   more to it.

   Both command-line options and scripting is organized in key/value pairs. 
   The main difference is that on the command-line there are restrictions on 
   syntax imposed by the shell and the command-line parser. E.g., no white 
   spaces. Therefore key/value pairs may be separated either by '=', ':' or
   empty spaces, or any combination of these. 
   Commant line options start with a dash '-' in front. Thus is, what may look
   like:

	key option1 option2=value

    in the configuration script, will end up as

	> ./minicrush [...] -key=option1|option2:value [...] 

    in the command line. Otherwise, they two ways of configuring are generally 
    identical to one-another. One exception is reading scans, which is done
    via the 'read' key in a script, whereas in command line, you simply
    list the scan number (or ranges or lists). I.e., 

	read 10056			# in script

	> ./minicrush [...] 10056	# on the command line.

    In the next section you'll find a description of the scripting keys.
    Now that you know how to use them also as command line options, you
    can choose scripting or command-line, or mix-and-match them to your
    liking and convenience...


    Startup Configuration
    ---------------------

    At launching, minicrush will parse 'minicrush.cfg' and 'default.cfg'
    to establish a default configuration set. Both these configurations
    are first parsed from the files in the minicrush directory, then 
    additional options or overriders are read also from the appropriate 
    instrument subdirectories, if exist. 
    Then, to allow for local (machine or user-dependent) configurations,
    minicrush will check, if the configuration files exists under the
    same name in ~/.minicrush directory of the user. If so, they will be
    parsed at this point. See more on this in the next section under the
    'config' option. 






#####################################################################
2. Basic Configuration
#####################################################################



2.1. Global configuration options
=================================

   Configuration options here are listed as scripting keys i.e., without
   a preceding dash. However, you can use the same options in the command
   line by adding the dash. (BTW, '=' can be replaced by a space in 
   scripting...) 

   Upon starting minicrush, first 'minicrush.cfg', then 'default.cfg'
   are parsed, establishing initial settings. (Remember, if these, or
   any of the invoked configuration files are found in '~/.minicrush/'
   then they will be read from there also after parsing these from the 
   default location in the program execution directory.

   	config=<filename>	Load the configuration file filename. 
				The file is looked for in the locations in the
				following order:

					1. ./
					
					2. ./<instrument>/

					3. ~/.minicrush/

					4. ~/.minicrush/<instrument>/

				Whenever a matching file is found its contents
				are parsed. Because of the ordering, it is 
				convenient to create overriding configurations.
				Thus instrument specific settings can be used 
				to override default settings, and user specific
				settings placed in ~/.minicrush/ can override
				shipping defaults. Whenever a configuration is
				parsed, there is a note of it on the console
				output so that one always knows which files 
				were read and in what order.
				E.g. when using,
				
				   > ./minicrush laboca -faint 12066

				the following configuration files will be
				loaded in order (provided they exist):

				   ./default.cfg
				   ./laboca/default.cfg
				   ~/.minicrush/default.cfg
				   ~/.minicrush/laboca/default.cfg
				   ./faint.cfg
				   ./laboca/faint.cfg
				   ~/.minicrush/faint.cfg
				   ~/.minicrush/laboca/faint.cfg
				   
				Each successively loaded file may override
				setting loaded before it.

	poll			Whenever unsure what options are set at any
	poll=<option>		given stage, you can poll the settings.
				Without an additional argument it will list
				all options to the standard output. When
				an argument is specified it will list
				all configuration settings that start with
				the specified string. E.g.

					./minicrush [...] -poll=iter
	
				will list all iteration based options that 
				are set including all the [...] options set
				prior to '-poll' in the command line.

	outpath=<path>		Set the directory into which output files
				(e.g. maps) will be written. Can use '~'
				for home directory and environment variables
				in {}'s. Thus,

					outpath=~/images

				and

					outpath={$HOME}/images

				are equivalent

	forget=<option>		Forget the priorly set values for <option>
				as if it was never defined. E.g. 
		     		
					forget=outpath

				will unset the 'outpath' option.
				You can specify more than one options as a
				comma-separated list. E.g.
	
					forget=outpath,project
				
				With unset both the 'outpath' and 'project' 
				options.	

	blacklist=<option>	Similar to 'forget', except it will not
				set options even if they are then specified
				at a later time. This is useful for altogether
				removing settings from the configuration.

	whitelist=<option>	Remove <option> from the blacklist, allowing
				it to be set again if desired.

	reservecpus=N		Instruct minicrush NOT to use N number of CPUs
				of the machibe. By default minicrush will try 
				to use all processors in your machine for 
				maximum performance. This option allows to 
				modify this behavior according to need. Note,
				that at least 1 CPU will always be used by 
				minicrush, independent of this setting.
				The number of actual parallel threads will be 
				the larger of the allowed number of CPUs and
				the number of scans processed.




2.2. Pipeline configuration
===========================

  a. Source types. The default reduction (see 'default.cfg') is optimized 
     for compact (up to a quarter of the array) sources in the S/N range of
     ~30-300. These options are useful if your source does not match that 
     range.


	bright		Use for bright sources (S/N > ~1000). This setting
			entirely bypasses all filtering to produce a very
			faithful map. The drawback is more noise, but
			that should not be an issue for such a bright guy :-)
			Will invoke 'bright.cfg'.

	faint		Use with faint sources (S/N < ~30) when the
			source is detected in a single scan. This setting
			applies some more aggressive filtering of the 
			timestreams, and extended structures. It invokes 
			'faint.cfg'.

	deep		Use for very faint sources which are not at all 
			detected in single scans, or if you think
			there is too much junk noise (baselines) in your 
			map to your liking. This setting results in the most 
			agressive filtering. Will load the configuration from
			'deep.cfg'. The output map is optimally filtered 
			('smoothed') for point sources.

	photometry	Reduce data taken in photometric observing mode. Instead
			of making a map, an appropriate point source flux is
			calculated and displayed at the end of the reduction.

	extended	Try to better preserve extended structures. This
			setting can be used alone or in combination with
			the above brightness options. See also -sourcesize=X 
			below. With the fainter settings the recovery of 
			extended structures becomes increasingly more 
			difficult. For bright structures recovery up to FOV 
			(or beyond!) should be possible, while for faint 
			structures ~1/4 FOV - FOV scales are maximally 
			obtainable (see more on this in the section below.)
  
	sourcesize=X	This option can be used instead of 'extended' in 
			conjunction with 'faint' or 'deep' to specify the 
			typical size of sources (FWHM in arcsec) that are 
			expected. The reduction then allows filtering 
			structures that are much larger than the specified 
			source-size...
			If 'sourcesize' or 'extended' is not specified, then 
			point sources are assumed in both 'faint' and 'deep' 
			modes.	
			Note, that the 'sourcesize=X' option will be ignored
			if 'extended' is also specified!

	ordering=a,b,c	Specify the order of pipeline elements as a comma
			separated list of keys.



2.3 Recovery of Extended Structures
===================================

	You can usually choose to recover more extended structures if you
	are willing to put up with more noise on those larger scales. This 
	trade-off works backwards too -- i.e., you can get cleaner maps if
	you are willing to filter them more.

	As a general principle, structures that are bright (>> 5-sigma) can
	be fully recovered to up to a few times the field-of-view (FOV) of the
	bolometer array. However, the fainter the structure, the more it will
	be affected by filtering.

	Generally, the fainter the reduction mode, the more filtering of faint
	structures results, and the more limited the possibility of
	recovering extended structures becomes. The table below is a rough
	guide of what maximum scales you may may expect for such faint feaures,
	and also, how noise is expected to increase on the large scales as
	these are added in 'extended' mode:


	 Table 1. Maximum faint structure scales for average S/N < 5


		     |	  (COMPACT)	extended      Noise power
		     |	  (default)		         index
	===========================================================
	bright	     |	
	(DEFAULT)    |	   FOV/2	 ~FOV		 ~l^2
                     |
		     |	2*sourceSize	 		  
	faint, deep  |	     or		 FOV/2		  ~l
		     |	   2*beam
	-----------------------------------------------------------





2.4. Configuring the source map
===============================


	projection=	Choose a map projection to use. The following 
			projections are supported:

				SIN  --  Slant Orthographic
				TAN  --  Gnomonic
				ZEA  --  Zenithal Equal Area
				SFL  --  Sanson-Flamsteed
				MER  --  Mercator
				CAR  --  Plate-Carree
				AIT  --  Hammer-Aitoff
				GLS  --  Radio

	name=		Specify the output image file name, relative to the
			directory specified by 'outpath'. When not given
			minicrush will chose a file name based on the source
			name and scan number(s), which is either

				<sourcename>.<scanno>.fits

			or
	
				<sourcename>.<firstscan>-<lastscan>.fits

	altaz		Reduce in Alt/Az mode instead of the default RA/DEC.
			This is useful for reducing pointing scans.

	unit=xxx	Set the output to units xxx. Currently 'uV/beam' and
			'Jy/beam' are defined.

	grid=X		set the map pixelization to X arcsec. Pixelization
			smaller than 1/2 beam is recommended. The default is
			1/3 beam map pixelization.

	smooth=X	Smooth the map by X arcsec FWHM beam. Smoothing
			helps improve visual appearance, but is also useful
			during reduction to create more redundancy in the data
			in the intermediate reduction steps. Also, smoothing
			by the beam is optimal for point source extaction from
			deep fields. Therefore, beam smoothing is default in
			with the 'deep' option (see '<instrument>/deep.cfg').
			Typically you want to use some smoothing during 
			reduction, and you may want to turn it off in the 
			final map. Thus, you may have something like:

			  smooth=9.0			# initially smooth to 9
			  iteration:2=smooth:12.0 	# smooth more later
			  iteration:last=forget:smooth  # no smoothing at last

	filter=method [blanking]   Filter extended structures using the 
				   specified method ('fft' or 'convolution'). 
			Optionally the filter scan skip over map points that
			are above the 'blanking' S/N level. Thus any structure
			above this significance level will remain unfiltered.
			Filtering is useful to get deeper in the map when 
			retaining the very faint extended structures is not 
			an issue. Thus filtering above 5 times the source size
			(see 'sourcesize') is default.
			Note the if 'extended' is instead specified, then 
			the filtering is disabled even in 'deep' mode.
	
	rcpgains	Specifies to use the source gains from the RCP file.
			This is BoA's way of doing things. Accordingly one
			should change the calibration factor to that of BoA's
			as well. Otherwise, MiniCRUSH uses source gains that
			are based on the correlated noise response (with an
			optional main-beam-efficiency correction). 

2.5. Scan-specific configuration
================================


      Some options relate to the scans, helping to configure and handle them
      These options are specified *before* the list or range of scans to
      which they apply, and remain valid for all scans read after, or until
      an overriding option is placed. E.g.

	  ./minicrush -option1=x 10218-10532 12425 -option2=y 11411 \
	              -option1=z 10496

      will set option1 to 'x' for all scans but the last one, which will have
      this option set to 'z'. And the last two scans will have option2
      set to 'y'.

      A detailed listing of all scan specific options is found in the
      file OPTIONS. Here's a few of the most commonly used ones. Here are
      a few of the most important ones.



	read scanNo	  Read scan number scanNo (scripting only).
			  in command line mode, ommit '-read='.
	read from-to	  Read the range of scan number between from and to.
	read X Y [...]    You can combine scan numbers and different ranges
			  in a space-spearated list...
			
			  As mentioned, the 'read' keys only apply to scripts.
			  Thus, 

				read 10755-10762                 # in script

			  and

				> ./minicrush [...] 10755-10762  # command line

			  are equvivalent.
				

	project=<id>      Set the project ID. Use Capitalize form. E.g.,

				project  T-79.F-0002-2007

	datapath=<dir>	  Start looking for raw data in directory <dir>
			  or in <dir>/<project>. Thus, if
				
				datapath /homes/data
				project T-79.F-0002-2007

			  then minicrush will try to find data first in
			  '/homes/data', then in '/homes/data/T-79.F-0002-2007'
			  This is convenient, as datapath can simply specify
			  a root for all project directories. :-)
 
	tau=X		  Specify a zenith tau value to use. When not used
			  minicrush will try to interpolate from
			  <instrument>/<instrument>-taus.master.dat if possible
			  or use 0.0 as default.

	tau=<filename>	  Alternatively, tau can specify a file-name with 
			  lookup information (usually containing averaged tau
			  values from the radiometer values and the skydips).
			  Tau values will be interpolated for each scan,
			  as long as the scan falls inside the interpolator's
			  range. Otherwise, tau of 0.0 will be used.

	center=dAZ,dEL	  Center the map at dAZ,dEL (like pcorr). 

	scale=X		  Scale the fluxes by X. With this option you can apply
			  calibration corrections to your data.

	scale=<filename>  Alternatively, scan specific scaling can be defined
			  by an appropriate calibration file, which among 
			  other things, contains the ISO time-stamp and	
			  the corresponding calibration values for each scan.



#####################################################################
3. Instrument Specific Configuration
#####################################################################




3.1. Instrument Definition Files
================================

There are various instrument configuration files. These reside in the 'laboca/'
and 'aszca/' subdirectories in the distribution directory. The RCP files are
standard APEX RCP files containing the position information of the bolometers
(other information is not used from therein). 

the <instrument>-pixel.dat file contains default gain, weight, and flag 
information. This can be overwritten by files produced via the 'pixeldata' 
option. 

Skydip data can be placed in '<instrument>-taus-master.dat'. Then reduction
will use the appropriate entries for interpolating zenith opacities whenever
the reduced data falls within the range covered by skydips.





3.2. Laboca specific options
============================

	he3=<source>	Use He3 temperature drift correction from <source>
			(either 'thermistor' or 'blank').

	boxes		Decorrelate on amplifier boxes.

	cables		Decorrelate on cables. (Microphonic pickup).

	amps		Decorrelate on amplifier boards.

	twisting	Solve for the signals that are associated with the
			band cables twisting. This helps to arrive at cleaner
			data.




3.3. ASZCA Specific options
===========================

	wafers		Decorrelate wedges.

	regions		Decorrelate along 12 horizontal strips.
	
	boxes		Decorrelate electronic boxes.

	cables		Decorrelate electronic cables.

	amps		Decorrelate squid amplifiers



#####################################################################
4. Advanced Options
#####################################################################



4.1. Advanced Pipeline Options
==============================


  b. Basic pipeline configuration

	beam=X			Set the instrument beam to X arcseconds.

	blacklist=clear		Clear the list of banished options.
	
	estimator=<type>	'median' or 'maximum-likelihood' estimators to
				use in deriving signal models. 'median' 
				estimators are less sensitive to the presence 
				of bright sources in the data, therefore it is
				the default for the initial gain iterations 
				(see 'gainrounds') and when 'bright' is 
				specified (see 'bright.cfg').
				
				When medians are used, the corresponding models
				are reported on the console output in []'s...
				(see the Console Output section).

	extendedopt=<opts>	Specify a |-separated list of options which
				should be set when reducing in 'extended' mode.
				E.g., 

	 			   extendedopts drifts 10.0 | blacklist sky

				will set the 1/f filtering timescale to 10 
				seconds and will never solve for sky-gradients.
	
	extendedopt=clear	Clear all extended options set thus far.

	rounds=N		Iterate N times.

	gains=<method>		Specify that gains should be solved where
				appropriate. (individual models may override 
				this when they have the 'nogains' oprion set).
				Additionally an estimation <method> can be
				specified ('median' or 'maximum-likelihood'). 
				
	weighting=<method>	Enable pixel weighting by measured noise. 
				The optional <method> can specify which way
				noise estimates are derived. The following
				methods are available:

				  rms	         Standard rms calculation.
				  robust         Use robust estimates for the
					         standard deviation.
				  differencial   Estimate noise based on pairs
					         of data separated by some
						 interval in time.

	scanweighting=  	'robust' or 'maximum-likelihood'.
				If specified, each scan gets an assigned weight
				with which it contributes to the composite map.
				This weight is measured directly from the noise
				properties of the produced map. Note, that this
				option conflicts the the 'extended' option, and
				is ignored when both are specified.

	iteration.N=[options]	Set the ; separated list of options when
				starting iteration N. For use in command-line
				mode, the '=' sign in the usual 'key=value' 
				pairs can be replaced by ':'. E.g.:
				-iteration:3=cables:true;despike:true
				This avoids a shell cry.

	iteration.last=...	Similar to the above, but setting the options
				for the last iteration, independently from
				how many iterations were specified.
	
	iteration.last-N=...	Similar to the options above but sets options
				for the N-before-last iteration.

	iteration.XXX=clear     Clear all setting for iteration XXX.

	iterations=[options]	Set [options] for all iterations explicitly
				defined thus far. E.g.
				  iterations=forget:despike
				will remove despiking from the iterations
				where it was defined...

	iterations=clear	Clear all iteration-based settings. 

	

	nefd-range [min],max	Specify a range of acceptable NEFDs in 
				Jy sqrt(s) units. All scans with NEFD outside
				this range will not contribute to the composite
				map.
				If the optional 'min' value is not given, then
				it is assumed as 0. ('-' may also be used in
				place of ',' for separating min and max 
				values.) 

        stability=X		Specify the instrument's 1/f stability time
				scale in seconds. This value is used for 
				optimizing reduction parameters when (e.g. the
				filtering time scale for the 'drifts' option)
				when these are not explicitly specified.

 
   c. Model enabling/disabling

      Each model in the pipeline can be disabled and enabled at will.
      A more detailed discussion of what models there are and what these do
      is discussed later. Here's just a very brief mention.      

	<model> [options]	Enables the model <model> in the pipeline
				with the optional [options].

	forget <model>		Disables the model.





4.2. Advanced source map configuration
======================================


	blank=X		Skip data from modeling over points that have a source
			flux exceeding the signal-to-noise level X. This may
			be useful in reducing the filtering effect around 
			bright peaks. See also -clip below.

	clip=X		In early generations of the source map, force map
			pixels with flux below signal-to-noise level X to zero.
			This may help getting lesser baselines, and filtering 
			artefacts around the brighter peaks.	

	noiseclip=X	Flag (clip) map pixel with a noise level that is more
			than X times higher than the deepest covered parts
			of the map.

	exposureclip=X  Flag (clip) map pixels whose relative time coverage
			is less than the specified value X. 

	scanmapdespike=X  Despike scan maps at the significance level X. 
			  Clearly you want to set X to be higher than the most 
			significant source in your map. Therefore it is only 
			really useful in 'deep' mode, where 5-sigma despiking
			is default (see 'deep.cfg').

	jansky=X	Specify the calibration factor from uV to Jy.


	scanmap-redundancy=N  Specify the minimum redundancy (N samples) that
			      each scan-map pixel ought to have in order to be
			considered valid. Pixels with redundancies smaller than
			this critical value will be flagged an not used in
			the composite source mapmaking.

	sources=<file>	Add sources as specified by a crush-style mask <file>.
			This is useful for runnign reduction on simulated data
			that can help understanding the systematics of the
			reduction. One thing to look out for is that the
			peaks here have to be specified in the instrument's 
			native units (i.e. volts), rather than Jy. This way
			any conversion related confusion is bypassed since the
			signal levels are unambigiously specified.


4.3. Advanced scan-specific options
===================================

	vclip=min[,max]	Clip data where the field scan velocity is outside
			the specified range (in arcsec/sec). The successfull
		 	disentangling of the source structures from the various
			noise terms relies on these being separated in 
			frequency space. With the typical 1/f type limiting 
			noise, this is harder when the scan speed is low s.t. 
			the source signals occupy the low frequencies. 
			Therefore, requiring a minimum scanning speed is a 
			good idea...
			On the other side, too high scanning speeds will smear
			out sources, if the movement between samples is larger
			than ~1/3 beam.


	aclip=X		Clip data when the telescope acceleration is above
			X arcsec/s^2. Heavy accelerations can put mechanical
			energy into the detector system, thereby generating
			false signals. Clipping data when there is danger of 
			this happening is a good idea. (see also the 'accel' 
			option for possible modeling of these signals)

	downsample=X	Downsample the data by a factor of N. At times the
			raw data is sampled at unnecessarily high frequencies.
			By donwsampling, you can ease the memory requirement
			and speed up the reduction.

	hipass=X	Highpass filter the timestream over X second timescales.

	frames=<from>-<to>  Read only frames <from>-<to> from the data. Maybe
			useful for quick peeks at the data without processing
			the full scan.

	shift=X		Shift positional information by X seconds. This is 
			meant to be used only when there is a timing problem, 
			whereby the data frames are misaligned from the 
			telescope position data.
	
	scramble	Make a map with inverted scanning offsets. Under the
			typical scanning patterns, this will not produce a
			coherent source. Therefore it is a good method for
			checking on the noise properties of deep maps.

	taufactor=X	One can specify a factor by which skydip-equivalent 
			tau values ought to be multiplied to obtain good 
			calibration. A gain dependece on loading can be 
			effectively modeled in this way. E.g. for LABOCA
			taufactor=1.3 provides approximately flat calibration
			in all weather conditions and at all elevations.

	scanmap-redundancy=N   Specify the minimum number of data points that
			       have to contribute to each pixel in a scan's
			       map in order for that pixel to be valid.

	flag=ch1,ch2,...chn    Specify a list of backend channels that ought
			       to be flagged for the successively read scans.

	pixeldata=<filename>   Load pixel defaults from <filename>. The format
			       is identical to the output of 'pixeldata=write'
			       option.

	rcp=<filename>	Use the non-default RCP file from <filename>.	

	uniform		Specify to use uniform pixel gains initially
			instead of the values read from the appropriate
			pixel data file (see 'pixeldata' key)...



4.4. Pipeline elements (Models) and their options
=================================================

	sourcemap [opt]	Solve for the source model.
			The optional argument [opt] can be:

				nosync		Do not sync the derived source
						increment to the data.
						Use only if you're sure you
						want this...
				mem		Maximum-Entropy correction. 

			MEM mapping can be used to suppress low-level noise
			but may cause some filtering on faint structures
			also. Nonetheless, it is an attractive method to
			produce very clean looking maps when for sources with
			S/N >> 1.

	offsets		Remove the residual DC offsets from the bolometer 
			signals (ignored when 'drifts' below is also 
			specified.)

	drifts=X	Filter low frequencies below the characteristic 
			timescale of X seconds. An effective way of dealing 
			with 1/f noise.

	decorrelate	Remove the correlated noise term accross the entire
			array. This is an effective way of dealing with most
			of the sky noise, grounf pickup and/or temperature 
			fluctuations
			
	gains		Solve for pixel gains based on their response to the
			correlated noise (above).

	weighting	Derive pixel weights based on the rms of the unmodelled
			timestream signals.

	despike=<method>,X[,...]  Use despiking method <method> at a S/N level 
	despike2=[...]            X (with possible further options). The
	despike3=[...]	          following methods are available:

				   neighbours  Despike only by comparing 
					       neighbouring data.

				   absolute    Flag data that falls above the
					       specified level.

				   gradual     Like 'absolute' but proceeds 
					       more cautiously.

				   features    Look for spikes wider than just
					       a sigle frame. The 'featurewidth'
					       option (below) controls up to
					       what timescale are spikes sought.

				  All methods can flag pixels and frames if 
				  these have too many spikes. The critical 
				  number of spikes pee channel is set by the 
				  larger of 'spikefraction' times the number of
				  frames in the scan, or 'spikecount'. 
				  The critical number of spikes per frame are
				  determined by the 'framespikes' option.
				  See these options below. 
	
	spikefraction=X		Tolerate (w/o pixel flagging) spikes up to 
				fraction X of the scan frames in each channel.

	spikecount=N		Tolerate (w/o pixel flagging) up to N spikes up
				in each pixel.

	framespikes=N		Tolerate up to N spikes per frame;

	spikewidth=X		When despiking using the 'features' method,
				spikes up to X second in width will be sought.

	sky			Model spacial sky noise as gradients accross
				the array.

	whiten			Use noise whitening algorithm. Careful, it is
				not yet well tested, and may affect the 
				calibration!!!
	
	neighbours		Decorrelate pixel neighbours on array. Note,
				this is effectively an extended structure 
				filter which gets more aggressive with 
				iterations. Therefore use only with care, if
				you must, and for 1 or few iterations only!!!





4.5. Further output options
===========================

	pixeldata=write	Write the pixel data file (gains, weights, flags). The
			putput will be 'pixel-<scanno>.dat'. You can use these
			files to update instrumental defaults in the instrument
			subdirectory. E.g., you can overwrite 

				laboca/laboca-pixel.dat

			to change what default pixel settings to use.


	covars		Write covariance data. The full pixel-to-pixel 
			covariance data is written, as well as versions of it 
			with pixels reordered in groups (boxes, cables, boards,
			squidgroups or squids) with the dead pixels discarded, 
			for convenient study of the correlated signals that 
			may be present.


	intermediatemaps  Write the maps made during the reduction into 
			  'intermediate.fits' (inside the minicrush directory).
			This option thus allows to keep an eye on the evolution
			of maps iteration-to-iteration. Each iteration will
			overwrite this 'temporary' file, and it will be erased
			at the end of the reduction.

	scanmaps	When specified, a map will be written for each scan
			(every time it is solved), under the name 
			'scan-<scannumber>.fits' in the usual output path.
			Best to use as:
			     
			     iteration:last scanmaps
 			
			to avoid the unnecessary writing of scan maps for every
			iteration (unless you really want that to be the case).

	spectrum [name][,][size]  Writes channel spectra (of residuals) in an 
				  ascii table. Optional arguments include the
			'name' of the window function to use (default is 
			'Hamming', and the window size (in powers of 2!, 
			default is 512). 


#####################################################################
5. Understanding the console output
#####################################################################


   Say you are reducing two scans with the command:

	./minicrush -deep 12065 12066

   The output begins with some minimalistic capture of the scans that are
   read in. This is reasonably straighforward.

   Then, you'll get some information on the type of reduction that has been
   selected by you. This refers to the brighness and extent of the source. 
   In this case, since '-deep' was specified without a further specification of
   source size, the reduction will assume deep-field-type sources that are
   point like.

   After this, some basic information is given on the source and the map
   parameters. 

   At last, the reduction begins. First there is an abridged loop meant to
   consolidate gain solutions (and provide some despiking) before the main
   loop is entered (see 'gainrounds').
   
   Then the reduction enters its main iterative phase. You'll see some cryptic 
   words and some numbers in a row. Each letter corresponds to an incremental 
   modeling of the time-series signals, while the integer numbers tell you how 
   many pixels remain unflagged after a step which can discard 'funky' pixels.
   Here's what the letters stand for:

   {n,m}:		processing scan n, subscan m from the list

	[]		bracketed models are solved via median estimators
	O		Solving for pixel offsets
	D[#]		Solving for pixel drifts (1/f filtering) on blocks
			of # frames.
	C		Solging for correlated noise and pixel gains
			Followed by the number of unflagged pixels.
	G		Solving for pixel gains.
	W		Estimating pixel weights.
			When done the average pixel NEV is printed (in
			units of V sqrt(s)).

	dA(%)		despiking absolute deviations.	
			In brackets it shows the percentage % of frames
			flagged in the data as spiky by any method.
	dG(%)		like above but proceeds gradually.
	dN(%)		despiking using neighbouring samples in the timeseries.
	dF#()		despiking wider features (up to # samples).

	S		Estimating atmospheric gradients accross the FOV.

	B		Decorrelating electronic boxes.
	c		Decorrelating electronic cables (LABOCA)	
	t		Solving for the twisting of band cables (LABOCA).
	a		Decorrelating amplifier boards.
	Q		Decorrelating wafers.

	A		Solving for the acceleration response of pixels.

   [whiten]		Noise whitening.
	
   [Source]		Solving for source map.
	[despike]	despiking scan maps before adding to composite.
	(level)		Zero levelling to map median.
	(smooth)	Smoothing map.
	(noiseclip)	Clip out the excessively noisy parts of the map.
	(filter)	Filtering large scale structures (i.e. sky-noise).
	(clip)		Clipping under-exposed map points.
	(blank)		Blank excessively bright points from noise models.
	(sync)		Removing the source model from the time-stream.
	

   Once the reduction is done, various files, including the source map, are
   written. This is appropriately echoed on the console output.






#####################################################################
6. Pointing Corrections
#####################################################################



Reducing the data with the correct pointing can be crucial (esp. when 
attempting to detect/calibrate faint point sources). At times the pointing
may be badly determined at the time of observation. Luckily, getting the 
exact pointing offset wrong at the time of the observation has no serious 
consequences as long as the source still falls on the array, and as long as 
the exact pointing can be determined later. If you, at the time of the data 
reduction, have a better guess of what the pointing was at the time the
data was taken (e.g. by fitting a model to the night's pointing data), you 
can specify the pointing offsets that you believe were more characteristic
to the data, by using the -center options *before* the scan 
numbers that should be reduced with the updated pointing. E.g.,

	-center=12.5,-7.3

Will instruct minicrush that the 'true' pointing was at dAZ=12.5 and dEL=-7.3
arcsec each (i.e. it works just like pcorr on APECS).

You can also make pointing changes after the reduction (alas, now in RA/DEC).
You can read off the apparent source position from each separately reduced
scan (e.g. by using 'show' and placing the cursor above the apparent source
center/peak). Then you can edit the individual FITS image headers (e.g.
using the '-changeKey' option of 'imagetool', by putting the apparent RA/DEC 
pointing offsets into the RAP and DECP fields. E.g.,

	> imagetool [...] -changeKey=RAP:3.0 -changeKey=DECP:-4.5 [... 

The above moves the map origin by 3" in RA and -4.5" in DEC.

Then, other crush tools (like coadd, imagetool etc.) will use these images 
with the proper alignment. Clearly, this method of adjusting the pointing is
only practical if your source is clearly detected in the map.

Thirdly, the tool 'jiggle' can be used to determine the above pointing offsets
through an interactive display in which the individual maps can be moved
w.r.t one-another. The cursor arrows will move the selected map in
the desired direction, while '+', and '-' can be used to change the map that
is selected for movement. Currently, 'jiggle' does not itself update the
RAP, DECP keys with the determined values. It only suggests what values
you will have to enter yourself in the FITS images.






#####################################################################
7. Examples
#####################################################################


   Reduce scans 12065-12069 and 12072 with zenith tau of 0.18:

	./minicrush -tau=0.18 12065-12069 12072

   Reduce scans 10562 and 10645 together, with the first scan observed at a
   zenith tau of 0.21, and the second at tau of 0.35 with.

	./minicrush -tau=0.21 10562 -tau=0.35 10645

   Say you realize that the pointing was off by -5.4" in AZ, and 2.4" in EL
   for the second scan. Then:

	./minicrush -tau=0.21 10562 -tau=0.35 -center=-5.4,2.4 10645

   Say scan 10049 is a scan of a bright source (e.g. Saturn) and the default
   reduction ends up clipping much of it. Then,

	./minicrush -bright 10049

   If the source still gets nipped from the resulting map, you can try 
   disabling pixel weigting altogether (this is the likely culprit) by:
	
	./minicrush -bright -forget=weighting 10049

   Perhaps you seem to filter too much in the default reduction (negative
   dip arond your source) in scan 10550:

	./minicrush -extended 10550.

   You can also try blanking and/or clipping at say the 5-sigma level:

	./minicrush -extended -blank=5.0 -clip=5.0 10550

   Try reduce some faint distant galaxy (scan 10057):

	./minicrush -deep 10057

   Maybe your galaxy has extended structure. Then try:

	./minicrush -deep -extended 10057.

   You can also fine-tune, what is the largest source-scale (more-or-less) 
   that you are interested in. Then the reduction will adjust its parameters
   accordingly. Say you expect your source to be a blob around 1' in diameter,
   then you can try:

	./minicrush -deep -sourcesize=60.0 10057.

   You can also play with the blanking clipping methods above, if there is
   annoying negative dips remaining around your brighter peaks. Note, that
   you porbably want to stick with -extended, or else such dips may be the
   result of the aggressive filtering settings applied to your specific
   source size (via -size).

   As mentioned before, command-line options and scripting are equivalent
   ways of configuring the reduction. Thus, the runnng the script 'test.cfg':

	deep
	extended
	tau 0.18
	center -3.2,4.8
	read 12065-12069
	center 2.3,-0.5
	read 12072
	blank 10.0
	clip 3.0
	iteration.last-1 forget clip

   via 
	./minicrush -config=test.cfg	

   is equivalent to the command line:

	./minicrush -deep -extended -tau=0.18 -center=-3.2,4.8 12065-12069 \
		-center=2.3,-0.5 12072 -blank=10.0 -clip=3.0 \
		-iteration.last-1=forget:clip  







#####################################################################
8. Further information
#####################################################################



MiniCRUSH is a spinoff from the SHARC-2 data reduction package CRUSH, to reduce
LABOCA data. It is a much simplified version of the full package. However, the 
output FITS image is fully compatible with the SHARC-2 version. Thus, some 
parts of the original CRUSH package can be useful for manipulating images post 
reduction.

Mainly, the CRUSH distribution provides tools for coadding images ('coadd'), 
image manipulation ('imagetool'), adjusting pointing in RA/DEC maps ('jiggle'),
or prducing histograms of the flux (signal-to-noise) distribution in maps
('histogram').

Downloading the CRUSH package, and further information on the FITS images
is available at:

	http://www.submm.caltech.edu/~sharc/crush

I hope you'll find this helpful and may you end up with beautiful LABOCA
images!!!



===============================================================================
(C) 2007 -- Attila Kovacs 

