            *** CRUSH User's Manual Extension for the SCUBA-2 ***

                              Attila Kovacs
                        <attila[AT]submm.caltech.edu>

                        Last updated: 8 Sep 2015



Table of Contents
=================

1. Introduction
  1.1 Computing requirements
  1.1 Performance
2. Setup and configuration
  2.1 Starlink
  2.2 Your personal settings
3. Quickstart guide
  3.1 Pointing and calibration
  3.2 Source brightness
  3.3 Extended sources
  3.3 Compact sources
  3.5 Smoothing 
4. Advanced Topic: Reading Scans
  4.1 FITS vs. SDF
  4.2 '450um' vs '850um' reduction mode
  4.3 Locating scans by date and scan number
  4.4 Specifying scans via file names
5. Advanced Topic: Calibration and Pointing
  5.1 Extinction Correction
  5.2 Calibration Correction
  5.3 Pointing Corrections
6. SCUBA-2 specific pixel divisions
7. Glossary of SCUBA-2 specific options
8. SCUBA-2 specific log quantities



#####################################################################
1. Introduction
#####################################################################

This document contains information specific to using CRUSH to reduce SCUBA-2
data. It is assumed that you are already familiar with the contents of the main
CRUSH README (inside the distribution directory), especially its Section 1
(Getting Started).

If you run into difficulty following this document, you will probably find it 
useful to study the main README a little longer.


1.1. Computing requirements
===========================

CRUSH does not require any special computing hardware to reduce SCUBA-2 data. 
You just need a run-of-the-mill PC or Mac (laptop, desktop or server) with a 
few GB or RAM (to fit the SCUBA-2 data), running any flavor of UNIX (such as 
Linux, BSD, or MacOS) or Windows. (CRUSH will also run on ARM-based platforms, 
but don't expect blazing speeds on these mobile-class devices...)

You will also need Java 6 (a.k.a 1.6.0) or later on the machine, but you
probably have that already... 

Your main limitation will likely be the RAM available, which will limit how
many scans you can reduce at once. But, you might be surprised just how far
a few GB will get you...


1.2. Performance
================

CRUSH is lean and mean when it comes to memory use and reduction speed. It is 
also thoroughly parallelized, and will use all processing cores in your machine 
effectively.

If your machine has N processing cores, you can expect to reduce a minute of
full 850um SCUBA-2 data (4 subarrays) in about an Nth of a minute. Thus, with 
a dual-core i5 mobile CPU, my Lenovo T430s laptop churns through a minute of 
850um SCUBA-2 data in around 30 seconds. An i7 desktop with 4 cores should get 
there in 15 seconds, while a 12-core Xeon should clock in at around 5 seconds 
reduction time per minute of observation (provided you are not slowed down by 
disk access when reading the data).



#####################################################################
2. Setup and configuration
#####################################################################

Start by installing <b>CRUSH</b>, as instructed by the main README.


2.1 Starlink
============

   CRUSH will readily read SCUBA-2 data in FITS format, when these are 
   available. More typically, though, your SCUBA-2 data comes in NDF (.sdf) 
   format of the Starlink suite, and so you will need to convert your files to 
   FITS for using with CRUSH. Fortunately, CRUSH can make the conversion 
   seamless, as long as the necessary Starlink tools are accessible. (Once you 
   have FITS files, you no longer need Starlink, or the  original .sdf files.)

   To set up the automatic NDF to FITS conversion, first make sure the Starlink
   suite is installled on your machine. (You only really need the 'ndf2fits' 
   tool included in Starlink...). You can now continue personalizing CRUSH.


2.2 Your personal settings
==========================

   Start by creating (or editing) '.crush2/scuba2/default.cfg' in your home 
   directory (this file will contain your personalized configuration entries). 
   E.g. on UNIX and Mac OS:

     > mkdir -p ~/.crush2/scuba2

   Then create/edit 'default.cfg' therein, e.g. with 'nano':

     > nano ~/.crush2/scuba2/default.cfg

   (Replace 'nano' with your preferred editor, such as 'vi' or 'emacs' or 
   'gedit'.) In this file you can specify your default values for the SCUBA-2 
   scan data directory ('datapath'), and where CRUSH should write its output 
   files, such as the reduced images ('outpath'). You might also want to 
   specify where the Starlink 'ndf2fits' conversion tool is. E.g.:

     datapath /home/data/scuba2
     outpath ~/images
     ndf2fits /usr/local/Starlink/bin/convert/ndf2fits

   (The 'ndf2fits' setting may not be needed if you already have FITS files, or
   if the STARLINK_DIR environment variable points to your Starlink 
   distribution folder...). Also, make sure the output directory actually 
   exists. :-)

   You can also specify and override any of these settings ad-hoc on the 
   command line later. E.g.:

     > crush scuba2 -datapath=/home/myData -outpath=../myImages [...]



#####################################################################
3. Quickstart guide
#####################################################################

   
Now, let's reduce some data. Suppose you want to reduce pointing scan #62 from 
2015-06-30, at 850um:

  > crush scuba2 -850um -date=2015-06-30 62  

(The '-850um' or '-450um' options select the imaging band for the reduction -- 
CRUSH will default to 850um imaging if the imaging band is not explicitly 
specified.) 

You may have to prepend the <path> to 'crush' when invoking. E.g. use 
'./crush [...]' (if running 'crush' from within its own directory), or 
'~johndoe/crush/crush [...]'.

After the reduction completes, you look at your images with CRUSH's own 'show' tool. E.g.:

  > show ~/images/URANUS.20150630.62.850um.fits

And, you can post-process output images using 'imagetool' (and also 'show',
which takes all of imagetool's options). E.g.:

  > imagetool -smooth=7.5 ~/images/URANUS.20150630.62.850um.fits

(To see a help screen with the available image processing options, run 
'imagetool -help' or check out the online manuals on the CRUSH site). Of 
course, you may also use your own favorite FITS viewer/tool, if you prefer.



3.1. Pointing and Calibration
=============================

   For pointing scans, CRUSH will automatically perform a pointing fit at the 
   end of the reduction, complete with peak and integrated (aperture) flux 
   measures. 

   Suppose, you find that the pointing was off by -1.2,4.6 arcsec (in Az/El). 
   And, suppose the flux (peak or integrated) you found was too low by a factor
   of 1.14. You can the apply these corrections (to this scan, or others) as:

     > crush scuba2 [...] -pointing=-1.2,4.6 -scale=1.14 [...]

   Now, you are ready to reduce some science scans from the same day (say scan 
   numbers 63-66, 70 with the above pointing and 81-83 with a different 
   pointing):

     > crush scuba2 [...] -pointing=-1.2,4.6 63-66 70 -pointing=2.3,1.8 81-83

   Similarly, you can apply different calibration scalings too for the 
   different scans. In general, any options will be applied to all scans after
   the option is set, redefined again, cleared (using -forget), or blacklisted 
   (via -blacklist) on the command line. Thus, global options should be up 
   front before any of the scans are listed...

   CRUSH comes with a rough built-in calibration for SCUBA-2, with around 10% 
   rms blind calibration error. It is based on a handful of primary calibrator 
   scans on Uranus and Neptune, and the temperature model in GILDAS/ASTRO. It 
   is possible that with a more thorough analysis and a larger representative 
   sample of calibrator scans, the blind calibration may be improved. If you 
   would like to help improve the blind calibration, please contact Attila 
   Kovacs <attila[AT]submm.caltech.edu>. 


3.2. Source brightness
======================

   The examples above produce a default reduction, which is meant to work with 
   a wide variety of sources (brightness and extent), but you may do better 
   than that for your particular target. For example, if your source is faint 
   you  may use the '-faint' option. For deep fields (very faint point sources)
   you can use '-deep' which will aggressively filter the large scales... E.g.:

     > crush scuba2 -faint [...]

   or

     > crush scuba2 -deep [...]

   For very bright sources (which might be flagged excessively by the default
   settings) you may try '-bright'.

   You can find more details on pointing and calibration in the sections below.
   A more detailed general overview is in the main README (e.g. the Quick Start
   guide there, and the various advanced topics).   


3.3 Extended sources
====================
  
   By default, CRUSH is optimized for reducing compact sources, and to provide 
   the cleanest possible images. This works well up to scales of 2 to 4 
   arcmins across (depending on how bright the source is). If you have a more 
   extended source, you may try the 'extended' option. E.g.:

     > crush scuba2 -extended [...]

   or 
 
     > crush scuba2 -faint -extended [...]

   (I.e., 'extended' may be combined with other options like 'faint' to tune 
   the reduction for your particular source). The 'extended' option will 
   recover more of the large scales (up to FoV or, for bright sources, beyond 
   the FoV -- see the corresponding section in the main README), but you will 
   pay the price of reduced sensitivity (increased noise, especially on the
   large scales).

   You can further fine-tune 'extended' mode reductions by specifying an 
   approximate source extent (FWHM) via the 'sourcesize' option (in arcsecs), 
   and by changing the number of iterations via 'rounds'. E.g.

     > crush scuba2 -extended -sourcesize=300 -rounds=20 [...]

   (The above will set the expected source size to 300" fwhm and will iterate 
   20 times.) The larger the 'sourcesize', and the more you iterate, the more 
   extended emission you will recover, but you will also see more and more 
   noise, especially on the larger scales (such as a wavy background). You need
   to make you own call as to what is best.

   You can find more information on the recovery of extended emission in 
   the main README, under the 'Advanced Topics' section.


3.4 Compact Sources
===================

   For compact sources (<~ 2') you may get a cleaner image if you remove
   gradients from the exposures. The rationale is that the gradient removal
   will reject sky-noise to first order. However, it will also toss out any
   structure that is comparable to, or larger than, FoV/2. However, structures 
   much smaller than that will remain largely unaffected. 

   Gradient removal is default for 'faint' and 'deep' mode reductions. If you
   want to turn on gradient removal otherwise, you can do it via the 
   'gradients' flag. E.g.:

     > crush scuba2 -gradients [...]


3.5 Smoothing
=============

   You have the option to smooth your images via the 'smooth' option. Beware,
   however, that CRUSH typically applies one smoothing for intermediate maps
   (to increase redundancy between iterations), and another for the final
   output map. Therefore, to smooth the output map by a 6" FWHM Gaussian, you
   will want to use:

    > crush scuba2 -final:smooth=6.0 [...]

   (The prepended "final:" directive instructs crush to apply this setting
   in the final iteration, overriding whatever setting was specified for it
   before.)

   Beyond specifying the size (FWHM) of the smoothing kernel, you can also
   set one of: 'minimal', 'halfbeam', '2/3beam', or 'beam' to specify the
   smoothing kernel relative to the beam size ('minimal' is 1/3 of a beam).
   E.g.:

    > crush scuba2 -final:smooth=halfbeam [...] 

   Default reductions are unsmoothed, whereas 'faint' images will be smoothed
   with '2/3beam', and 'deep' images will be 'beam' smoothed, by default.

   To disable smoothing your output image, you can use:

    > crush scuba2 -final:forget=smooth [...]

   To learn more about the effects of smoothing, please refer to the section
   'Pixellization and Smoothing' under 'Advanced Topics' in the main README.




#####################################################################
4. Advanced Topic: Reading Scans
#####################################################################


4.1 FITS vs. SDF
================

   The SCUBA-2 data are recorded in the proprietary SDF format of Starlink.
   While the SDF files can be accessed with the Starlink software suite, the 
   format itself is not easily readable independently. Therefore, CRUSH 
   accesses these data via the Starlink suite.

   The Starlink suite provides a utility for converting SDF files to FITS,
   which is the format expected by CRUSH. The 'ndf2fits' tool usually resides 
   in the 'bin/convert' subdirectory of the Starlink installation directory. 

   You have two choices for handling the conversion. You can either do this
   manually, or let CRUSH convert the files on-the-fly, as needed. In either
   case, you will need access to the Starlink software suite. 

   Manual Conversion
   -----------------
   If you opt for manual conversion, make sure to use the 'proexts' option
   of 'ndf2fits'. E.g.:

	> ndf2fits <scan-name>.sdf <scan-name>.fits proexts

   You should name the FITS files exactly like the originals SDF ones, changing
   only the extension from '.sdf' to '.fits'.

  
   On-the-Fly Conversion
   ---------------------
   The alternative is to let CRUSH handle the conversion automatically, by 
   calling on 'ndf2fits' internally. If 'ndf2fits' is in your path, you should
   be set to go. Otherwise, you will have to instruct crush where to find the 
   'ndf2fits' executable. You can do this by editing 'default.cfg' in the 
   'scuba2' subdirectory of crush. Change the value of the 'ndf2fits' keyword 
   to indicate the location of the executable. E.g.:

        ndf2fits /usr/local/starlink/bin/convert/ndf2fits

   Alternatively, if the STARLINK_DIR environment variable is set, and points
   to the Starlink directory, as recommended by the Starlink installation, then
   the default setting (in scuba2/default.cfg):

        ndf2fits {@STARLINK_DIR}/bin/convert/ndf2fits

   will also work. If you set this up correctly, CRUSH will manage the 
   conversion of SDF files automatically, writing the FITS files in the same
   directory as the SDF.

   

4.2 '450um' vs '850um' reduction mode
=====================================

   When reducing SCUBA-2 data with CRUSH, you should specify the wavelength, in
   which you are interested. This way CRUSH will load settings that are
   appropriate for mapping in that band. You can choose between '450um' or 
   '850um' reduction modes. E.g., on the command-line:

      > crush scuba2 -450um [...]
   
   If you do not define the wavelength, then CRUSH defaults to '850um' 
   reduction.
   (The wavelength does not have to be the first argument, but should precede
   the first scan number or name in the argument list).


4.3 Locating scans by date and scan number
==========================================

   The preferred way of reading SCUBA-2 data is by 'date' and scan number.
   The convention of SCUBA-2 is to spread the scan data over several files.
   Data from each subarray is dumped at some intervals (typically 30s), which 
   are referred to as subscans. Thus, with two subarrays, the data for a 10 
   minute scan are typically spread over 40 files.

   (The file names begin with the subarray id, 's4a' for the current single 
   450um subarray, and 's8d' for the single 850um subarray, then continue with
   the date ('YYYY-MM-DD' or 'YYYYMMDD' format), scan number, and finally the 
   subscan number.)
   
   Thus, given the location of the data ('datapath' option), the reduction 
   wavelength ('450um' or '850um'), the date, and the scan number, CRUSH can 
   easily find all files necessary. E.g.:

      > crush scuba2 -450um -datapath=/home/data/scuba2 -date=2010-03-01 36

   The above line will load all usable subscans of scan #36. Some subscans
   may be designated unusable by CRUSH, either because the subscan is a dark
   measurement, or because it contains no coordinate data from the telescope, 
   or because the subscan is shorter than a preset minimum length (3s). These 
   will be dropped from the reduction queue. You can override the minimum 
   length requirement by changing or commenting the 'subscans.minlength' option
   in 'scuba2/default.cfg'

   Scan ranges can be specified as usual, e.g.:

      > crush scuba2 -date=2010-03-01 39-42 45-48 -date=2010-03-02 12-15 17
   
   
4.4 Specifying scans via file names
===================================

   The alternative is to specify file names directly. You can use both full
   pathnames (which can contain wildcards and other shell-expansions) or
   give exact file names (without wildcards, etc.) relative to 'datapath'. 
   Locating scans by file names is not recommended in general, since it allows
   for processing 450um scans in '850um' mode (and vice versa) or even mixing
   the data from the different wavelengths. On the other hand, you may want
   to use file-names for locating scans, if:
	
	* your scans are named differently from the standard convention, or

	* you want to control which subscans are processed.

   For example:
	
      > crush scuba2 -450um /data/scuba2/s4a20100301_00036_0002.sdf

   or, equivalently

      > crush scuba2 -450um -datapath=/data/scuba2 s4a20100301_00036_0002.sdf

   will both process only the 2nd subscan of scan 36 (taken on 2010-03-01) in 
   '450um' mode.




#####################################################################
5. Advanced Topic: Calibration and Pointing
#####################################################################	

The default configuration of CRUSH provides approximate blind calibration, 
expected to be good to ~10% rms, when using aperture fluxes in apertures large 
enough to contain the SCUBA-2 error beams of the emission.

When using peak fluxes, the blind calibration accuracy is expected to be worse
due to the variations in main-beam efficiency with focus, elevation and
the surface quality of the JCMT.

To improve on the calibration accuracy, you can switch to a smoothed 183 GHz 
tau value based on the JCMT 183 GHz archive ('tau=jctables'), and/or provide 
order-of-unity corrections based on observations of nearby calibrator sources
(via 'scale'). E.g.


5.1 Extinction Correction
=========================

   The JCMT records both 225GHz and 183GHz opacities from radiometer 
   measurements. These are stored in the data files. By default, CRUSH uses the
   183GHz value with an appropriate scaling relation to the SCUBA-2 bands, to 
   obtain a guesstimate of the in-band opacities for the atmospheric extinction 
   correction. 

   You can get a more accurate local average value of the 183 GHz tau using the
   the 'tau=jctables' option (requires an online connection!).

      > crush scuba2 -tau=jctables <scans> [...]

   The above will download the table of measurements from the online JCMT 183 
   GHz tau archive for the date(s) of each scan, and calculate a local average
   around the time of observation within a Gaussian window (see 'tau.window'
   option in the GLOSSARY and in 'scuba2/default.cfg'). The resulting tau 
   should be more precise than the default, but is noticeably slower to obtain
   (due to the size of the tables that need to be downloaded on demand) and 
   requires a online connection.


5.2 Calibration Correction
==========================

   Even if the opacity is perfectly determined, there are remaining 
   uncertainties in the calibration. These are mainly due to the beam-quality 
   and variations thereof, due to imperfect focusing, gravitational flexing of 
   the JCMT primary with elevation, or changes in surface quality (e.g. before 
   and after a holography run). 

   Because the JCMT beams are not perfectly Gaussian (esp. at 450um), the 
   calibration scaling is also dependent on the type of flux measure used. You
   will need a different flux scaling when using apertures (of different sizes)
   or peak fluxes (for point sources). To calibrate your science target 
   properly you should first determine an order-of-unity calibration correction
   using the same flux measure (peak flux or integrated flux in the *same* 
   aperture) on a known calibrator source. You can use 

      > crush scuba2 -point [...]

   to measure both peak fluxes and fluxes in 98% inclusive apertures on compact
   calibrator sources. You can then determine a appropriate calibration scaling
   as the ratio of expected flux to observed flux. To multiply your science
   data with this scaling, use the 'scale' option of crush. E.g.:

      > crush scuba2 [...] -scale=1.23 <scans> [...]

   The blind calibration of CRUSH is based on apertures large enough to capture
   98% of the integrated source flux of point sources (including the first 
   error beam), because this measure is much less dependent on beam quality. 
   Therefore, if you are to use peak fluxes (e.g. for deep field source 
   extraction) you should pay special attention to derive appropriate 
   calibration scalings using the same method.


5.3 Pointing Corrections
========================

   You can specify incremental pointing corrections as AZ,EL offsets for every
   group of scans, the usual way using the 'pointing' option. E.g.:

      > crush scuba2 [...] -pointing=-2.3,1.6 <scans> [...]




#####################################################################
6. SCUBA-2 specific pixel divisions
#####################################################################

For a general overview of channel divisions, please consult Section 3.
(Correlated Signals) in the main README document.

   subarrays     A grouping of pixels by subarrays.

   rows          Pixel grouping by detector columns. There are 40 columns on
                 each subarray. Incidentally, the channels in a row are all
                 read out through the same SQUID mux. Thus, the group 'cols'
                 is also the grouping by SQUID multiplexer.

   cols          Pixel grouping by detector rows. There are 32 rows on each
                 subarray.

   blocks        Pixel grouping by user-specified geometric blocks of n by m
		 pixels. The size of the blocks is specified via the 'block'
		 option.   



#####################################################################
7. Glossary of SCUBA-2 specific options
#####################################################################


   450um		Select 450um imaging mode.

   850um		Select 850um imaging mode.

   block=<n>
   block=<n>x<m>	@Advanced
			Set the block size (pixels) for 'correlated.blocks'.
			The decorrelation block size will be <n> by <n>, or
			<n> by <m> pixels, depending whether there are 1 or 2
			arguments (separated by 'x').
			@See: 'correlated.blocks'

   blocks		@Alias: 'correlated.blocks'

   cols			@Alias: 'correlated.cols'
                        @Expert
                        Specifies the grouping of channels by geometric columns
			on the array, following the storage and array geometry
			convention of 32 rows by 40 columns. Since the 32 
			pixels of a 'column' are read through the same SQUID
			MUX, decorrelating on 'cols' is effectively 
			decorrelates channels that share a MUX.
			@See: 'correlated.<?>'

   darkcorrect		@Expert
			@Since: 2.31
			Attempt to use dark SQUID data to remove common mode
			noise on SQUID MUX. In practice, the use of this option
			may not improve reduction quality, and may in fact
			degrade it. Therefore, the option is disabled by
			default.

   date=YYYY-MM-DD
   date=YYYYMMDD	Specify the the date of observation for locating scan
			data. When used together with 'datapath' and scan 
			numbers, CRUSH can locate all the necessary input files
			easily. E.g.:

			  > crush scuba2 -date=2014-05-30 75

			@See: 'datapath'

   he3			@Alias: 'correlated.he3'
			Remove correlations with the detector temperature (at
			the He3 stage -- hence the name). This may, or may not
			improve the sensitivity of the data.

   mirror		@Expert
			Indicate a mirrored focal plane layout. (Since one
			subarray is seen in transmission, the other in 
			reflection, through the beam-splitter, one of the
			subarray layouts is a mirror image of the other...

   ndf2fits=<path>	Specify the path to the Starlink 'ndf2fits' executable
			(usually in bin/convert subdirectory of the Starlink
			distribution. You may not need to change this option
			if the STARLINK_DIR environment variable is set to
			point to the Starlink installation folder on your 
			system, as recommended by the Starlink installation
			instructions. Otherwise, you can set this e.g.:

			  ndf2fits /usr/local/starlink/bin/convert/ndf2fits

   pixelmm=X		@Expert
			Specify the physical pixel size (separation) in mm,
			on the detector arrays.
			@See: 'platescale'

   platescale=X		@Expert
			Specify the scaling from mm on the focal plane to
			arcsec on the sky.
			@See: 'pixelmm'

   rows			@Alias: 'correlated.rows'
			@Expert
			Specifies the grouping of channels by geometric rows on
			the array, following the storage and array geometry
			convention of 32 rows by 40 columns.
			@See: 'correlated.<?>'

   <sID>.position=dX,dY	@Expert
			Set the position of pixel 0 (first row, first column)
			of subarray <sID> to dX, dY pixels from the optical 
			center. E.g.

			  s8a.position -37.3, -35.3
			
			The offset will be applied *after* any rotation.
			@See: '<sID>.rotation'

   <sID>.rotation=X	@Expert
			Set the orientation angle (degrees) of subarray <sID>
			on the focal plane. E.g.

			  s8d.rotation -90

			@See: <sID>.position'

   <sID>.scale=X	@Expert
			Set the subarray scalings. Scaling the data from the
			different subarrays such that the pixels on a subarray
			show the same response, on average, from one subarray
			to the next, is useful for maintaining calibration when
			not all subarray data is available, or when reducing
			just a select set of subarrays. E.g.:

			  s8a.scale 0.63

			@See: 'subarray'

   subarray=<list>	@Expert
			Restrict the reduction to just the specified subarrays
			at the current wavelength. The argument is a list of
			subarray letter codes (a thru d) that specify which
			subarray data to use. E.g.:

			 >crush scuba2 -850um -subarrays=a,c [...]

			Will reduce only the 850um subarrays a and c.	
			@See: '<sID>.scale'



#####################################################################
8. SCUBA-2 specific log quantities
#####################################################################

Coming soon...


   dir

   filter

   foc.X

   foc.Y

   foc.Z

   obsmode

   obspattern

   obstype  


#####################################################################
9. Credits
#####################################################################

This re-incarnation of the CRUSH support for SCUBA-2 would not have happened 
without the essential help from others. The thanks go to Tony Mroczkowski for 
the persistent prodding to create this module, and for seeking out the 
necessary training data for configuring it; and Ed Chapin, without whom I would
not have made enough sense of the data themself. If you like the outcome, I 
suggest you invite them (and me) for a beer or two, if you happen to bump into 
them (or me). :-)



-----------------------------------------------------------------------------
Copyright (C)2015 Attila Kovacs <attila[AT]submm.caltech.edu>
