# CRUSH: SOFIA

Author: Attila Kovacs <attila[AT]sigmyne.com>

Last updated: 23 Sep 2018


----------------------------------------------------------------------------

#### Table of Contents


1. __Introduction__
     - 1.1. A brief description of of what CRUSH does and how...

2. __Installation__
     - 2.1. Installation from tarball (POSIX/UNIX, incl. Mac OS X)
     - 2.2. Optional system-wide installation.i
     - 2.3. Optional startup environment & Java configuration

3. __Quickstart guide__
     - 3.1. Locating scan data
     - 3.2. A few common options to use with SOFIA instruments

4. __Technical details for expert users__
     - 4.1. Glossary of SOFIA specific options
     - 4.2. SOFIA specific log quantities

-----------------------------------------------------------------------------



## 1. Introduction


This document contains information specific to using CRUSH-2 with a SOFIA 
instrument in general, without discussing the specific use for the particular
instrument per se. Separate READMEs cover the particulars of each instrument.

You may use this as a standalone guide, with instructions for 
installation and basic use, or in conjucntion with the main CRUSH-2 README. 
Either way, it is recommended that you also familiarize yourself with 
the contents of the main CRUSH-2 README and its Section 1 (Getting Started) 
especially. 



### 1.1 A brief description of what CRUSH does and how...

CRUSH is a reduction pipeline, designed mainly to remove correlated signals 
(correlated noise) in the detector time-streams to arrive at clean & 
independent bolometer signals, which are then used to derive a model of the 
source (usually an image).

As such it is not an interactive reduction software (e.g. as opposed to e.g. 
BoA). The term _scripting_ in CRUSH mainly means defining configuration options
(in the command line or through configuration files) which are parsed in the 
order they are read.

During the reduction CRUSH aims to arrive at a consistent set of solutions for 
various correlated signals, the corresponding gains and the pixel weights, as 
well as tries to identify and flag problematic data, and applies appropriate 
spectral filtering.

This means a series of reduction steps, which are iterated a few times until 
the required self-consistent solutions are arrived at.

To learn more about the details please refer to Kovacs, A., "CRUSH: fast and 
scalable data reduction for imaging arrays," Proc. SPIE 7020, 45, (2008). If 
that does not satisfy your curiousity, then you can find yet more explanation 
in Kovacs, A., PhD thesis, Caltech (2006).





## 2. Installation


### 2.1. Installation from tarball (POSIX/UNIX, incl. Mac OS X)

#### Step 1.
    
  Install Java (if necessary), e.g. from www.java.com. If you already have 
  Java, check that it is version 1.8.0 (a.k.a. Java 8) or later, by typing:

    > java -version

  Note, that The GNU java a.k.a. `gij` (default on some older RedHat and Fedora 
  systems) is painfully slow and unreliable, and will not always run CRUSH 
  correctly. If you need Java, you can download the latest JRE from

   http://www.java.com

#### Step 2.
  
  Unpack the tarball in the desired location (e.g. under `~/astrotools/`):

    > cd ~/astrotools
    > tar xzf crush-2.xx-x.tar.gz
   

#### Step 3.
    
  Verify that CRUSH works:

    > cd crush
    > ./crush

  You should see a brief information screen on how to use CRUSH.



### 2.2. Optional system-wide installation

 To create system-wide access to the crush executables, you may optionally 
 wish to run `install.sh` (as root or with `sudo`). It will link the
 executables to `/usr/bin`, and install man pages.

    > cd crush
    > sudo bash install.sh

 You can check the success of the above optional step by typing:

    > man crush

 If all is in order, you should see the UNIX manual page on the crush. 



### 2.3. Optional startup environment & Java configuration
   
 CRUSH ships with a default Java configuration. On Windows and the most common 
 UNIX  platforms (Linux, Mac OS X, BSD, and Solaris), it will automatically 
 attempt to set an optimal configuration. On other platforms, it comes with 
 fail-safe default values (default java, 32-bit mode and 1GB of RAM use).

 You can override the defaults by placing your settings in arbitrary files 
 under `/etc/crush2/startup` or `~/.crush2/startup` (for the equivalent
 configuration under Windows, please refer to `README.windows`) 

 (Any  settings in the user's home under `~/.crush2/startup` will override the 
 system-wide values in `/etc/crush2/startup` or `C:\Program Data\startup`. 
 If multiple config files exist in the same location, these will be parsed by 
 the shell in non-specific order).

 E.g., placing the following lines in `~/.crush2/startup/java.conf` overrides
 all available settings: 

    JAVA="/usr/java/latest/bin/java"
    USEMB="4000"
    JVM="server"
    EXTRAOPTS="-Djava.awt.headless=true"

 Upon startup CRUSH will find and apply these settings, so it will use
 `/usr/java/latest/bin/java` to run CRUSH with 4GB of RAM, using the HotSpot 
 `server` VM, and in headless mode (without display, mouse or keyboard).

 Below is a guide to the variables that you can override to set your own
 Java runtime configuration:

    JAVA           Set to the location of the Java executable you want to use. 
                   E.g. `java` to use the default Java, or
                   `/usr/java/latest/bin/java` to use the latest from Oracle or
                   OpenJDK.

    USEMB          Set to the maximum amount of RAM (in MB) available to
                   CRUSH. E.g. "4000" for 4GB. On a 32-bit OS, or with a 32-bit
                   Java installation no more than 2GB of RAM may be accesssed. 
                   In practice, the maximum 32-bit OS/Java values range from 
                   around "1000" (32-bit Windows / Java) to "1900" (32-bit 
                   Linux / Java).    

    JVM            Usually set to `server` for Oracle or OpenJDK. If using 
                   IBM's Java, set it to "" (empty string). On ARM platforms, 
                   you probably get better performance using `jamvm` or 
                   `avian`. o see what VM options are available, run 
                   `java -help`. The VM options are listed near the top of the 
                   resulting help screen.

    EXTRAOPTS      Any other non-standard options you may want to pass to the 
                   Java VM should go here. Typically set to "".


 You can also specify environment variables, and add shell commands (bash),
 since these configuration files are in fact sourced as bash scripts before
 launching Java / CRUSH. For example you can add:

```bash
 CRUSH_NO_UPDATE_CHECK="1"
 CRUSH_NO_VM_CHECK="1" 

 echo "Will try to parse my own configuration now... "
	  
 if [ -f ~/mycrushconfig.sh ] ; then
     echo -n "OK"
     source ~/mycrushconfig.sh
 else
     echo -n "Not found"
 fi
```

 The above will disable update checking (not recommended!) and VM checking
 (also not recommended!) and will source the contents of  `~/mycrushconfig.sh` 
 if and when such a file exists.
 
   

## 3. Quickstart guide


### 3.1. Locating scan data


#### By file name/path

  The default method of locating files is by file name, which may specify 
  either an absolute path, e.g.:

    > crush hawc+ /data/2016-12-03_HA_F354_009_CAL_unk_HAWD_HWPOpen_RAW.fits

  or it can be filename/path relative to `datapath`
 
    > crush hawc+ 2016-12-03_HA_F354_009_CAL_unk_HAWD_HWPOpen_RAW.fits

  The two are equilalent assuming that `datapath` is set to `/data/hirmes` in 
  the second case, e.g. in the user configuration file 
  `~/.crush2/hirmes/default.cfg`, or on the command-line.


#### By flight and scan numbers

  Often the simpler way of locating input files is by a combination of flight
  and scan numbers. This is often shorter, and allows to specify multiple scans
  and ranges with more ease.

  Scan lookup by flight and scan number requires you to set `datapath` to point
  to the data directory. E.g., by placing the line in the user configuration 
  for HAWC+ (E.g. `~/.crush2/hawc+/default.cfg`):

    datapath /data/hawc+

  Now, you may simply reduce scan 9 from flight 354 as:

    > crush hawc+ -flight=354 9

  You can also reduce multiple scans, from multiple flight together. E.g.:

    > crush hawc+ -flight=354 104-105 129 -flight=362 13 16 33-35

  The above will co-reduce 3 scans (104, 105, 129) from flight 354 with 5 
  scans (13, 16, 33, 34, 35) from flight 362.

  __Note.__ Locating scans by flight and scan number may not be possible for
  simulated data, or data that has been renamed from it's original file name.
   

### 3.2. A few common options to use with SOFIA instruments

 You may specify reduction options as command-line arguments to CRUSH. Each 
 option in the line begins with a dash `-`, and should not contain white 
 spaces, unless these are enclosed in quotes. E.g. below is a valid option 
 line, with three options, one of which contains a white-space characters:

   > crush hawc+ -faint -name="My HAWC+ image.fits" -projection=TAN ...

 Below is a brief guide to a few useful options, grouped by functionality.


#### Source type options
   
  The default reduction is generally OK for sources smaller than the field of 
  view (<FoV/2) with S/N in the 10--1000 range. Outside this range, the 
  following options can be used to obtain better results:


    -bright             Reduce very bright sources (S/N > 1000).

    -faint              Reduce faint sources (S/N < 30).

    -deep               Optimized for the extraction of deep-field point 
                        sources. It is similar to `-faint` (above) but it also 
                        spatially filters the map, removing all signals above a
                        few beam scales. Use only for very faint point sources.
    
    -extended           Reduce extended sources (>FoV/2). The retention of
                        scales larger than ~FoV/2 come at a price of increased 
                        map noise (on those scales). See Section 2.4 of the 
                        main README on the 'Recovery of Extended Emission'. Can
                        be used on its own (assuming the default brightness 
                        setting) or together with `-faint` or `-bright`.

     -sourcesize=X      Can be used together with `-extended` (above), to
                        tweak just how extended the source is expected to be.
                        The argument is an approximate source diameter in
                        arcseconds. E.g. `-sourcesize=300.0`. 

     -source.sign=Z     By default, CRUSH assumes that sources are seen in
                        emission (+), and biases the reduction as such to 
                        produce images without filter bowls or other negative 
                        reduction artifatcs. However, your object may contain 
                        absorption features, for which you may want ot bias in 
                        the opposite direction (-), or not bias at all (0).
                        Set the sign accordingly to +, -, or 0. E.g.
                        `-source.sign=+`


#### Output map options
  These options change how the output FITS image will be constructed. All 
  these options have reasonable default values, and so you should only use them
  to override those if needed.


    -grid=X             Set the pixelization of the map to X arcseconds. (The 
                        default pixelization is chosen to be be around 1/5th of
                        a beam for each HAWC+ band).

    -projection=XXX     Change the WCS spherical projection. The following
                        projections are supported:

                                SFL  --  Sanson-Flamsteed
                                SIN  --  Slant Orthographic
                                TAN  --  Gnomonic
                                ZEA  --  Zenithal Equal Area
                                MER  --  Mercator
                                CAR  --  Plate-Carree
                                AIT  --  Hammer-Aitoff
                                GLS  --  Global Sinusoidal
                                STG  --  Stereographic
                                ARC  --  Zenithal Equidistant

                        The default is SFL (Sanson-Flamsteed). E.g. 
                        `-projection=TAN`.

    -ecliptic           Produce maps in ecliptic coordinates, instead of the
                        default equatorial (same as `-system=ecliptic`).

    -galactic           Produce maps in galactic coordinates, instead of the
                        default equatorial (same as `-system=galactic`).

    -gzip               Compress the output (e.g. FITS) with GZIP, if possible.
                        (The output file name will have .gz extension added as
                        needed.)

    -supergalactic      Produce maps in super-galactic coordinates, instead of 
                        the default equatorial (same as 
                        `-system=supergalactic`).

    -horizontal         Produce maps in horizontal coordinates, instead of
                        the default equatorial (same as `-system=horizontal`).

    -focalplane         Produce maps in focal-plane coordinates, instead of the
                        default equatorial (same as `-system=focalplane`).

    -final:smooth=X     Smooth the final map by a Gaussian with X arcsec FWHM.
                        Alternatively, X can be one of `minimal`, `halfbeam`
                        `2/3beam` or `beam`. E.g. `-smooth=8.0` or 
                        `-smooth=2/3beam`. To turn smoothing off completely
                        use `-final:forget=smooth`.



## 4. Technical details for expert users

From this point on, the documentation is of more technical nature, intended
for expert users only.


### 4.1. Glossary of SOFIA specific options

    atran.amcoeffs=<list>       A list of comma-separated polynomial 
                        coefficients for the airmass correction term in the 
                        Vacca et al. ATRAN-based static atmospheric model, for 
                        the polynomial expansion around 41 kft altitude and 45 
                        deg elevation.
                        @See: 'atran.altcoeffs', 'atran.reference'

    atran.altcoeffs=<list>      A lust of comma-separated polynomial
                        coefficients for the altutide correction term in the
                        Vacca et al. ATRAN-based static atmospheric model, for
                        the polynomial expansion around 41 kft altitude and 45
                        deg elevation.
                        @See 'atran.amcoeffs', 'atran.reference'

    atran.reference=X   The typical transmission at the reference point of 41
                        kft altitude and 45 deg elevation for the Vacca et al.
                        ATRAN-based static atmospheric model. Together with
                        'atran.amcoeffs', 'atran.altcoeffs', this defines
                        a standard atmospheric correction model for SOFIA.
                        The option 'tau=atran' activates this model as the
                        one used for calculating atmospheric corrections in
                        CRUSH.
                        @See: 'tau', 'atran.amcoeffs', 'atran.altcoeffs'

    calibrated          Set this option when the reduction includes the final
                        calibration (opacities and other calibration
                        corrections) to produce Level 3 data. Otherwise, CRUSH
                        will produce Level 2 output.

    flight=N            Set the flight number for locating data files using
                        scan numbers only. Remember to set 'datapath' also to
                        specify the folder where scan files are stored.
                        @See: 'datapath'

    organization        Specify the organization at which CRUSH is being used
                        for reducing data. The value of this option is stored
                        directly in the FITS ORIGIN header key as required by
                        the DCS. If you want the ORIGIN key to be set properly
                        you might consider adding the organization option to
                        `~/.crush2/sofia/default.cfg`, e.g. as:

                          organization SOFIA Science and Mission Ops 

     pwv41k=X           For a simple static atmoshpere model, set the typical
                        precipitable water vapor (PWV) level to X microns at
                        41 kft altitude.
                        Based on it, and on 'pwvscale', CRUSH will calculate
                        and appropriate PWV value for each scan. The PWV
                        value can then be converted into a typical zenith tau
                        value, provided the 'tau.pwv.a' and 'tau.pwv.b' scaling
                        parameters are defined. One may then select this tau 
                        value to use for opacity corrections via the 'tau=pwv'
                        option. 
                        @See: 'pwvscale', 'tau.pwv.a', 'tau.pwv.b', 'tau'

     pwvscale=X         Set the scale height (kft) for the PWV, locally around
                        the reference value at 41 kft (see 'pwv41k').
                        @See: 'pwv41k'

     tau=atran          Selects the static ATRAN-based atmospheric model by
                        Bill Vacca for the opacity corrections.
                        @See: 'atran.amcoeffs', 'atran.altcoeffs'
                              'atran.reference'

     tau=pwv            Selects the built-in static atmospheric model of CRUSH
                        for the opacity corrections.
                        @See: 'pwv41k', 'pwvscale', 'tau.pwv.a', 'tau.pwv.b'

### 4.2. SOFIA specific log quantities.


    ac.airspeed         (km/h) Airspeed.
    ac.alt              (m) Altitude.
    ac.altkft           (kft) Altitude.
    ac.dir              (deg) Direction of heading.
    ac.gndspeed         (km/h) Groundspeed
    ac.lat              (rad) Geodetic latitude
    ac.latd             (deg) Geodetic latitude
    ac.lon              (ra) Geodetic longitude
    ac.lond             (deg) Geodetic longitude
    ac.trkangle         (rad) Tracking angle

    array.sibsx         (pix) Instrument boresight pixel in x.
    array.sibsy         (pix) Instrument boresight pixel in y.

    chop.amp            (arcsec) Chop amplitude
    chop.angle          (deg) Chop angle
    chop.flag           'C' if chopping or '-' if not
    chop.frequency      (Hz) Chop frequency
    chop.profile        Chop profile
    chop.sys            Chop coordinate system
    chop.tilt           (deg) Chop tilt angle
    chop.tip            (deg) Chop tip angle

    dfoc                (um) Focus offset.

    env.tamb            (C) Ambient temperature
    env.pwv             (um) Precipitable Water Vapor level
    env.sect            (C) Secondary mirror temperature.
    env.t1              (C) T1 temperature of the primary.
    env.t2              (C) T2 temperature of the primary.
    env.t2              (c) T3 temperature of the primary.
    
    inst.cfg            Instrument configuration
    inst.datatype       Type of data produced by instrument.
    inst.exp            (s) Total exposure time
    inst.inttime        (s) Ttoal on-source integration time
    inst.mode           Operating mode
    inst.slit           Slit element name
    inst.spec1          First spectral element (from SPECTEL1)
    inst.spec2          Second spectral element (from SPECTEL2)
    inst.wave           (um) Central wavelength

    mssn.id             SOFIA Mission ID.
    mssn.leg            Flight leg number
    mssn.plan           Flight plan ID.

    nod.amp             (arcsec) Nod amplitude
    nod.angle           (deg) Nodding angle
    nod.dwell           (s) Nod dwelling time
    nod.n               Nod cycles
    nod.pattern         Nod pattern style
    nod.pos             Nod beam position
    nod.settle          (s) Nod settling time
    nod.style           Nod style
    nod.sys             Nod coordinate system name

    obs.aor             SOFIA AOR ID
    obs.aot             SOFIA AOT ID
    obs.dict            FITS keyword dictionary version ID.
    obs.fgid            File group ID.
    obs.imgid           Image ID
    obs.obj             Object name
    obs.objtype         Object type
    obs.obsid           Observation ID.
    obs.src             Data source

    orig.creator        Creator software & version. E.g. 'crush v2.42-1'
    orig.file           Original file name at creation
    orig.observer       Name(s) of observer(s) or PI(s).
    orig.operator       Name(s) of the operator(s).
    orig.org            Organization where file was created / reduced.  

    pnt.dSIBSX          (pixels) SIBS pointing increment in X
    pnt.dSIBSY          (pixels) SIBS pointinh increment in Y 

    proc.stat           Header key status. E.g. 'ORIGINAL'
    proc.level          Product level
    proc.nspec          Number of spectra
    proc.product        Product type
    proc.q              Product integer quality
    proc.quality        Product quality string.

    scan.Ax             (arcsec) X amplitude
    scan.Ay             (arcsec) Y amplitude
    scan.angle          (deg) Scan position angle
    scan.cross          Is cross scanning?
    scan.dec            (deg) Declination
    scan.dY             (arcsec) Raster scan step size
    scan.frel           Lissajous Y/X frequency ratio
    scan.iters          Number of iterations
    scan.nsubs          Number of subscans
    scan.PA             (deg) Current scan position angle
    scan.pattern        Scanning pattern
    scan.phi0           (deg) Lissajous Y/X relative phase.
    scan.ra             (hour) R.A.
    scan.speed          (arcsec/s) Scanning speed.
    scan.strips         Number of raster strips
    scan.sys            Scan coordinate system name.
    scan.T              (s) Scan duration
    scan.t0             (s) Lissajous time shift.
    scan.trk            Is tracking enabled?
    scan.type           Type of scan, e.g. 'LISSAJOUS'.
    scan.X              (arcsec) Raster scan length

    spec.bw             (MHz) Total spectral bandwidth.
    spec.df             (MHz) Frequency spacing of channels.
    spec.obsfreq        (GHz) Observing frequency at the reference channel.
    spec.restfreq       (GHz) Rest frequency at the reference channel.
    spec.tsys           (K) System temperature (if applicable).
    spec.vsys           Radial velocity reference system.
    spec.vframe         (km/s) Radial velocity of refence frame wrt observer.
    spec.vrad           (km/s) Source radial velocity wrt observer.
    spec.z              redshift of source.

    tel.focus           (um) Telescope z-focus value
    tel.bdec            Boresight declination
    tel.bra             Boresight RA
    tel.cfg             Telescope configuration
    tel.el              Telescope elevation
    tel.epoch           Epoch for equatorial coordinates (e.g. 2000.0)
    tel.fbc             FBC status
    tel.los             Telescope line-of-sight angle
    tel.rdec            Requested declination
    tel.rew             Timestamp of last rewind.
    tel.rra             Requested RA
    tel.trkerr          Had tracking errors?
    tel.trkmode         Tracking mode
    tel.vpa             Vertical position angle (i.e. parallactic angle)
    tel.xel             Telescope cross-elevation angle
    tel.za              Telescope zenith angle

------------------------------------------------------------------------------
Copyright (C)2018 -- Attila Kovacs <attila[AT]sigmyne.com>


