Data Reduction Page:


General Remarks
Updates 
most recent
28.01.98
06.04.98 a
06.04.98 b
17.08.98
20.08.98
06.09.98
10.09.98
06.10.98
08.10.98
10.11.98
19.01.99
07.04.99 a
07.04.99 b
18.04.99
27.05.99
04.06.99
05.06.99
20.07.99
26.07.99
17.09.99
27.09.99
30.09.99
26.11.99
31.01.00
04.03.00
17.04.00
26.05.00
03.05.01

MPE-3D Pages

MPE-3D Homepage

The Instrument

References and Links

SPECIAL NOTES FOR SPECIFIC
RUNS

Public FTP-Site

Recent Results from 3D

 
     

    3D Data Reduction Software

    • Data reduction for 3D is based on GIPSY. Routines which are specific for 3D data reduction are available via anonymous ftp. Note that this collection of software is constantly being updated!  Watch this space for update messages. 

    •  
    • A (partial) draft data reduction manual (manual-part2.ps.gz) is also available. Added 28 Jan 1998. 

    •  
    • Calibration Files are being assembled in our public ftp directory. The calibrations are being assembled in a scattershot fashion, so please check back if the one you want is not yet available. Added 06 Apr 1998. 
    • NOTE: see the entry from 03 May 2001 for an update on this feature!
       
    • The updated version of 3dmegacal (required for wavelength calibration) is now available. The file has been updated to allow for wavelength calibration of data from all of the AAT runs to date (November 1997, December 1997, February 1998, and March 1998). Be sure to also download the required auxilliary files. These files should be placed in the PATH pointed to by $gip_tsk. See the file 000_README for further details. Added 06 Apr 1998. 

    •  
    • There is a host of new routines which are for use in reducing or analyzing data. The whole lot can be downloaded as a single gzip'd tar file, or downloaded individually (see the Table below). These routines are as follows: 

    •  
      New 3D Software (added 17 Aug 1998)
      Problem Cause
      Solution
      Function
      Flux calibration of 3D data when no flux calibrator was observed, but when broadband aperture measurements exist in the literature. Accidental (or intentional) oversight.
      3dbootflux.c
      Bootstraps flux from literature flux measurements.
      Flux calibration of 3D data when a flux calibrator was observed.
      3dbootflux.c
      Outputs a conversion factor which can then be applied to the source .
      Atmospheric standard star exhibits strong H recombination lines. Accidental oversight or intentional observation!
      3dastar1.c
      Fits and removes H lines.
      You need to extract a spectrum from a region of your datacube which is circular, seeing-weighted, or a non-integer number of pixels in size.
      3dextract.c
      Returns the desired spectrum
      The seeing was so good that the automatic bad pixel finding routine thinks that your source is a series of bad pixels, and sets the mask accordingly. We never anticipated such good seeing!
      3dclip.c
      Column-wise flags pixels as bad if they exceed n absolute deviations from the median-subtracted column.
      Post-merging pixel-to-pixel noise is present in your spectra. On some occasions during the 97B and 98A AAT runs the dispersion direction was not perfectly perpendicular to the spatial dimension.
      3drowtil1.c
      Rectifies the tilt of the dispersion direction to be vertical (and can be used to laterally shift)
      Post-merging pixel-to-pixel noise is present in your spectra. On some occasions during the 97B and 98A AAT runs the dispersion direction was not perfectly perpendicular to the spatial dimension.
      3dmerge2.c
      Adds another level of "tweaking" to correct lateral offsets between the two piezo frames
      You want to include 3drowtil1, 3dmerge2, and 3dclip into your source reduction.
      3dcenter3.col
      Does the same as 3dcenter.col, but with those additions and substitutions.
      You want to include 3drowtil1 and 3dmerge2 into your atmospheric standard reduction.

      3dstar2.col
      Does the same as 3dstar.col, but with those additions and substitutions.

    • There is a slightly revised version of dpix.  The revised version has a hidden KEYWORD called BLANKFRAC which is used as a threshold for detecting BLANK pixels.  The revised version of dpix is available in  the usual place.  Added 20 Aug 1998. 

    •  
    • Special Note: At MPE we often see a need to produce model atmospheric transmission spectra.  In particular, we need to introduce these spectra into GIPSY.  For that task we have written a routine which serves as a GIPSY-interface for the atmospheric modelling program ATRAN (S.D. Lord, ``A New Software Tool for Computing Earth's Atmospheric Transmission of Near- and Far-Infrared Radiation,'' NASA TM-103957, 1992.).  This interface, 3datran, requires a local version of ATRAN (in particular, it requires that it reside in the path $gip_root/ATRAN).  If you want to make use of ATRAN in the context of GIPSY (or any other context, for that matter) please contact Steve Lord at IPAC to obtain a copy.  Added 06 Sep 1998. 

    •  
    • A (slightly) revised version of 3dstat is now available.  The revised version now writes the value of the KEYWORD VELZ on the plot (if PGLINES=y).  Added 10 Sep 1998. 

    •  
    • A (slightly) revised version of gipenv is now available.  The revised version fixes a "feature" which prevented it from working on (at least) Solaris 5.6 (in any event it works on HP-UX 10.20).  Added 06 Oct 1998. 

    •  
    • (Slightly) revised versions of 3dspec and 3dcal are now available.  The revised version of 3dspec fixed a typo which somehow crept into the version in the web software collection and affected high resolution H band data. The revised version of 3dcal fixes a "feature" which prevented it from working on (at least) Solaris 5.6 (in any event it works on HP-UX 10.20).  Added 08 Oct 1998. 

    •  
    • A new (and better!) 3dcenter COLA script is now ready for release. The new routine 3dcenter2000.col, is based on the original 3dcenter.col, but has a few more features built in: 

    •  
      1. 16x16 pixel images are rebinned to a finer 32x32 pixel scale. This improves the relative registration of the individual frames. 

      2.  
      3. The parameter TWEAK allows you to select the level of tweaking done before and during the piezo merging process. 

      4.  
      5. The parameter DPCOR allows you to select the method used for finding dead pixels which are not listed in the default dead pixel list 

      6.  
      An additional feature which is still undergoing debugging is one which would allow you to throw away "bad" data on-the-fly. This data editting will be initiated within 3dfit64, when it queries you for the finetuning center position (either because you've selected manual finetuning [FCEN=0 or 1], or because 3dfit64 cannot find a suitable peak). When prompted for the center position you will be able to specify a position outside of an acceptable range. From that point on that particular subset will be ignored (that is, as soon as I get the routine(s) debugged). Note that at present specifying a center position outside of the acceptable range results in the entire data set being corrupted! NOTE: see the entry from 07 April 1999 for an update on this feature!

      If you must throw away data you must note the subset(s) in question and rerun 3dcenter2000.col in a piecemeal fashion such that the parameters START and END are chosen to avoid the "bad" data. Finally, in order to use 3dcenter2000.col you will also need to have the routines 3dcalc64.c, 3dfit64.c, 3dmean64.c, 3drebin1.c, 3dshift64a.c, and 3dsmoo.c. These routines as well as 3dcenter2000.col itself are available as a single bundle (see the README file).  Added 10 Nov 1998. 
       

    • A bug was found in 3dshift64a.c which fouled up the nature of the peak finding (for cases when fcen was either 2 or 3). The revised version is available in  the usual place.  Added 19 Jan 1999.

    •  
    • It is now possible to throw away source data on-line! While this may sound nonsensical (at best), anyone who has reduced a set of data with a bad patch in the middle will really appreciate this. The routines which must be updated for this to work for you are 3dfit64.c and 3dmean64.c. The data reduction proceeds as follows: 
       
      1. All of the data from a particular set are thrown into 3dcenter2000.col, and are processed in parallel (flatfielding all on-off pairs, dead pixel correcting all on-off pairs, ...). 

      2.  
      3. When the reduction reaches the stage of registering individual on-off pairs with respect to each other, the process is controlled (as always) by the value of fcen that you use, and your ability to throw away data depends on value of fcen and/or the nature of the data. This is summarized in the Table below 

      4.  
        fcen
        Value
        Centering Notes
        Coarse Fine
        0 man man This is the easiest of cases, as you have complete control over everything. When bad data is encountered in this case enter anything for the position of the coarse peak (at the Center= prompt of 3dshift64a), and one or more values outside of the range [-8,8] at the CENTER= prompt of 3dfit64.
        1 man auto When bad data is encountered in this case enter a position for the coarse peak (at the Center= prompt of 3dshift64a) which is as non-Gaussian as you can find. Hopefully, this will cause the automatic procedure within 3dshift64a to fail. As a result of this failure you will be prompted for the fine peak (with the Center= prompt of 3dshift64a). At this point give one or more values outside of the range [-8,8].
        2 auto man When bad data is encountered in this case enter one or more values outside of the range [-8,8] at the CENTER= prompt of 3dfit64.
        3 auto auto This is the the most difficult of cases, as you have to be lucky enough for the automatic procedure within 3dfit64 to fail in fitting a Gaussian to whatever coarse peak 3dshift64a found. There are many cases where you will not be so lucky and for that reason if you suspect that the set contains one or more bad on-off pairs do not use fcen = 3.

      5. For ease of computation, all data, including the on-off pairs identified as being bad, will continue to be processed beyond the 3dfit64 point. However, the mask values for the bad data are set to zero and the data will not be added to the final cube (and the final mask will not include the corresponding integration time. 

        Note that this entry supercedes the cautionary note in the entry of 10 Nov 1998.  Added 07 Apr 1999.

    • Fundamental changes have been made to 3dstar2.col such that it more resembles the structure and function of 3dcenter2000.col (by adding the Keywords TWEAK and DPCOR). Added 07 April 1999.

    • A pair of bugs were found in 3dmean64.c which affected the subset rejection procedure and the handling of data at very small flux levels. The revised version is available in the usual place.  Added 18 Apr 1999.

    • The latest version of megacal3d.c now includes entries for the following run/band combinations:
       

      Run
      Band
      Description
      lp94
      KN
      La Palma 1994, Knormal grism
      ls94
      KN
      La Silla 1994, Knormal
      ca95
      KN
      Calar Alto 1995, Knormal
      HN
      Calar Alto 1995, Hnormal
      lp96
      KN
      La Palma 1996, Knormal
      HN
      La Palma 1996, Hnormal
      ls96
      KL
      La Silla 1996, Klong
      a972
      KN
      AAT December 1997, Knormal (piezo 1)
      a972a
      KN
      AAT December 1997, Knormal (piezo 2)
      a972b
      HN
      AAT December 1997, Hnormal (piezo 1)
      a972c
      HN
      AAT December 1997, Hnormal (piezo 2)
      a984
      KN
      AAT October 1998, Knormal (piezo 1)
      a984a
      KN
      AAT October 1998, Knormal (piezo 2)
      a985
      KN
      AAT November/December 1998, Knormal (piezo 1)
      a985a
      KN
      AAT November/December 1998, Knormal (piezo 2)
      a985b
      HH
      AAT November/December 1998, Hhigh (piezo 1)
      a985c
      HH
      AAT November/December 1998, Hhigh (piezo 2)
      a991
      KN
      AAT March 1999, Knormal
      a992
      HH
      AAT April/May 1999, Hhigh
       
      NOTE: See updates from 04 June 1999, 27 September 1999, 31 January 2000, and 04 March 2000! The latest version is always available in the usual place.  Added 27 May 1999.

    • A bug was recently discovered (and fixed!) in megacal3d.c which precluded using RUN values with 5 characters.  The revised version is is available in the usual place.  Also, 3dstar2.col was found to be rife with bugs.  These, too, have been repaired and the revised version can be found in the usual place.  Finally, a minor bug was discovered in 3dmkflat.colHere is the corrected version.  Added 04 June 1999.

    • A (slightly) revised version of 3dcubin.c is now available.  The revised version fixes a "feature" which compromised its performance (by producing funny spectral "dropouts") on (at least) Linux 2.X, and possibly Solaris 5.6 (in any event it works on HP-UX 9.X and 10.20).  Added 05 June 1999.

    • A (slightly) revised version of 3dmean64.c is now available.  The revised version fixes a bug in the data selection/rejection routine.  Added 20 July 1999.

    • A minor typo bug was found in the DPCOR=2 subsection of 3dstar2.col. The revised version is located in the usual place.  Added 26 July 1999.

    • The input files for 3dcubin for the September/October 1999 Calar Alto run (3dcubin.993g and 3dcubin.993k) are now available. In addition, be sure to pick up the updated copy of 3dcubin.c itself.  Added 17 September 1999.

    • The revised version of 3dmegacal.c (with additions for the September/October 1999 Calar Alto run) is now available.  Added 27 September 1999.

    • A new version of 3dlin.col is now available. This new version includes the additional SEQU keyword options 6 and 66 for observations taken in LPSOU mode.  Added 30 September 1999.

    • A new version of dpix.c is now available. This revised version fixes a bug in the dimensioning of the neig (neighbor) array. Thanks go to Stefan Hippler (MPIA, Heidelberg) for bringing this to our attention.  Added 26 November 1999.

    • A revised version of 3dmegacal.c is now available. This revised version now includes calibration numbers for the H-band for the 1996 La Palma run.  Added 31 January 2000.

    • A revised version of 3dmegacal.c is now available. This revised version now includes calibration numbers for the K-band for the February 2000 Calar Alto run.  Added 04 March 2000.

    • A new version of 3dshift64a.c is now available. This revised version fixes a bug which would result in cubes sometimes being shifted to the upper right corner of the reference image rather than to its maximum. Thanks go to Jack Gallimore (NRAO) for bringing this to our attention.  Added 17 April 2000.

    • We now provide the routine required to convert "raw" 3D data into usable GIPSY-format data. This routine, gipcond.csh, is a simple C-shell script which operates on either an individual ".00" file, or a file containing a list of ".00" files. It is run simply by issuing the command source gipcond.csh.  Added 26 May 2000.

    • Due to system upgrades and security measures implemented, this page had to be reconstructed essentially from scratch. As a result, we unfortunately no longer have the calibration files referred to in the entry of 06 Apr 1998. We apologize for any inconveniences this might cause.  Added 03 May 2001.

       This page was last modified on 03 May 2001. 


      MPE-3D Homepage The Instrument References and Links
      SPECIAL NOTES FOR SPECIFIC
       RUNS
      Public FTP-Site Recent Results from 3D