Planetary radar scripts

05nov10


Summary of high level processing scripts:
The data files and their suffixes:
Hierarchy of calls that scripts make
The drv.dat configuration file
Links to individual scripts:
drv_all.sc
drv_fft_pwr.sc
drv_mapmsc.sc
drv_scale.sc

Summary of high level processing scripts:

    These are the scripts called directly by the user:
Script
 Description
 Routines used:
 inpSuf
 outSuf
drv_all.sc
  • decode data
  • transpose, fft, compute power, avg by range bin
  • drv_decode.sc
  • drv_fft_pwr.sc
 .rawfN
 .dcdfN
.mapfN
drv_scale.sc
  • scale maps to sigmas
  • scaletosigma
 .mapfN .mapsfN
drv_mapmsc.sc

  • Merge multiple .map files into a single map.
  • optionally
    • rescale scale to sigmas using a subset of image
    • avg input maps
    • rotate along ranges
    • create fits file
  • drv_scale.sc
  • drv_rotrange.sc
  • drv_mapavg.sc
  • drv_mergemaps3.sc
 .mapsfN

 .imgN.M
or
.fits

The data files and their suffixes: (top)

    Different datafiles used by the processing are identified by their suffix. These suffixes are defined in the drv.dat config file. The datafiles with suffixes are:

SUFFIX
 DESCRIPTION
 OUTPUT BY:
INPUT BY:
.rawfN
 raw input datafiles containing the packed complex voltages from the ri or pfs.
input:radardecode
.dcdfN
 after decoding by drv_decode.sc this file contains the complex decoded data.
The user can specify a subset of the rangebins to keep (via numbins in drv.dat).
The data is still in ipp order (it hasn't been transposed yet).
 output:radardecode
input:radarfft
.mapfN
 The unscaled delay doppler maps. One for each look. The data is now in freq order by rangebin.
 output:radarfft
input:drv_scale.sc
.mapsfN
 The scaled delay doppler maps. The .mapsfN files are scaled to units of sigma
 output:drv_scale.sc
input:drv_mapmsc.sc
.fit
 Fits file containing mosaic of scaled maps.
 output:makefits

Hierarchy of calls: (top)


The drv.dat config file (top)

    The scripts use the filename in the $DRVSB environment variable (set by the sbdrv command) to determine how to process the data. Each line in this file contains a keyword, value, and optional comments (separated by one or more spaces). Any line that begins with # is a comment line. The scripts search the file looking for a particular keyword and extract the value in column 2 (the keyval.sc script does this).

    Users will normally copy a template of this file from the scripts directory to the directory where they want the processed files. They would then edit the file for the particular experiment.

The file contains:

KEY
 VALUE
 Description
fbase
 2003UV11.p05
 base name for all files used. If no directory path than used current directory.
tapedev
 /share/aeron5/SN12.001
 file location for the input data. used by drv_tapetodisc.sc to create the .rawfN files
codelen
 65535
 length of pncode to use for decoding
bits
 2
 number of bits the input data is backed to.
numpol
 1
 number of polarizations in the input .rawfN data.
poltouse
 1
 If numpol=2 then this is the polarization to process (1 or 2). 1 is normally the OC.
bin1
 29761
 after decoding the data you can save only a subset of the range bins using bin1 and numbins. bin1 is 1 based.
numbins
 200
 number of range bins to keep starting at  bin1.
smpperbaud
 1
 Number of samples per baud on input data.
machine
 pfs
 what type of data this is: ri, pfs,cbr,bytes, floats or unpacked
dcdpacked
 0
 always 0. Originally this was for the hardware decoder.
dcdfftlen
 262144
 This is the fftlen to use for decoding. If not a power of 2, the script drv_decode.sc will round up to the nearest power of 2. The cpu has to be able to allocate a few complex arrays of this length. So don't make it larger than a fraction of the available memory.
numthreads
 1
 if > 1 then use radardecode_thr which uses a threaded version of fftw to decode the data. The number of threads to use depends on the dcdfftlen and the number of processors on the cpu. For a 128k dcdlength the cpu utilization stopped increasing after 5 threads (on an 8 processor nehalem system).
Setting numthreads to 1 uses  radardecode which has the unthreaded fftw routines.
codestodecode
 99999999
 Maximum number of codes to decode in a look.
removedc
 1
 1 will remove dc prior to decoding. It does this using dcdfftlen complex points at a time.
cohavg
 1
 if > 1 then coherently add this many decoded ipps after decoding.
spcfftlen
 7932
 Length of fft to use when computing spectra.
spcfftkeep
 400
 Number of spectral channels to keep after spectra computed.
Dc will always be located at spcffttokeep/2 + 1 (1 based).
compcodeprog
No longer used
codetype
 aopnc
dsnpnc
barker
 Type of code used for coding. The default is ao pncodes.

Suffix defintions
sufraw
 raw
 input datafiles: to drv_decode.sc
sufdcd
 dcd
 decoded files: output from drv_decode.sc, inp drv_fft_pwr.sc
sufsps
 map
 unscaled delay doppler maps: output from drv_scale.sc, inp to:drv_mapmsc.sc
sufscl
 maps
 scaled delay doppler maps: from drv_scale.sc
sufpha
 pha
 used by drv_phase to compute phase of range bins
sufeng
 eng
 Used for engineering tests:drv_decodepwr,drv_power


Individual script descriptions:

top
NAME
drv_all.sc
 decode and then compute spectra
SYNTAX
 drvall.sc -d firstfile numfiles
ARGS
-d
 Delete the input files (.rawfN) when done. By default the input files are not touched.
firstfile
 first filenumber to start processing. eg .rawfM. M defaults to 1
numfiles
 Number of rawf input files to process.
DESCRIPTION
 This is the main driver script called by the user. It will process numfiles  .rawfN  input files starting at firstfile. For each file it will: decode the file, transpose the matrix, compute the unscaled delay doppler map. The .dcdfN and .mapfN files are created by this routine. This script can gobble up lots of disc space (depending on the number freq and range bins kept

SCRIPTS USED
 drv_decode.sc
drv_fft_pwr.sc
SUFFIXES USED
 Input: .rawfN
Output: .dcdfN, .mapfN
EXAMPLE
 To use this routine:
  • cd to the directory where the output files will be stored.
  • make sure your path has the directory of the planetary radar scripts.
  • edit the drv.dat file that you are  going to use
  • sbdrv drvdatfilename  sets the environment variable $SBDRV to this filename
  • Split the data into separate looks (see drv_tapetodisc.sc)
  • Run the program. Suppose there are 51 files starting from file 1 and you want to keep the input  files on disc:
    • drv_all.sc  1 51
    • If will create 102 files:
      • fbase.dcdf1 to fbase.dcdf51 (where fbase is defined in drv.dat)
      • fbase.mapf1 to fbase.mapf51
logfile
 fbase.hdrfN is written to by drv_decode.sc and drv_fft_pwr.sc:
drv_decode.sc START            :Wed Nov  3 19:16:57 AST 2010
    decoding program            : radardecode
    1st range bin kept         : 30060
    number of range bins kept  : 200
    samples per baud           : 1
    numPol, pol Used           : 1,1
    fftlength used for decoding: 262144
    remove Dc                  : 1
    coherent average per FFT   : 1
    machine                    : pfs
    codetype                   : aopnc
    maximum # codes to decode  : 99999999
    input file                 : 2003UV11.p05.rawf1
    output file                : 2003UV11.p05.dcdf1
drv_decode.sc  END             :Wed Nov  3 19:17:52 AST 2010
drv_fft_pwr.sc START       : Wed Nov  3 20:03:56 AST 2010
    samples/rbin input     : 3354
    samples/rbin used      : 7932
    length fft             : 7932
    dcd packed data        : 0
    ffts added incoherently: 1
    freq points kept       : 400
    input file             : 2003UV11.p05.dcdf1
    output file            : 2003UV11.p05.mapf1
drv_fft_pwr.sc END         : Wed Nov  3 20:04:03 AST 2010
Done with 2003UV11.p05.dcdf1

top
NAME
drv_fft_pwr.sc
 compute delay doppler maps from decoded data.
SYNTAX
drv_fft_pwr.sc         
firstfile numberOfFiles                                               
ARGS
firstfile
 first filenumber to start processing. eg fbase.dcdfN
numfiles
 Number of .dcdfN  files to process.
drv.dat info
key
 use
bin1
 First range bin to use. Count from 1.
numbins
 Number of range bins to use (starting at bin1)
spcfftlen
 fftlen to use when computing delay dopper map
spcfftkeep
 Number of frquency bins to keep. Dc is always in channel spcfftkeep/2 + 1 (count from 1)
xoff
 apply frequency offset.
DESCRIPTION
 drv_fft_pwr.sc will compute the delay doppler maps from the decoded files. Data is processed 1 look (.dcdfN) file at a time. For each file:
  • transpose a block of numbins*spcfftlen decoded samples
  • For each of numbin range bins compute the fft and then power.
  • keep spcfftkeep frequency bins about DC.
  • repeat for all sets  of numbins*spcfftlen blocks of data. Incoherently add the power of each block.
  • Any extra data at the end of the file is discarded.
  • if spcfftlen is less than the number of ipps, the program will zero fill each fft to be spcfftlen samples.
The output file is the .mapfN file.
SCRIPTS USED
 radarfft: program to transpose, fft, and compute power.
selectpnts: to keep only spcfftkeep frequency channels
SUFFIXES USED
 Input: .dcdfN
Output: .mapfN  unscaled delay dopper maps.
EXAMPLE
 To use this routine:
logfile
 fbase.hdrfN is written to by this script. the output is:
drv_fft_pwr.sc START       : Wed Nov  3 18:19:04 AST 2010
    samples/rbin input     : 3354
    samples/rbin used      : 7932
    length fft                     : 7932
    dcd packed data        : 0
    ffts added incoherently: 1
    freq points kept          : 400
    input file                     : 2003UV11.p05.dcdf1
    output file                   : 2003UV11.p05.mapf1
drv_fft_pwr.sc END         : Wed Nov  3 18:19:11 AST 2010
top
NAME
drv_mapmsc.sc
 mosaic a number of scaled maps into one image file.
SYNTAX
drvmsc.sc {-s c1 c2} {-S r1 r1} {-r bins} firstfile numfiles filesToAvg mapsPerRow
 {newFreqLen} {newRangeLen} {freqColOffset}
ARGS
-s c1 c2
 Rescale to sigmas using column c1 thru c2 (1 based). This lets you rescale the map using data points that don't include the echo. The .mapfN files are processed and the .mapsfN files are recreated.
-S r1 r2
 Rescale to sigmas using row r1 thru r2 (1 based). Note that you can specify both -s and -S.
-r bins
 rotate the maps by bins range bins. A positive number pushes to farther range bins. This is done after any rescaling.
firstfile
 first filenumber to start processing. eg fbase.mapsfM.
numfiles
 Number of mapsf  files to process.
filesToAvg
 number of .mapsFN files to average for each output file. If numfiles does not divide evenly by filesToAvg, then the files at the end are not included.
mapsPerRow
 The number of averaged maps to put on each row of output.
newFreqLen
 The number of frequency bins to output. The default is the number on input. They are selected about freqColOffset bins from DC.
newRangeLen
 The number of range bins to output. The are  selected about the central range (after the -r bins option is applied).
freqColOffset
 If newFreqlen is specified then the freq bins are selected about DC. If freqColOffset is also supplied then the center is moved from DC  by freqColOffset frequency bins before selection. A positive number moves to frequency bins to the right of DC.
DESCRIPTION
 drv_mapmsc.sc will make a mosaic of the .mapsfN files and output to a .fits file. The script can optionally:
  • recreate the .mapsfN files from the .mapfN files using a subset of the map for the scaling  to sigmas. This is done with the -s c1 c2 -S r1 r2 options.
  • average maps together before outputting.
  • select a subset of the input maps for output. This allows you to recenter the image on the map:
    • newFreqLen and freqColOffset select the subset in frequency. The first freq bin is computed as:  (freqBins-newFreqLen)/2 + 1 + freqColOffset (1 based)
    • -r bins and newRangeLen are used to select a subset of range bins.
      • -r bins first rotates in range.  positive numbers push to farther range.
      • the first range is then rngSkip=(rangeBins - newRangeLen)/2 (this is after the rotation.. 1 based).
The output file:
  • Will be written the the Fits/ directory below the current directory. You may need to create it.
  • The fits filename is:
    • if numfiles == 1 then fM.fit where M is  the firstfile parameter
    • if numfiles > 1 then  fRxC.fit where R is the number of rows output and C is the number of columns output.
SCRIPTS USED
 drv_scale.sc (if -s or -S are supplied)
drv_rotrange.sc (if -r bins is supplied)
drv_mapavg.sc  (if filesToAvg > 1)
drv_mergemaps3.sc (to do the merging).
SUFFIXES USED
 Input: .mapsfN,(.mapfN if -s or -S)
Output: .fit (written to Fits/  subdirectory).
EXAMPLE
 To use this routine:
  • First run drv_all.sc, drv_scale.sc
  • mkdir Fits  .. If is doesn't already exist.
  • Suppose you have 51 files starting at 1 and you want to average 8 at a time. Make the output map 3 maps per row and two rows. 6*8=48 so the last  3 maps will not be used.
    • drv_mapmsc.sc 1 51 8 3
    • the output file is: Fits/f2x3.fit
logfile
 fbase.hdrmsc is written to by this script. the output is:
drv_mapmsc.sc START       :Thu Nov  4 08:43:25 AST 2010
   basefile               : 2003UV11.p05
   fits filename          : ./Fits/f2x3.fit
   1st file, numfiles     : 1 51
   orig frq,range bins    : 400 200
   kept frq,range bins    : 400 200
   frq bin offset,smp/baud: 0 1
   rotate range bins      : 0
   maps avged             : 8
   maps per row           : 3
drv_mapmsc.sc END         :

top

NAME
drv_scale.sc
 Scale the .mapfN files to sigmas
SYNTAX
 drv_scale.sc firstfile numfiles firstcolumn lastcolumn  {firstRow lastRow}
ARGS
firstfile
 first filenumber (.mapfN) to use
numfiles
 number of .mapfN files to process
firstcolumn
 first column number (freq bin) to use for computing avg,sigma. count from 1.
lastcolumn
 last column number (freq bin) to use for computing avg,sigma.
firstRow
 first range bin to use for computing avg,sigma. the default is 1
firstRow, lastRow are optional. If not supplied use all of the rows of the file.
lastRow
 last range bin to use for computing avg,sigma. The default is the number of range bins kept.
DESCRIPTION
 Using the fbase.mapfN files as input,for each file:
  • compute the average, and sigma for the subset of the .mapfN file defined by the first,last keywords above.
  • remove the average
  • scale the data to units of sigma.
  • output to fbase.mapsfN
  • Note that average and sigma can be computed over a subset of the file, but the entire file is input, processed, and then output.
SCRIPTS USED
 scaletosigma (program)
SUFFIXES USED
 Input: .mapfN
Output: .mapsfN
EXAMPLE
 To use this routine:
  • create the fbase.mapfN files with drv_all.sc
  • suppose there are 51 files to processes starting with f1:
    • drv_scale.sc 1 51
logfile
 Written to fbase.hdrfN:
drv_scale.sc   START       : Thu Nov  4 08:43:20 AST 2010
    average removed        : 2152531444.801440
    1 sigma scaling        : 11127334442.139343
    1st/last frqbin        : 1 400
    1st/last rangebin      : 1 200
drv_scale.sc END           : Thu Nov  4 08:43:20 AST 2010


planetary_radar_page:
 home_~phil