Planetary radar scripts

05nov10

Summary of high level processing scripts:
The data files and their suffixes:
Hierarchy of calls that scripts make
The drv.dat configuration file
Updates
Links to individual scripts:
drv_all.sc
drv_fft_pwr.sc
drv_mapmsc.sc
drv_scale.sc

Summary of high level processing scripts:

These are the scripts called directly by the user:


Script	Description	Routines used:	inpSuf	outSuf
drv_all.sc	decode data transpose, fft, compute power, avg by range bin	drv_decode.sc drv_fft_pwr.sc	.rawfN	.dcdfN .mapfN
drv_scale.sc	scale maps to sigmas	scaletosigma	.mapfN	.mapsfN
drv_mapmsc.sc	Merge multiple .map files into a single map. optionally rescale scale to sigmas using a subset of image avg input maps rotate along ranges create fits file	drv_scale.sc drv_rotrange.sc drv_mapavg.sc drv_mergemaps3.sc	.mapsfN	.imgN.M or .fits

The data files and their suffixes: (top)

Different datafiles used by the processing are identified by their suffix. These suffixes are defined in the drv.dat config file. The datafiles with suffixes are:

SUFFIX	DESCRIPTION	OUTPUT BY: INPUT BY:
.rawfN	raw input datafiles containing the packed complex voltages from the ri or pfs.	input:radardecode
.dcdfN	after decoding by drv_decode.sc this file contains the complex decoded data. The user can specify a subset of the rangebins to keep (via numbins in drv.dat). The data is still in ipp order (it hasn't been transposed yet).	output:radardecode input:radarfft
.mapfN	The unscaled delay doppler maps. One for each look. The data is now in freq order by rangebin.	output:radarfft input:drv_scale.sc
.mapsfN	The scaled delay doppler maps. The .mapsfN files are scaled to units of sigma	output:drv_scale.sc input:drv_mapmsc.sc
.fit	Fits file containing mosaic of scaled maps.	output:makefits

Hierarchy of calls: (top)

drv_all.sc

drv_decode.sc

main_decode.sc

selectpnts: if ri and 2 pols this selects the pol to use.
radardeoode: to decode data or radardecode_thr to use threaded version (determined by numthreads > 1).
avgdata: if cohavg specified
selectpnts: to grab subset of rangebins if requested

drv_fft_pwr.sc

radarfft: transpose, compute spectra
selectpnts: to keep subset of freqbins if requested

drv_scale.sc

scaletosigma: scales mapfN files to sigmas.

drv_mapmsc.sc

drv_scale.sc: if -s or -S options requested to rescale .mapf files
drv_rotrange: if -r nbins requested to rotate range bins
drv_mapavg.sc: if mapstoavg > 1.
drv_mergemaps3.sc: main routine for merging.

dd|selectpnts to create any subset of map requested
mapsmerge : program to merge the maps into 1 image.
makefits: program to create fits file in Fits/ subdirectory.

The drv.dat config file (top)

The scripts use the filename in the $DRVSB environment variable (set by the sbdrv command) to determine how to process the data. Each line in this file contains a keyword, value, and optional comments (separated by one or more spaces). Any line that begins with # is a comment line. The scripts search the file looking for a particular keyword and extract the value in column 2 (the keyval.sc script does this).

Users will normally copy a template of this file from the scripts directory to the directory where they want the processed files. They would then edit the file for the particular experiment.

The file contains:

KEY	VALUE	Description
fbase	2003UV11.p05	base name for all files used. If no directory path than used current directory.
tapedev	/share/aeron5/SN12.001	file location for the input data. used by drv_tapetodisc.sc to create the .rawfN files
codelen	65535	length of pncode to use for decoding
bits	2	number of bits the input data is backed to.
numpol	1	number of polarizations in the input .rawfN data.
poltouse	1	If numpol=2 then this is the polarization to process (1 or 2). 1 is normally the OC.
bin1	29761	after decoding the data you can save only a subset of the range bins using bin1 and numbins. bin1 is 1 based.
numbins	200	number of range bins to keep starting at bin1.
smpperbaud	1	Number of samples per baud on input data.
machine	pfs	what type of data this is: ri, pfs,cbr,bytes, floats or unpacked
dcdpacked	0	always 0. Originally this was for the hardware decoder.
dcdfftlen	262144	This is the fftlen to use for decoding. If not a power of 2, the script drv_decode.sc will round up to the nearest power of 2. The cpu has to be able to allocate a few complex arrays of this length. So don't make it larger than a fraction of the available memory.
numthreads	1	if > 1 then use radardecode_thr which uses a threaded version of fftw to decode the data. The number of threads to use depends on the dcdfftlen and the number of processors on the cpu. For a 128k dcdlength the cpu utilization stopped increasing after 5 threads (on an 8 processor nehalem system). Setting numthreads to 1 uses radardecode which has the unthreaded fftw routines.
codestodecode	99999999	Maximum number of codes to decode in a look.
removedc	1	1 will remove dc prior to decoding. It does this using dcdfftlen complex points at a time.
cohavg	1	if > 1 then coherently add this many decoded ipps after decoding.
spcfftlen	7932	Length of fft to use when computing spectra.
spcfftkeep	400	Number of spectral channels to keep after spectra computed. Dc will always be located at spcffttokeep/2 + 1 (1 based).
compcodeprog		No longer used
codetype	aopnc dsnpnc barker	Type of code used for coding. The default is ao pncodes.
xoff	0.	freq correction value (Hz) done in radarfft (called by drv_fft_pwr.sc). When specifying a non 0 value for xoff, you must also specify the tmsmp (time sample in seconds) eg. assume 1usec baud, 1smp/baud .. (see radarfft.c apply_linear_phase() for the actual code) tmsmp 1e-6 t=0 For each voltage sample *phaseIncr= txoff then phase rotate v(t) by phaseIncr t=t+tmsmp endfor**
tmsmp	0.	sample time in secs. If 1usec baud, 1 smp/baud the tmsmp 1e-6
	Suffix defintions
sufraw	raw	input datafiles: to drv_decode.sc
sufdcd	dcd	decoded files: output from drv_decode.sc, inp drv_fft_pwr.sc
sufsps	map	unscaled delay doppler maps: output from drv_scale.sc, inp to:drv_mapmsc.sc
sufscl	maps	scaled delay doppler maps: from drv_scale.sc
sufpha	pha	used by drv_phase to compute phase of range bins
sufeng	eng	Used for engineering tests:drv_decodepwr,drv_power

Updates

190815

scripts in /pkg/aosoft/common/bin were out of date relative to the copies in svn version control. Some things were added new in /pkg/aosoft and some new things had been added in svn repository.
The svn repository for the scripts

located at: URL: http://svn.naic.edu/repos/aosoft/trunk/plrdr/scripts
a working copy that i have is at /share/megs/phil/svn/aosoft/plrdr/scripts/ .. but owned by phil
maybe we should checkout a working copy in /proj/radar somewhere so radar people can checkout, edit , and update the scripts from the repository.

switched scripts to use bash script keyval.sc rather than ex.awk to search drv.dat for keyword values
drv_all.sc added -c option.. if supplied will do complex radarfft output (for stokes)
drv_cw.sc put under svn version control
drv.dat mods

codeprog changed to codetype
numthreads is now used by drv_decode.sc .. if you don't want a multi-threaded fft for decoding, set numthreads to 0 or 1

main_decode.sc will call radarfft or radarfft_thr depending on numthreads.

xoff if not 0. then this will be a frequency correction to the raw data (done in radarfft.c.. called by drv_fft_pwr.sc)
tmsmp if xoff is not 0., then you need to specify the sampling time (in seconds for the input data).

cohavg will now coherently avgerage decoded ipps (if > 1) .. done in main_decode.sc.
drv_mapmsc.sc fixed bug.. if -s col1 col2 not specified if would take row1,row2 and the cols to use for scaling.
drv_mergemaps.sc fixed bug. If -F was specfied, it would ignore -f that followed (had left out shift if -F present).
radarfft.c - added -c option for complex output. output 1 block of fftlen*numbins of complex data (no pwr or averaging).

Individual script descriptions:

top

NAME

drv_all.sc

decode and then compute spectra

SYNTAX

drvall.sc -d -c firstfile numfiles

ARGS

-d	Delete the input files (.rawfN) when done. By default the input files are not touched.
-c	drv_fft_pwr.sc will output complex values rather than spectral power.
firstfile	first filenumber to start processing. eg .rawfM. M defaults to 1
numfiles	Number of rawf input files to process.

DESCRIPTION

This is the main driver script called by the user. It will process numfiles .rawfN input files starting at firstfile. For each file it will: decode the file, transpose the matrix, compute the unscaled delay doppler map. The .dcdfN and .mapfN files are created by this routine. This script can gobble up lots of disc space (depending on the number freq and range bins kept.
It then calls drv_fft_pwr.sc [-c] to do the transforms

SCRIPTS USED

drv_decode.sc
drv_fft_pwr.sc

SUFFIXES USED

Input: .rawfN
Output: .dcdfN, .mapfN

EXAMPLE

To use this routine:

cd to the directory where the output files will be stored.
make sure your path has the directory of the planetary radar scripts.
edit the drv.dat file that you are going to use
sbdrv drvdatfilename sets the environment variable $SBDRV to this filename
Split the data into separate looks (see drv_tapetodisc.sc)
Run the program. Suppose there are 51 files starting from file 1 and you want to keep the input files on disc:

drv_all.sc 1 51
If will create 102 files:

fbase.dcdf1 to fbase.dcdf51 (where fbase is defined in drv.dat)
fbase.mapf1 to fbase.mapf51

logfile

fbase.hdrfN is written to by drv_decode.sc and drv_fft_pwr.sc:
drv_decode.sc START            :Wed Nov 3 19:16:57 AST 2010
    decoding program            : radardecode
    1st range bin kept         : 30060
    number of range bins kept : 200
    samples per baud           : 1
    numPol, pol Used           : 1,1
    fftlength used for decoding: 262144
    remove Dc                  : 1
    coherent average per FFT   : 1
    machine                    : pfs
    codetype                   : aopnc
    maximum # codes to decode : 99999999
    input file                 : 2003UV11.p05.rawf1
    output file                : 2003UV11.p05.dcdf1
drv_decode.sc END             :Wed Nov 3 19:17:52 AST 2010
drv_fft_pwr.sc START       : Wed Nov 3 20:03:56 AST 2010
    samples/rbin input     : 3354
    samples/rbin used      : 7932
    length fft             : 7932
    dcd packed data        : 0
    ffts added incoherently: 1 ... or
    cmplx ffts output      : 200     if -c used
    freq points kept       : 400
    input file             : 2003UV11.p05.dcdf1
    output file            : 2003UV11.p05.mapf1
drv_fft_pwr.sc END         : Wed Nov 3 20:04:03 AST 2010
Done with 2003UV11.p05.dcdf1

top

NAME

drv_fft_pwr.sc

compute delay doppler maps from decoded data.

SYNTAX

drv_fft_pwr.sc

-c firstfile numberOfFiles

ARGS

-c	if present then output complex spectra (no power computation)
firstfile	first filenumber to start processing. eg fbase.dcdfN
numfiles	Number of .dcdfN files to process.

drv.dat info

key	use
bin1	First range bin to use. Count from 1.
numbins	Number of range bins to use (starting at bin1)
spcfftlen	fftlen to use when computing delay dopper map
spcfftkeep	Number of frquency bins to keep. Dc is always in channel spcfftkeep/2 + 1 (count from 1)
xoff	apply frequency offset.
tsmp	if xoff used

DESCRIPTION

drv_fft_pwr.sc will compute the delay doppler maps from the decoded files. Data is processed 1 look (.dcdfN) file at a time. For each file:

transpose a block of numbins*spcfftlen decoded samples
if -c not supplied then

For each of numbin range bins compute the fft and then power.

keep spcfftkeep frequency bins about DC.

repeat for all sets of numbins*spcfftlen blocks of data. Incoherently add the power of each block.

if -c is supplied then

compute numbins ffts of spcfftlen and output the complex values
Need to check what happens if more than numbins*fftlen points in file.

Any extra data at the end of the file is discarded.
if spcfftlen is less than the number of ipps, the program will zero fill each fft to be spcfftlen samples.

The output file is the .mapfN file.
if -c is not supplied then a scaled to sigma set of files will be in .mapsfN

SCRIPTS USED

radarfft: program to transpose, fft, and compute power.
selectpnts: to keep only spcfftkeep frequency channels
drv_scale.sc to compute scaled maps

SUFFIXES USED

Input: .dcdfN
Output: .mapfN unscaled delay dopper maps.

EXAMPLE

To use this routine:

This routine is called by drv_all.sc

logfile

fbase.hdrfN is written to by this script. the output is:
drv_fft_pwr.sc START       : Wed Nov 3 18:19:04 AST 2010
    samples/rbin input     : 3354
    samples/rbin used      : 7932
    length fft               : 7932
    dcd packed data        : 0
    ffts added incoherently: 1 or
    cmplx ffts output        : numbins if -c option used
    freq points kept          : 400
    input file               : 2003UV11.p05.dcdf1
    output file           : 2003UV11.p05.mapf1
drv_fft_pwr.sc END         : Wed Nov 3 18:19:11 AST 2010

top

NAME

drv_mapmsc.sc

mosaic a number of scaled maps into one image file.

SYNTAX

drvmsc.sc

{-s c1 c2} {-S r1 r1} {-r bins} firstfile numfiles filesToAvg mapsPerRow
{newFreqLen} {newRangeLen} {freqColOffset}

ARGS

-s c1 c2	Rescale to sigmas using column c1 thru c2 (1 based). This lets you rescale the map using data points that don't include the echo. The .mapfN files are processed and the .mapsfN files are recreated.
-S r1 r2	Rescale to sigmas using row r1 thru r2 (1 based). Note that you can specify both -s and -S.
-r bins	rotate the maps by bins range bins. A positive number pushes to farther range bins. This is done after any rescaling.
firstfile	first filenumber to start processing. eg fbase.mapsfM.
numfiles	Number of mapsf files to process.
filesToAvg	number of .mapsFN files to average for each output file. If numfiles does not divide evenly by filesToAvg, then the files at the end are not included.
mapsPerRow	The number of averaged maps to put on each row of output.
newFreqLen	The number of frequency bins to output. The default is the number on input. They are selected about freqColOffset bins from DC.
newRangeLen	The number of range bins to output. The are selected about the central range (after the -r bins option is applied).
freqColOffset	If newFreqlen is specified then the freq bins are selected about DC. If freqColOffset is also supplied then the center is moved from DC by freqColOffset frequency bins before selection. A positive number moves to frequency bins to the right of DC.

DESCRIPTION

drv_mapmsc.sc will make a mosaic of the .mapsfN files and output to a .fits file. The script can optionally:

recreate the .mapsfN files from the .mapfN files using a subset of the map for the scaling to sigmas. This is done with the -s c1 c2 -S r1 r2 options.
average maps together before outputting.
select a subset of the input maps for output. This allows you to recenter the image on the map:

newFreqLen and freqColOffset select the subset in frequency. The first freq bin is computed as: (freqBins-newFreqLen)/2 + 1 + freqColOffset (1 based)
-r bins and newRangeLen are used to select a subset of range bins.

-r bins first rotates in range. positive numbers push to farther range.
the first range is then rngSkip=(rangeBins - newRangeLen)/2 (this is after the rotation.. 1 based).

The output file:

Will be written the the Fits/ directory below the current directory. You may need to create it.
The fits filename is:

if numfiles == 1 then fM.fit where M is the firstfile parameter
if numfiles > 1 then fRxC.fit where R is the number of rows output and C is the number of columns output.

SCRIPTS USED

drv_scale.sc (if -s or -S are supplied)
drv_rotrange.sc (if -r bins is supplied)
drv_mapavg.sc (if filesToAvg > 1)
drv_mergemaps3.sc (to do the merging).

SUFFIXES USED

Input: .mapsfN,(.mapfN if -s or -S)
Output: .fit (written to Fits/ subdirectory).

EXAMPLE

To use this routine:

First run drv_all.sc, drv_scale.sc
mkdir Fits .. If is doesn't already exist.
Suppose you have 51 files starting at 1 and you want to average 8 at a time. Make the output map 3 maps per row and two rows. 6*8=48 so the last 3 maps will not be used.

drv_mapmsc.sc 1 51 8 3
the output file is: Fits/f2x3.fit

logfile

fbase.hdrmsc is written to by this script. the output is:
drv_mapmsc.sc START       :Thu Nov 4 08:43:25 AST 2010
   basefile               : 2003UV11.p05
   fits filename          : ./Fits/f2x3.fit
   1st file, numfiles     : 1 51
   orig frq,range bins    : 400 200
   kept frq,range bins    : 400 200
   frq bin offset,smp/baud: 0 1
   rotate range bins      : 0
   maps avged             : 8
   maps per row           : 3
drv_mapmsc.sc END         :

top

NAME

drv_scale.sc

Scale the .mapfN files to sigmas

SYNTAX

drv_scale.sc firstfile numfiles firstcolumn lastcolumn {firstRow lastRow}

ARGS

firstfile	first filenumber (.mapfN) to use
numfiles	number of .mapfN files to process
firstcolumn	first column number (freq bin) to use for computing avg,sigma. count from 1.
lastcolumn	last column number (freq bin) to use for computing avg,sigma.
firstRow	first range bin to use for computing avg,sigma. the default is 1 firstRow, lastRow are optional. If not supplied use all of the rows of the file.
lastRow	last range bin to use for computing avg,sigma. The default is the number of range bins kept.

DESCRIPTION

Using the fbase.mapfN files as input,for each file:

compute the average, and sigma for the subset of the .mapfN file defined by the first,last keywords above.
remove the average
scale the data to units of sigma.
output to fbase.mapsfN
Note that average and sigma can be computed over a subset of the file, but the entire file is input, processed, and then output.

SCRIPTS USED

scaletosigma (program)

SUFFIXES USED

Input: .mapfN
Output: .mapsfN

EXAMPLE

To use this routine:

create the fbase.mapfN files with drv_all.sc
suppose there are 51 files to processes starting with f1:

drv_scale.sc 1 51

logfile

Written to fbase.hdrfN:
drv_scale.sc   START       : Thu Nov 4 08:43:20 AST 2010
    average removed        : 2152531444.801440
    1 sigma scaling        : 11127334442.139343
    1st/last frqbin        : 1 400
    1st/last rangebin      : 1 200
drv_scale.sc END           : Thu Nov 4 08:43:20 AST 2010

planetary_radar_page:
home_~phil