Content-type: text/html Manpage of RADARRANGING

RADARRANGING

Section: Misc. Reference Manual Pages (1L)
Updated: 16 August 1999
Index Return to Main Contents

NAME

radarranging - process radar ranging data.(ao)

SYNOPSIS

The following scripts are used to do the radar processing of ranging data. They are broken down into a basic and other group. The basic group does the standard processing from raw data to unscaled maps. The other group contains scripts that perform more specialized functions. In each script, numfiles specifies how many files to process starting from firstfile.

Basic Processing scripts

drv_all.sc -n firstfile numfiles

: Decode the raw data, transpose the matrix, and then compute the maps. This script calls drv_decodes.sc and drv_fft_pwr.sc. The output consists of the decode and map files. If -n is not suppled, then the raw input files will be deleted when done. The data should already have been put on disc with drv_tapetodisc.sc.

drv_tapetodisc.sc firstScanNum firstOutfileNum numscans

: Skip forward from the current position on the tape until scan firstScanNum is found and then copy numscans scans from tape to disc, stripping off the header and naming the outputfiles starting with file number firstOutfileNum

drv_decode.sc [-d] firstfile numfiles

: Decode numfiles files starting with file number firstfile. If -d is specified then delete the raw input file when done.

drv_fft_pwr.sc [-d] firstfile numfiles

: Using the decoded data as input, transpose the data, compute the spectra, power, and then incoherently average the power in each file.

Other processing scripts

drv_scale.sc firstfile numfiles firstcol lastcol [firstrow lastrow]

: Scale the map files to sigmas. Use columns firstcol thru lastcol and firstrow thru lastrow to compute the standard deviation of the noise. If the rows are not specified then all of the rows are used. The output files will have the maps suffix.

drv_findmax.sc firstfile numfiles [filetype]

: Find the range and frequency bin for the maximum value in each map. filetype can be map for an unscaled map (the default) or maps if the input is to come from the scaled maps. The output is written to stdout.

drv_mapmsc.sc {-s c1 c2} {-S r1 r2} {-r bins} firstmap nummaps mapsToavg mapsPerRow {newRowLen newColLen colOff}

: Create a mosaic using nummaps scaled maps (the .maps files) starting with map firstmap. Average mapsToAVg maps together and then place mapsPerRow maps per horizontal row. There will be a blank line to the right and below each set of maps. If -s c1 c2 -S r1 r2 is included, then the scaled maps files will be created from the unscaled map files using columns c1,c2 and rows r1,r2 for the noise. If -r bins is included then each averaged map will be rotated in range by range bins (where a positive number pushes the echo back farther into the map). The output map name will be a fits file created in the ./Fits directory (you need to create the directory first). newRowLen will output newRowLen rows about the central row. newColLen colOff will output newColLen columns about the new central col which is the center column plus the colOff. These three arguments are in units of pixels. The filename will be ./Fits/fRxC.fit where R and C are the number of rows and cols in the output. If there is only 1 output image in the mosaic then the name will be ./Fits/fN.fit where N is the file number.

drv_makefits.sc file1 numfiles {maps}

: Make numfiles fits files from the maps starting with file1. If maps is included, use the scaled maps files. If not, use the .map files. The fits files are written to the ./Fits directory and are named ./Fits/fN.fit where N is the file number.

drv_power.sc firstfile numfiles

: Using the decoded files as input, compute numfiles energy plots starting with file number firstfile. The output will be floating pointing numbers with numfiles times codelen numbers.

PROCESSING DESIGN GOALS

The radar ranging software processes the data taken with the sband data acquisition program and produces delay doppler maps. In developing the software, the following design goals were applied:

The software should be flexible.

It should allow for easy extensions and modifications. When debugging the software or hardware you should be able to step in at any point and examine the results of a processing step.
This led to a design based on small programs or filters. Each filter performs a small well defined operation on its input stream and sends the output to its standard output (Unix jargon). The shell programming language with its scripting cababilities (command files) and pipes is then used to build a hierarchy of proccessing steps. Modifications are simply an edit of the command file with a text editor.

The drawbacks with this method are that few people understand the shell programing language, there is more overhead running multiple small filters rather than one large program (due to the context switch overhead), and real*4 format takes up twice as much disc space as integer*2.

Take advantage of multi-processing.

In the past, radar processing has been tied to the availability of an AP. With our investment in workstations, it is now possible to process radar data on workstations in a reasonable amount of time if multiple workstations are run simultaneously.

To keep things simple, the processing unit for a workstation should be a complete run. The input and output files must also come from disc. The limited number of tape drives would make it difficult for multiple workstations to process the data simultaneously.

The drawbacks to this approach are that you need lots of disc space. Processing 1 bit sampled data as 4 byte reals increases the disc space by a factor of 32. This requires that the input files be small and/or you only keep a limited number of range bins.

Process multiple runs automatically.

You should be able to process multiple files of data without an operators continued attention.

To implement this, a filenaming convention was adopted that uses a basename (tied to the observation), a suffix (tied to the processing step), and a filenumber. The scripts are setup to process a user specified number of files keying off of the filenumber. This filenaming convention also makes multi-processing easier. Computer A can process files 1-5 while computer B processes files 6-10 etc..

Keep a log of the processing.

It is import to keep track of what processing has been done to the data. This is implemented by the scripts using header files.

The header from each run is extracted from the tape, converted to ascii, and then written to disc when the processing starts. As each processing step is run, a summary of what was done to the data is appended to the header file.

DESCRIPTION

Overview:

The processing consists of filters that process the data. Scripts string multiple filters together to perform a specific task. The information describing the type of data (codelen, baudlen, etc.) and how to process it (decodelen, fftlen, rangebins to keep, etc) is stored in an ascii disc file that the user creates (lets call it drv.dat). The file format is keyword value. The scripts search this file for the keywords they need and extract the corresponding value. Before running the scripts the user must set an environment variable telling the scripts the location for there drv.dat file.

The radar data is read from disc and then written back out to disc (except for drv_tapetodisc which reads from the tape). The filenaming convention used for the disc files consists of a basename, suffix, and filenumber. The format is basename.suffixFn.

The basename is taken from the drv.dat file. It would typically be something like: ven12aug_U4.2 where ven is the name of the observed object, 12aug is the date of observation, and U4.2 is the a 4.2 microsecond baud. There is no limitation on what the basename can be.

The suffix varies depending on the processing step.

raw: This is the raw data. It is written by drv_tapetodisc.sc and read by drv_decode.sc.
dcd: This is the decoded data. It is written by drv_decode.sc and read by drv_fft_pwr.sc and drv_power.sc.
map: This is the unscaled map data. It is float data stored in freqbin order with DC in the center (or one pixel to the right of center if an even number of frequency bins). It is written by drv_fft_pwr.sc. It is read by drv_makefits.sc, drv_findmax.sc, and drv_mapmsc.sc.
maps: This is the scaled map data taken from the .map file and scaled to sigmas. It is written by drv_scale.sc or drv_mapmsc.sc and read by drv_makefits.sc, drv_findmax.sc, and drv_mapmsc.sc.
: The normal mode for processing the data would take the raw data as input and produce delay doppler maps as output. It is also possible to paritially process the data to do such things as produce energy plots or check the phase stability.

Normal processing:

The normal processing takes the raw data and outputs delay doppler maps. In this mode of operation, the intermediary files get deleted after they are used. When processing is done, you will have the decoded data files, the map files, and the header files still on disc. The default processing can be broken down into 5 steps:

1.: Make sure your path environment variable contains ~phil/datatak/astro/rdrN (this is the location for the scripts). Put the alias alias sbdrv 'setenv DRVSB `pwd`/!:1' in your .cshrc file. It is used to defile the name and location of your drv.dat file. Copy the master drv.dat file located in ~phil/datatak/astro/rdrN/drv.dat to your local directory giving it whatever name you want (say drvP2.dat). Edit this file to reflect the processing you are about to do. Enter sbdrv drvP2.dat to set the environment variable DRVSB to your drv.dat filename.
2.: Input the raw data from tape to disc. This is done with drv_tapetodisc.sc.
3.: Select out the polarization to process, decode the data, and then select out the range bins that are to be kept. The drv_decode.sc script does this.
4.: Transpose the decoded data , compute the spectra, power, and then sum for the entire run. This is done by the drv_fft_pwr.sc script.
5.: Search for the echo in the maps, make maps in sigma units, and create fits file mosaics of multiple maps or single fits files. You can do these things with drv_findmax.sc, drv_scale.sc, drv_mapmsc.sc, and drv_makefits.sc.
: Steps 3 and 4 above can be done with the drv_all.sc script. You can look at the float images (.map, .maps) with saoimage -r4 numFrqBins numRangeBins filename The fits files can be examined using iraf and its commands rfits and disp.

Multi processing:

The data can be processed with multiple cpus simultaneously. Assume you want to use cpu1, cpu2, and cpu3. rlogin to each cpu and cd to your processing directory. Be sure and enter sbdrv "yourfilename.dat" on each cpu. On cpu1 enter: drv_all.sc 1 5, on cpu2: drv_all.sc 6 5, and on cpu3 enter: drv_all.sc 11 5. This will process 3 files simultaneously. This form of processing works best when the processing time is much greater then the i/o time (1 or 2 bit data where you don't keep all the range bins).

Selective processing:

When searching for the echo in the range bins or when debugging, you may not want to perform the normal processing. Below are listed a few different ways of looking at the data:

Energy plots: Decode the data with drv_decode.sc and then use drv_power.sc to compute the energy plot. This method requires the decoded data to be out on disc for its input.

The drv.dat file:

The scripts use the filename in the DRVSB environment variable (set by the sbdrv alias above) to determine how to process the data. Each line in this file contains a keyword, value, and optional comments (separated by one or more spaces). Any line that begins with # is a comment line. The scripts search the file looking for a particular keyword and extract the value in column 2 (the ex.awk script does this). The file contains:

#
# key value
#
fbase /share/cuca7/radar/ml/15apr99/ml15apr_U.3 basefile
tapedrv /dev/nrst12 Tape device to copy from
codelen 16383 For data to process
bits 2 This would be 4 bit packed data
numpol 2 Number of polarizations recorded
poltouse 1 Polarization to use (1 or 2)
bin1 8000 1st range bin to keep after dcding
numbins 2000 Number of range bins to keep
smpperbaud 1 samples per baud
machine ri ri,cbr, or unpacked data
dcdpacked 0 if decoder data then 1, 2 , or 4
# samples per 4 byte word.
dcdfftlen 131072 Length of fft for decoding.
codestodecode 9999999 Number of codes to decode.
removedc 0 if 1, remove dc before decoding.
cohavg 0 Codes to coherently avg.(used by
# drv.power
spcfftlen 2048 Length fft for spectral analysis.
spcfftkeep 400 number of freqs to keep in map.
#
# Below are the suffixes used for the different input/output
# files. You can change them (although it isn't necessary).
sufraw raw
sufdcd dcd
sufspc map
sufeng eng
sufpha pha
sufscl maps
# end of drv.dat

You should copy this file to your working directory before running any of the scripts via:

cp ~phil/datatk/astro/rdrN/drv.dat yourfilename.dat

You can then use an editor to make the needed changes in the value column.

The Scripts in more detail:

drv_all.sc [-n] firstfile numfiles

What it does:

: This scripts performs the normal processing. It decodes the data, transposes the matrix, computes the spectra, and then makes unscaled delay doppler maps. If the -n option is set, the raw input files are not deleted after they are used (if -n is used, be sure that you have enough disc space). You must run drv_tapetodisc.sc to put the data on disc before running this script. When done with this script, the .dcd and .map files will be out on disc.

What it uses:

: This script calls other drv_ scripts. It does not use values from drv.dat explicitly. It uses the scripts: drv_decode.sc and drv_fft_pwr.sc.

Things to watch out for:

: When running this script make sure there is enough disc space (see the computing disc usage section below).

drv_tapetodisc.sc firstScan firstfile numfiles

What it does:

: This script will copy numfiles files starting with scan firstScan from tape to disc. The output disc file names will be fbase.rawfN. where fbase comes from the drv.dat file and N starts counting at firstfile.
For each file copied to disc, a header file will be generated with the name fbase.hdrfN. It will contain an ASCII printout of the header from the tape. All succeeding scripts will append information to these header files.

What it uses:

: This script uses: fbase, tapedrv, and sufraw from the drv.dat file. The filters used are dd and splitScan

Things to watch out for:

: The input records are limited to 70k bytes.

drv_decode.sc [-d] firstfile numfiles

What it does:

This script decodes the data. It will decode numfiles files starting with file number firstfile. It does this by calling the script main_decode.sc numfiles times. Each time, the requested polarization is extracted (if two are used), the data is decoded via the radardecode filter, and then the range bins bin1 thru bin1 + numbins - 1 are output to disc. If the -d option is set, the input files are deleted from disc after the output file is created ( you should normally run with the -d option to guarantee that the output files will fit on disc).

What it uses:

: This scripts uses: fbase, bits, codelen, smpperbaud, numpol, poltouse, bin1, numbins, dcdfftlen, codestodecode, removedc, machine, sufraw, sufdcd from the drv.dat file. If codestodecode is set to a large number, the decoding will stop when the end of file is hit. The filters and other scripts used are: main_decode.sc, radardecode, and selectpnts. The input files are named fbase.rawfN and the output files are fbase.dcdfN.

Things to watch out for:

The data output from this stage can be large. The data expansion factor is:

(32) * (numbins/codelength) / (bits*numpol)

Since each input file is deleted after it is decoded, the disc space will not grow as long as you keep the output file about the same size as the input file. This can be done by adjusting the numbins (the number of range bins kept).

drv_fft_pwr.sc firstfile numfiles

What it does:

This script computes the delay doppler maps from the decoded data. The harmonic analysis is done one file at a time.

Each block of numbins * spcfftlen of data is transposed and then for each range bin the data is fft'ed, the power is computed, and then the power spectra for a range bin are summed. spcfftlen in drv.dat determines how long a transform is done while spcfftkeep determines how many frequency bins are to be kept (centered on dc). The data is output one range bin at a time starting with the first range bin (the first range bin was determined back in drv_decode.sc by the bin1 parameter of drv.dat).

When done, there will be numfiles files each containing a delay doppler map. The maps are stored with the lowest range bin coming first. For each range bin there will be spcfftkeep frequencies with the DC component of the spectra in position spcfftkeep/2 + 1 (one place to the right of the middle). The maps contain real*4 data. The maps have not had the mean removed or been scaled to units of sigma (see drv.scale.sc to do this). The decoded input file can not be deleted by this script.

What is uses:

: The script uses: fbase, sufdcd, sufspc, numbins, spcfftlen, spcfftkeep, numpol, poltouse, and dcdpacked from drv.dat. The filters used are: selectpnts, radarfft, and rotate. The fourier transform is the mit algorigthm that allows for many different lengths. The rotate filter is used to move DC to the right of center. The input files are fbase.dcdfN while the output files are fbase.mapfN.

Things to watch out for:

: If the number of time points in the file is not a multiple of the fftlen and the number of time pnts is greater then the fftlen , then the extra points at the end are ignored. If the number of time points is less then the fftlen then the transform will be zero exteneded.

Misc. things
Disc Usage.

: When decoding data, the amount of disc space needed for output is: (32) * (numbins/codelength) / (bits*numpol) time the input file size. When transposing the matrix in radarfft you will need enough memory to hold 5 * numbins *spcfftlen * 4 bytes (complex input buf, complex fftbuf, and read power buf for averaging). When drv_all.sc is run the -d option is enabled by default to remove the raw input files when done (use -n to override this).

Checklist of processing steps:

When processing the data you should:

1.: Find a directory with lots of disc space. You will do all the processing from this directory.
2.: Copy ~phil/datatk/astro/rdrN/drv.dat to the directory. Edit it to reflect the data that you are about to process.
3.: Setup your $path variable so Unix knows how to get at the scripts. set $path=( $path ~phil/datatk/astro/rdrN ) if you are using the c-shell.
4.: Copy the files from tape to disc using drv_tapetodisc.sc. Be sure the correct tape device is in drv.dat .
5.: Run drv_all.sc 1 numfiles to process the files.
6.: As the processing goes along, make sure there aren't any problems. The decode files should all be the same size. If someone else fills up your disc partition while you are running then you may lose the last part of a run (I know this from experience!).
7.: After creating the .map files, use drv_findmax.sc to check that you have an echo and that it is in the correct range and frequency bins. Use drv_scale.sc to convert the map files to maps files (in units of sigma). You can then reuse drv_findmax.sc on the maps files to see how strong the echo is in sigmas for all the files.
8.: Make single fits files with drv_makefits.sc or mosaics with drv_mapmsc.sc.

FILES

Most executable programs (filters) are stored under /usr/S2local/bin. The source for these filters is in ~phil/util/filters. Some of the general purpose scripts may be under /usr/local/bin. These filters and scripts include:

avgdata       filter to average input points and/or ipps.
comppncode    filter to compute pncode.
datastat      compute max, min, mean, and sigma for a dataset 
dd            Unix supplied program to input and output data.
ex.awk        awk script to search file for keyword & return value
fftfilter     filter to compute fourier transforms
filesize.sc   script to return filesize in bytes.
maxinipp      filter to find max value in an ipp.
mapsmerge     merge multiple maps into one large map.
phase         filter to compute the phase given  sine and cosine as inputs. 
power         filter to compute power on complex input points.
printhdrV     filter to print a binary data header. 
radardecode   filter to decode radar data
radarfft      transpose, transform, compute power, and average.
scaleto8bitsr filter to scale real numbers to 8bits. 
scaletosigma  filter to remove mean and scale to sigma units.
selectpnts    filter to pass on a subset of the input bytes
splitscan     filter to split a datastream of scans into files.

The radar scripts are under ~phil/datatk/astro/rdrn.

miscellaneous files:

~phil/datatk/astro/rdrN/drv.dat  file that controls what scripts do.

ENHANCEMENTS

The drv_ scripts should have the option of putting them all together in one long pipe. There would then only be the raw input file and the map output file. The processing might also be faster if the context switch time for the pipes is less than the i/o time needed to read/write the intermediary disc files.

This document was created by man2html, using the manual pages.
Time: 21:00:10 GMT, November 03, 2010