clp processing program

jan09

Intro.
clp - mutltithreaded fftw
clp1- 1 thread per 10sec block (.rdev data)
clp1shs - 1 thread per block (.shs data)
    Selecting the clp ipps for the .shs data.
Compute times using 45 threads on rserv1.
09dec13: clp processing times on wombat with 15 and 30 threads.

See also: Misc coded long pulse info


Intro:

    The clp processing programs input the clp data from the telescope, decode each ipp, and then accumulate for a specified number of ipps (usually 1000 or 10 seconds). The output is a 2D image of spectral density vs range.

Types of input data:

The different flavors of the clp processing programs are:


clp -  multiThreaded fftw (top)

    What it does:
    Info:
    File locations:

clp1 - single Threaded fftw, multi threaded processing.  (top)

    Info:

    File locations:


clp1shs - single Threaded fftw, multi threaded processing for .shs files.  (top)

    Info:


File locations:



Selecting the clp ipps for the .shs files.  (top)

    The echotek card tries to write only clp ipps to disc. It's algorithm lets a few mracf, power, topsd ipps slip through. An algorithm to discard any non-clp ipps used for the .shs files is:
Deciding if an ipp is a clp ipp:
Data from the file t2573_28dec2011_000.shs was processed to see what a reasonable threshold would be. The ipp used was 10 milliseonds. For each ipp  20 usecs of data (100 samples) was averaged about 150, 430, and 490 usecs (this file had mracf,power, and clp).

The plots show the power at each Tx position versus the ipp in the file (.ps) (.pdf):

Summary:

processing: x101/120111/testclp.pro
 

Using rserv1: 48 core amd machine:  (top)

    rserv1 is a 48 core amd machine. clp1shs data was processed using 45 cores. Some notes on the processing are:
Plots were made of the daily processing of .shs clp data. Data is plotted vs processed block number (each block normally 10 seconds of data).
The plots show:

Processing times by date:

Date
Plots
proj
thrProc
avg secs
4K fftAvg
Usecs
pwrAccum
avg
Notes:
20111220
(.ps) (.pdf) t2574 921 +/-39.6
85.7 +/-1
13.7 +/-3.9

20111222
(.ps) (.pdf) t2574 917+/-27.8
86.9+/-1.5
11.7 +/-2.6

20111223 (.ps) (.pdf) t2574 887+/-19.2
83.9+/-.6
11.5 +/-2.2
blk 1200 to 1600 input took 20 secs.
20111224 (.ps) (.pdf) t2574 913+/-28.9
85.8 +/-.7
12.8 +/- 3.0

20111228 (.ps) (.pdf) t2573 864.9+/-17
83.0 +/- .4
9.8 +/-2.0

20111229 (.ps) (.pdf) t2573 870.4+/-22.1
83.0 +/-.4
10.4+/-2.6

20111230
(.ps) (.pdf) t2573 864.9+/-18.0
83.9 +/-.4
9.9 +/-2.1
first 45 blocks ran 40 seconds faster than rest
20120112 (.ps) (.pdf) t1193 864.9 +/- 18.0
83.0 +/-.4
9.8
frist 45 blocks ran 40 secs faster. Looks like the pwr and accum
code started to take longer.
20120113 (.ps) (.pdf) t1193 866.2 +/- 19.2
83.1 +/- .3
9.9+/- 2.3
1st 45 blocks ran 40 secs faster then pwr,accum slowed down.
processing: x101/120114/clp1shstmingchk.pro, tmingchk.pro

09dec13: clp processing times on wombat

    The wombat computer was installed 06dec13. It has:

    I ran the clp code on wombat using 15 and 30 threads.
The plots show the processing times for clp on wombat (.ps) (.pdf):
processing times,throughput
cpu
threads
clp processing
4K fft processing


tm 1 blk
secs
1 thread ratio
(cpu/rserv1)
(tm 1 blk)/nthreads
secs
tm 1fft
usecs
1 thread ratio
cpu/rserv1
throughput
nthreads*ffts/sec
throughput ratio
cpu/rserv1
wombat
15
180
5
12
10
8.3
1.5e6
3
30
300
3
10 *
20
4.15
1.5e6
3
rserv1
46
900
1
19
83
1
.536
1

* wombat 30 threads processes 10 secs of data in 10 seconds.. It can keep up with real time.

Summary:

processing: x101/131207/clpwombat.pro

 home_~phil