Pdev diagnostics

05aug10

Terminology
Intro
The diagnostics

ltest
rtest
rcomp
rdcomp

Running the tests
Diagnostics results
FPGA test signals. PNcode and noise src (with amplitudes), pncode

Terminology

configuration files:

Used to configure the registers in the spectrometer/fpga. look in /dump/pdev/files on pdevs1.

DLPF:

Digital low pass filter. This reduces the bandwidth of the input data.

File server:

The spectrometer connects to the file server via Ethernet.

polyPhase filter bank

spectral code in fpga. include complex mixing, dlpf,pfb,ff

spectrometer

The box containing the A/D, fpga, and power pc. connects to file server via Ethernet

Intro.

The pnet spectrometer control program written by jeff mock contains options to perform diagnostics on the spectrometer boxes and attached file servers. The diagnostics make use of a pncode generator built into the fpga as well as noise and cw tone sources built into the spectrometer fpga. The tests include:

ltest: pncode on fpga read into spectrometer and verified

test fpga and dma fpga to power pc on spectrometer

rtest: pncode on fpga read into spectrometer, send to file server and verified on file server.

ltest plus tests power pc to file server dma

rcomp:pncode on fpga sent to spectrometer and then pdevs1 file server. pdevs1 then compares the values from all the spectrometers.

pretty much the same as rtest except that all spectrometers are merged to pdevs1

rdcomp: user sets up internal noise or cw source. This data is passed thru the sp fpga code (spectral processing). All spectrometers send their data to first file server in configuration file (boxesDiag.conf ). This fileserver (in psrv.c) compares the bytes from all the configured spectrometer. Since the noise source is the same in all spectrometers, they should generate identical outputs.

This tests the "sp" code of the fpga (dlpf, pfb,fft) etc. You need to tailor the config file for the sp config you want to test.

A debugging method would be:

ltest: check dma fpga to spectrometer box to ensure this works.
rtest: check dma fpga to spectrometer box then file server
rdcomp: check fpga sp code. noise source to sp then spectrometer and file server. You can run this on various configuration files.

The tests:

--ltest=iorateMb/sec

fpga generates pn code, gets dma'd to spectrometer, and then data verified on spectrometer.
routines used:

Pnet command code: T,t, args: locdma, n to config pn in fpga(see below)
pnet routine: pnet_dmatest with locdma=1
prun.c : prun_ltest

Output:

Actual datarate is 5.00 MB/s for 2477.

fpga spits out 9 64 bits words every n cycles. 2477 is n used to get 5mb/sec at 172. Mhz clock.

Info from SPs:

b0s0g0: 0 0.066 4.84 0 6553

0 - index (output is once a second)
0.066 Total Mbytes xfered
4.84 measured i/o rate Mb/Sec
0 max length of fifo that was used in dma.
65536 max length for all reads.

Notes:

i/o rate is sum of both fpga's on spectrometer. fpgu to powerpc about 110 Mb/sec. this can be 1 fpga, or 55Mb/sec for each of two fpgas.

--rtest=iorateMb/SEc

fgpa generates pn code, gets dma'd to power pc then file server. psrv routine on file server checks the data.

Each spectrometer sends data to it's own file server.
The pserv is only using a 4kb buffer so the i/o rate is less than when a 20 mb buffer is used.
One psrv for each band that is used.

routines used:

pnet command codes:

prun: T,t args locdma, n to config pn in fpga, M,m for remote dma.
psrv: Y,y to get ready to check data.

pnet routine: pnet_dmatest with locdma=0
prun.c

prun_rtest. just starts dma to file server.
prun_rmtdma - reads data from fpga and sends it to file server.

psrv.c : prn_recv

output:

Info from SPs:

b0s0g0: 3372.188 34.91 3584 53936128
This comes from prun_rmtdma()
Last number looks like it is the sum of the two last maxreads.

--rcomp=iorateMb/sec

using pn code have all sp's send their output to pdevs1. compare each value from all the sp's. This uses the merged code in psrv.c
since all the i/o streams are multiplexed onto pdevs1 Ethernet you need to keep the individual io/rate low enough so that psdevs1 Ethernet stays around 70 mb/sec.

for 1 group 5mb/sec*2(fpga/box)*7 boxes=70mb/sec

Routines used:

pnet cmds/routines: B,A,G,T,M

B pnet_connect_psrv() send B to psrv to merge all input streams into pdevs1
A pnet_connect_psrv() tells all sp's to output to pdevs1
G pnt_rmtcomp-> psrv_process_merge_cmd() psrv_merge_compare.

get ready to merge and compare

T pnet->prun setup pn code generator .. with locdma=0
M pnet->prun start dma
only arg is requested i/o rate.

prun.c

prun_rtest
prun_rmtdma

psrv.c

B psrv_merge_connect (only on pdevs1)

generate socket, listen
tell pnet name of socket
pnet will then tell all sp's to connect to this socket.

psrv_merge_command() called at end of psrv_merge_connect..

psrv_process_merge_cmd() to process each command.
G psrv_merge_compare()

this reads and compares al lthe bands.

--rdcomp

uses config file to setup the sp fpga code. You need to select the noise or test tone. All the sp's are merged into pdevs1 file server where it is compared.
Since all the streams are merged to pdevs1 the aggregate i/o must be < 70 Mb/sec

The i/o rate is set by fftlen,fcnt, decimation, and sp*2

ioRate1Sp= fftlen*(4float)*(2pols)/(tm1spc)/fcnt/decimation
fcnt=10000, fftlen=8192, clk=172.032 Mhz gives:

ioRate1Sp= .068 mb/sec6 mb/sec
ioRate1grp= 14*ioRate1Sp= .9 mb/sec
the limiting factor may be the cpu time for the compare.

Routines/cmds used:

pnet

B pnet_connect_psrv() send B to psrv to merge all input streams into pdevs1
A pnet_connect_psrv() tells all sp's to output to pdevs1
G pnt_rmtcomp-> psrv_process_merge_cmd() psrv_merge_compare.

get ready to merge and compare

M pnet->prun start dma

prun.c

prun_rtest
prun_rmtdmd

psrv.c

B psrv_merge_connect (only on pdevs1)

generate socket, listen
tell pnet name of socket
pnet will then tell all sp's to connect to this socket.

psrv_merge_command() called at end of psrv_merge_connect..

psrv_process_merge_cmd() to process each command.
G psrv_merge_compare()

this reads and compares al lthe bands

Running the tests:

Config files to run the tests are on pdevs1.

slogin pdevs1
cd /dump/pdev/files
use rundiag.sc {-g 0/1} {-t ltest rtest rcomp rdcomp} to run

Edit boxesDiag_g0/1.conf boxes file to select the sp's to test
The .conf files are:

ltest,rtest,rcomp

Uses pnetDiag_g(n).conf
this selects the correct boxes file and then included pnetDiagIncl.conf

rdcomp

uses pnetRdcomp_g(0,1).conf
selects the group boxes file and then includes pnetRdcompIncl.conf

use pdev/pdev/datatk/pdevsyntest to update the synth freq

pdevsyntest

clk stat .. to get status
clk freq 172.032e6
crtl-d to exit

ltest:

pnet --conf=pnetDiag_g0/1.conf --ltest=ioRate
io/Rate is Mb/Sec. It is fpga to spectrometer. The max i/o rate is 110 Mb/sec for a single band in spectrometer, 55 Mb/sec if using 2 bands in the spectrometer.

rtest:

pnet --conf-pnetDiag_g0/1.conf --rtest=iorate
i/o rate is limited by spectrometer to file server. For a single band per spectrometer this is about 50Mb/sec. for dual bands it's about 35-45 Mb/sec (see table below). This is slower than the actual datataking speeds because it is using a smaller read buffer (4k rather than 20Mb).
When running it works better if you run 1 group at a time. With both groups fails sooner (shouldn't since i/o is to each fileserver).

rcomp:

pnet --conf=pnetDiag_g0/1.conf --rcomp=iorate
All spectrometer bands are multiplexed onto the first file server in boxes.conf. The i/o rate for all of the bands needs to pass thru the Ethernet cable for the first file server. Should keep the i/o rate (per band) small enough so that the sum is < 70Mb/sec.

rdcomp:

pnet --conf=pnetRdcomp_g0/1.conf --rdcomp
Edit pnetRdcompIncl.conf for the sp configuration that you want.

Be sure and set A/D inputs to 4 (noise source) and set the noise source amplitude (TS_NOISE_A/B).
Be sure the dlpf file for this decimation is on /dump/pdev/files of pdevs1. If not, run cima with this bandwidth to generate it.

pnetRdcomp_g0.conf is for group 0
pnetRdcomp_g1.conf if for group 1

These two just select the different boxes files for g0,g1
they both include pnteRdcompIncl.conf that has the main conf

edit boxesDiag_g0.conf or boxesDiag_g1.conf to add ,subtract boxes

Some things that can cause the tests to fail:

rdcomp sends all the data to the first fileserver in the list

If this fileserver is busy i've seen the tests fail with errors
When it fails, sometimes you can make it work again by decreasing the number of the sp's being tested

This seems to help even though the i/o rate is .068 mb/sec/box.
It may be that the server cpu gets too busy to compare the numbers
Or maybe the file server input from the various sp's is dropping packets?..

But it never fails on the rcomp merge which uses just the pncode.

fpga test signals:

The fpga contains a pncode generator, noise generator, as well as cw generator.

Pncode generator:

Used by --ltest,--rtest, --rcomp. It does not pass through the fpga sp code. I'm not sure what the levels are.

Noise source generator:

The noise source is enabled with the TS_NOISE_A,TS_NOISE_B keywords. The value determines the strength of the noise source (0 to 65535). The table below shows the mean value of the output spectrum (average channels 2000-6000 in an 8K spectrum) for various settings of the noise generator and spectrometer configuration:

TS_NOISE_X value	hr_shft	decimation	Fcnt	time Rec secs	spc Mean counts	spcMean 1 spc
65535	12	2	21000	2	2.4e8	11428
65535	12	2	10500	1	1.2e8	11428
65535	11	2	10500	1	3.1e7	2952
32767	12	2	10500	1	3.1e7	2952
32767	11	4	10500	2	1.33e7	1266
32767	11	4	5200	1	6.6e6	1269
32767	-	full bandwidth	21000	1	6.5e7	3095

Diagnostics results:

Pmax. maximum number of fifos bufs used??
maxread: max length of i/o read buf?
Individual days
Summary tables of all runs

ltest
rtest
rcomp
rdcomp

Dates:

19sep11 rdcomp

b5s1g0 failed .. chris had trouble with this on previous nite.j
b1s1g1 failed all the time
b4s0g1 failed two or 3 times in 5 minute run
If i disable b5s1g0 and b1s1g1, then i could run all the rest without trouble.

except b0s6g0 very rarely had a bit error.

20sep11 after b5sxg0 104 -> 112

ltest - all ran together (both groups) at 55 mb/sec
rtest- when ran groups together tended to fail. ran better 1 group at a time.

group 0:

all ran at 40mb/sec
at 45 mb/sec ran for awhile then b6s0g0 overran.

group 1:

b5s1g1 failed after awhile at 40 mb/sec
Commented out b5s1g1 then reran at 40mb/sec

b5s0g1: 4544.594 39.96 384 17891328
The I think 384 is the maximum length of the dma buffer used
Normally this is 130000... Max size of dma buffer. It is small because with only 1 fpga it doesn't have to share the dma to the power pc or the ethernet to the file server.

After a few tries all ran at 35 mb/sec.

rcomp:

group 0: all ran ok at 5mb/sec
group 1: all ran ok at 5mb/sec

rdcomp:

clock	grp0	grp1
172.032	7mb b6s1g0 2 errors	4.4mb then b1s1g1 caused abort. too many errs. 3.6mb then b4s0g1caused abort. 9.6mb no errors.
170.00	7mb noerrors	7mb noerrors
165.00	7mb noerrors	7mb noerrors
160.00	7mb noerrors	7mb noerrors

clock 170.032

group 0: ran for 5mb. only 1 error on b0s0g0

group 1:

b2s0g1 caused failure first couple of starts
ran for 5 mb then b2s0g1 caused it to die with too many errors.

clock 160.00 mhz

grp0,1 ran for 7 mb with no failures in either.

clock 165 Mhz

grp0,grp1 for 7mb no errors

clock 170 Mhz

ltest table

bsg	pdevNum	maxI/o	PMAX	MAXREAD	DATE
b0s0g0	110	110	40880	327680	100806
b0s1g0	110	110	41072
group 0 both bands
b0s0g0 b0s1g0 b1s0g0 b1s1g0 b2s0g0 b2s1g0 b3s0g0 b3s1g0 b4s0g0 b4s1g0 b5s0g0 b5s1g0 b6s0g0 b6s1g0	110 103 107 118 116 104 113	55	4496 39728 4560 39952 4544 40192 5040 40512 5088 40224 4768 40080 4640 40224	131072 131072	100806
group 1 both bands
b0s0g1 b0s1g1 b1s0g1 b1s1g1 b2s0g1 b2s1g1 b3s0g1 b3s1g1 b4s0g1 b4s1g1 b5s0g1 b5s1g1 b6s0g1 b6s1g1	119 108 122 106 115 117 120	55	same as above		100806

rtest table

Tests fgpa, pfifo, pwrpc, Ethernet to file server, and pserv reading in and checking.
acquire at least 1gb of data. There is no disc i/o for this test.
I had arun stop the rsyncing of data so the results would be more reliable.
changed i/o rate in steps of 5mb/sec
19sep11: ran 1 group at a time.. both groups together would fail

Pmax

bsg	devnum	MaxI/O(perfpga) Mb/Sec	Pmax	maxRead	date
b0s0g0	110	50	512	62390272	100805
group 0 two bands
b0s0g0 b0s1g0 b1s0g0 b1s1g0 b2s0g0 b2s1g0 b3s0g0 b3s1g0 b4s0g0 b4s1g0 b5s0g0 b5s1g0 b6s0g0 b6s1g0	110 103 107 118 116 104 113	35 40 40 40 45 40 40	5248 16896 44544 112768 15360 458752 41344 102528 50816 192384 44928 114304 44928 97536	19857408 12976128 20054016 15990784 786432 589824 589824 655360 7733248 1114112 655360 655360 655360 720896	100805
group 1 two bands
b0s0g1 b0s1g1 b1s0g1 b1s1g1 b2s0g1 b2s1g1 b3s0g1 b3s1g1 b4s0g1 b4s1g1 b5s0g1 b5s1g1 b6s0g1 b6s1g1	119 108 122 106 115 117 120	40 40 40 40 40 25 40	25984 83072 34176 45312 28288 51328 41728 106240 35968 112128 128 128 38528 59776	589824 589824 589824 786432 589824 589824 524288 720896 655360 655360 57212928 39976960 589824 655360	100805
b5s0g1	117	35	128	43384832	100805
b5s1g1	117	30	0	23986176	100805

rcomp table

using pn code generator, merge all sp streams to pdevs1. Let psrv then compare each value from each sp to make sure they are the same.

bsg	devNum	i/orate Mb/sec (each sp)	Pmax	maxRead	date
group 0
b0s0g0 b0s1g0 b1s0g0 b1s1g0 b2s0g0 b2s1g0 b3s0g0 b3s1g0 b4s0g0 b4s1g0 b5s0g0 b5s1g0 b6s0g0 b6s1g0	110 103 107 118 116 104 113	5	0	3997696 11796480 5439488 10747904 4587520 6094848 5570560 12124160 5177344 15532032 8192000 21889024 12845056 10813440	100805
group 1
b0s0g1 b0s1g1 b1s0g1 b1s1g1 b2s0g1 b2s1g1 b3s0g1 b3s1g1 b4s0g1 b4s1g1 b5s0g1 b5s1g1 b6s0g1 b6s1g1	119 108 122 106 115 117 120	5	0

Rdcomp table:

usually fcnt=1000,8192,polA,polB
tm spc= 8192/clk * fcnt * deckk
i/o rate:8192*4*2/tmpSc * 14 (1 grp)
When errors would abort it, i'd remove the offending board and rerun until i could run it with no errors for about 15mbytes.

date	clk	dec	Duration(Mb)	grp	errors
17apr12	164.6592	2	19.268	0	none
		2	15.0	1	none
		1	21.2	0	none
		1	22.6	1	none

	172.032	2	15	0	immediate failure: b5s0g0,b0s0g0,b1s1g0
		2	15.	1	b5s1g1 .. aborts after a few secs b1s1g1 ..intermittant errs. aborted after 11 mb. b3s0g1 .. aborts after 1.5 mb. b6s1g1 .. intermittant failures. ran till 16mb. b2s0g1, b4s0g1 aborts after 2.5 mb
		1	15.35	0	none
		1	17	1	none
01oct12	172.032	2		0	b0s0g0 fails immediately b0s1g9 fails immediately rest run vor 6mb no errro
	172.032	2	6mb	1	b4s0g1 1 error b0s0g1 2 errors
	163.84	2		0	b0s0g0 failed after 2mb commented out b0s0g0 . ran for 6 mb no errors then put all back in and ran for 6bm no errors.
05nov12	165.00	2	0	1	b1s1g1 polA failed at most clocks. pdev108 was bad
05nov12	165.00	2	20mb	1	ran for 20 mb without errors with pdev-101 replacing pdev=108 at b1sxg1 prior to 20mb run there were a few errors at b4s?g1
28mar13	172.032	2	abortedd 13mb	0	b5s0g0 failed immediately. After removing this one, it ran for 17.8 mbytes with only occasional errors from b6s?g0
	172.032		15mb	1	occasional b6s1g1 error but ran until i stopped it at 15.7 Mbytes
	170.0		aborted 15.5mb aborted	0	ran for 3.9 mbytes then faild b1s1g0 and a few b6s1g0 errors I then reran, and it ran for 15.5 Mbytes no errors later reran. b6s1g0 occasional error, b1s1g0 errs. quit after 5 mbytes.
	170.0		15.2mb	1	occasional err b6s1g1.. but ran till i aborted it.
	165.0		aborted	0	failed after 3.5 mbytes. b5s1g0
	160		33mb	0	ran with no errors
	160			1	ran with no errors,

rdcomp Notes:

17apr12:

164.6592

runs ok at dec=1,2. no errors

172.032

dec =1 runs ok
dec=2 ..errors ..grp 1 worse than grp 2.

23mar13:

strange that it ran ok with 172.032 mhz then failed at 170?
joanna rankin had trouble the previous day with one scan. b1s1g0 had lots of jumps.
looks like 160 Mhz always works
looked at the ethernet errors. There were some dropped packets. For this test all the data comes into eth0 of pdevs1.

<- page up
home_~phil