Glitch in ri A/D system Dig Q1
Diego janches had problems with his meteor data taken
07jul03. His setup was:
I,Q sample both channels (fifo 12) at 1 usec
input voltage levels about 13 mV rms.
Start the sampler running and leave it running for 12 hours (4mb/sec, 172
How the ri works:
The ri has 4 12 bit a/d converters (I1,Q1,I2,Q2). Each
12 bit sample is sign extended into a 16 bit number. The data from each
digitizer is clocked into a 32K by 16 bit fifo by a delayed sample pulse.
The data sits in the fifo until a read request comes from the vme bus.
At that time, a rdFifo pulse is generated and a sample at a time
is read out of the fifo. Each 16 bit by 32k fifo is built by combining
two 8bit by 32k fifo chips.
The problem occured with digitizer Q1. The data
values (a/d counts with 1 count=1.2 mV) ranged from +20 to -20 counts.
At some point there was a glitch and it would jump from a negative number
to +250 and then -250 (about 250..) and then back to +/-20. This jumping
would then continue once it started for the rest of the run. The problem
did not show up when the test patterns were run through the system (zeros,
toggle, staircase tests were run for many hours without a failure).
of the meteor data shows that the voltage distribution is skewed. There
are about 15e6 samples in the distribution.
To debug the problem, we took the same base band noise
and passed it through 4 opamps and then into the 4 a/d converters. The
signals should then be pretty close for all 4 a/d converters (the offsets
and gains can be a little different). When digitizer Q1 jumps, we can look
at the other digitizers to see what it should have done. The plots show
glitch with the same signal input to all 4 digitizers:
Fig 1 Top: Vertical scale is full scale counts. The vertical scale shows
+/- 40 a/d counts. The black line is skewed for levels -10 to 0.
The other a/d 's have jumps about 0.
Fig 1 bottom: The vertical scal is limited to 1e6 counts. The horizontal
scale is extended to show a/d values +/- 300. There are peaks at +/- 250.
Digitizer Q1 counts around zero have been moved out here.
The key to the error is that once the error occurs,
it continues for hours. This implies that there is a memory somewhere that
remembers that the error has occurred. This is what made us suspicious
of the fifo chips. The table below shows the data samples around
the start of a failure.
Fig 1 Top: 50 samples with the glitch evident at sample 11. Black
is digitizer Q1.
Fig 1 Bottom: a blowup of the top plot showing that the 4 digitizers have
the same signal (with differing offsets). They follow each other until
Fig 2: Plot of sample[i+1]-sample[i] for 20 samples. You can see the black
curve diverge at sample 6. The yellow curve is the Q1 data throwing out
the low Q1 byte at sample 6. All the Q1 low8 samples after this are shifted
forward in time.
The table below shows the data values (in hex and decimal) where the failure
Col 1 . This is the relative sample number as it appears on the data file
Col 2. This shows the corresponding a/d samples in the data file.
The two numbers are the samples coming out of fifo Q1Hi and fifo Q1Lo.The
problem occurs at sample 6. The rdfifo pulse did not last long enough
for Q1Lo to cause the fifo to output an 8bit value. The computer
ends up reading 0xff for Q1lo on sample 6 since the fifo output is tristated
hi. Sample 6 Q1Lo stays in the fifo and gets read out on sample number
7. The two fifos are now out of sync.
Col 3. These are the 16 bit data samples in Hex. The leftmost two
hex digits are from Q1Hi, the rightmost two hex digits are from Q1Lo.
The data at sample 6 should have been FF FA.
Col 4. This is the col 3 data in decimal.
Col 5. This has the col 3 data after correction. I removed sample 6 Q1Lo
and moved all the samples below this up by one. There are no longer any
Col 6. this is col 5 data in decimal.
17 16bit samples Q1 digitizer at failure
||FF FA SKIP FF LO8
What is the problem:
The problem is that the rdFifo pulse for fifo Q1lo was
not long enough to reliably clock the fifo output (we could see this when
we put a scope on the pin). When the clock pulse was missed, the computer
still read the output databus. The "missed" sample would come out on the
next fifo read. The high, low 8 bits would then be out of sync.
When did it first occur.
Diego has data from feb02 that has the jumps. Data from
Apr03 does not have the jumps. So it has been around awhile.
Who else might see this jump??
When the two bytes (hi,lo) get out of sync, data
that goes from negative (0xff nn ) to positive (0x00 mm) will show jumps.
If the voltage levels are small then these jumps will be obvious. If the
voltage levels are large, then you won't be able to distinguish these jumps
(of order 255) from the actual noise.
Most aeronomony/sband radar programs will take data
continuously for 10 seconds to several minutes. Each new cycle of datataking
will clear the fifo (resyncing the hi/lo data if an error has occurred).
Since diego took data continuously for hours and he had very low voltage
levels, his experiment was more likely then most to show the problem.
When I tried to generate the error using continuous
sampling, it took on average about 200e6 samples before the error occurred.
On 16jul03 the rdfifo pulse for all the fifos was lengthened
(by 7.5 ns??). After this i ran the ri for 6 hours continuously (about
5*e10 samples) and saw no glitches. I've also setup a macro for diego (rdloop)
that will take data for a specified number of buffers, stop, and then restart
taking ri data . This will clear the fifo and not let any errors continue.
Diegos data can be corrected. You need to search
through the data till you start seeing these jumps. At that point shift
the hi8,lo8 bytes to be in sync again. Continue reading data, shifting
until you start to see jumps after the shift is applied. This is where
another missed fifo pulse occurred. You need to do this for all of the
data. You also need to be careful that you don't mistake rfi/meteors for
the jumps. That shouldn't be too hard since the jumps occur very often
after they start (everytime the voltage crosses 0 volts).
processing: x101/030715/doplot.pro, usr/t1748/doit.pro