Removing rfi in clp data
How the rfi is removed
Looking at the results:
- clp coding spreads out the ion and plasma line. so
removing spikes in the spectral domain does not affect
- When using uncoded pulses (or tpsd data), the ion line (and
maybe a strong plasma line) will be clipped by the rfi removal
(since they stand out in the frequency domain).
rfi removal in clp data can take
advantage of the decoding step when trying to remove frequency
clp processing for each 10 second block of data (1000 ipps)
If rfi occurs with a finite bandwidth in an ipp:
- For each ipp:
- code the transmitted signal with a 2 usec pncode (2^39 code
length). The 2 usec baud defines the height resolution.
- sample the 440 usec coded transmitter pulse at .2 usecs
(2200 samples). Use this for decoding the returned echo.
- 440 usecs is pushing the duty cycle of the pulsed
- sample the height data at .2 usecs (5 MHz bandwidth). keep
22000 samples for each ipp
- for each of the 3961 heights (spaced by 1 usec):
- multiply 2200 height samples by the conjugate of the
code, then 0 extend to 4k.
- do a 4k xform
- compute power
- accumulate this spectrum in the corresponding height of
the average power buffer (4k,nhghts)
- continue this average the 3961 height spectra for the 1000
ipps (10 secs).
- multiplying by the code in the time domain is the same as
convolving the voltage spectrum by the transform of the code.
- This will spread narrow band rfi to 1/baudlen or .5 MHz. It
will also alternate the phase of these spread out samples
- this will decrease the strength of the rfi.
- very strong rfi will still appear in the spectrum spread out
over .5 MHz
How the rfi is removed: (top)
RFI can be removed by clipping large
signals in the frequency domain (prior to decoding). The process is:
- transform the 22000 height samples to the frequency domain
- compute the power (spcV*conj(spcV))
- find rfi in this power spectrum
- find the rfi outliers
- replace these outliers in the spcV (voltage spectrum)
- Replace the values with 0's or maybe with noise
- The notch that is put in the spcV by the 0's (or
noise) will get convolved by the transform of the code
(during the decoding step).
- This will smear out this discontinuity in the final averaged
- transform back to the time domain.
- proceed with regular clp decoding.
Looking at the results: (top)
Two 5 MHz bands were accumulated for 10
seconds centered at 435.5 and 439.5 MHz. The 10sec block chosen
- rfi in the lower block that sits on top of the plasma line
- rfi in the lower block sits above the plasma line
Height vs frequency
The table below shows the averaged data processed with no
clipping, clipping with 0 replacement, clipping with noise
spectra over all heights and 10 seconds. (top)
The plots show the average spectra for
the 10 seconds and all heights (.ps) (.pdf):
- Page 1 is channel one (centered at 435.5 MHz). Page 2 is
channel two centered at 439.5 MHz.
- Top: spectrum of undecoded data.
- The spectra of the 1000 ipps were averaged
- The rfi is narrow since the decoding has note been applied
- The vertical scale is in units of Tsys
- Middle: blowup overplotting raw and decoded spectra.
- black. Average of undecoded spectra (same as top plot)
- red: average spectra of decoded data
- The decoding has spread the rfi in frequency
- You can see sidelobes of the code. These show up in
the images as vertical stripes (they wrap around on page 2).
- Not sure why the decoded rfi is not centered at the
- Green. average of decoded spectra. Replace outliers with 0's
- blue: average of decoded spectra. Replace outliers with
noise before decoding
- Bottom: Blowup of average spectra with rfi outliers removed
- green: remove outliers with zeros
- blue: remove outliers with noise
- The two plots track each other pretty well.
- On page two the dip around the rfi is deeper for the
spectra with zero's removed.
noise with rfi in and outside of the plasma line freq range (top)
The below plots show a 10 second average
spectrum at a particular height that includes the plasma line.
Some have rfi outside of the plasma line, other plots have rfi on
top of the plasma line. The data was taken from
the 10 second average spectrum at a given height (.ps)
- Page 1: band 2 (439.5MHz) Plasma line cut off: height: 282.2
- Top: 440.5 - 441.2 MHz
- black: no rfi removal
- The rfi starts around 441.4 MHz.
- red: replace rfi with zeros
- green: replace rfi with noise.
- The solid line is the same data filtered by a 71 channel
- The plasma line is at 441.2 MHz
- Middle: normalize the bandpass by the median filtered
- overplot the 3 spectra
- green overlays the red... There is no difference
between the rfi zero fill, and rfi with noise fill.
- bottom: blowup in frequency, scale to sigmas.
- 440.5-441.0 was used to compute the rms. This was then
used to scale the plot.
- spectra with and without rfi removal are the same.
- Page 2: band 1 (435.5MHz) rfi in plasma line. Height: 558.5
Km, 432.28MHz. upshifted line
- top: data scaled to Tsys
- black: no rfi removal
- The large bump is the rfi at 434.18 spread by the code.
- red,green:( remove with zeros, noise)
- The solid lines are the data filtered by a 71 channel long
- the dotted red line is the location of the plasma line
- .bottom: data scaled to sigmas.
- the amplitude of the line is larger in the unfiltered
data, but it has many more noise outliers,.
- Page 3: band 1 (435.5 MHz).downshifted plasma line at
- Top: the 3 spectra scaled to Tsys
- bottom: the 3 spectra scaled to sigmas.
- In this case, the filtered data is larger than the
- Page 4: Mean,rms values of the undecoded spectra for each ipp.
- black is chan 1 (435.5), red is chan 2(439.5 MHz)
- Top: the mean value of the spectra for each ipp
- 2nd: the rms value of the spectra for each of the 900 ipps.
- rfi in channel 1 jumped up around ipp 730
- rfi in channel 2 jumped down around ipp 850
- 3rd: rms/mean for spectra of each ipp.
- Only 1 spectra has been accumulated, so rms/mean should
equal 1 (the actual value is 1.2 .. this may be from the
channel filter width.. a little larger than
- ipps with rfi have rms/mean > 1/
- 4th % of points discarded in an ipp because of rfi
- This used a 4 sigma clipping level to identify rfi
- black chan1:
- .7% of the points were discarded until ipp 730 when it
dropped down to close to .1%
- When the rfi appeared, the sigma increased, so points
> 4sigma were fewer.
- red: chan2:
- .3% of the points were excluded until ipp
- bottom: how many times iterated the rms.
- If the rms decreased by 2 after throwing out all points
> 4sigma, then the rms was recomputed using those points
- This occurred for most ipps with strong rfi.
Mike's plots (top)
- compare plasma line freq vs time:
processing: x101/atm/clpshs/cliprfi/chknseclip.pro shsclp.pro
- replacing points in the ipp spectrum with values > 4sigma
by zeros or noise reduces the decoded plasma line images.
- Replacing the points with zeros, or noise showed no difference
in the results.
- .2% to .7% of the points were modified in each ipp.
- the lower value .2% occurs with strong rfi.
- This is not getting rid of all of the rfi in the ipp.
- for Gaussian noise, 4 sigma should have only 1 or 2 points in
- We find 44 to 176 points. part of this may be the bandpass
correction .. although i think the noise is probably not
- Decoded rfi is not centered at the rfi freq (off by a .1-.2
MHz). Mike thinks this could be from the transmitter pulses used
to decode the data.
- I'll implement this in the multi threaded code, with an option
so specify the sigmas to use as replacement.
up to aeron