Removing lightning from ips data

16sep19

Intro: (top)

Interplanetary Scintillation (ips) data has total power spectra with useable information to maybe 10Hz max. When taking the data, we compute spectra at a 500 or 1000 Hz rate so any radar ipps do not alias back down into the 0-10 Hz region of interest. rfi frequency spike removal is done on the millisecond sampled spectra.
The rms/mean by freq channel is computed in 10 second blocks. Any channels with rms greater than 4 sigma are not included in the total power computation.
This works well for rfi that is limited in bandwidth.
Lightning rfi occurs within a single millisecond sampled spectra and it it broad band. It can also be very strong, causing the spectra to be saturated. These narrow spikes can occur in bunches that last up to a second or more.

Attempts at removing lightning

A first attempt at removing lightning threw out any total power samples above a specified threshold. This did not appreciably improve the ips total power spectra since lots
of weaker lightning flashes were falling below the threshold.

A second attempt computed the rms over blocks of 10 total power samples (10ms for lbw data, 20ms for other receivers). If the rms was greater than 4 sigma then the
entire 10 samples of total power data were replaced with the median total power over the entire 180 seconds of the scan. This method was used to generate the following plots.

Plots of lightning and removal

Various days were examined to see how the lightning removal performed. Some days had strong lightning, other days had very little.
For each set of data examined:

For a given receiver only the first freq band is examined

327 - only 1 band
lbw - 1160 MHz band
sbh - 3120 MHz band

The plots show the total power time series without and with lightning removal

Black: no removal
red : with removal
Purple dashed lines: the 10 sec intervals used to average the total power spectra.
Green dashed line (in top plot): the area blown up in the bottom frame.
Top frame: all 180 seconds
bottom frame: blowup around a particular set of spikes.

The horizontal lines are where the 10 sample blocks with large rms are replaced by the median value.

The .jpg images show the standard ips total power spectra.

Columns 1 and 2: spectra by pol and frequency block (10 MHz each).

The first column has polA data.
The 2nd column has polB data.
For these 2 columns

the top row uses the average of the 18 10 second spectra
the bottom row uses the median of the 18 10 second spectra
The colored lines are the spectra without lightning removal
The white lines are the spectra with lightning removal.
If you don't see any colored lines, then the white lightning removed spectra exactly overlay the spectra with no removal.

Column 3

Top: the average and median spectra for all freq bands and pols.

white: average with no lightning removal
green: average with lightning removal
red : median with no lightning removal
blue : median with lightning removal.
--> when comparing the no removal with removal:

compare avgNoRemoval with avgAndRemoval: white with green
compare medNoRemoval with medAndRemoval: Red with Blue

bottom: modulation index (only for the no lightning removed data).

date	src	epsilon	rcvr	plots	jpegs	notes
190808	B1005+077	16.2	327	(.ps) (.pdf)	.jpg	lightning AT 327 removal improves spc
190819	B1025+154	8.8	lbw	(.ps) (.pdf)	.jpg	weak lightning at lbw. must be far away removal still shows residual problems
190819	B1040+123	12.1	327	(.ps) (.pdf)	.jpg	lightning at 327. strong removal still shows residual problems.
190903	B1005+077	10.0	327	(.ps) (.pdf)	.jpg	327. no lightning spectra with and without removal match.
190906	B1005+077	12.7	327	(.ps) (.pdf)	.jpg	327. very little lightning (hadn't started yet). spectra with and without match
	B1040+123	6.9	lbw	(.ps) (.pdf)	.jpg	some strong lightning at lbw removal improves average a lot, improves median a little.
	B1040+123	6.9	sbh	(.ps) (.pdf)	.jpg	sbh weak lightning. only 1 set. removal makes little difference sbh was done before lbw, so lightning may not have kicked in yet.
	B1107+110	5.0	lbw	(.ps) (.pdf)	.jpg	lbw some strong spikes removal improves average a lot, improves median a little.
	B1107+110	5.0	sbh	(.ps) (.pdf)	.jpg	sbh one set of spikes visible removal makes little difference This was taken after lbw, so the lightning was active.
	B1138+015	11.7	327	(.ps) (.pdf)	.jpg	327 lots strong lightning removal improves average and median.
190910	B1005+077	16.5	327	(.ps) (.pdf)	.jpg	lbw. only 1 small spikes removal makes no difference
190913	B1005+077	19.3	327	(.ps) (.pdf)	.jpg	lbw. only 1 small spike removal makes no difference

Summary:

Lightning removal

improves the average spectra a lot
improves the median spectra if lightning occurs often enough
does not noticeably affect the spectra when no lightning is found.
for weaker lightning that occurs a lot, it does not do a complete job of removing the lightning from the spectra

The rms threshold would have to be made low enough to affect the uncontaminated spectra.

We see lightning

at 327
at lbw
and a little at sbh.

I haven't decided whether it is ok to apply this to the normal ips processing.

We need to fit the spectra with and without removal and see whether we can get meaningful parameters after removal.

processing: x101/projects/ips/testlightning/lighnting.pro, x101/projects/ips/pro/ips_lightning.pro

home_~phil