azimuth enc1 failure during lightning
The encoder failure
Pointing errors while using azimuth2
Pointing check after everything
encoder failed and the pointing errors from using az2 encoder (.ps)
errors before/after the encoder board failure(.ps) (.pdf)
The Failure. (top)
On 14oct06 at 15:56:03 (ast) a lightning strike caused
azimuth encoder 1 to fail. This is the azimuth encoder that is used to
point the telescope. When the failure occurred, the software automatically
switched to the az2 encoder (on the carriage house side). This caused a
degradation in pointing (see point errors).
We ran pointing with az2 encoder until 18oct06 at 16:00 AST.
Debugging the azimuth 1 encoder failure:
Since port one looks like it is getting zapped by lightning
We saw "az1 encoder failure" as a yellow (non fatal) fault on the
ocu display. This error signals that encoder 1 has failed and the software
has switched to the az2 encoder (on carriage house side) for pointing.
On the ocu the az1 encoder reading would be correct for awhile and then
jump by 100's of degrees.
We replaced the az1 encoder with a spare. The problem persisted.
We replaced the encoder interface board. The problem persisted. The interface
board has 4 ports. Each port reads one of the 4 encoders:
Port 1: az1 encoder. Used for pointing the azimuth. (top position of board).
Port 2: az2 encoder. Not used for pointing. It's main purpose is to measure
the azimuth arm bending.
Port 3: gregorian encoder.
Port 4: the carriage house encoder.
We plugged az1 encoder into encoder interface port 2. Az1 encoder began
to read ok (through port 2). So the az1 encoder was ok.
We moved az2 encoder from port 2 to port 1. When it was plugged into port1,
az2 encoder failed. So the problem was the encoder interface port 1.
We had the same problem with port 1 on both encoder interface boards (it
took us awhile to realize that both boards were bad).
We looked at the clock and data lines for encoder board port 1 on
the breakout connector behind the lcu computer. The clock lines were fine
(from interface board to encoder). There were no data line signals (encoder
to interface board).
When we looked at encoder interface port 2, the clock and data lines were
fine. This was true for all encoders plugged into port 2.
The encoder documentation (vertex 3rd party vendor documentation) stated
that the encoder was short circuit and reverse polarity protected. The
data lines disappearing meant that encoder interface port 1 has a
short on the data lines. This has to be at the board input or in the sn75175
differential line receiver chip. The failure must be on both encoder boards.
The software switched to encoder port 2 for azimuth pointing on 14oct06
15:56:03. At 1:00 am 19oct06 we plugged az1 encoder into port 2 of the
interface board. This got rid of the pointing errors caused by using the
az2 encoder. The software was still using port 2 of the encoder board,
but we now had the correct "az1 encoder)" plugged into it.
We did replace az1 encoder in the debugging process. This lost the az encoder
position. We used the index at 270 degrees to realign the new encoder.
After doing this we needed to check the azimuth pointing (since this alignment
is hard to do down to the arc second level).
We looked at the spare encoder interface board in the lab. The sn75175
differential receiver chip for port 1 had bad data line inputs. The resistance
between vcc,ground to the data line inputs was not the same as the other
We ordered new chips and socketed the line receiver chip.
On 26oct06 (around noon) we installed the spare encoder board
(with the new line receiver chip). The system worked correctly. Az1 encoder
was reinstalled in encoder board port 1. The encoder board that had been
running also had a bad line receiver chip (which is being fixed).
On 27oct06 we did a pointing check
to verify that the pointing was ok.
What probably happened.
Port 1 on both interface boards were bad.
The cable to az1 encoder is always plugged into port 1.
The lightning is probably coming down the az1 encoder cable and blowing
out the receiver chip. The cable to the az2 encoder is a lot shorter (right
next to the vertex shelter). The gr and ch encoder cables are real long
so length may not be the problem. It may be that the shield for the az1
encoder cable is no longer well grounded at the vertex shelter.
The spare encoder board was probably blown out and replaced some time in
the past. In hind sight, the 11nov05
encoder failures was probably when the spare encoder board port
1 was damaged. My guess is that there was an encoder and an encoder board
failure. When they installed a new encoder board, the problem did not go
away. When they then installed a new encoder the problem was solved. This
may have led them to think that only the encoder was bad.
What we should do:
Check the shield on az1 encoder. Make sure it has a good ground to the
Have a preventative maintenance that looks at the data lines on all 4 encoder
boards. We might see a degradation in time if smaller strikes are affecting
the receiver chips.
At 15:56:03 the az1 encoder failed and the plc
software switched to use the az2 encoder for pointing the azimuth. This
encoder is on the carriage house side of the azimuth arm. This switch caused
a pointing error:
Pointing errors caused
by running on az2 encoder: (top)
The plot shows when the az1
encoder failed and the pointing errors from using az2 encoder (.ps)
The az1 encoder is normally used to point the azimuth. It is the encoder
that is used when the pointing models are made.
The difference between the az1 and az2 encoders is not a constant 180 degrees.
As we drive in one direction or another, the azimuth arm bends causing
the encoder difference to vary.
The encoder rack gear is also not perfectly round (probably not even close
to perfect .. see az
encoder rack gear runout). The encoders are measuring arc length rather
than angle. So if the rack gear is not round, the two encoder will run
at a different rate causing their angular difference to vary.
From 14oct06 16:00(AST) thru 18oct06 16:00 AST we ran
with the azimuth pointing error shown at the bottom of page two.
Page1: Azimuth position on day of failure. This uses az1 encoder.
The position started jumping around 16:00 AST. The bottom plot is
a blowup in time when the failure occurred. The encoder read the
correct value for a few seconds and then jumps away.
Page2: The az1-az2 encoder difference for 01oct06 thru 14oct06.
Top: az1-az2 encoder versus azimuth in encoder units (.52 asecs per
encoder unit). The two tracks in azimuth correspond the clockwise and counter
clockwise motion. The difference comes from the bending of the azimuth
Middle: (az1-az2)*sin(za). This is the error on the sky. The units are
Bottom:(az1-az2)*sin(za) versus za. This shows the sin(za) dependence.
At za=20 the pointing error on the sky can differ from -60 to +10
The encoder board failed on 14oct06. We
ran using az2 encoder until 16:00 Ast 18oct06 when we plugged az1 encoder
into encoder port 2 (that was being used for pointing). We reinstalled
the repaired encoder board on 26oct06. The az1 encoder was replaced a few
times during this period. When we replaced the az1 encoder, we had
to reset the az encoder pointing using the mechanical index. This has a
pointer (mounted next to the encoder) that slides out and aligns with a
mark at azimuth 270.
Pointing check after
reinstalling encoder board. (top)
The encoder is about 60 feet from
the center bearing. A 1 arcsecond error at 60 feet is about .003
inches along the arc. Including the sin(za) term, a 1 arcsecond error on
the sky is about .01 inches at za=20 degrees and .02 inches at za=10deg.
So the alignment of the pointer with the mark is non-trivial.
To verify the alignment, data was taken
on 27oct06. These were the same sources taken on 14oct06 prior to the failure.
The plots show the pointing
errors before/after the encoder board failure(.ps) (.pdf):
The azimuth error before and after is
about the same. The za error varies by about 10 arcseconds for the source
B1040+123. This source transited around 9:40 AM AST. There may have been
problems with the tiedowns going loose. It looks like the azimuth
position if fine.
The black * are taken 14oct06 prior to the failure. The red
* are taken 27oct06 after everything was replaced.
The top two plots show the za error versus za for the two
The bottom two plots show the azimuth error (great circle
) vs za for the two sources.