On 08mar06 the encoder 2 jumped again. It was
replaced
by a new encoder on 09mar06
There have been a number of servo failures with
the azimuth amplifiers recently. The error message is (amp xx not
ready).
When the amplifier leds are inspected, an over speed condition is
present.
The problem occurred at more than 1 az, za position but it was
repeatable
whenever we used az=334.6 and za=2. The sequence that caused the
problem
was:
- moving in increasing azimuth (at say .1 deg/sec) close to
334.6, dome
at
low za so most of the weight is on the wheels opposite the dome side.
- user switches receivers. the trkinit routine is called and it
does:
- pnt x vme 3. This command transfers control to the vme system
(even
though
it already had it). A side affect is that the motors go to stop mode.
- A second or two after this command is executed, a pnt
mode tr 3
command
is sent. This requests the az and gr to move to program track
mode.
People were placed near each azimuth wheel when this sequence was
executed. We were thinking that maybe the wheels were spinning in this
area. They reported that:
- The wheels would slow down and they could hear the brake come
on.
- While the brake was on, they could hear a noise in the motor.
It
sounded
like the motor was fighting the brake.
- After about 5 seconds there was a large bang and then the wheel
would
spin
fast and then stop.
The Figures
show the motor torques and i/o bits during this sequence. The x-axis is
the time in seconds from midnight. The data is sampled once a second so
transitions can be occurring that are not being plotted.
- Figure 1 shows the 8 motor torques versus time (about 11
seconds). The
torques are offset for plotting. They take about 4 seconds to rise to a
maximum value of 20 amps. This is happening with the brakes on. On the
fifth second they plunge to zero.
- Figure 2 shows some of the i/o bits during this time period.
The most
interesting
bits are:
- IvelAz. This controls the PI error integration. It remains on
for the
entire
sequence. Bit 4 of digital output byte 10 (sum_n controller) also stays
on so the output of the PI loop is being sent to the motors.
- BrkG48Rel,BrkG15Rel show that the brakes were on at 60982
seconds.
- horn. This went off at 60981 and back on at 60982. This
implies that a
new request to move was being activated.
- The green line with * over plots the motor torques for az
motor 11
during
this period.
- Figure 3 shows the transition from azimuth moving at .1
degrees/sec to
stop without a restart request. From bottom to top the bits plotted
are:
drvsEnabled, brksReleased, then output bytes: Q10,Q8,Q9. These output
bytes
control the relays that cause different portions of the velocity
request
to be included in the value sent to the amplifiers. The average
velocity
is also over plotted in green (*). [note: it bothered me the first time
i saw the drvenable going low before the brkrelease had gone low (the
drives
should never be disabled without the brakes on). After searching the
code
it looked like the drvenable bit high implied that the drives had
finished
the drive up sequence (or they had not yet started the drive down
sequence)].
The time sequence for the failure (figs 1,2) was:
- 60978 - The transfer control command was probably sent. This
started
the
slowdown/stop sequence.
- 60981 - request brakes M11,M12 to be on. The horn was turned
off here
(this
is on when in motion). The DCS az 0 speed signal has become true.
- 60982 - request brakes M51,52,41,42,81,82 to be on. The read
back shows
that all brakes are now on. The program track request was probably sent
here too since the horn has been re-activated (this occurs pending new
motions). The torques begin to rise at this point.
- 60986 - the torques reach a maximum of about 20 amps.
- 60987 - a servo failure occurs (over speed). The horn is turned
off.
- 60988 - the DCS az I portion of PI loop is turned off.
-
The likely sequence of events was then:
- The azimuth was asked to stop so it started slowing down.
- motors m11,m12 stopped first. The other motors stopped 1 second
later.
In between this time a new request to move arrived. The
differential
stop times may have been because some of the wheels were slipping.
- The brakes are on and dcs spd0 is true. This disconnects the
input
velocity
request in dcs brd 4a1 via relay K7.
- There must have been a non-zero difference between the grounded
velocity
request and the actual speed value read back from the motors. This
value
integrated up for the 5 seconds that it took to startup. With the
brakes on, this caused the noise and vibrations that the people
heard.
For some reason, the velocity error integrator was not turned off
during
the 5 seconds.
- When the startup time had completed, the brakes were released.
The 20
amps
of torque now had 0 load. It caused the loud noise and the motor wheels
to spin rapidly. The rapid spinning made the amplifiers go over speed
and
shut the system down.
The az,gr,ch information is recorded once a second. I searched
the files from april through august and found more occurrences of this
problem: 2 in april, 5 in august, 11 in june, 8 in july, 5 in august. I
never saw it occur in the gr or ch. This is why i think that a
wheel
slippage may be related to this.
I've changed the trkinit code so that it doesn't try to take
control
of the axis if it already has it. This will cut down on the number of
times
that this condition should occur. It should still be fixed since users
can request the azimuth to stop and start without going through the vme
system (via the lcu or ocu).
This problem will stress the motors/amplifiers (removing the load
so
quickly). The brakes should also be inspected. Jose maldonado pointed
out
that wheels spinning on the track will remove the grease and could
cause
damage to the rail.
01sep01 update. Motor/amp 12 on the azimuth had an output
voltage
of 300 millivolts when the azimuth was not moving. All of the other
motors/amps
were under 10 millivolts. This value was averaged with the other 7
motors
and caused the velocity error to be non zero and integrate up while the
brakes were on. Fixing this will decrease the chances of this happening
again. The code should still be changed so that the integrator is
disabled
whenever the brakes are on (at least for the azimuth).