Topics:

A brief introduction.
history/problems
The vertex software
Daily monitoring of the status
hydraulic brake on the dome and carriage house.

Monitoring info

Measurements:

A brief introduction.  (top)

1. The motors are controlled by the DCS . It is an analog control system.
2. The DCS is commanded by the Siemens cpu928 PLC controller. The PLC controller sits in a Siemens crate that also contains the LCU, PCU interface, and i/o boards. The PLC  is programmed with the STEP 5 programming language from Siemens.  Internally the PLC is a 186 computer. The software  runs on a 10 ms update cycle.
3. The PLC communicates with the LCU (a 486 computer running MS DOS. also called a Siemens cp581) via a shared memory that sits on the LCU. This interface is proprietary to the Siemens crate.
4. The LCU runs MS DOS 6.2. It talks to the PLC via the shared memory.  The LCU's task is to take external requests, process them, and then pass them on to the PLC. External requests can come from:
1. A keyboard / monitor connected to the LCU
2. The OCU (located in the control room). It  interfaces to the LCU via a serial port.
3. The  AO computers   interface to the LCU via a dedicated ethernet link.
The LCU contains a ISA bus that has an irig card, hard/floppy discs, and an ethernet card. Onboard the LCU is a 1 Mb silicon disc. Normal  operation of the LCU uses the silicon disc. The system can also be run from the hard disc (switch able from the prom monitor setup).

Daily monitoring of az,dome,ch status  (top)

• Azimuth daily summary plots (.pdf)
• Gregorian dome daily summary plots (.pdf)
• Carriage house daily summary plots (.pdf)

How the status bits respond to commands.  (top)

• The system started out under vme control with the motors on.
• The command sequence was:
• reset
• pwr on (even though it was already on)
• reset
• Program track 3 (dome and azimuth)
• stop
• These commands are the pntpwrdip sequence.
• Plot 1 General status. These commands did not change any of these bits. In particular, the stop command does not turn off drive power (the main contactors remain closed).
• Plot 2 Dome axis status. The only affect is the stop command. It affects:
• BrkRel: The brake release bit goes from released to brakes on.
• drvEnable: The amplifier drive enable bit goes from enabled to not enabled. This occurs because the plc drives the drive enable request to false once the brakes are on.
• Plot 3 Dome axis mode, equipment status. The only affect is that the system goes from program track mode to stop mode when the stop command is sent. The dome contactors do not open on a stop request.
• Plot 4 Dome amplifier status: The amplifier status does not change because of these commands. The amplifiers remain ready and there are no faults generated.

17jan13:az encoder 2 replaced  (top)

    On 17 jan 13 aeronomy was doing azimuth spins when encoder 2 failed. The encoder was replaced.

The plots show the azimuth encoder difference before and after the replacement (.ps) (.pdf)

• Page 1: 17jan13 azimuth encoder difference before and after the change:
• Top frame: encoder units
• bottom frame: degrees
• black is CW new encoder
• Red CCW new encoder
• green CW old encoder
• blue CCW old encoder
• The dome was at  za=15 and the ch was at za=0
• Page 2: 22jan13 az encoder difference with new encoder
• za ch=15, zadome=8.79

23jan12:az encoder 1 becomes loose. (top)

• The motion was:
• dome=10.976 degrees, ch at stow.
  
• 270 to 0 at .4 degrees/sec
• 0 ->360 at .4 degrees/sec
• 360->270 at .4 deg/sec
• Page 1:The black trace is the clockwise motion. The red trace is the counter clockwise spin.
• Top: azEnc1 - azEnc2 in encoder units.
• Bottom: azenc1-azEnc2 in degrees. The radius is about 60 feet so one degree is about 1 foot at the encoder radius.
  
• Page 2: compare az encoder dif with 19jan12 before coupling tightened.
• If the azenc1 moves and everything else remains constant, then the encoder difference will also change. To check this, an az swing during the world day (19jan12) was used. On 19jan12 the come was at za=15 degrees (ch at stow) so it was not exactly the same as 23jan12.
• Top: Az difference clockwise spin comparison.
• black 23jan11, green: 19jan11
• Bottom: az difference ccw spin comparison.
• red 23jan11, green 19jan11.
• Comparing the 2 dates, the az encoder diff has shifted about .01 to .02 degrees. This is most likely from the different za of the dome.
• Looking back on 25nov11 (see below) the dome za was actually moving during the measurement so it probably can't be used for the comparison.
  
• For reference:
• .01deg change at 60 feet is about .12 inches.. this is probably the accuracy we can position the pointer at 270.
• .01Deg az error on the rack gear is on the sky:
• 20deg: 12 arc seconds
• 10 deg: 6.2 arc seconds.
• Need to redo an az swing at dome=15 to compare with 19jan12.
• When ever we do the az swings to check the az dif we should use a fixed dome za (say 11) and a fixed ch za (say stow).
  

25nov11: az encoder 2 replaced. (top)

• The motion was:
• 270 to 0 at .4 degrees/sec. dome at 5 degrees, ch at stow
  
• 0 ->360 at .4 degrees/sec. dome at 5 till  az=110 then dome moved to 10 deg za (got there when az=240)
  
• The black trace is the clockwise motion. The red trace is the counter clockwise spin.
• Top: azEnc1 - azEnc2 in encoder units.
• Bottom: azenc1-azEnc2 in degrees. The radius is about 60 feet so one degree is about 1 foot at the encoder radius.
  
• The az encoder difference for the swings has a negative offset of about 140 Encoder counts.
• To center the difference, azenc2 should have it's value decreased by 140 counts.
• This is probably not critical since i think the az bending limit is set to a value larger the 400 encoder counts.
  
• The PI control loop uses the azEnc1 encoder to control the motion.
• The plots show that the azEnc2 (on the opposite side)  leads the azEnc1 a little (probably some offset in voltages sent to these motors).

12apr09: mot52 fail at az 326 11/12apr09  (top)

• The failure occurred near the same time on both days.
•  This must be because carl was tracking the same source.
• The dome was moving down from 19 degrees at .002 deg/sec and the azimuth was increasing at .01 deg/sec.
  
• It was then commended to slew to a new position. The failure occurred 5 seconds into the slew.
• To replicate the failure:
• pnt tr 325 18.92 -cx
• wait for the telescope to get there.
• pnt tr 327 17 -cx
• this should move it thru az 326 at slew like the failures.

07mar06 azimuth encoder 2 jumps  (top)

• Top: the azimuth position (encoder 1) versus hour of day.
• Bottom: The azimuth encoder difference (azGr - azCh) in degrees.

    On 08mar06 the encoder 2 jumped again. It was replaced by a new encoder on 09mar06

---

20jun02: dome motor amp glitching when changing direction.  (top)

Fig 1 top. Torques versus hour of day. The zenith angle is plotted in black at the top. The green spikes are the torques from motor 21.

Fig 1 bottom. This is a blowup of the top plots. The sampling is once per second so the glitch is on the order of  1 second.

Fig 2. The top is torque versus velocity (degrees/second), middle is torque versus acceleration (deg/sec^2), and the bottom plot is torque versus za. The green crosses are motor 21. The glitches only occur at positive acceleration.

---

11nov05 azimuth encoder failures. (top)

• 11nov05 (.ps): 16:00 enc2 fault. 16:20 enc1 fault. replaced encoder 1 with a spare encoder. Used the az index to align the azimuth arm.
• 13nov05 (.ps): 11:07 encoder 1 fault. Kept running
• 14nov05 (.ps): early morning . encoder 1 (the one that was installed on friday) failed. Moved enc2 -> enc1. put bad encoder on encoder 2.
• 16nov05: replaced encoder 2 with a new encoder.

---

12aug04: dome vibrations while moving at slew rate at low za.  (top)

• Fig 1. The top two plot are the azimuth position (black) and azimuth velocity (red). For the azimuth .4 degrees/second is the slew rate. The bottom two plots are the dome position (black) and the dome velocity(red). The dome slew rate is .04 degrees/second.
• Fig 2. The top plot are the azimuth motor torques for the 8 motors. The red graph at the top is the azimuth velocity. The solid black line is a scaled version of the azimuth position. The bottom plot is the torques for the 8 dome motors. The solid black line is the dome zenith angle while the red lines show the dome velocity.

24aug01: az servo failures. Motor torques build up while stopped. (top)

    There have been a number of servo failures with the azimuth amplifiers recently. The error message is (amp xx not ready). When the amplifier leds are inspected, an over speed condition is present. The problem occurred at more than 1 az, za position but it was repeatable whenever we used az=334.6 and za=2. The sequence that caused the problem was:
1. moving in increasing azimuth (at say .1 deg/sec) close to 334.6, dome at low za so most of the weight is on the wheels opposite the dome side.
2. user switches receivers. the trkinit routine is called and it does:
• pnt x vme 3. This command transfers control to the vme system (even though it already had it). A side affect is that the motors go to stop mode.
• A second or two after this command is executed, a  pnt mode tr 3 command is sent. This requests the az and gr  to move to program track mode.

 
 People were placed near each azimuth wheel when this sequence was executed. We were thinking that maybe the wheels were spinning in this area. They reported that:
• The wheels would slow down and they could hear the brake come on.
• While the brake was on, they could hear a noise in the motor. It sounded like the motor was fighting the brake.
• After about 5 seconds there was a large bang and then the wheel would spin fast and then stop.

 
The Figures show the motor torques and i/o bits during this sequence. The x-axis is the time in seconds from midnight. The data is sampled once a second so transitions can be occurring that are not being plotted.
1. Figure 1 shows the 8 motor torques versus time (about 11 seconds). The torques are offset for plotting. They take about 4 seconds to rise to a maximum value of 20 amps. This is happening with the brakes on. On the fifth second they plunge to zero.
2. Figure 2 shows some of the i/o bits during this time period. The most interesting bits are:
• IvelAz. This controls the PI error integration. It remains on for the entire sequence. Bit 4 of digital output byte 10 (sum_n controller) also stays on so the output of the PI loop is being sent to the motors.
• BrkG48Rel,BrkG15Rel show that the brakes were on at 60982 seconds.
• horn. This went off at 60981 and back on at 60982. This implies that a new request to move was being activated.
• The green line with * over plots the motor torques for az motor 11 during this period.
3. Figure 3 shows the transition from azimuth moving at .1 degrees/sec to stop without a restart request. From bottom to top the bits plotted are: drvsEnabled, brksReleased, then output bytes: Q10,Q8,Q9. These output bytes control the relays that cause different portions of the velocity request to be included in the value sent to the amplifiers. The average velocity is also over plotted in green (*). [note: it bothered me the first time i saw the drvenable going low before the brkrelease had gone low (the drives should never be disabled without the brakes on). After searching the code it looked like the drvenable bit high implied that the drives had finished the drive up sequence (or they had not yet started the drive down sequence)].

 
The time sequence for the failure (figs 1,2) was:
• 60978 - The transfer control command was probably sent. This started the slowdown/stop sequence.
• 60981 - request brakes M11,M12 to be on. The horn was turned off here (this is on when in motion). The DCS az 0 speed signal has become true.
• 60982 - request brakes M51,52,41,42,81,82 to be on. The read back shows that all brakes are now on. The program track request was probably sent here too since the horn has been re-activated (this occurs pending new motions). The torques begin to rise at this point.
• 60986 - the torques reach a maximum of about 20 amps.
• 60987 - a servo failure occurs (over speed). The horn is turned off.
• 60988 - the DCS az I portion of PI loop is turned off.
• 
  
The likely sequence of events was then:
• The azimuth was asked to stop so it started slowing down.
• motors m11,m12 stopped first. The other motors stopped 1 second later. In between this time a  new request to move arrived. The differential stop times may have been because some of the wheels were slipping.
• The brakes are on and dcs spd0 is true. This disconnects the input velocity request in dcs brd 4a1 via relay K7.
• There must have been a non-zero difference between the grounded velocity request and the actual speed value read back from the motors. This value integrated up for the 5 seconds that it took to startup.  With the brakes on, this caused the noise and vibrations that the people heard.  For some reason, the velocity error integrator was not turned off during the 5 seconds.
• When the startup time had completed, the brakes were released. The 20 amps of torque now had 0 load. It caused the loud noise and the motor wheels to spin rapidly. The rapid spinning made the amplifiers go over speed and shut the system down.


The az,gr,ch information is recorded once a second.  I searched the files from april through august and found more occurrences of this problem: 2 in april, 5 in august, 11 in june, 8 in july, 5 in august. I never saw it occur in the gr or ch. This is why  i think that a wheel slippage may be related to this.

I've changed the trkinit code so that it doesn't try to take control of the axis if it already has it. This will cut down on the number of times that this condition should occur. It should still be fixed since users can request the azimuth to stop and start without going through the vme system (via the lcu or ocu).

This problem will stress the motors/amplifiers (removing the load so quickly). The brakes should also be inspected. Jose maldonado pointed out that wheels spinning on the track will remove the grease and could cause damage to the rail.

01sep01 update. Motor/amp 12 on the azimuth had an output voltage of 300 millivolts when the azimuth was not moving. All of the other motors/amps were under 10 millivolts. This value was averaged with the other 7 motors and caused the velocity error to be non zero and integrate up while the brakes were on. Fixing this will decrease the chances of this happening again. The code should still be changed so that the integrator is disabled whenever the brakes are on (at least for the azimuth).

---

History/Problems (top)

• 17jan13: az encoder 2 bearing froze up. replaced encoder. enc dif is a little off but probably ok.
  
• 18jun12: azStpCC, servoFail error
• These 2 error occurred together. No other errors were visible. It was caused by the 120 volt breaker (behind the computer monitor rack) tripping. This breaker gives the 120 the motor  amps and to the cable car az limit switch.  The power is pd2208  See pd.22 in schematics .
  
• 26jan12: az encoder 1 replaced.
  
• 25nov11: encoder 2 failed.. replaced.
  
• 23oct11: power dip, amp 81 not ready. lcu parmeter menu encoder offset has 281 deg. Reset to 0.
  
• 09dec08: klaus comes for drive tuneup.
  
• 11jun08: lcu azimuth encoder offset went 1 degree -> 0.
  
• 21feb08: lcu azimuth encoder offset went from 0 to 1 degree. (returns to 0 11jun08)
  
• 05feb08: reset pots on az motor amplifiers to standard position. The cmdScale, current limit of some were not set correctly.
  
• 10aug07: azbending error 9:06am. corner 8 inside painting containment. Grit got into mount spring mechanism that pushes encoder into the rack gear. encoder had come away from rack gear causing bending problem. Manually reset encoder 2 position to same encoder difference as before the failure. cleaned encoder mount.
  
• 08apr07: azimuth encoder 1 fault.
• 04apr07:azimuth encoder 1 and 2 reset.
• 13feb07:amplifier 21/22 goes offline on dome. Amplifier 21 and later 22 of the dome have been going offline when starting to move downhill at high za. This has occurred on 13,15, and 16 feb07.
• 14nov06: az encoder1 fails. replaced. Had to reposition az using pointer.
• 14oct06: encoder board port 1 (az enc1 ) fails. replaced board 18oct06.
• ??sep06: az encoder 1 failed and was replaced (not sure about date).
• 16aug06: aux encoder replaced with newer version.
• 06apr06: ch brakes changed.
• 05apr06: greg brakes m11,m12 replaced.
• 04apr06: greg brakes m42,m41,m32,m31,m22,m21 replaced.
• 10oct05: telescope only moves to smaller az when put in tracking mode: The azimuth 2 encoder was changed in the morning (because it was reporting encoder failure). In the afternoon when the telescope was put in tracking mode, the telescope would only moved toward azimuth of zero. Preset and rate mode worked ok. This problem has occurred before (but i don't have any notes on it). During the morning maintenance, the plc was powered off then back on. After the problem jorge tried a reset and then an overall reset of the plc. The problem continued. He then tried replacing the plc. The azimuth started working again.
• 26jan05 cam limit switch dome: found that the cam limit switch for gregorian is bad. May have a loose tooth or something inside. This generates the hardware version of the pre and final limits (there is also a software version that comes from the encoder). The lower limit was creeping up from 1.06 -> 2.5 -> 3.6 degrees over the last week or so. Readjusted the upper limit to be 19.5 degrees. Should order a new unit.
• 19jan05 vertex shelter overheats: air conditioners vertex shelter tripped off. Trouble with pointing. Only moves in negative az in tracking mode. Turned out that the plc wasn't working (replaced). Other problem was that the negative direction seemed to come from a 1 degree az offset that installed. This comes from f8 params menu in ocu. Gets stored in the file params.dat.. Looking back at old data , this offset has been there since last jun04. With it in now, it is causing the tracking to fail. May be that the old version of the software was ignoring this??. Reset to 0 via f8 ocu. Every time lcu rebooted need to do this. Looks like the new value is not getting set in the file params.dat (we are running on the silicon disc...). Need to figure this out.

<- page up
 home_~phil