Vertex drive systems (az,ch, gregorian).

june, 2001

Topics:

Monitoring info

Daily monitoring of az,dome, ch status.
Average motor torques by month.

Measurements:

10apr17: az enc1 faults, value jumps by > 1000 degree. need to realign it using pointer.
11oct16: testing when system returns to enc1 after enc1 fault.
10oct16: az encoder 1 fault for 1 day.
01oct16: az encoder 1 fails. is replaced
10jan16: az encoder 1 fails. is replaced.
22jun15: az encoder 1 fails.. is replaced.
26nov13: gregorian amp32 failure.
01jul13: fit gregorian motor torques (after hydraulic brake). Dome weight
19jun13: amp/mot22 failure
17jan13: az encoder 2 replaced.
23jan12: az encoder dif after encoder 1 coupling tightened.
30mar10: computing the unbalanced moments in the az arm
17mar10: dome za that balances az arm with ch at stow
07mar10: Dome weight
31jan10: azimuth motion failures.
08aug09: azimuth encoder1 failures 27jul09->04aug09
12apr09: az mot52 fail az=326 on 11/12apr09
07dec08: gregorian motor torques while galfacts scanning.
23jun08: on 11jun08 mot82 torques read non-zero when power off.
21feb08: az motor torques for group 1 and group 2.
05feb08: motor torques vs az after amp pots adjusted.
jan08: variation in az motor torques.
Dec07: az motor torques vs az position
11dec07: recommissioning the telescope after the painting project.
10dec07: az bending errors.
11apr07: azimuth encoder 1 fault.
09dec07: azimuth motor torques
08apr07: azimuth encoder 1 fault.
04apr07: Both azimuth encoders reset, dome local disable cable breaks.
13feb07: amplifiers 21,22 of dome going not ready
28nov06: pnt checks after az encoder failure. Az error and tiedown offsets.
14nov06: az1 encoder fails.
14oct06: az1 encoder failure from lightning (encoder board failure)
14mar06: azimuth encoder difference, rack gear error at az=160
07mar06: azimuth encoder 2 (ch side) jumps
27jan06: check the azimuth error after the nov05 encoder failure.
11nov05: azimuth encoders fail
12aug04: vibrations in dome when moving at slew rate at low za.
14oct03   : azimuth encoder jumps.
15sep02  : azimuth encoder jumped.
25jun02: gregorian torques moving 2<->20 deg za.
20jun02: dome motor amplifier glitching when changing direction.
24aug01: az servo failures. Motor torques build up while stopped.
jan00       : azimuth encoder rack gear error.

A brief introduction.  (top)

     The vertex drive systems control the azimuth, gregorian, and carriage house.  The control hierarchy is:
  1. The motors are controlled by the DCS . It is an analog control system.
  2. The DCS is commanded by the Siemens cpu928 PLC controller. The PLC controller sits in a Siemens crate that also contains the LCU, PCU interface, and i/o boards. The PLC  is programmed with the STEP 5 programming language from Siemens.  Internally the PLC is a 186 computer. The software  runs on a 10 ms update cycle.
  3. The PLC communicates with the LCU (a 486 computer running MS DOS. also called a Siemens cp581) via a shared memory that sits on the LCU. This interface is proprietary to the Siemens crate.
  4. The LCU runs MS DOS 6.2. It talks to the PLC via the shared memory.  The LCU's task is to take external requests, process them, and then pass them on to the PLC. External requests can come from:
    1. A keyboard / monitor connected to the LCU
    2. The OCU (located in the control room). It  interfaces to the LCU via a serial port.
    3. The  AO computers   interface to the LCU via a dedicated ethernet link.
    The LCU contains a ISA bus that has an irig card, hard/floppy discs, and an ethernet card. Onboard the LCU is a 1 Mb silicon disc. Normal  operation of the LCU uses the silicon disc. The system can also be run from the hard disc (switch able from the prom monitor setup).

Vertex manuals

    The table below contains the most recent electronic version of the manuals from klaus (vertex). I've included the original .doc file as well as an .odt version (created by libre office). I need to check if the paper versions have more recent updates.
The original location of the manuals was /share/megs/vertex

volume
manual
section
.doc
rev,date
1




2
operation and maintenance manual
control system
1
Operation,Maintenance, and troubleshooting
om5034.odt  om5034.doc
2.2
sept1997
2
LAN interface description
(i have a corrupted version of this..LAN5034.TXT
2.1
1997
3
programmable controller siemens simatic s5-135u
Description of Internal interfaces
if5034.odt    if5034.doc
2.3
dec1999
4
modifications and adjustments
adj530.odt   adj530.doc
2.3
dec1999
3
installation manual
control system

drawing list
2.1
sept1997

single line diagram and control circuit diagram
(the electrical schematics)

list of parts

interconnection diagram

cable list

cable diagram

location diagram
4
maintenance manual
control system
1
Vendor publications
md5034.odt  md5034.doc
2.2
sept1997
     
misc
PLC Simatic S5-135U
list of plc routines

progl034.odt   progl034.doc
dec1999

dcs device hardware interface pinouts

 dcs_pinouts.doc
sept 1995




Daily monitoring of az,dome,ch status  (top)

    Each morning  summary plots of the previous days status are made (via a cronjob on megs2).
processing: x101/agc/mon/agcchkdaily.sc


How the status bits respond to commands.  (top)

    The plots show how the vertex status bits respond to various commands (.ps) (.pdf):
processing: x101/070219/chkpnt.pro

10apr17 - azimuth encoder 1 fails,jumps by > 1000 deg.

    On 10apr 17 the following sequence occurred:

The plots show what happened (.ps) (.pdf):

Summary:

processing: x101/170411/azencerr.pro


11oct16:when system returns to enc1 after enc1 fault

    We did some tests to see when the azimuth drive system returns to enc1 after an enc1 fault. The normal sequence is:
What we did:
The result is that after an encoder 1 failure, the system continues to use encoder 2 until the operator resets the encoder1 fault:
processing:x101/161011/chkenc1_2.pro


10oct16: az 1 encoder fault (top)

    Azimuth encoder 1 had encoder faults on:

    When encoder 1 fails, the system switches to encoder 2 for the  pointing. The pointing model is made with azimuth encoder 1. The difference encoder 2 - encoder 1 changes as the azimuth goes 0 to 360 degrees. The maximum error is:

    We also record the difference between the 2 encoders (for the azimuth bending). If encoder 1 fails, then the difference is fixed  (since the value of encoder 1 is not changing).

    During the azencoder 1 fault,  the az encoder difference was still being recorded. So the encoder 1 faults looks like it was a glitch. The error message gets latched until the operator resets it (so we can see that the error occurred).

    On 11oct16 we verified that after an encoder 1 fault, the system continues to use encoder 2 until the fault is reset. So we had degraded pointing 14:54 10oct16 till 07:00 11oct16.

The plots show the pointing error from using az encoder 2 (.ps) (.pdf)

processing: x101/161011/en1flterr.pro


01oct16: az 1 encoder failure. (top)

    Azimuth 1 encoder failed at 08:16:31 on 01oct16.
Willie replaced the encoder  and then used the pointer at 270 degrees to align the encoder. The steps were:

    After replacing the encoder and moving it to the stripe, willie took some pictures of the alignment:

    We didn't have a ruler against the line so not sure exactly how far it is off. The previous pictures showed that the scribe mark is about 1 mm wide.

    Position error.

    Down in the lab we opened the encoder. The glass disc with the  position info was broken.
This was the encoder that was installed 10jan16.

processing: x101/160910/encfail.pro
   

22jun15: az 1 encoder  failure (top)

    Az encoder 1 failed atround 18:00 22jun15. The telescope continued to track (since it  switches to use enc2 for the pointing).

    azimuth status 22jun15 (.ps) (.pdf)

    azimuth status (blowup when motion stopped) (.ps) (.pdf)

    gregorian dome status during failure (.ps) (.pdf)


On  23jun15 around 10:00 osvaldo replaced the encoder1. It was pretty dirty... things then started to work.

Aligning new encoder.

processing: x101/150623/azencflt.pro

17jan13:az encoder 2 replaced  (top)

    On 17 jan 13 aeronomy was doing azimuth spins when encoder 2 failed. The encoder was replaced.

The plots show the azimuth encoder difference before and after the replacement (.ps) (.pdf)

processing: x101/130124/chkazenc.pro


23jan12:az encoder 1 becomes loose. (top)

    There was an az encoder 1 fault (the encoder used for pointing) on 21jan12 during a world day. We continued running for 2 days using encoder 2. On 23jan12 the coupling holding encoder 1 was found to be loose. The coupling
was tightened. The azimuth position at az=270 was not reset using the pointer on the azimuth arm.
    On 23jan12 around 11am Az swings were then done to measure the encoder difference.
the plots show the encoder difference for the swings (.ps) (.pdf):
processing: x101/120124/chkazenc.pro

25nov11: az encoder 2 replaced. (top)

    On 25nov11 the az2 encoder (on the carriage house side) failed and was replaced. The azEnc1 is the one used for pointing. The encoder value for azEnc2 was set with the azimuth at 270. After replacement, azimuth swings were done to check the azimuth encoder difference (azEnc1 - azEnc2).

The plots show the azimuth encoder difference for the azimuth swings after the azEnc2 replacement (.ps) (.pdf):
processing: x101/111125/azenc2.pro

12apr09: mot52 fail at az 326 11/12apr09  (top)

    The was a servo failure: motor 52 not ready on 11apr09 and 12apr09. Both of these occurred at an azimuth of 326 degrees. carl heiles was observing on both days.
plots of 11apr09 status (.ps) (.pdf)
plots of 12apr09 status (.ps) (.pdf)
processing: x101/090412
   

07mar06 azimuth encoder 2 jumps  (top)

  On 07mar06 azimuth encoder 2 (carriage house side) jumped around 21.1 arcseconds.
    The plots show the az encoder values (.ps)  (.pdf):     The azimuth arm had been sitting at az=359.650 for about 18 minutes when the az2 encoder decided to jump. So the problem was not because the gear jumped a tooth.

    On 08mar06 the encoder 2 jumped again. It was replaced by a new encoder on 09mar06

processing: x101/060307/agc.pro


20jun02: dome motor amp glitching when changing direction.  (top)

    The torque on motor 22 was glitching for about  1 second when the dome slowed down or speeded up. The amplifier was moved to motor 21 and the problem moved to motor 21 (so it was the amplifier). This amplifier was replaced and the glitching continued on motor 21. When the spare amplifier was replaced with yet another amplifier, the glitching went away (so both amps had the problem). The plots show the torques while the dome was moved between 13 and 15 degrees za at .02 deg/sec (half slew rate). The hydraulic brakes were not connected during this test. The motor numbering is from uphill to downhill 11,12,21,22,31,32,41,42.
  • Fig 1 top. Torques versus hour of day. The zenith angle is plotted in black at the top. The green spikes are the torques from motor 21.
  • Fig 1 bottom. This is a blowup of the top plots. The sampling is once per second so the glitch is on the order of  1 second.
  • Fig 2. The top is torque versus velocity (degrees/second), middle is torque versus acceleration (deg/sec^2), and the bottom plot is torque versus za. The green crosses are motor 21. The glitches only occur at positive acceleration.
  • processing: x101/agc/ampglitch20jun02.pro


    11nov05 azimuth encoder failures. (top)

        The azimuth encoders started to fail on 11nov05. The links show the daily summary of the azimuth for. The timeline was:


    12aug04: dome vibrations while moving at slew rate at low za.  (top)

        At 11:45 am on 12aug04 some tests were being conducted with the dome, azimuth while the azimuth was at a low zenith angle. Two member of the electronics department who were in the dome at the time reported large vibrations/shaking of the dome structure while this movement was taking place.
        I looked at the az/za positions, velocities, torques that are recorded once a second. The plots show the actual motion of the dome and azimuth (not the requested motion) during the shaking period. After 46 minutes (after 11 am) the motion of the dome was at a low zenith angle (about 2 degrees za). The velocities that were being requested were plus/minus slew rate for both axis (probably because they were moving large distances).
    The torques for the dome are large (20 ft-lbs per motor) when moving downhill at low zenith angle. This is caused by the hydraulic braking system installed on the dome. When moving downhill the fluid creates a resistance that is proportional to the velocity squared. At normal tracking positions (above say 6 degrees za) the fluid resistance is reacting against the component of gravity that is pushing the dome downhill. At 2 degrees za, there is not gravitational component to counteract.  I wonder if the shaking that was felt is an interaction between the PI loop and the hydraulic brakes that is not normally seen with gravity present. We need to check this out further.
    processing: x101/040812/doit.pro


    24aug01: az servo failures. Motor torques build up while stopped. (top)

          There have been a number of servo failures with the azimuth amplifiers recently. The error message is (amp xx not ready). When the amplifier leds are inspected, an over speed condition is present. The problem occurred at more than 1 az, za position but it was repeatable whenever we used az=334.6 and za=2. The sequence that caused the problem was:
      1. moving in increasing azimuth (at say .1 deg/sec) close to 334.6, dome at low za so most of the weight is on the wheels opposite the dome side.
      2. user switches receivers. the trkinit routine is called and it does:
       People were placed near each azimuth wheel when this sequence was executed. We were thinking that maybe the wheels were spinning in this area. They reported that: The Figures show the motor torques and i/o bits during this sequence. The x-axis is the time in seconds from midnight. The data is sampled once a second so transitions can be occurring that are not being plotted.
      1. Figure 1 shows the 8 motor torques versus time (about 11 seconds). The torques are offset for plotting. They take about 4 seconds to rise to a maximum value of 20 amps. This is happening with the brakes on. On the fifth second they plunge to zero.
      2. Figure 2 shows some of the i/o bits during this time period. The most interesting bits are:
      3. Figure 3 shows the transition from azimuth moving at .1 degrees/sec to stop without a restart request. From bottom to top the bits plotted are: drvsEnabled, brksReleased, then output bytes: Q10,Q8,Q9. These output bytes control the relays that cause different portions of the velocity request to be included in the value sent to the amplifiers. The average velocity is also over plotted in green (*). [note: it bothered me the first time i saw the drvenable going low before the brkrelease had gone low (the drives should never be disabled without the brakes on). After searching the code it looked like the drvenable bit high implied that the drives had finished the drive up sequence (or they had not yet started the drive down sequence)].

      4.  
      The time sequence for the failure (figs 1,2) was: The likely sequence of events was then:


      The az,gr,ch information is recorded once a second.  I searched the files from april through august and found more occurrences of this problem: 2 in april, 5 in august, 11 in june, 8 in july, 5 in august. I never saw it occur in the gr or ch. This is why  i think that a wheel slippage may be related to this.

      I've changed the trkinit code so that it doesn't try to take control of the axis if it already has it. This will cut down on the number of times that this condition should occur. It should still be fixed since users can request the azimuth to stop and start without going through the vme system (via the lcu or ocu).

      This problem will stress the motors/amplifiers (removing the load so quickly). The brakes should also be inspected. Jose maldonado pointed out that wheels spinning on the track will remove the grease and could cause damage to the rail.

      01sep01 update. Motor/amp 12 on the azimuth had an output voltage of 300 millivolts when the azimuth was not moving. All of the other motors/amps were under 10 millivolts. This value was averaged with the other 7 motors and caused the velocity error to be non zero and integrate up while the brakes were on. Fixing this will decrease the chances of this happening again. The code should still be changed so that the integrator is disabled whenever the brakes are on (at least for the azimuth).


    History/Problems (top)


    <- page up
     home_~phil