Crossing midnight at the end of the
The data taking uses irig from the trak clock to keep track
of time. The irig standard does not include the year with the
information that it distributes. To get around this, the datataking
stores the year information in the file:
This is an ascii file with the first line containing the current
year as a 4 digit number (eg. 2006).
This file is read by the pointing program and
the irigServ program when they are started. Both of these programs
are running on the vme pointing crate.
Procedure for crossing midnight:
The instructions for crossing midnight at the start of a new
Note: i changed the instructions on 01jul15
- . Instead of rebooting the pointing crate, we just stop the
pointing program and restart it.. this gets around a
race condition when the pnt clock is being syncronized and the
pointing program is asking for the utcToUt1 info..
- When is midnight:
- Leap seconds are transmitted 0 hr 01jan UTC. This is 20:00
hours at arecibo:AST.
- The clocks that we run at the AO (TRAK, old eeco, etc) are
running on AST time. They will perform operations (addition of
leap second, leap year) when they read 0 hrs (in our case AST
- The UTC to UT1 conversion
(/home/online/vw/etc/Pnt/utcToUt1.dat) uses a linear
approximation to the drift. The file contains many sets of
timestamped polynomials that get updated when the linear
approximation is large or a leap second is added. Pointing
searches for the most recent set of coefficients (the one
whose start time has passed and is closest to the current
time). The timestamp in the file is mjd but the comparison
uses AST time so that leap second additions will occur at 0hrs
AST (to be in sync with our clocks).
- So midnight in this document refers to 0hrs AST.
- There is a potential problem when leap seconds are
added. Some of the computers in the datataking are using ntp
(network time protocal). This will insert the leap
second at 0hrs UTC (20hrs ast).
- between 20hrs and 24:00 on the day of a leap second, the
start times in the headers will be without the leap second.
Pulsar observers need to take this into account.
- The pointing should be ok since it runs off of our irig
that switches with our clocks.
- Make sure that the file /home/online/vw/etc/initDat/year
has the new year in the first line of the file (eg. 2006).
- You can update this file before midnight since the pointing
program only reads it when it starts.
- I wouldn't worry about the pointing crate rebooting
prematurely from a power dip since it is on a UPS.
- Shut off the tiedowns using the tiedown disable switch.
- When you reboot the pnt crate (below), the tiedowns will
lose communications with the pnt crate. This causes the
tiedowns to release their tensions. When you disable the
tiedowns with the switch, they do not release (since you've
cut the power).
- Stop any data taking that is running.
- If the gui is running, it's probably best to exit it.
- You don't have to exit the other vxWorks (vw%)
- pre jul15 verion.. (no
- At the operators terminal in the window with the vw%
prompt enter: exit
- This window is where the operators enter the pointing
commands. It is running on the pnt crate that we are going
to reboot below.
- At the laser ranging pc (on the wall to the right of the
observer2 workstation) update the dos time using the date
- New instructions (after jun2015)
- from vx% prompt
- rsh pnt pntMProgStop
- rsh pnt i
- continue doing this till the programs
pntMProg pntXform are no longer in the list
- rsh pnt pntMProgStart
- pre jul15 instructions..
(no longer used) this has a race condition .. see
things to think about below..
- Go to the pointing
crate in the clock room and push the red reset button.
This will reboot the pointing (pnt) crate.
- It takes about 1 minute for
the crate to reboot.
- Wait for the pointing crate reboot to complete.
- You can monitor this by looking at the error message in
the lower part of the telescope display (next to the
operators and observers windows). While the crate is
rebooting, there will be an error message saying:
agcProg Error or scramNet error. When this error
goes away, the reboot is complete.
- Re-enable the tiedowns by turning the tiedown disable switch
back to enabled.
- Restart the operators vxWorks window.
- At the operators terminal (in the same window you typed
exit in above) enter: pointing This will
restart the operators vw% program.
- In the operators vw% window enter:
- pnt td reset
- pnt td time 0
- pnt tur time 0
- pnt reset 15
- If you want the tiedowns to track the platform height then
- Make sure that the other vxWorks windows are connected to the
pointing crate after the reboot.
- In the receiver room on the cpu by the oscilloscopes enter
connectall in the window with the vw%
- If the correlator vw% window is also in use, enter connectall
in its window.
- If you want to restart the gui, go ahead.
Things to think about:
Path to this page: ~phil-->datatking software --> crossing
midnight at the end of the year.
- It is possible to go through this sequence without
rebooting the pointing crate but you would have to also worry
about some other things. The tiedowns and turret have their own
little star cpus. These computers receive time stamped requests
to move to different positions. They keep track of time by
counting 1 second ticks. When they start (and when the pnt xx
time 0 command is sent), they resync their time to that of the
pointing program. This is not really necessary during a midnight
crossing unless a leap second has been inserted. In that case
you would have to:
- Have the pnt program and irigServ program reread the year
- After the one second tick has been inserted, execute the
pnt td time 0 , pnt tur 0 commands to update the second count.
- If programs are taking data while this is occurring then
you need to worry about race conditions where datataking does
something in the middle of the above sequence.
- It's simpler just to reboot the pnt crate.
- You don't have to reboot the cor or da vme crates. Each crate
is counting 1 second ticks. Every 10 seconds they talk to the
irigServ program and ask the time for the current tick. When a
leap second occurs, all of the crates will jump to the current
time (within 10 seconds after midnight).
- 01jul15: found race condition in pntProg startup and
synchronizing with irig clock.
- i rebooted the pointing crate 30 seconds after midnite. We
had the new time, but the point program did not get the latest
utcInfo (offset,rate) from the utcToUt1.dat file. The next
morning i stopped, restarted pointing and it got the
- the routine is passed year,daynum,dayNum .. these come from
- the dayno,,secs come from the irig server via syncClock
(queries irig server every 10 secs). the problem is:
- in the startup script for the cpu we do:
- extTickInit (ext tick running, but dayno, sec
probably 70,1 (initial values) with year correct (comes
- irigServStart (on pnt computer)
- syncClStart(10) this start the synchronizing.. this is a
daemon that stays running. so we return immediately...
- more startup commands
- start point program.. if the syncClock has not yet
finished synchronizing with the irig, the call to
utcInpInfo(_) will have the correct year but daynum will
probably be 11mar (guess who's birthday that is...).
- This is a possible race condition..
- tfor leap seconds A solution would be to not reboot the
crate, but just restart the pointing program:
- rsh pnt pntMProgStop
- rsh pnt i ... till pntMProg has gone away
- rsh pnt pntMProgStart
- This way we will not be rsynchronizing the clocks. you
should wait for say 20 secs after midnite before doing
- This race condition exists any time we reboot the pnt crate.
The changes i made to get around this are:
- when pntMProg starts, i check to see if the syncClock()
has done at least 1 successful synchronize with the irig
server.. If not, i initialize the year,daynum, secmid from
observer2. This just needs to have the correct
year,daynumber.. eventually the syncClock will succeed, and
the seconds will be correct....