INACCURATE TIMESTAMPS OF DATA IN ARECIBO FITS-HEADERS
Data affected: all previous FITS-files up to 2005-Feb-22
=====================================================

A bug affecting the timestamping of the data has been found and corrected.

The effect of the bug has been that timestamps indicating when data
was taken (FITS-header keyword CRVAL5) may be off by one second. This
bug has randomly affected one or more WAPPs.

This problem affects FITS-files generated up until 22 February 2005,
although it does not affect all files due to the "random" nature of
the problem. It is a concern for all projects depending on accurate
timestamping of the data, i.e. projects using drift modes or mapping
modes.

A modified timestamping algorithm was installed 22 February 2005.  The
version number of the CIMAFITS-files has been updated from '1.00' to
'1.01' to reflect this correction.

For FITS-files obtained prior to this correction, the recommendation
is to not trust the CRVAL5 value, if exact timing of the data is
needed. The true time could instead be taken from the keywords
OFF_TIME or ENC_TIME (although the latter should be rounded to the
nearest integer) - provided that the WAPPs have been run using
1-second subscans.

Note that CRVAL5 as well as OFF_TIME and ENC_TIME all were modified in
different ways (unit, format, timezone) with the introduction of
CIMAFITS version '1.00' on 3 February 2005. For more information, see
the CIMA homepage at http://www.naic.edu/~cima.



DETAILED DESCRIPTION
--------------------

AFFECTED KEYWORD:

The affected keyword is:

CRVAL5             (time stamp indicating start time of subscan)


Synchronisation errors between the 1PPS pulses and the NTP-controlled
system clocks have frequently caused the timestamps indicating the
start of each subscan (CRVAL5) to be off by one second. The problem
has been that one or several WAPPs have started one second before the
reported start time. The problem occured at the start-up of a scan and
then persisted throughout the entire scan since the timestamping of
the data has been done by incrementing the start time.

As an example, assume that an observer has made a ON-OFF observation
consisting of a 300 second ON-scan and a 300 second OFF-scan. Assume
that WAPP3 was affected by the problem when starting the ON-scan but
not when starting the OFF-scan. In that case, all 300 subscans taken
during the ON-scan from WAPP3 will have CRVAL5-values that are off by
one second while all the subscans taken during the OFF-scan will have
correct timestamps.

The synchronisation error has occurred in a more or less random
way. Certain days it has not occurred at all, while it has occurred
frequently on other days. Also the number of affected WAPPs have been
varying randomly. There are occasions when all four WAPPs have been
affected simultaneously.

The randomness has been caused by drifts and jitter in the system
clocks on the different WAPPs. The problem has been much worse during
and after big transfers of files out from the WAPPs, since this kind
of activity introduces serious drifts of the system clocks. Drifts of
up to 500 milliseconds have been recorded during half an hour.

A typical symptom of the problem has been that when plotting the
telescope position for a certain (CRVAL5) timestamp, the telescope
appears to be in two locations at the same time. The reason is that
parts of the data indeed were taken at two different times
(positions). Checking for discrepancies in telescope positions is,
however, not the best way to look for the problem, since it can happen
that all four WAPPs are affected simultaneously. Proper testing should
be done by comparing CRVAL5 with OFF_TIME or the rounded value of
ENC_TIME. Note that this check will not work for observations done
with subscans of any other length than 1 second. While 1 second
subscans are the normal mode while observing, calibration scans
typically use a single subscan of larger length.

Note that this problem is independent from another problem which had
the WAPPs start at different times which was also reported in the log
as different start times. In that case the CRVAL5 values were
correctly assigned - unless the WAPPs happened to be affected by
synchronisation error at the same time. That problem has also been
fixed.



					Mikael Lerner

					Arecibo, 28 February 2005