AO scramnet

29aug2004

Description
25nov14: scramnet driver on observer cpu fails.

    Scramnet is a reflective memory system consisting of nodes (sbus,vmebus cards) connnected in a ring configuration via fiber optics. Each node contains memory (128kb in our case). When any node writes into its memory, the information is propagated around the ring so it appears in the same memory location in all the other nodes on the ring. Ring transfer rates are 5 to 12mb/sec
    . The data acquisition system at AO uses scramnet to pass information between 3 vme crates (pnt,da,cor) and 2 sun workstations (observer,control). The memory is broken into different blocks. Each block holds status information about a particular device or function. The observer workstation takes the data from scramnet memory and places the data in a shared memory block.

     The use of "scramnet" was extended at a later time:

The data from the reflective memory is placed in shared memory on the observer workstation.
A program (multiexec) takes some of the blocks from shared memory and multicasts them out the ethernet.
3 other programs (cima exec shared memory, alfa monitoring and position, wapp info) take various blocks of information and multicast them out the local ethernet.
All of these broadcast packets are now called "scramnet info".

Blocks that originate from reflective memory:

The reflective memory broadcast blocks are defined as C structures.
disc locations:

/home/phil/vw/devs/scrm/scrmLoad.h - defines the blocks that make up reflective memory
/home/phil/vw/h/scrmBlkXXX.h defines an individual scramnet block. Usually just contains a single structure definition.

eg pntProgState.h

Blocks that go through reflective memory
block	Struct defined in /home/phil/vw/h	DataSrc	Notes
pointing	pntProgState.h	vme pnt	pointing info, ra, dec,etc
vertex (az,ch,gr)	agcProgState.h	vme pntp1	az,dome,ch device info. actual encoders, state,etc.
turret	ttProgState.h	vme pntp2	rotary floor info
tiedowns	tieProgState.h	vne pntp3	tiedown info
Downstairs iflo (IF2)	if2ProgState.h	vme da	synths, power, switches for downstairs if/lo
Upstairs iflo (IF1)	if1ProgState.h	vme da	synths, power, switches for upstairs if/lo
Doppler correction (vxWorks only)	dopState.h		not loaded
pattern procedure info	procState.h		not loaded
planetary radar info	plrdrState.h	pntp3	not yet loaded
sb723 trransmitter info	sb723State.h	pntp3	not yet loaded?
tertiary			not used
Blocks that originate from a program
block	Struct def:	DataSRc	Notes
Cima executive info	execshm.h	observer2	Only runs when cima gui is active. Holds experiment setup info: projid, etc.. see multiexec
alfa monitoring and motor position	alfashm.h	aeroncpu	see alfa_mon
Wapp dataking info	wappshm.h	wapp	see multiwapp

Notes:

multiexec.c - multicast broadcast of cima execuitve info

Source: /home/cima/Wapp/Software/Svnwork/Sources/Tools_linux/
includeFile: /home/cima/Wapp/Software/Svnwork/Includes/execshm.h
Only runs while cima gui is running

alfa_mon.c - multicast alfa dewar temps as well as alfa motor position (which it gets from alfa_mot.c).

Source:/home/cima/Wapp/Software/Svnwork/Sources/Utilities/
Include: /home/cima/Wapp/Software/Svnwork/Sources/Includes/alfashm.h

multiwapp.c - multicast wapp observation data.

Source: /home/cima/Wapp/Software/Svnwork/Sources/Wapp
include: /home/cima/Wapp/Software/Svnwork/Sources/Includes/wappshm.h

scramShm.c - grabs multicast packets for all blocks and places them in shared memory (phil's version).

The shared memory is configured for the mock spectrometers. The general idea of grabbing the blocks is here.
source:/share/megs/phil/svn/pdev/pdev/datatk
Uses libbp (buffer pool library,). libphil ( bunch of utility routines).

scramshm.c - grab multicast packets for agc or pnt and place in local shared memory (jeff hagens version).

Old writeup on the structure of reflective memory scramnet used at ao (.ps)

25nov14: scramnet driver on observer cpu fails.

On 09nov14 around 5am the ups for the observer computer failed. This caused the observer cpu to reboot.

After reboot, the message log contained the error message:
Nov 9 05:15:02 observer mshmD: program started
Nov 9 05:15:02 observer automountd[135]: server observer1 not responding
Nov 9 05:15:02 observer mshmD: Err error 69992452 calling scrmAoInit. Must be on a scramnet cpu
Nov 9 05:15:02 observer mshmD: Err error 69992452 calling scrmAoInit. Must be on a scramnet cpu
Nov 9 05:15:02 observer mshmD: daemon exiting

about 6 minutes later it looks like someone logged in and manually restarted the scramnet shared memory daemon:

Nov 9 05:21:43 observer sshd[375]: log: ROOT LOGIN as 'root' from observer2
Nov 9 05:50:59 observer sshd[447]: log: ROOT LOGIN as 'root' from observer2
Nov 9 05:53:03 observer mshmD: program started

Looking back at things, it looks like mshmD did not start the first time because the scramnet board did not initialize properly (i'm not sure why it started the 2nd time).

:The interim correlator data files have header information that is loaded from the da computer (the if/lo synth frequencies)
and the pointing crate (the telescope position, az, za, etc). This information is passed between the computers on the scramnet
memory/fiber link. The connection topology is:

daCrate -> pntCrate -> observerComputer -> corCrate ->back to daCrate

Looking at the interim cor data files, I see that the if/lo headers (from the da crate) and the pointing hdr (from the pntCrate)
remained constant starting at 6:00am 09nov14. They remained constant until 25nov14 17:00 when i rebooted the observer
computer.

It looks like the problem was that the observer computer was not passing the scramnet packets on to the cor crate. When i rebooted the
observer computer on 25nov14, it started working again.

This means that:

all correlator data files 09nov14 6:00am thru 25nov14 17:00 have incorrect data for

the if/lo (mainly synth freq)
the pointing informatio (ra,dec and telescope position).

All data taken on the daCrate (sband and aeronomy) will have incorrect pointing information in the header.

The pnt header info had to go pnt -> observer ->cor -> then to da
The if/lo header on the da crate is ok. Since it originates in the da crate, it does not need the scramnet.

It also means that all data taken with the wapps, or the mock spectrometers were not affected:

The wapp and the mock spectrometer get the header information from the scramnet header packets that are broadcast by the observer computer over ethernet.

The observer computer was receiving the packets from scramnet and then broadcasting them out over ethernet without any problems.
The observer computer was not forwarding these packets over scrament fiber to the cor ,da computers.

How the data were affected:

The setup and running of the experiments is done via ethernet. So all of the experiments were setup and run correctly.
The iflo and pointing monitor data that was loaded into the header was incorrect.

Interim correlator spectra:

The requested frequencies, velocities, all originate in the cor crate (that is who cima talks to) so they are ok.
The synthesizer frequencies are used to determine if the band is flipped or not (odd number of high side lo's).

The bad data iflo data in the header always said that the frequency bands were not flipped.
If you used the 750 IF (most users) then the frequencies are normally flipped (1 high side lo to go rf to if1, 1 hiside lo to got 750 to 260, and 1 more flip to get to the cor sampler) So when you plot your data, the frequency labels will be in a reversed order ..
The interim cor idl routine: corhflippedh(hdr,sbcnum) is used by most interim cor idl routines (corget,coronoff,corplot,etc) to tell if the band is flipped.
I can provide a version of this that always returns 1(flipped) You could then use this when processing the bad files (.. and not use it after that!).

The pointing information is wrong. The az will always be about 303 degrees and the za is 10 degrees.

You may use this information for az,za dependent gain corrections. If so, i will have to provide the az,za for every scan from the pointing archive files (not something i'm looking forward to).

sband radar:

the sband radar uses the da crate.. so all of the frequencies are ok (since they come from the da crate).
the pointing positions will be wrong for the date range above.

astronomy Observations that were affected by the problem:

for each project .1,.2 are the files corfile.ddmonyy.projId.n
a1312:

24nov14: .1

a2048:

18nov14: .1 ->.12
21nov14: .1 ->.6
22nov14: .1 -> .12
I talked with robert minchin.. two of nites had other troubles.
he used the 10ghz upconverted so there is no flip problem on the other night

a2812:

10nov14: .1
11nov14: .1
12nov14: .1
13nov14: .1
14nov14: .1
17nov14:.1
18nov14:.1
22nov14:.1

a2892:

09nov14: .2
10nov14: .1,.2
12nov14: .1
13nov14: .1
14nov14:.1
15nov14: .1
16nov14:.1
17nov14:.1
18nov14:.1
19nov14:.1
22nov14:.1
24nov14:.1

a2894:

13nov14: .1
14nov14: .1
15nov14: .1
16nov14: .1
17nov14: .1

a2899:

23nov14: .1 -> .36
24nov14: .1 -> .36

a2902:

09nov14: .1
15nov14: .1
16nov14: .1
22nov14:.1
23nov14:.1

a2903

15nov14:.1
22nov14:.1
23nov14:.1

x111

i deleted all of the x111 files affected

x102

i deleted 2 nights of x102 data.. 15,16nov14 since they didn't have position info

x101

<-page up

home_~phil