HOW TO MAKE PALFA OBSERVATIONS: A Beginner's Guide. Summary ------- Before the observations: ------------------------ 1. Prepare a PALFA schedule (at present, contact Jason for this) 2. Ensure there is enough disk space 3. Ensure WAPP disk links are set up correctly Observing: ---------- 4. VNC instructions for REMOTE observations 5. Set up observing 6. Quicklook processing 7. End observations 1. Before PALFA run ------------------- Prepare an observing schedule for run: [Jason is the black-belt with this. In general he will have made/make schedule files for the observing day if you ask nicely.] First, make sure you have palfa group permissions for your account. Then log in to fusion00 as yourself and run: ~palfa/bin/fill_observed [y; passwd: 305m_adm] (to update database) ~palfa/bin/plot_pointings (to see pointings that have been done) ~palfa/bin/makeobs_new (to generate the observation list) The default is to choose sources for MJD=current and |b|<1; should you wish to tailor this for specific mjd or b then try ~palfa/bin/makeobs_new mjdstart bmin bmax A schedule file "autosched.cmd" is written locally and a postscript file is generated. Copy to the observing runtime directory: cp autosched.cmd -> /share/obs4/usr/pulsar/p2030/autosched_yymmdd.cmd 2. Ensure that there is sufficient disk space --------------------------------------------- Data rate is 8 MB/s/beam, so we need 58GB/WAPP[2 beams]/hr free. Check space manually by logging into each WAPP with 'ssh wapp@wappn' and doing 'cd /share/wappn1; df -h .' We are currently writing directly to the WAPP RAID disks. For now, coordinate with Arun to make sure there is enough free disk space. In future these instructions will explain how to do that yourself. If there is enough disk space on the WAPP RAIDs, move to step 4. now. _If_, in an emergency, there isn't enough free disk space on the WAPP RAIDs, and you haven't been able to reach Arun (who may be moving data off the RAIDs even as you write to them!) you can still write directly to Firewire disks on the WAPPs, although this is not as reliable as writing directly to the RAID disks. The rest of this section explains how to do this. If you don't need to do this, jump ahead to step 4. now. Otherwise... As user alfa on aspmaster (ssh alfa@aspmaster), execute the script "check_wapp_space" (to check free disk space on the Firewire WAPP disks; doing a df /export/firen1 and df /export/firen2 [n = 1, 4]) If there is not enough space, then a new Firewire disk will need to be formatted and swapped for one of the old ones. To do this on wappred1, Log into wappred1 as root. Unmount Firewire disk (if one is already mounted): umount /dev/sdd1 (do a dmesg if you are unsure of device name) Remove and replace with disk to be used for observing. wappred1 will recognize disk and tell you device name (e.g. /dev/sdd1). Mount Firewire disk: mount /dev/sdd1 to check if it needs new filesystem. Make a new filesystem: umount /dev/sdd1 mkfs.ext3 -T largefile4 -L VOLUME_ID /dev/sdd1 (VOLUME_ID is "palfa_c105" or similar) Unmount disk: umount /dev/sdd1 and so on, until all disks have new filesystems. To replace full disks on the WAPP, for each WAPP in turn, as root: Save the "ls -l" output for the to-be-removed Firewire disk in a file to be saved to ~palfa/public_html/disks/AsWrittenAtAO. Unmount existing Firewire disk: umount /export/firen2 (or similar) Remove and replace with new disk. Set aside old one for shipment to Cornell. Mount new disk: mount /export/firen2 (Sometimes there is a bit of a delay before the WAPP will recognize the new disk, so be patient!) Change the ownership to wapp: chown wapp /export/firen2 3. Ensure soft links are set up on the WAPPs -------------------------------------------- This enables data to get written to the correct disks (this is generally needed only if writing directly to Firewire disks since by default we write to WAPP RAIDs): In each of the 4 WAPPs in turn, check that the soft link /share/wappn1 (where n=1,2,3,4) points to the WAPP disk that you would like to write to (e.g., /export/firen1 or /export/firen2) If not, then do the following as root: cd /share rm wappn1 ln -s wappn1 /export/firen2 and so on for the other WAPPs. If there are non-PALFA observing sessions that will use the WAPPs before the next PALFA run, switch the links back to /export/wappn1 (etc) at the end of the observing session to change back to writing directly to WAPP RAIDs (but you wouldn't likely be doing this step if there were a lot of free disk on WAPP RAIDs in the first place...) 4. For remote observing, start up a VNC session ----------------------------------------------- If you're not doing remote observing, jump ahead to step 5. now. Otherwise, you may begin by consulting AO remote observing instructions (http://www2.naic.edu/~astro/remote.shtml). However, if you are going to use VNC (HIGHLY recommended!!), the instructions below tell you all that you should need to know. Let's get started. From your computer away from Puerto Rico: %ssh -X @remote.naic.edu [That -X may not be needed, depending on how your Xforwarding is set up, but it doesn't hurt.] $ssh fusion00 $vncserver -geometry 1280x1024 The geometry parameters can be anything you like (they don't have to be "normal" values like 1024x768 etc. They can be 1200x900 or 1500x1100, etc). It is best to make it slightly smaller than the resolution of the monitor that you are working on. [If you've never done this, it'll ask you for a passwd; make it one that you won't have a problem sharing with others - in other words, _not_ your usual passwd!] [This will return something like fusion00:4 <- that number is important.] Back at your computer off Puerto Rico, %vncviewer -via @remote.naic.edu fusion00:4 [instead of 4, use whatever number you got from running vncserver in fusion00; also, you can change the geometry parameters.] [For Mac, Windows and some Linux, the -via option is not available. On a Mac, run ssh -f -L 590N:fusion00:590N YOURLOGIN@remote.naic.edu sleep 14400 in a terminal window. Then open Vncviewer (just use the icon) and select display > open -- localhost:N in both lines, N is the important number from above. For Linux, use the same first line; then run vncviewer localhost:N For Windows, download Putty, set Session > Host name = remote.naic.edu Session > Protocol = ssh Connection > Auto-login username = YOURLOGIN Connection > Tunnels Source port = 590N Destination = fusion00:590N And connect. In VNCViewer, open localhost:N ] You'll be asked for 2 passwds; the 1st is your regular Arecibo passwd; the 2nd is the one you made in fusion00 when running vncserver, and that you'll be comfortable sharing with others. Now you should have a big window on which you can open xterms, etc, and soon you'll start cima and do the observing from here. Very usefully, anyone else can hook into your vnc session (if you give them your vnc passwd) and see all that you're doing (or do it for you!) and offer advice, by doing the following from their computer: vncviewer -geometry 1200x900 -via @remote.naic.edu fusion00:4 At the end of the session, you log out of CIMA as per the observing instructions, within the VNC session. Then you kill the VNC window (by clicking on the x at the top right hand corner of your large window, or something similar). Finally, you must log into fusion00 from an xterm in your computer and do the following, in order to not monopolize resources: $vncserver -kill :4 5. Set up actual observing -------------------------- a. Call the operator and let them know that you'll be observing: 1-787-878-2612 x211 b. Log in to observer2 console in control room as dtusr. The password is written on a sticker on the monitor, or ask the operator if observing remotely. If observing remotely, "ssh dtusr@observer2" (from an xterm within VNC). c. Start CIMA ([normal] version): using the right mouse button if at AO or by typing "cima" if observing remotely. You will be prompted for some information. Enter your name(s), the project number ("p2030"), hit "Pulsar" button. If observing remotely, you first have to enter your name, project number, and phone number in the initial widget, 'call' the telescope operator, and get a reply from him; you then hit "Pulsar" on the widget that pops up along with a few other windows. In "Preferences" (button on lower right), make sure the option to write daily log files is set to "Yes". Note (especially for remote observers): sometimes the above window prompting for information might be buried under the chatterbox, so check carefully (e.g., minimize or move the chat window). The Receiver widget will pop up by itself. Click on: "ALFA", "disable quick tsys", "Select Receiver Now", and "Dismiss". d. Select a bright "test" pulsar: Choose "Pointing Control". Use default psr.cat (or choose psr_bright.cat), click on a test pulsar (e.g. B1933+16, J1906+0746), click on "Point". Telescope will slew to source. e. Monitor the telescope's status: If observing remotely, run "aostatus" from an xterm (in the control room this is always on display). This will display the telescope pointing status, etc. f. Load the PALFA configuration: Click "Load/Save State" from the "Observer's Interface" window. Click "Load a Saved State". Select "alfa_psr_scan.1440.gui", "ok", and "Dismiss". g. Apply the PALFA configuration to the IF/LO system: Click "Receiver IF/LO Control". Choose "New Improved IFLO setup" (blue). "Apply This Setup", "Dismiss", and "Dismiss this window". h. View the IF/LO settings: If observing remotely, "ssh buye" and then do "/usr/local/bin/rxview" on that machine (in the control room this window is always visible). Make sure the IF-LO setup is correct on the IF/LO monitor (rxview): Center = 1440, bands = 1390-1490 (among other things) NOTE: The output of the above rxview is sometimes different from that displayed in the control room. Just be aware that you may have done everything right - and everything _may_ be right - but that the rxview display could be misleading. This display, while pretty, isn't quite entirely reliable. You may prefer to run /usr/local/bin/monif from a Sun - its display isn't as pretty (and is a bit harder to interpret), but is usually correct. If you have reason to believe that the IF-LO setup is incorrect, redo "New Improved IFLO setup". i. Verify that we are using beam "0" for pointing: Bring up "Pointing Control" again. ALFA Center should be "0" by default, but you can explicitly choose it. j. Balance the WAPPs and check the levels: Click "Pulsar Observing", leave it open. Configure the WAPPs for "2 chan, 3-level auto", "Copy WAPP 1 to all". "Sanity Check" - and check that it passes in 'AO Observer Display'. "Start Monitor" and "Auto Attenuation". To view all 14 channels, run "alfaobswin &" from a dtusr@observer2 window. The levels on alfaobswin should be ~1 for both poln chans for all beams. Back in the WAPP GUI, "Stop Observation". k. (OPTIONAL) Run a SNAP session to view the data in real-time: NOTE: for very strong pulsars you may well see the odd single pulse; but then again you mightn't. In any case, you'll soon get to see the reduced data in step 6., so this may be more trouble than is worth your while. However it is a nice check to see the 21 cm line booming through at the expected freq! If in control room: run SNAP from WAPP GUI: "More"; select 'wapp' computer (instead of 'dataview'); "Start Snap"; "Dismiss". Remotely, "ssh wapp@wapp" and then "snap". In plot window "Select" menu, choose "Raw Band" and select "Continuous" in WAPP Quicklook window. When you start taking data, you will (hopefully) see the pulsar in the waterfall plots. Toggle from Raw Band to Ref Band to get an integrated bandpass which can help you to verify that the 100 MHz filters are in and centered at 1440 and that the 21 cm line is as expected (but invert the band first or freq labelling is upside down!). In quicklook window, cycle through all 4 wapps and look for 21 cm line to be sure all bands are working. l. Take a "test" scan (or scans) on a bright pulsar: In the WAPP window, change to "sum chans, 3-level auto". "Copy WAPP1 to all", "Sanity Check". After telescope settles (Tracking is green in "aostatus"), select the integration time of your choice (67 s default is ok) and click "Start Observation". When the realtime processing starts (see 6. below), you should see the test pulsar appear in the beams that you observed the pulsar with. If you want (this should be done the first session of a new observing epoch, but is optional afterward), you can cycle through beams, by going to 'Pointing Control' in the CIMA main window, and choosing the "ALFA Center" beam (from 0-7) of your choice. Then, "Start Observation". If you are using SNAP, change the WAPP setting as appropriate (e.g. beams 2 & 3 are on wapp2). Use pull down menu in upper left of 'WAPP Quick Look - Snap'. When you are finished with the test pulsar, you must quit SNAP. IMPORTANT: WHEN YOU ARE FINISHED, MAKE SURE TO SET "ALFA Center" BACK TO BEAM "0" IN THE "TELESCOPE POINTING CONTROL" WINDOW!!!!!!!!!!!!!! m. Choose the pointing catalog for the day's observing: In the main panel ("CIMA Observer's Interface), select "Command file Observing". Click on the "Load" button below. This makes a widget appear that allows you to select the command list file. After selection, click on "open" button in that widget. You will see many of the lines of the command file now displayed in "command file observing" widget. Press the button that says "Run". At first this may take a while since the telescope is moving to new source and is rotating ALFA - check action in 'AO Observer Display'. Should you have any problem/error at this stage with the ALFA feed rotation, check with the operator if it is turned on (by default it should be turned on). If not, ask the operator to turn it on before you proceed. If you need stop the observation (to change attenuators, etc) you can click "Stop" (to finish at the end of present scan) or (more drastically) "Abort" (stop taking data _and_ command file observing now). While the observing is going on, you should monitor the levels periodically in the WAPP window. Things are probably "bad" if they go below 0.8 and/or above 1.2. If that happens, you should "End at Next Loop", re-balance the system as in step (j) above, and then re-start the catalog observing (on an even line number! - see above), as per steps (l) and (m) above. 6. Now that observations are running, start quicklook ----------------------------------------------------- a. Log in to aspmaster and start the realtime software: (Remote observers: It is much faster over vnc.) "ssh alfa@aspmaster" and then "bin/realtime" In a few seconds, a status window will pop up. Give it a little while to update, then: At the beginning of an observing epoch the processing will likely be stopped; "Clear Files" and confirm ("Yes") to clear the ASP disks (takes ~1 min). Then hit "Quick processing" and copying/processing will begin. NOTE: The interface can be slow, especially when nodes are busy! Click each button once only and wait. We usually hit "Stop processing" only when, e.g., ASP needs to be used for observing. When nodes are idle (likely prior to starting to observe), it's ok to "Clear Files". Multiple instances of the interface can run at once. BUT wait for any action (i.e. button click) to be completed before clicking a button on the same or another interface! b. View candidates: Using a browser on a local AO machine (or in your VNC session), open http://aspmaster.naic.edu:8080/alfa/viewer.php More details in DOCS: asp_processing.txt 7. To end observations ---------------------- a. In the "AO Obs Display" select "End at Next Loop". b. In the "CIMA Observer's Interface window" select "Exit normally". c. Hit "Quit monitor" button on ASP nodestatus screen. NOTE: The realtime processing will lag the observations by up to 1hr+. d. Look for candidates in all quick look data, at least on inner Galaxy scans. (On telecon of 19 Dec, 2005, we decided that this should be the observer's responsibility. It works ok over vnc.) e. If there is anything important to report or in log file (/share/obs4/usr/pulsar/p2030/p2030.cimalog), place it in ~palfa/public_html/observing.notes/ (and make sure group and permissions are correct).[also update the link to this file on webpage: /home/palfa/public_html/observing_notes/index.html ] There should be at the least a session summary here for each session. ****************************************** *** Routine checks during observations *** ****************************************** - If power levels change by ~20% or more away from 1.0, it's probably time to rebalance the levels: stop the observation at the end of the present scan (or abort), change back to 2IFs' mode, copy wapp1 to all, sanity check, start monitor, auto attenuation, stop observation, back to summed IFs' mode, copy wapp1 to all, sanity check, and restart observation - on correct schedule file (i.e., even) line. - Check 'aostatus' once in a while: Close to the zenith (< 1 deg ZA), the azimuth arm can not swing fast enough to track. But even at somewhat larger ZAs, the ring of alfa feeds rotates more than a half beamwidth on the sky, thereby greatly diminishing the sensitivity of all but the central beam. From Phil Perillat's Fig 4 at www2.naic.edu/~phil/mbeam/mbeamrotation.html, - An anticenter (~2.5 min duration) observation should not be done with za<3 deg, if we wish to keep beam skidding on the sky < 1/2 beamwidth. - An inner (~4.5 min duration) observation should not be done with za<5 deg. (See red and green curves respectively on Phil's Fig 4 for details.) In these cases, you may choose to advance a few pointings in the schedule file (see 5.[m] above). *********************** *** Troubleshooting *** *********************** - If there was a nearby radar run, ask to make sure ALFA cover is off and the filterbank has been reset to the standard setup. - If there are a lot of 'overflow' warning errors, sometimes doing a "Sanity Check" when the WAPPs are idle and restarting the observations helps. - When all else fails, restart the WAPPs: "More" on WAPP GUI, then "Restart ALL WAPPs". Sometimes restarting the specific WAPP having problems works even better: To restart WAPPn, ssh wapp@wappn and type start_wapp (which stops and starts it). ----------------------------------------------- Notes by DJN, 2 Aug 2004 Updated by NDRB, 5 Sep 2004 Updated and expanded to Beginner's Guide by SC, 9 Apr 2005 Updated by DJC, 7 May 05 Updated by FC,MAM,DRL,RR,FCr, 2 Aug 2005 Updated by FCr, 18 Aug 2005 Formatting changes, SMR, 28 Nov 2005 Updated by FC, 30 Nov 2005 1440 MHz is now the default mode, PCCF, 16 Dec 2005 Added vnc tunneling info, JvL, 19 Dec 2005 Added misc, JMW, 20 Dec 2005 (JMW being a relative novice added some novice suggestions, and also pointed out the serious parallactic rotation near zenith.) Updated by FC, 28 Dec 2005 Updated by NDRB, 22 Apr 2006 Updated by PF, 03 Jul 2006, no more need for setting up "alfafb".