PUPPI Observing and Support Guide

Introduction
IF Paths
PUPPI Hardware
-- Analog Components
-- Digital Components
-- Computer Hardware
Software Components
Observing Modes
-- Incoherent Mode (Observing instructions)
-- Coherent Mode (Observing instructions)
-- Baseband Mode (Observing instructions)
-- Typical Setups
-- The puppi_observe script
Software Tools for PUPPI
Troubleshooting

Introduction

PUPPI is the Puerto Rico Ultimate Pulsar Processing Instrument, the newest backend at the Arecibo Observatory. It is a clone of GUPPI, the Green Bank Ultimate Pulsar Processing Instrument, and this page is a near-clone of the GUPPI support guide.

The software running on PUPPI to record data is also the same as what runs on GUPPI. The names of various commands described below containing "guppi" remain the same when running on PUPPI.

If you are a GBT user, you would have used GUPPI via Astrid, the GBT telescope control software. At Arecibo, we expect to integrate PUPPI control into CIMA very soon. Until then, you will be running CIMA to point to a source and track, and control PUPPI from the command line to take data. This is easy!

PUPPI Hardware

Analog Components

  • IF conditioner - The initial signal path input to PUPPI. This contains amplifiers and attenuators to set input power levels appropriately. There are no software-controllable settings.
  • Clock synthesizer - The synthesizer is used for the main PUPPI ADC clock. It should be set to one of the allowed PUPPI bandwidths (800, 200, or 100 MHz). The synth is locked to the site 10 MHz maser reference frequency. The synth frequency is set via software control.

Digital Components

  • ADCs - Two CASPER iADC boards, one per polarization. The ADCs are run in interleaved mode (so clock rate = bandwidth), and sample with 8 bits. Each ADC board is attached to an IBOB.
  • IBOBs - CASPER single-FPGA boards. For PUPPI, the IBOBs simply read the ADC output and send it to the BEE2 over XAUI connections.
  • BEE2 - CASPER FPGA board containing 5 FPGAs. The BEE2 reads sampled data from the IBOB, and splits it into frequency channels using a digital (polyphase) filter bank. In some modes, the spectra are detected and accumulated in the FPGA, while in others the channelized voltage data are output directly. The output is over four 10-gigabit ethernet lines to the PUPPI 10GbE switch.
  • Power strip- A software-controllable power strip that can be used to power-cycle the IBOBs and/or BEE2.
  • Power-cycling the IBOBs is done routinely whenever the clock frequency is changed (for different bandwidth modes). This takes ~5 seconds.

Computer Hardware

Software Components

  • Controller- Internal software component responsible for getting/setting HW and SW parameters for PUPPI.
    • Python-based, written mainly by R. DuPlain and P. Brandt.
    • Main server runs on puppimaster. Also has processes running on GPU nodes, and the BEE2.
    • Client programs connect to the controller to get/set parameters. A python-based interpreter (the "guppi prompt") is one such program, useful for debugging and setting PUPPI parameters outside of CIMA.
    • This code loads/configures FPGA personalities on the BEE2.
    • Code available at github - http://github.com/nrao/guppi-controller
  • DAQ software- also called guppi_daq, this code is responsible for receiving, processing and recording the data.
    • C-based, written mainly by P. Demorest and S. Ransom
    • Runs on puppimaster and on gpu nodes.
    • Uses shared memory segments for: storing observing parameters ("status shared memory"), and ring buffers for data processing ("databuf shared memory")
    • Also available on github - http://github.com/demorest/guppi_daq.
  • In addition to these major components, several smaller scripts are always running in the background on puppimaster:
    • supervisord - This daemon is responsible for (re)starting all the other software components.
    • gbtstatus_loop - Periodically reads info from the GBT status database and fills it into the guppi_daq status shared memory.
    • guppi_gpu_push - Distributes configuration info from puppimaster to the GPU nodes.

Observing Modes

PUPPI has two main modes of operation, incoherent filterbank mode, and coherent dedispersion. Each has a large amount of flexibility in terms of number of channels, etc. Baseband recording is also possible but is still under testing.

Incoherent Mode

Sometimes this is also called "search mode" or "filterbank mode". This mode is just like a normal spectrometer except the integration times are very short (typically tens of us). In this mode, the spectra are accumulated in the FPGA hardware, then output over the network and recorded to disk on puppimaster. Observing instructions for incoherent mode. The available parameters are:
  • Total bandwidth: 100, 200, or 800 MHz
  • Number of channels: Any power of two from 32 to 4096.
  • Output: 8 bits only.
  • Polarizations: Total intensity (summed) or full-Stokes. 4096-channel mode is total intensity only.
  • Minimum spectrum integration times:
    • For 100 and 200 MHz bandwidth the minimum integration is given by 4 * (# channels) / BW.
      • This is a FPGA hardware limit.
      • e.g., for 200 MHz, 2048 channels, minimum integration time is 40.96 us.
      • In "Fast4K" mode the minimum is 2 * 4096 / BW.
    • For 800 MHz, the minimum is 16 * (# channels) / BW. This is a disk write speed limit (max 200 MB/s to the RAIDs on puppimaster).
  • Maximum spectrum integration time:
    • The accumulation length (T_int * BW / N_chan) should not exceed ~1000 (typically ~ms).
    • The 8-bit output values do not have enough dynamic range for this.
  • Data format: Search mode PSRFITS.
  • There is a real-time folding mode available, but it is hardly ever used now due to the availability of coherent dedispersion mode. An exception is when observing the pulsed noise cal, the data are often folded in real time at the 25 Hz standard cal period.
  • Note that in this mode, the true achievable time resolution depends not just on PUPPI setup but also on dispersion smearing. This in turn depends on the pulsar's DM and the observing frequency.

Coherent Mode

For observing known pulsars, a dispersion measure-specific filter can be applied to completely remove dispersion smearing.This is often done for MSP timing. In this mode, voltage data are output from the FPGA hardware to the gpu cluster computers. The GPUs apply the dedispersion filter and either fold or integrate in real time. Observing instructions for coherent mode.
  • Total bandwidth: 100, 200, or 800 MHz
  • Number of channels: Powers of two from 32 to 2048.
  • Polarization: Total intensity or full-Stokes.
  • Fold mode:
    • Folds the data modulo the current pulse period in real time.
    • Outputs time-integrated profiles, minimum dump time is ~1 second.
    • Output is floating point profiles in PSRFITS format.
    • The time (or pulse phase) resolution is given by N_chan / BW.
  • Cal mode: Special case of fold mode, folds at a constant 25 Hz period for the cal signal.
  • Coherent search mode:
    • Coherently-dedispersed spectra are accumulated in time rather than folded.
    • Spectrum integration time is subject to the same total data rate limit (200 MB/s max) as incoherent mode.
    • Data are re-quantized to 8 bits and output in search-mode PSRFITS format
  • Note, for all types of coherent dedispersion observations, each gpu node processes and independently records 1/8 of the total BW. The 8 sub-bands are assembled into a single output file after the observation is over.

Baseband Recording Mode

For some applications, e.g. cyclical spectroscopy of pulsars, raw voltage can be recorded with PUPPI. In practice, the desirable data rate is too high for only one GPU to do the data recording, so data taking uses a relatively small number of very wide channels that get split between GPUs. The user then recombines them to recover the raw voltage data stream. Observing instructions for baseband mode.
  • Total bandwidth: 100 or 200 MHz
  • Number of channels: 32 or 64

Typical Setups

While there are a potentially large number of combinations of BW, # channels, etc available, a few setups get used much more than others:
  • 800 MHz BW, 2048 channels, 40.96 us, incoherent. This is used for searches at higher frequencies (L-band, S-band).
  • "Fast4k" - 100 MHz BW, 4096 channels, 81.92 us, total intensity, incoherent. This is used for low-freq searches (350 MHz; GBNCC, etc).
  • 800 MHz BW, 512 channel, coherent. For MSP timing at L-band and higher.
    • Fold mode is usually used for single pulsars (eg NANOGrav timing).
    • Search mode is used for multiple pulsars in the same beam (double pulsar, globular clusters).
  • 200 MHz BW, 128 channel, coherent. For MSP timing at 820 MHz.
There are also a set of standard frequencies and bandwidths for the commonly used pulsar receivers:
  • 327 MHz receiver: 327 MHz center, 100 MHz BW.
  • 430 MHz Gregorian receiver: 430 MHz center, 100 MHz BW.
  • L-wide receiver: 1380 MHz center, 800 MHz BW.
  • S-low receiver, bottom half of band: 1700-1880, 2040-2400 MHz filters, 2030 MHz center, 800 MHz BW.
  • S-low receiver, top half of band: 2600-3100 MHz filter, 2030 MHz center, 800 MHz BW.

The puppi_observe script

Until PUPPI can be controlled from CIMA, puppi_observe is the script that an observer runs on puppimaster in order to take data with PUPPI.
  • Usage: puppi_observe [options] parfile
  • Options:
    • -h, --help (Show this help message and exit)
    • --test (Test mode - don't actually start an observation)
    • -c, --cal (Take a cal scan)
    • -b BAND, --band=BAND (Observing band: 327, 430, L, S. Default: L)
    • -f FREQ, --freq=FREQ (Override default freq)
    • -T LENGTH, --scan-length=LENGTH (Total scan length in seconds. Default: 90 for cal, 3600 otherwise)
    • -m MODE, --mode=MODE (Observing mode: COHERENT_CAL, COHERENT_FOLD, SEARCH, CAL, or RAW)
    • -n NCHAN, --nchan=NCHAN (Override default number of channels)
    • -w BW, --bw=BW (Override default bandwidth)
    • -p POL, --pol=POL (Override default polarization)
    • -P PROJID, --projid=PROJID (Project ID number, e.g. P1234)
    • -I, --onlyI (Record total intensity only. Default:full-Stokes)

Software tools for PUPPI

  • GUPPI environment: this is set up via the bashrc file that is read in when you log in to puppimaster.
  • guppi_status - Run on puppimaster or gpu nodes, this shows a text window of the current status shared memory. Look for the observer's name, project id, source name, etc to be set correctly on the screen. guppi_status gets updated once approximately every second and so things which should normally change at that cadence should be changing (such as LST, AZ, ZA). If you only see about half of this information or if the information is stale, see the troubleshooting section below.
  • guppi_gpu_status - Run on puppimaster, shows an overview of the status of all gpu nodes.
  • guppi_adc_hist - Run on puppimaster, shows a histogram of count values from the PUPPI ADCs.
  • guppi_monitor - Run on puppimaster, shows a real-time bandpass display only for incoherent modes.
  • pav and psrplot - Run on puppimaster or gpu nodes to view data file contents. These are data display programs that are part of the PSRCHIVE data analysis software. These will only work on fold-mode or cal-mode files.

Troubleshooting

If PUPPI appears not to be working correctly, there are several things to check, depending on what exactly the issue is.

guppi_status is not updating every second or has old or incorrect looking data in it

The guppi_status daemon on puppimaster needs to be restarted. This can only be done by root on puppimaster via the supervisorctl command. Call Phil or Arun.

PUPPI is dropping a lot of packets in an incoherent mode

If guppi_status or the guppi log files show that a significant fraction of the data are being consistently dropped, there can be several issues.
  • You may be running low on disk space on /data. use "df -h" to check and re-configure with the other data directory "guppi.datadisk" and re-start the scan.
  • There could be data transfer happening by one or more people (using "rsync" typically) which are affecting the available I/O rate. Use "ps aux | grep rsync" and/or "top" to see if there are any active rsync users, especially those with an associated sshd process as well. Using the ssh-protocol accidentally over the 1gig (as opposed to 10gigE) network for internal AO transfers can easily cause many dropped packets. Also check to see if the processes are "nice'd". If you find this, either call Phil or Arun, or even better, contact the person running the rsync to have them stop.
  • There may be people doing significance data analysis using the same /data directory that you are writing to. You can either contact the person and have them "nice" their processes or ask Phil or Arun to kill the processes. Note that data processing by 1 or 2 processes, properly niced does not usually cause many dropped packets.

In a coherent observation mode, one or more of the GPU nodes is not working

The first thing you should do is decide if you want to abort your observation or not in order to fix the problem (which may or may not be fixable). If you decide to do it, here are a couple steps to take:
  • Can you ping the node in question? If so, ssh to it as gpu (e.g. "ssh -l gpu gpu01") and run guppi_status on the node. If the status looks good, and is updating, there is probably a problem with PUPPI itself or another hung GPU node. You can check for the latter by checking the uptime's (for instance) on all the GPU nodes:
    • Try "gpu_broadcast uptime" or equivalently "gpu_run uptime". If any of the uptimes are very short, it could indicate that a node crashed and re-started incorrectly. If that has happened, the guppi_status screen will not be complete or updating properly. You can restart the GUPPI daemons on a node using the command "/home/gpu/gpu_run" as root. Note that this requires Phil or Arun.
  • If you can't ssh into the node, the node needs to be (properly) rebooted. You can ask the operator to hit the reset switch on the GPU node. When it reboots, ssh to it as gpu and check guppi_status to see if it looks OK.
  • If guppi_gpu_status shows one of the nodes in a strange mode, perhaps with one of the statuses as "Unknown" or in a state very different from the other nodes (like "Exiting" when all the others are running), the GPU node may be in a strange hung state. If this happens, the node may need to be rebooted. Call Phil or Arun, or in an emergency, ask the operator to simply hit the reset switch.