Minutes of PALFA Telecon 19 December 2005 20 December 2005, revised 28 December 2005 In attendance: Joeri, Julia, Joel, David N., Ingrid, Andrea, Jason, Vicky, Shami, Bryan, Fernando, Maura, Dunc, Jim, Scott 1. 1906 update: - paper in press (March '06) - timing proposal accepted at AO - timing proposal to Parkes resubmitted for next term - timing at other telescopes (Jodrell, Parkes, GBT) - Press release: Dunc and Jim working on it; customizable for PALFA institutions; time it for AAS in DC; check with Bob Brown 2. PALFA mini-meeting at AAS: - Tentative day/time: 1 pm on Thursday 12 Jan; location TBD (NB: JMC has to meet with the Senior Review panel at 2:15pm that same day, so we should either start earlier than 1pm or be efficient. SR wants to meet with (NAIC) for about an hour.) 3. New pulsars page at AO (Dunc): - includes timing solutions from Jodrell timing - about 1/2 of new pulsars are timable at JB (L band) 4. Timing strategies: - FC and others argued that we should devote PALFA time to timing new pulsars at the level of one 19h session per month or so. This is an interim situation until we receive explicit telescope allocation for timing at AO. - eventually need to sort out which ones are timed at AO regularly with those timed at JB and elsewhere - purposes of timing: * short term timing to figure out pdot, binarity * long term monitoring (binaries, MPSs) * should focus AO time on both of these, all pulsars initially, only the important ones for long-term timing - Full Stokes or not: * many of us have emphasized the importance of full Stokes for precision timing => should use full Stokes for 1906, etc. * however, full Stokes not necessary for preliminary timing * however, need to worry about overhead for Stokes vs. summed-pol timing * action item: identify calibration protocol and code for Stokes timing possibilities: > JMC, Ramesh Bhat code for WAPP data > Rankin code > Han code - for next AO session: defer Stokes, do summed-pol data acq mode; if all timing in single session, use L-band wide system 5. Search code pipeline status: - McGill/NRAO code: * port sigproc etc. to 64 bit Opteron clusters * single pulse code: a version exists within sigproc * need to sync Maura's and Jim's version of code (NB: Maura's takes a matched filtering approach as in Cordes & McLaughlin 2003 paper; Jim's does the same and also applies a friends-of-friends algorithm; the f-o-f algorithm is immune from the effects of having a maximum-width matched filter as mentioned in the RRATS paper to appear in Nature). - Cornell code: * running on Cornell CTC Windows NT Itanium machines * evolution: original linux port -> Windows NT Windows NT code has evolved but with intention of re-porting to Linux Re-port to linux can happen any time and should be "straight forward" but may require a lot of small fixes to make compatible - Fast folding algorithm: * should include to detect slow pulsars * a version exists in sigproc * need to include in piplines - RFI excision/mitigation algorithms: * Laura@UBC will work on this * RFIfind needs to be explored - Code management: need version control, installation scripts - expect sigproc/presto code package in next few weeks - Cornell package (which uses sigproc + other code): can begin linux re-port any time; will take about one month 6. Data integrity and management at AO: - latest batch of disks from AO with "AOGD" numbers show file system errors when read on linux machines in Cornell Space Sciences Building and at CTC on their linux box - reminiscent of problems with GBT/Spigot data having to do with linux kernel version issues - action item for all: send JMC and Julia (and cc others) anecdotal + quantitative evidence for disk problems, either while writing at AO or reading elsewhere; we need to develop a record of problems - possible solutions: * need to keep Arun in the loop re solutions and to reduce his work load - data crunch at AO; remote observing; Arun's workload, etc: * solution: buy more RAID for AO * attach to ASP? or attach to another non-WAPP machine? * probably the latter given workload on ASP and to maintain flexibility. - action item: how much RAID to buy, how to configure at AO? contributions: Joel ?k Fernando ?k Jim ?k Others ?k we need about 10 TB of RAID or about $20k: please contribute 7. Data transport to processing sites and CTC: - disks: which kind to purchase? see action item under 6. as first step toward answering this question - shipping scenarios: I. AO -> CTC -> processing sites (other than Cornell) pro: gets data into archive (2nd copy) asap if we ship original con: personnel/work-load/shipping-cost issues at CTC; risk of losing data in shipping if we ship original II. AO -> processing sites -> Cornell pro: gets data to processing sites asap reduces workload at CTC con: delays getting archive copy at CTC until after second ship phase; risk of losing data in shipping if we ship original; increased workload at AO if we make copies there - short term: go with scenario I. long term: need to address costing issues for shipping - possibilities: NAIC may be able to cover shipping costs for 1 or 2 legs of a triangle; other costs borne by processing sites/consortium - shipping costs for FedEx: may need accounts set up by different institutions for charges at CTC, AO or elsewhere - which data to ship? need a database to select from: * CTC database currently shows data that have been processed at the CTC * we have requested a database of data now on the robotic tape archive at CTC (=> action item) * database at AO of positions observed + file names - what kinds of disks to purchase? * we have trouble of some kind with all kinds of disks (IDEs in trays, LaCies etc.) True? * we have had successes with all kinds of disks * suspect we have problems intrinsic to some kinds of disks but due to kernel version and driver issues in other cases * action item: Julia/Arun to write text re recommendation for disk purchases (NB: any type can be handled at the Cornell CTC). - Fast networks: Canarie: Canada (Joeri) successful tests between UBC and the Netherlands ~1 TB/3hr transfer rate National Lambda Rail: US (Cornell is the Northeast Lambda Rail host and will have fast data pipes going to NYC and to Chicago via Syracuse; Cornell has bought in early at US$1M/yr). The Northeast loop will be commissioned ~ Jan 1 '06. * Cornell, UVa on the NLR. UCB also? who else within PALFA? 8. observing in the next two months: 21 Dec Joel 22 Fernando 23 Scott 24 David C. 26 Joeri 19-22 Jan Wouter (Jim?) 2-7 Feb Froney + students 9. Setting up observing: - need a web page for setting up schedule to enable volunteering - similar (same?) web page to account for quick-look inspection 10. To quick look or not: - We decided that we should continue to use the ASP/QL capability for (a) data integrity and (b) rapid detection of known/new pulsars - it is the observer's responsibility to run QL and scan the output - while some argued that it was less important for the outer Galaxy, JMC points out that we discovered two anticenter pulsars with the QL analysis and the data integrity issue applies to all data - we should all use the same ID in the QL inspection so that we can tell which scans have been looked at 11. Detection rates: ok? (Fernando) - need to analyze QL output to assess detection rates - vary with time according to episodic RFI? due to equipment? - action items: * assess S/N of QL output on known pulsars to quantify goodness of survey (Fernando) * compare quick look and full-resolution output (e.g. Cornell analysis output) to check scalings, data quality, etc. 12. Workshop at McGill: - to discuss code pipelines and compare output from various codes - tentative dates: 6-7 March '06 13. Next telecon: 23 Jan (Monday) 1pm EST Note: this is Martin Luther King day in the US; not at holiday at Cornell; problem for some? ACTION ITEMS: 1. identify calibration protocol and code that can be used for Stokes timing; write up procedures for use by observers. Seek input from Bhat, Rankin, Han (JMC + Joel) 2. send JMC and Julia (and cc others) anecdotal + quantitative evidence for disk problems, either while writing at AO or reading elsewhere; or at GBT, etc.; we need to develop a record of problems (ALL). 3a. how much RAID to buy, how to configure at AO? (JMC to ask Arun) (e.g. RAID connected to non-WAPP, non-ASP machine, but with GBE connections to ASP) 3b. contributions to RAID purchase (Joel, Fernando, JMC offers so far) 4. get database entries for all data on CTC robotic tape archive (Julia, JMC) 5. recommendation for disk purchases based on reliability and compatibility (Julia, Arun) 6. investigate use of Canarie, NLR for data transport (Joeri, Ben, JMC, Scott ...) 7. Analyze S/N, of quicklook output on known pulsars to quantify goodness of survey; also discovery rate vs. epoch for both known and new pulsars (volunteers needed for this). 8. Analyze/compare quicklook output and full-resolution output for common, processed data sets (Julia, JMC) 9. Develop scheme for choosing and tracking data to be shipped to processing sites either from AO or CTC (Arun, JMC, Fernando, others) (using dbs at AO and CTC) 10. Ask Arun if he wants to be part of PALFA for publications, etc. (JMC) 11. Send Arun a summary of the temporary data storage issue as we see it and ask him about the RAID solution and mechanisms for migrating data from RAIDs to external disks (JMC) 12. Confirm McGill meeting dates and organize (Vicky) 13. Identify meeting time/place at AAS meeting in DC on 12 Jan (JMC) Note 1: the PALFA session ends at 11:30 and Blain's invited talk is from 11:40 - 12:30. Note: The Senior Review panel has requested that NAIC reps meet with them at 2:15pm that same day, so we should either start earlier than 1pm or after about 3:30pm (allowing for SR time slippage). 14. ASAP: Send Fernando material for his PALFA talk at the AAS session on 12 Jan (all)