Summary:

• USB3 is 2.5 times faster than USB2
• The i/o rates for NTFS and XFS were similar
• NTFS has way more cpu overhead than xfs
• The benchmark  108 MBytes/sec approached the maximum sustained i/o rate quoted by western digital:123 MBytes/sec

12mar15: wombat disc array. can it  handle usrp clp datarates?

    The wombat server has:

• model name      : Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
• the 2680 is an 8core 20Mbyte cache chip. Wombat has 2 chips (16 cores in total, 32 if you count hyperthreading).
  
• 8x4tb toshiba (MG03ACA400) disc hardware raid. 22 Tbytes available raid level 6. stripe=256K. xfs filesystem

• The program opens the file with O_SYNC so the program should wait for each rec to be flushed to disc.
• Kb =1e3, Mb=1e6,Gb=1e0 (not 2^n)
  

Summary:

• The array can do > 350 Mb/sec for recsizes >= 2Mbytes.
• The 200Gb file is greater than the 128Gbytes of memory.
  
• The times are the same as the 20Gb file, so i don't think the cache is skewing the measured values.
• # copies=2 --> 2 separate copies of the benchmark were run simultaneously.
• The dual copy versions gave about 1/2 the single copy rates
• So the single copy rate is probably the disc i/o rate
• The single version 200Kb reclen version had a much lower rate: 286Mbyes/sec
• The average time for each of these  records was 700 usecs.
  
• The decreased rate may come from the kernel overhead of setting up the write and then waiting for completion
• The dual version 200Kb rate did not decrease.. but it was probably running on 2 separate cores
• (1): For this test i ran:
• 1  of the disc benchmark (20Mb reclen) and rsync
  
• 2 copies of the disc benchmark (200Kb reclen) and  rsync.
  
• The rsync:
• rsync -av   inpDir outDir   both on the raid . inpdir was about 660 Gbytes.
  
• This was to emulate a processing program reading and then writing the processed data back to the raid while data was being taken.
  
• rsync i/o rate
  
• with the single disc benchmark: 50Mbytes/sec (input and then output)
• with the dual disc benchmark : 44 Mbytes/sec (input and then output).
• How much of the aeronomy  usrp recording can the raid handle?
• at 100Mb/sec usrp rate for 1 beam:. this raid can simultaneously handle writing both the dome and the ch data.
• It can probably also simultaneously process the data via the gpu.
  

20mar15: PNY 128 Gb flash drive i/o rates

• dell xps-15 laptop. 2Ghz, quad I7, 8gb memory, usb ports
• File system:
• ntfs (i used windows 7 to format the file system)
• OS: centos 7 (linux)
• I ran 2 tests:
• Note ( 1Mbyte is 1,000,000 bytes not 2^20)
  
• 300 20Mbyte records (6Gb)
• 600 10Mybte records (6Gb)
• Using clock_gettime, (with the real time clock) i measured the time for each record
  

• The black line shows the 20Mbyte reclength test. the red line shows the 10Mbytet reclen test.
  
• both record lengths go at 170 to 190 Mbytes/sec until about 1.2 Gbytes.
• The i/o rate after 1.2Gbytes drops to about 40 Mbytes/second.
• I had set the OSYNC option, so the program should wait for the record to clear the cpu buffer cache.
• Looks like drive has an internal buffer.

ntfs partition table (parted)

Discwr2: disc write benchmark

    The discwr2 benchmarck write N M byte files to disc. It uses the OSYNC option so the program should wait for data to be written to disc before returning. The time for each record write is recorded. After all records are written, the min,max,rms, and avg i/o time is computed.

the table below holds the results from various measurements:
The devices are:

• evo 840 ssd 1tb drive. This is the internal disc for the philpc laptop
• flash memory sticks
• SanDisk extreme 64 Gb  drive
• PNY 128Gb drive. Further investigations showed that this drive probably has a 1Gb buffer in it. After 1Gbyte the speed goes from 160Mb/sec to around 38Mb/sec.

• vfat only allows up to 4Gb  files.  
  

<- page up
home_~phil