Skip to main content

A Week in the Life of an I-LOFAR Cheif Observer

This is a more simplified version of some of the pages David made.made and is meant to on board new chief observer's (CO) onto. It details the pipeline used to turn voltages to filter-banks, clean and search them before archival on DIAS or Archie. 

Disk Space 

There is a lack of disk space locally in Birr. By default the observing is stored in /mnt/ucc1_recording2/data/observations/. This  is one of the reasons that UCC1 shouldn't be used during observations. Keep an eye on the data rates throughout observation, if there is no space then there is no observations. This can be done through the terminal using df -h . Also there is a script setup to email the observing mailing list when the space on the disk drops below 20%.

The data rates for the observations are as follows, this is a good way to see if you have enough space left for the remaining observation time.

Observation 
 Data Rate (GB/hr) 
BF-ed HBT 
 ~550
KBT (Sun) 
 ~700

This means that for a given full day of observation, about 14 TB is needed, KBT dependent.

NOTE: It is important that the disks remain free, if the disk looks like it will fill before the observations end, please email the CO list. 

Processing 

Each week the collected data needs to be processed. This can be started during obs as it takes place on the other UCC nodes. This step should be started at the latest right after observations end. (usually 3pm on Wednesdays).

Filter-banking & Folding 

First things first is to do some folding on the known pulsars that were observed. To begin the processing you will need to log into either ucc3 or 4 as obs. 

ssh obs@ucc4

The next thing to do is setup a tmux session, these sessions are just terminal instances that persist when you close your local terminal.

tmux new -t processing

Then cd to the directory where you want to generate the filterbanks, currently this is done in /mnt/ucc4_data2/data/David under a sub folder with the observation date. You will need to make a new directory for the most recent observation run, i.e. mkdir 2024_02_09.

Next thing is to load the docker with all the processing software. This is done using dckrgpu command. If successfully loaded you will see the following output. 

:: initializing oneAPI environment ...
   BASH version = 4.4.20(1)-release
:: mkl -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: ipp -- latest
:: mpi -- latest
:: inspector -- latest
:: compiler -- latest
:: ippcp -- latest
:: vpl -- latest
:: dpl -- latest
:: advisor -- latest
:: tbb -- latest
:: ccl -- latest
:: clck -- latest
:: dpcpp-ct -- latest
:: itac -- latest
:: vtune -- latest
:: oneAPI environment initialized ::

Then the cdmtProc.py script needs to be copied to current directory,directory (where the observations are stored), this calls CDMT and digifil and makes shell scripts for non-Solar observations to create filterbanks. 

for i in {0..4}; do python cdmtProc.py -i /mnt/ucc1_recording2/data/observations/20231114T073100/ --extra ${i}; bash cdmtProc_${i}.sh; python ../generatePrepfolds.py; pushd folds; bash fold_${i}.sh & popd; done

You also can add a sleep statement (sleep $delay) such that the processing starts when observations end. This takes about the amount of time it took to take the observation (~1:1 processing time). 

RFI Cleaning 

This is interactive but fast, you also need X display for this. So when you ssh you need to use the -X flag. 

ssh -X obs@ucc4
cd /mnt/ucc4_data2/data/David/..... 
python3.6 ../generateHeimdall.py

After a minute or two (during which your stdout will show some progress bars zipping along, reading filterbanks), if you have X, this will load up a bandpass plot like below, Screenshot 2024-02-12 at 16.02.59.png

Some bad channels are automatically identified and marked in red. You need to mark any others that were missed. You can just click on the channel(s). You can cover a range by clicking on the left side of that range and then moving your mouse right and typing ‘a’. This will zap every channel between the click and the point where you clicked ‘a’. Note this works left to right only, not right to left! When you are happy with your zapping just click the x on the window to close it. You will keep getting fed such bandpasses until you’ve gone through all the filterbanks. 

 

Back to top