By default, the script wakes up 3 minutes before the first (and any `STOPSTART`) observations to configure the telescope into software level 3. Ensure you consider this time when scheduling observations.
Each entry roughly follows the format ``` # Generic Beamformed Target YYYY-MM-DDTHH:MM - YYYY-MM-DDTHH:MM :The `transfer.sh` script depends on the `sleepuntil.sh` script located in the same folder, if you make a copy of `transfer.sh` to modify the target directory, be sure to copy `sleepuntil.sh` to the same folder to ensure it can run.
The transfer.sh script takes 3 inputs: the path of the script to transfer, the date to transfer (as YYYYMMDD) and the time to start trying to transfer (HHMMSS). As an example, to transfer a script for handover occurring at 2022-02-02T08:00:00 the following command could be used to start trying to transfer file 30 minutes prior to handover. In the case that we receive the station early, this will allow for the file to be transferred and station to be configured as soon as possible. ```shell ilofar@LGC ~/David $ tmux new -s transferLCuScript < enter tmux shell> ilofar@LGC ~/David $ bash transfer.sh lcu_script_name.sh 20220202 073000 < Control d, b to exit > ``` #### Preparing the LGC In the case that Solar/KBT observations are requested, a script, `VerifySolarKBT.py` can be run on the LGC to validate that the KBT system has initialised correctly and BST files are being generated. The required command will be produced based on your input schedule. As an example, an observation that runs from 2022-02-22T12:22:00 to 2022-02-22T16:22:00, will produce a command with one entry, which can then be ran from the standard "obs\_schedule" folder on the LGC. As a reminder on how to find and execute the script, ``` ilofar@LGC:~$ cd Scripts/Python/obs_schedule_py3/ ilofar@LGC:~/Scripts/Python/obs_schedule_py3$ python3 VerifySolarKBT.py -d 2022-02-22T12:22:00,2022-02-22T16:22:00 ``` This script will start up 1 minute after the set start time, and send the standard "KBT OK" or "KBT FAILED" emails to the observers mailing list, and produce end-of-observation plots after the observation has ended. #### Recording on UCC1 Given `ucc1` should always be available, the script can be transferred to the recording drive and executed as needed. We currently perform all recording on the `ucc1_recording2` drive which can be easily accessed via the `cdr2` alias on the `obs` account. After that, you can get the required scripts in the scripts folder, and make a sub-directory for the given source category and perform the observation. ```shell # Connect to ucc1 ilofar@LGC ~ $ ucc1 # Navigate to the recording directory obs@ucc1 ~ $ cdr2 # Make your recording folder obs@ucc1 /mnt/ucc1_recording2/data $ mkdir rrats/2022_02_02 # Copy the recording scripts to the folder obs@ucc1 /mnt/ucc1_recording2/data $ cp ./scripts/* ./rrats/2022_02_02/ # Enter the folder obs@ucc1 /mnt/ucc1_recording2/data $ cd rrats/2022_02_02 # Add your ucc_script_YYYYMMDD.sh script to the folder via rsync, scp, nano + copypaste, etc # Open a new tmux shell obs@ucc1 /mnt/ucc1_recording2/data/rrats/2022_02_02 $ tmux new -s recordingUnless you have used the `-u` flag, the script will give you a prompt to confirm the start and stop times of the observation when you initially run the command. The observation will not start until the prompt has been accepted.
Such a command would look like this: ```shell $ python ScheduleSolarKBT_DMcK_wip.py -e rcu357_1beam_datastream -r ``` In the case you wish to **schedule an observation**, you will need to add the `-d` flag, and optionally start (`-t`) and stop (`-p`) times. ``` $ python ScheduleSolarKBT_DMcK_wip.py -e rcu357_1beam_datastream -r -d YYYY-MM-DD -t HH:MM -p HH:MM $ python ScheduleSolarKBT_DMcK_wip.py -e rcu357_1beam_datastream -r -d 2022-02-22 -t 09:30 -p 14:55 ``` Additionally, if you have to split up the observations, the `-u` flag may be of use, as it will allow the script to run without any input to confirm the start/end times. For example, if there are intermediate observations at 11:00 - 11:30 and 14:15 - 14:45, a schedule\_kbt.sh script could be written like this ```shell python ScheduleSolarKBT_DMcK_wip.py -u -e rcu357_1beam_datastream -r -d 2022-02-22 -p 10:55 python ScheduleSolarKBT_DMcK_wip.py -u -e rcu357_1beam_datastream -r -d 2022-02-22 -t 11:31 -p 14:10 python ScheduleSolarKBT_DMcK_wip.py -u -e rcu357_1beam_datastream -r -d 2022-02-22 -t 14:46 ``` ## Processing the Solar Observations When run with the -r flag, the `ScheduleKBT` script will launch a recording process on `ucc1` to record the raw voltage correlations from the station. Once the observations for local mode have been completed, you will need to process these voltages in order to convert them into Stokes parameters and down-sample them in order to save on space.Do not start processing solar observation until after the station has been handed back to ASTRON. The process is entirely disk limited, and running the processing script while other data is still being recorded will cause packet loss.
This is performed by using software within a Docker container on `ucc1`. Scripts are copied to each Solar recording directory to simplify the process. To start, you will need to connect to `ucc1` and then go to your solar observations. ```shell obs@ucc1~ $ cdr2 # cd to the recording drive obs@ucc1:/mnt/ucc1_recording/data $ cd sun/YYYYMMDD_sun357 # Enter the solar data folder for a given observation ``` Afterwards, you will want to enter a tmux shell and launch the Docker container to get access to the software you will need, your current directory will be stored in $LAST\_DIR to make getting back to it easier. ```shell obs@ucc1:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 $ tmux new -s solarProcessing < enter tmux shell > obs@ucc1:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 $ bash enter_docker.sh < Docker container with processing software will load > root@aaaaaaa:/home/user $ cd $LAST_DIR # Your previous directory was saved in $LAST_DIR, go back to to root@aaaaaaa:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 # ``` You will then be able to run the processing script, which will hand processing the data, then compressing the outputs. The results can then be transferred wherever they will need to be sent to and the container and shell can be closed. ```shell root@aaaaaa:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 # bash process_data.sh # Run the processing script < allow processing to complete, it may take a few seconds for the inital output to appear in ./output_fils/ > < exit the container, tmux shells, notify someone to transfer the data to DIAS > root@aaaaaa:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 # exit obs@ucc1:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 $ exit obs@ucc1 ~ $ exit ``` #### Command Flow Summary ```shell ilofar@LGC~ $ cd ~/Scripts/Python/obs_schedule ilofar@LGC ~/Scripts/Python/obs_schedule $ tmux new -s KBT < enter tmux shell > ilofar@LGC ~/Scripts/Python/obs_schedule $ python ScheduleSolarKBT_DMcK_wip.py -d YYYY-MM-DD \ # Date for scheduled observation -r \ # -r will enable beamforming on ucc1 -e rcu357_1beam_datastream \ # Name of experiment (no need to change) -t HH:MM \ # Start Time (not always needed) -p HH:MM \ # End Time (not always needed) # Optional -u \ # Unattended mode, does not require confirmation of times to start script < Control+b, d to exit tmux shell once things are running as intended > < Allow all observations to complete, return later, exit the tmux shell with `exit` > ilofar@LGC~ $ ucc1 < ucc1 is an alias to connect to ucc1 > obs@ucc1~ $ cdr2 # cd to the recording drive obs@ucc1:/mnt/ucc1_recording/data $ cd sun/YYYYMMDD_sun357 # Enter the solar data folder for a given observation obs@ucc1:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 $ tmux new -s solarProcessing < enter tmux shell > obs@ucc1:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 $ bash enter_docker.sh < Docker container with processing software will load > root@aaaaaaa:/home/user $ cd $LAST_DIR # Your previous directory was saved in $LAST_DIR, go back to to root@aaaaaaa:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 # bash process_data.sh # Run the processing script < allow processing to complete, it may take a few seconds for the inital output to appear in ./output_fils/ > < exit the container, tmux shells, notify someone to transfer the data to DIAS > root@aaaaaa:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 # exit obs@ucc1:/mnt/ucc1_recording/data/sun/YYYYMMDD_sun357 $ exit obs@ucc1 ~ $ exit ``` # (Archived) Executing a Schedule and Processing Data ## Running Observations #### Automatic Configuration on the LGC The observing process has been simplified to only require the use of a schedule file and a single command on the `LGC`. The current working directory can be accessed via the ilofar used on the LGC and running the alias `sched` in the console. The current directory is `ilofar@LGC:~/Scripts/Python/sched_bf/tmp_dmckenna`. Once in the directory, you can create your schedule file as described above, and execute the `scheduleAndRun.sh` with an input file name to automate transferring the scripts to the `LCU` and `UCC1`, and setup the beamforming and and recording processes as needed. This command does not need to be run in a `screen/tmux` and will exit early if there is an issue with the input schedule file. Issues can be debugged using the `quicksched.py` script and the `--no-write` flag. ```shell bash scheduleAndRun.sh sched_YYYY-MM-DD.txt # Run into issues? Run the quicksched.py script separately to debug the file python3 quicksched.py sched_YYYY-MM-DD.txt --no-write ``` #### Manually Configuring the LCUThe `transfer.sh` script depends on the `sleepuntil.sh` script located in the same folder, if you make a copy of `transfer.sh` to modify the target directory, be sure to copy `sleepuntil.sh` to the same folder to ensure it can run.
The transfer.sh script takes 3 inputs: the path of the script to transfer, the date to transfer (as YYYYMMDD) and the time to start trying to transfer (HHMMSS). As an example, to transfer a script for handover occurring at 2022-02-02T08:00:00 the following command could be used to start trying to transfer file 30 minutes prior to handover. In the case that we receive the station early, this will allow for the file to be transferred and station to be configured as soon as possible. ```shell ilofar@LGC ~/David $ tmux new -s transferLCuScript < enter tmux shell> ilofar@LGC ~/David $ bash transfer.sh lcu_script_name.sh 20220202 073000 < Control d, b to exit > ``` #### Recording on UCC1 Given `ucc1` should always be available, the script can be transferred to the recording drive and executed as needed. We currently perform all recording on the `ucc1_recording2` drive which can be easily accessed via the `cdr2` alias on the `obs` account. After that, you can get the required scripts in the scripts folder, and make a sub-directory for the given source category and perform the observation. ```shell # Connect to ucc1 ilofar@LGC ~ $ ucc1 # Navigate to the recording directory obs@ucc1 ~ $ cdr2 # Make your recording folder obs@ucc1 /mnt/ucc1_recording2/data $ mkdir rrats/2022_02_02 # Copy the recording scripts to the folder obs@ucc1 /mnt/ucc1_recording2/data $ cp ./scripts/* ./rrats/2022_02_02/ # Enter the folder obs@ucc1 /mnt/ucc1_recording2/data $ cd rrats/2022_02_02 # Add your ucc_script_YYYYMMDD.sh script to the folder via rsync, scp, nano + copypaste, etc # Open a new tmux shell obs@ucc1 /mnt/ucc1_recording2/data/rrats/2022_02_02 $ tmux new -s recording