pre_average step non-descript error

[SimonP2207]

Hi,

I'm having trouble with a rather non-descriptive error thrown during the pre_average step by python_plugin which I have copied below. It only occurs during some facets. My dataset is multi-epoch (3 nights) and I removed the highest frequency band's .MS on one of the epochs.

2020-06-12 16:23:02 ERROR   node.lofar.ap.dias.ie.python_plugin: '0-58'

The supporting 'failed on localhost' message is:

2020-06-12 16:23:02 ERROR   facetselfcal_cc6691_patch.executable_args: Remote process python /opt/lofarsoft/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py ['[/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk0.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk1.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk2.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk3.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk4.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk5.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk0.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk1.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk2.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk3.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk4.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk5.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk2.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk3.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk4.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk5.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk6.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk7.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk8.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk9.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk10.shift_cal]', '/home/purser/.local/lib/python2.7/site-packages/FACTOR-2.0-py2.7.egg/factor/scripts/pre_average_time.py', ['[/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk0.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk1.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk2.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk3.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk4.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689452_SB001_uv_12D1EFC53t_121MHz.pre-cal_chunk5.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk0.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk1.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk2.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk3.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk4.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L689466_SB001_uv_12D19BAEFt_121MHz.pre-cal_chunk5.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk2.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk3.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk4.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk5.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk6.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk7.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk8.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk9.shift_cal,/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_121MHz.pre-cal_chunk10.shift_cal]', 'DATA', 'DATA', 'WEIGHT_SPECTRUM', '0.5', '/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch/L751489_SB001_uv_12EE70D56t_168MHz.pre-cal_chunk10.calc_bl'], {}, '/mnt/purser_data/LOFAR_Data/LC11_004/S255IRS3/MSs/all/factor_run/results/facetselfcal/cc6691_patch', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'gnu', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'LOFARDATAROOT': '/opt/lofarsoft/data', 'OMP_NUM_THREADS': '8', 'PYTHONPATH': '/opt/lofarsoft/lib/python2.7/site-packages:/home/purser/Dropbox/SimonsPythonModules', 'LOFARROOT': '/opt/lofarsoft', 'PATH': '/opt/lofarsoft/sbin:/opt/lofarsoft/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/sbin:/bin:/home/purser/.local/bin', 'LD_LIBRARY_PATH': '/opt/lofarsoft/lib:/.singularity.d/libs:/home/purser/.local/miriad/linux64/lib:/usr/local/lib'}] failed on localhost

(apologies for the lengthy error)

Do you have any idea what could be causing this?

Many thanks,

Simon.

[darafferty]

Indeed, the error message is rather unhelpful, though I suspect it is related to the missing MS file. Does the L751489_SB001_uv_12EE70D56t_168MHz.pre-cal* MS file referenced in the error message exist, or was this the one that was removed?

One quick workaround is to disable the preaveraging step by setting preaverage_flux_Jy = 0.0 under the [calibration] section of the parset. This might reduce the SNR of the solutions a bit, but probably won't make a huge difference.