Calibrator in selfcal and facet images

[duyhoang-astro]

I have a bright calibrator in the field which should be subtracted properly with direction-dependent gain. The calibrator in the selfcal images looks fine, but not ok in the facet image. As I would expect they are the same since they are corrected with the same solutions. Would it be something wrong during the transferring solutions to the facet data? The other sources in the facet image seem not to be properly deconvolved either.

(left is selfcal calibrator, right is calibrator in facet image, the two images have similar resolutions, color scale is the same for both images)

Factor branch: master (current version)
LOFAR version 2.9
wsclean version: 2.5

[rvweeren]

The other sources in the facet image seem not to be properly deconvolved either.

I think the deconvolution is -not- a real problem. Currently it will not fully deconvolve because of the artifacts. If the artifact problem (suggesting sort of an 'applycal' / correction issue) is solved I suspect the deconvolution will work fine.

The last image of the calibrator in the DDE selfcal cycle should just be very similar to the full facet image with that calibrator. After all the solutions do not change anymore for that facet and the data should be the same as well, except that some additional sources are present (apart from some different data averaging settings, but that should not change how the calibrator source looks, otherwise there is something wrong...)

[darafferty]

I suspect something has gone wrong when the selfcal solutions are merged. Duy, can you send me the *.merge_selfcal_parmdbs and *.convert_merged_selfcal_parmdbs parmdbs in the results/facetselfcal/facetname directory?

[duyhoang-astro]

Sure, here is the link to the parmdbs: https://owncloud.strw.leidenuniv.nl/index.php/s/0NUJ0fHlg5atpWc

[darafferty]

Indeed, there is a difference between the original selfcal solutions and the merged ones (in which all effects have been combined into a single gain). See below: left is merged, right is original -- there is an offset of 0.5 radians or so between them. I'm trying now to figure out where the offset is coming from...

[darafferty]

Nevermind -- I was confused about which frequency parmdbplot.py uses to convert TEC to phase. Once I plotted it correctly, the two are identical. So, still searching...

[rvweeren]

David, do you I in general observe that the calibrator in the facet image roughly has the same quality as in the last DDE selfcal image?

Wondering if the averaging is too aggressive..as I notice some sort WSClean aliasing issue? Or there is a mixup with calibration tables between blocks...? Renormalization of gains going wrong? (error pattern in the image sort of suggests something with the slow-gain solutions not being the same between the last DDE selfcal image and the facet image)

[darafferty]

Yes, usually the calibrator and facet image are very similar, but I have seen a few cases where the facet image looks worse (though not as bad as this one).

I see what you mean about the aliasing (the artifacts to the north and south of the bright source, right?). The averaging can be controlled with the max_peak_smearing parameter (under the [imaging] section of the parset), so it might be worth experimenting with that.

At any rate, I agree that the artifacts suggest something's wrong with the amplitudes but they look fine in the parmdbs that Duy provided. I'm checking them now in more detail to see if they're really the same.

[rvweeren]

I see what you mean about the aliasing (the artifacts to the north and south of the bright source, right?).

Yes.

[duyhoang-astro]

The averaging can be controlled with the max_peak_smearing parameter (under the [imaging] section of the parset), so it might be worth experimenting with that.

Below are the images of the calibrator in the selfcal (left) and in the facet image (right). The first row is from the run with max_peak_smearing = 0.15 (default value), the second row is from the run with max_peak_smearing = 0. The results are similar (beside very small difference in the statistics within the calibrator region).

Here is the full facet image including the facet layout.

[twshimwell]

And you think this is calibration rather than deconvolution? What do the model images and masks look like for the selfcal and facet imaging steps?

[rvweeren]

Wondering about the normalization, since this quite of looks like an amplitude normalization error. David you do not calculate these normalization factors separately per timechunk (or freqblock)?
(because that will certainly lead to problems....)

[duyhoang-astro]

And you think this is calibration rather than deconvolution? What do the model images and masks look like for the selfcal and facet imaging step?

Tim: I did a test with the deconvolution: multiscale clean as we discussed. Below are the model of the calibrator in the selfcal and facet images without and with multiscale clean. The model of the calibrator in the facet model using single scale clean seems not to be deconvolved properly. It has just a few clean components. But with multiscale clean, there are more clean components in the facet model (close to the model in the selfcal model). And the facet image with multiscale clean seems to be a bit better. But still lots of artifacts, maybe something else causing the problem too. Maybe I should try to add a proper mask, as this facet cleaning uses the entire facet mask?

David: is it possible to input a mask in Factor to replace the facet mask (*premask)? I can add CASA region file in the direction file. But I am not sure if it also accepts CASA mask. Another thing is that the the multiscale options (e.g. selfcal_multiscale_scales_pixel = [0, 7]; facet_multiscale_scales_pixel = [0, 7]) in the factor parset do not seem to propagate through the pipeline when it generates the selfcal parset (results/facetselfcal/facet_patch_163/pipeline.parset). I had to change in the factor/pipeline/parsetsfacetselfcal_pipeline.parset. (I am not sure if this is due to my local environment settings though)

[darafferty]

David you do not calculate these normalization factors separately per timechunk (or freqblock)?

Yes, the solutions for all the time and freq chunks are combined before doing the normalization. However, there is a step in which the amplitudes are all set to 1.0: the one that makes the "preapply" parmdb that is preapplied to all the other directions before selfcal. I wonder whether it could be the one used in facet imaging for some reason instead of the normal parmdb? Duy, can you check the contents of results/facetimage/facetname/mapfiles/expand_merged_parmdbs.mapfile? It should have the *.convert_merged_selfcal_parmdbs file and not the *.create_preapply_parmdb one.

David: is it possible to input a mask in Factor to replace the facet mask (*premask)?

No, unfortunately it only accepts a casa region file.

One other thing you might try is to use the other facetimage pipeline by setting automask_facet_image=False (or True if you already have it as False). This should help to rule out the possibililty that the deconvolution is the issue, as the two facetimage pipelines use different ways to do the masking (automasking vs. pybdsf).

Another thing is that the the multiscale options (e.g. selfcal_multiscale_scales_pixel = [0, 7]; facet_multiscale_scales_pixel = [0, 7]) in the factor parset do not seem to propagate through the pipeline when it generates the selfcal parset (results/facetselfcal/facet_patch_163/pipeline.parset).

I'll check the multiscale options. Must be a bug...

[duyhoang-astro]

I wonder whether it could be the one used in facet imaging for some reason instead of the normal parmdb? Duy, can you check the contents of results/facetimage/facetname/mapfiles/expand_merged_parmdbs.mapfile? It should have the *.convert_merged_selfcal_parmdbs file and not the *.create_preapply_parmdb one.

Yes, it does. The results/facetimage/facetname/mapfiles/expand_merged_parmdbs.mapfile contains *.convert_merged_selfcal_parmdbs, not the *.create_preapply_parmdb one.

[duyhoang-astro]

One other thing you might try is to use the other facetimage pipeline by setting automask_facet_image=False (or True if you already have it as False). This should help to rule out the possibililty that the deconvolution is the issue, as the two facetimage pipelines use different ways to do the masking (automasking vs. pybdsf).

Do you mean that setting automask_facet_image=False to make the facet image that is in the, e.g., facetimage_c1.5r-0.25t0.0u200.0? And this will use pybdsf to make a mask instead of automasking with wsclean?

This option, automask_facet_image=False, is not used for imaging the facet during the selfcal, right? As I tried this option, but the automask is still on in the results/facetselfcal/facet_patch_xxx/pipeline.parset:

wsclean_image_full.argument.auto-mask = 3
wsclean_image_full.argument.auto-threshold = 1.0
wsclean_image_full.argument.local-rms-window = 50
wsclean_image_full.argument.local-rms-method = rms-with-min