incorporating ground control points

[joemcglinchy]

Hello, is it possible to incorporate ground control points, either directly via a list of lon/lat + image coordinates or a known base layer, into the processing? I'm looking to see if there's a way to improve the georeferencing of the resulting DSM, or if I need to improve the georeferencing of the input image RPCs prior to running through CARS. I couldn't find anything specific to that in the documentation, so please forgive me if it is located there somewhere!

[dyoussef]

Hello @joemcglinchy,

In the case of 3D restitution with more than two images, we have developed a script called “cars-bundleadjustment” that allows the models in the images to be made coherent with each other. The script is available by installing (there is documentation via the -h option) but no reference in the readthedocs documentation at this time (WIP).

I get the impression that what you need is more of an adjustment in comparison to an external reference. We don't have that in CARS yet: however, if it is indeed a misalignment compared to another reference MNS, you can try using XDEM (https://github.com/GlacioHack/xdem). The software is developed by Glaciohack in collaboration with CNES. It includes the “Nuth and Kaab” method, which allows you to estimate the planimetric offset. Then the difference between the two MNS would only be an altimetric bias. XDEM is installed with CARS, so if you have the latest version, you already have XDEM.

Regarding the documentation: it is very comprehensive: you can find it here: https://xdem.readthedocs.io/en/stable/ : If you have any questions about using XDEM, feel free to ask them in this thread. One of the XDEM developers will be able to answer your questions 👋 @belletva, @rhugonnet @adehecq, @adebardo

It is important for CARS to realign the DSM with an external reference: I think we will do a short tutorial on this in the future, perhaps based on the discussions with you in this thread.

Regards,
David

[joemcglinchy]

Hi @dyoussef , thank you for all this information. That's great! I definitely agree a bundle adjustment is going to help, i was not aware of the utility. I essentially added in that block to my configuration file, and ran with cars-bundleadjustment config.json, is that how you meant to run it? when doing that, I see this error:

Processing Tiles: [ epi_matches_left ] ...: 100%|█████████████████████████████████████████████████████████████████████████| 320/320 [41:05<00:00,  7.70s/it]
Processing Tiles: [ epi_matches_left ] ...:  67%|████████████████████████████████████████████████▊                        | 214/320 [28:04<18:53, 10.70s/it]Processing Tiles: [ epi_matches_left ] ...: 100%|█████████████████████████████████████████████████████████████████████████| 320/320 [44:33<00:00,  8.35s/it]
Processing Tiles: [ epi_matches_left ] ...: 100%|█████████████████████████████████████████████████████████████████████████| 325/325 [53:29<00:00,  9.88s/it]
pair: ['one', 'two']: 2478056 matches
pair: ['one', 'three']: 1338554 matches
pair: ['two', 'three']: 2630010 matches
Traceback (most recent call last):
  File "/home/jmcglinchy/miniconda3/envs/pycrown/bin/cars-bundleadjustment", line 8, in <module>
    sys.exit(cli())
             ~~~^^
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/cars/bundleadjustment.py", line 707, in cli
    cars_bundle_adjustment(**vars(args))
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/cars/bundleadjustment.py", line 598, in cars_bundle_adjustment
    refined_rpcs = new_rpcs_from_matches(
        conf_as_dict["inputs"]["sensors"],
    ...<2 lines>...
        **bundle_adjustment_config,
    )
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/cars/bundleadjustment.py", line 447, in new_rpcs_from_matches
    os.path.join(config_directory, sensors[key]["geomodel"])
                                   ~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'geomodel'

my config.json does not have a geomodel key, do i need to add the RPC*.xml filepath from the pleiades data bundle for each image within the config.json? I had assumed it to grab the RPCs directly from the image path using the DIM*.xml file, which is how I've been generating DSMs previous to now, after seeing this in the docs for basic configuration:

geomodel: If the geomodel file is not provided, CARS will try to use the RPC loaded with rasterio opening image.

[dyoussef]

Hi @joemcglinchy, I think there is a small bug that we need to fix (the cars-bundleadjustment code is still a bit ‘experimental’).

Could you try putting “geomodel”: ‘path to your image’? Then CARS will try to use the RPC loaded with rasterio opening image. If that works correctly, I will push a change for the next version so that it is done automatically in the code.

[joemcglinchy]

Absolutely, thanks for the tip. I'll report back!

[joemcglinchy]

@dyoussef got just a touch further i think:

Processing Tiles: [ epi_matches_left ] ...: 100%|█████████████████████████████████████████████████████████████████████████| 320/320 [40:08<00:00,  7.53s/it]
Processing Tiles: [ epi_matches_left ] ...: 100%|█████████████████████████████████████████████████████████████████████████| 320/320 [43:51<00:00,  8.22s/it]
Processing Tiles: [ epi_matches_left ] ...: 100%|█████████████████████████████████████████████████████████████████████████| 325/325 [42:27<00:00,  7.84s/it]
pair: ['one', 'two']: 2478056 matches
pair: ['one', 'three']: 1338554 matches
pair: ['two', 'three']: 2630010 matches
Traceback (most recent call last):
  File "/home/jmcglinchy/miniconda3/envs/pycrown/bin/cars-bundleadjustment", line 8, in <module>
    sys.exit(cli())
             ~~~^^
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/cars/bundleadjustment.py", line 707, in cli
    cars_bundle_adjustment(**vars(args))
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/cars/bundleadjustment.py", line 598, in cars_bundle_adjustment
    refined_rpcs = new_rpcs_from_matches(
        conf_as_dict["inputs"]["sensors"],
    ...<2 lines>...
        **bundle_adjustment_config,
    )
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/cars/bundleadjustment.py", line 451, in new_rpcs_from_matches
    matches_df = estimate_intersection_residues_from_matches(
        geomodels, matches_df
    )
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/cars/bundleadjustment.py", line 117, in estimate_intersection_residues_from_matches
    matches_dataframe[["col_" + key, "row_" + key]].values,
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/pandas/core/frame.py", line 4108, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/pandas/core/indexes/base.py", line 6200, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jmcglinchy/miniconda3/envs/pycrown/lib/python3.13/site-packages/pandas/core/indexes/base.py", line 6249, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['col_three', 'row_three'], dtype='object')] are in the [columns]"

if it's a simple fix I can try to plug into the installation on my end, seems maybe an indexing/off-by-one error, but i say that without digging into it.

[dyoussef]

Hi @joemcglinchy,

if it's a simple fix I can try to plug into the installation on my end, seems maybe an indexing/off-by-one error, but i say that without digging into it.

Exactly! You need to remove the pair [“one”, “three”] from the configuration: since the "key" already exists, the match merging crashes. I need to find a way to fix this bug.

[joemcglinchy]

perfect! Thanks @dyoussef the cars-bundleadjustment utility ran through and I could produce a set of config.json files with updated geometry models. I assume those .geom files will override the RPCs in the DIM*.xml files that define the image. I am noticing some improvement in the georeferencing of the resulting DSM, but still several meters off from identifiable features in previous data for the area I'm working in.

I'll take a look at xdem next to see if we can align the tristereo surface to some of our existing data. Am I correct in understanding that may be the preferred approach, outside of a separate workflow to improve the RPC's geolocation through ground control points from another data source? seems like we can either:

1. use xdem to 'realign' the tristereo surface to an existing surface of known higher spatial accuracy, or
2. use a GCP workflow outside of the CARS framework to try and improve the initial RPC geolocation and re-run cars-bundleadjustment+cars on those resulting files, and hope 😄 it's closer. It does not seem like there is a way to input pixel coordinates + longitude/latitude(/elevation) coordinates into the bundle adjustment at this stage (please correct if wrong there!)

[dyoussef]

cars-bundleadjustment executable only makes the (geo)models coherent. I think that if you use it after trying to add GCPs, it may find a different solution (and therefore a worse "absolute" fit). I'll see about doing a full tutorial: what do your GCPs look like? to find out how I could adapt the script.

[joemcglinchy]

Maybe this is something to be done outside of this toolset, then? It sounds like it makes sense to run the bundle adjustment as currently implemented and try to incorporate GCPs post-hoc. I think the GCPs could be organized something like below and then passed to rpc_fit in a similar way the bundle adjustment uses rpc_fit to update each geomodel. This would require good knowledge of these pixel locations and corresponding lon/lat/alt triplets, and they'd likely need to be enough of them to span the image domain.

lon	lat	alt	row_1_init	col_1_init	...	row_init_N	col_init_N	row_1_gcp	col_1_gcp	...	row_N_gcp	col_N_gcp

• lon: longitude of GCP
• lat: latitude of GCP
• alt: altitude of GCP (referenced to ellipsoid for RPCs)
• row_1/N_init: initial row in image for camera 1 (up to N cameras) for lon/lat/alt triplet (incorrect / estimated pixel location)
• col_1/N_init: initial column in image for camera 1 (up to N cameras) for lon/lat/alt triplet (incorrect / estimated pixel location)
• row_1/N_init: initial row in image for camera 1 (up to N cameras) for lon/lat/alt triplet (true pixel location of triplet)
• col_1/N_init: initial column in image for camera 1 (up to N cameras) for lon/lat/alt triplet (true pixel location of triplet)

[dyoussef]

I'm going to try to develop something before the end of August. We'll be in touch!

[dyoussef]

We haven't developed the feature yet, but we haven't forgotten about it. We'll post a message here when it's developed.
If you have a suggestion for a change, we can also integrate it.

[joemcglinchy]

hi @dyoussef, thank you for the follow up! I have an idea of how it could work, but haven't been able to dedicate time to see if it works well or not. In the current bundle adjustment code, can you remind me, are the off-nadir images essentially adjusted to the nadir image? or is the anchor image determined a different way?