Why can't we download Discrete return LiDAR point cloud from by_tile_aop

[bw4sz]

howdy @cklunch. The current aop_download.py prevents the user from specifying DP1.30003.001 (or DP3.30003.001, just as a check). I'm not sure what about the functionality would prevent the download structure to be limited here. Was trying to upgrade from our old source code to this.

[feature request] crop the tile by the input bounds for the data product, not just download it.
[feature request] nu.by_tile_aop requires a 'year' argument, it would be nice if it iterated over available years, docs suggest this possible in other functions.

thanks!

[bw4sz]

I wasn't able to make this really work, I think its all there, but couldn't get it integrated cleanly.

xmin
np.float64(542494.8049999989)
ymin
np.float64(4136741.686999999)

nu.by_tile_aop(dpid=id, 
                site=site, 
                easting=xmin,
                northing=ymin,
                year=2023,
                token=read_neon_token(),
                include_provisional=True,
                check_size=False,
                savepath=os.path.join("/orange/ewhite/NeonData/", site),
                verbose=True,
                )
Warning, the following coordinates fall outside the bounds of the NEON site, so will not be downloaded:
542000,4136000
Downloading 2 NEON data files totaling approximately 324.0 MB

at MLBS, it says that is outside the bounds, but I know thats not true.

2023_MLBS_6_542000_4136000_image.tif
2023_MLBS_6_542000_4137000_image.tif

both exist on the portal.
Also, if it is outside of the bounds, what is it downloading?

[feature request] return the path of objects downloaded, its difficult to debug if you have to guess at what was/wasn't downloaded. Not clear to me if it skips download if it exists (probably), verbose=True doesn't quite provide enough information.

[cklunch]

@bw4sz Thanks for the alert about this!

On the feature requests - we've typically kept a little extra friction in the system for AOP downloads, to make it slightly harder to download huge amounts of data that aren't actually needed. That's why 'year' can only be a single year in the AOP download functions, unlike in the tabular data downloads. But I'm open to reconsidering.

Download by area, rather than by coordinates, is on the list of possible improvements, and it would make sense to incorporate a cropping option at that point. I'm not sure what the timeline will be on that, there may be some back end changes coming in the API that would make this more straightforward.

I think the Lidar point cloud is an oversight, it's the only Level 1 data product available in tiles. @bhass-neon can you take a look at that, and at the error Ben is getting in the MLBS download?

[bw4sz]

Sounds good.

[bhass-neon]

@bw4sz @cklunch - looking into these AOP download errors, thanks for catching! The tiled lidar point cloud (DP1.30003.001) is an oversight. I think the bounds issue with the camera files is because the file names for camera are different than from our other products (a known issue for a long time) ... I will fix! and will add the camera and L1 lidar to our tests.

[cklunch]

@bw4sz Fixes are in from @bhass-neon . We'll probably put in a few more tweaks before the next push to PyPi, so for now the fix is only on GitHub. Give it a try and let us know if you run into any problems!

[bhass-neon]

@bw4sz - following up on @cklunch 's comment, fixed these issues in #21. In the MLBS example you shared, the tile should actually be downloaded in the DP3.30010.001\neon-aop-products\2023\FullSite\D07\2023_MLBS_6\L3\Camera\Mosaic folder, but the warning message was incorrectly saying that the tile was out of bounds. We also included the [feature request] to return the path of objects downloaded for verbose=True. The AOP download functions download all data and metadata files that match the input criteria, so in the example you showed, it found one data tile and one README file, which is why it said 2 files. In the next version, verbose will print out the files and paths that will be downloaded.

As for your other feature requests, currently it's not possible to download only a portion of a tile, but as Claire mentioned, if NEON makes back-end changes to the API, this could potentially be an option. Would download by area (to download all tiles intersecting a polygon instead of requiring eastings and northings as inputs, e.g.) be of interest? We left this out for now in order to keep neonutilities more light-weight (especially given common gdal conflicts with different versions of Python), but have considered this as an option or as part of a separate AOP-focused package.

Thanks again for reporting these issues!

[bw4sz]

thanks, i'll have a try this week.