Improve integration with xarray

[s-kganz]

Hello,

I'm interested in using deepforest to identify alive/dead trees in the Lower Teakettle NEON site. Overall this package is very impressive! I was able to run segmentation and start classifying trees from scratch in about an hour. However, I had two pain points in this notebook that I wanted to bring up:

1. As it stands the deepforest API doesn't work well with imagery held as an xarray DataArray. Functions will either fire a warning, or in the case of predict_tile, fail completely because BoundingBoxDataset expects a non-null image path. I managed to hack a workaround by iterating over crops myself, but I think this is a missed opportunity because xarray works with so many backends. So if deepforest can work with xarray, then it can work with lazy dask arrays, zarr, imagery on cloud storage, STAC, and so on. There's been some success already in linking xarray with PyTorch models here.

2. The alive/dead classifier doesn't seem to give reasonable results. I've tested it all over TEAK and every tree is classified as alive with confidence >99% even though there's some clear gray-stage mortality where I'm running it. I saw the note that the classifier is probably overfitted, but TEAK is in the training data in this repo so I thought it would be ok. Not really relevant to the title but I'm including this here in case there's some gotcha about channel order or image transformation that I'm missing. If I end up having to make my own dataset then oh well.

I'd be happy to contribute changes related to 1, and I'm hoping that there might be something I'm missing related to 2. Thanks for your great work on this package!

[s-kganz]

And just following up on the alive/dead classifier, looks like that model is just overfitted. There's only 16 crops from TEAK in the DeepTreeAttention repo and when I run the model on those it classifies all of them as alive even though there are 3 dead ones. Looks like I have some labeling in my future :)

[bw4sz]

The repo does not hold the data, this is going to take you down an incorrect path.

[bw4sz]

Thanks for opening a discussion, let's run through some points here.

1. Can you tell me a bit about your use case and what you hope to achieve to help orient me? Have a look at Nicole's work at teak. https://scholar.google.com/citations?user=-EirMQYAAAAJ&hl=en&oi=sra

2. Let's make a minimal example with a sample piece of data with xarray and make a feature request. I have played with it in context of dask and parallel generation of crops, but did not have a use case for it. Let's make it happen. Open a new issue, make a minimum example and we will problem solve together. We are working towards 2.0, so now is a great time to add something.

3. "The alive/dead classifier doesn't seem to give reasonable results. I've tested it all over TEAK and every tree is classified as alive with confidence >99% even though there's some clear gray-stage mortality where I'm running it."

This strikes me as a preprocessing error in your pipeline, we know that can't be right, looking at our NEON predictions there are millions of dead predictions in the landscape and the model looks very reasonable. Be careful not to assume its the model, its almost certainly in the way you are feeding the images in.

Let's seperate the xarray and the alive dead model issues, if you are using NEON tiles, open with rasterio and we can make a minimum example. The overfit is in the calibration scores, not the model performance.

I saw the note that the classifier is probably overfitted, but TEAK is in the training data in this repo so I thought it would be ok.

Not quite right, that repo is species classification, and contains no data besides testing. Thousands of dead trees were used, including many from TEAK, looking at DeepTreeAttention, which is a HSI model, will just distract you.

Not really relevant to the title but I'm including this here in case there's some gotcha about channel order or image transformation that I'm missing. If I end up having to make my own dataset then oh well.

This is almost certainty where you went wrong What about the documentation made you need to develop your own dataset structure, let's walk through that together. Overall we've got alot going on in this discussion. Let's make a new discussion on cropmodel and dataset structure (might need to wait till 2.0 and pull from the github repo not pypi, we made it a bit better and faster)

[s-kganz]

Hi again,

Thank you for all the information. My use was to identify live/dead trees over a few years of imagery in TEAK. It turns out that Nicole has already done that, so I will just use those results instead of duplicating effort.

Regarding data structure and the alive/dead model, I forked the repo and went through the setup instructions for the developer's guide. The API in the current version vs. the one on pypi changed such that I think my above comments about data structure aren't relevant anymore. If the model objects accept a data loader then that's enough flexibility to use all kinds of data sources, including xarray. Apologies for distracting, I should have checked the dev version first.