Inherit from Dask Dataframe classes

[mrocklin]

This allows us to remove a large amount of boilerplate code.

This requires changes upstream both in Dask DataFrame to enable better
subclassing and in PyGDF to better match the Pandas API.

[mrocklin]

See also:

• Improve sub-classability of dask dataframe dask/dask#4015
• Dask gdf fixes cudf#262

cc @kkraus14

[scopatz]

Please remove dead code

[mrocklin]

This isn't dead. I had to comment it out to get tests to pass. I prefer to leave it here for now until we decide what the plan is on this.

[scopatz]

Ok! Can you please put a comment here to that effect. This is something we do in libgdf etc for "future code"

[mrocklin]

I'm happy to actually fix the test. Sorry, I should have mentioned above that this PR is a work-in-progress. I expect that I'll end up doing more changes before it's done. I've pushed up the first round of PRs just so that folks can take a look at what I'm proposing. I don't think that the time has come for scrutiny on the level of style or comments.

[mrocklin]

So, some reasons not to do this:

1. dask-gdf becomes more strongly tied to upstream
2. we'll need to implement NotImplemented methods to cover functionality that we don't implement. These classes just got a ton of new methods, many of which don't currently work.
3. ... ?

[kkraus14]

So, some reasons not to do this:

1. dask-gdf becomes more strongly tied to upstream
2. we'll need to implement NotImplemented methods to cover functionality that we don't implement. These classes just got a ton of new methods, many of which don't currently work.
3. ... ?

1. Given that we're already heavily tied to dask and distributed and will likely be tied very tightly to upstream as we add functionality to dask and distributed for GPUs I don't see this as an issue.
2. I'm okay with this and it will give us additional functionality targets that we should have anyway. That being said there is some places we diverge from the Pandas API and those situations would need to be handled: multi-index, additional parameters for things like Joins / GroupBys, etc.
3. I'd be +1 in favor of extending Dask.DataFrame to work with additional DataFrame classes underneath.

[mrocklin]

OK. I'll work on cleaning up things on the dask side and I'll add a couple checks to the pygdf side. Would we want to merge this if both of those go in? That would require us to wait on a release until the next time dask-core releases (probably at least a week or two away) and would pin us above that version for the future.

[kkraus14]

Does dask-core publish nightlies to Conda? Given the "alpha" state of dask-gdf I'd be happy to have a dependency on pre-release Dask versions as we've done with Numba in the past for PyGDF as we identify functionality that should be added or bugs get fixed.

[mrocklin]

Does dask-core publish nightlies to Conda

No, but it's pure Python, so we typically pip install git+https://github.com/dask/dask

[kkraus14]

I think relying on master until needed changes / features make it into a release is okay at this point.

[mrocklin]

OK. I'll add a new line to the CI system in this PR once other PRs finish up.

…

On Wed, Sep 26, 2018 at 11:34 PM Keith Kraus ***@***.***> wrote: I think relying on master until needed changes / features make it into a release is okay at this point. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszJ5U4YSYai1nJJAya4ggHC_iz9IPks5ufEcsgaJpZM4W7cuD> .

[mrocklin]

Closing in favor of #48