Decouple from Django/Flask

[jvanasco]

this looks like an interesting improvement to celery , but the django/flask requirement is annoyingly limiting.

since the only thing you rely on from these packages is their cache, have you considered using a standalone caching library like Dogpile ( https://pypi.python.org/pypi/dogpile.cache ) ?

[winhamwr]

Hello!

the django/flask requirement is annoyingly limiting.

I certainly agree. I would much prefer a solution that works easily with Django and Flask, but that doesn't require them. I would absolutely like to support a general python caching option, if one exists.

Looking at Dogpile, it appears to be a specialized caching library. Even so, if Dogpile is what you're currently using and you'd like to use Jobtastic, then I'd be very happy to accept a pull request that either allows the necessary hooks to add support, or adds it as a specific option.

Thanks
-Wes

[winhamwr]

It looks like at least one other person shares your interest via #36. How do you feel about the Beaker library? I'm still interested in Dogpile, if that's your preferred alternative, but Beaker might kill two birds with one stone.

[jvanasco]

Beaker is bad and in maintenance mode, with a recommendation to update.

Dogpile is basically Beaker 2.0, without cookie support. It was written by one of the two core authors.

One of the roadblocks in trying to write a patch ties into issues #8 and #10 -- it looks like you're just piggybacking onto the cache namespaces of Django and Flask, and expecting the application to have set everything up. I was having a hard time wrapping my head around a good way to write a patch.

That gets really messy when thinking about how to support other frameworks. I think it would be cleaner if you had something like a 'CachingApi' object that would get instantiated on setup, then just proxy requests through that. If someone is using Django/Flask, it might behave in a similar way. But if someone is using Pyramid,Bottle,CherryPy,Tornado,etc/etc/etc, the library could set up a dogpile cache and send requests into it.

[winhamwr]

Dogpile is basically Beaker 2.0, without cookie support. It was written by one of the two core authors.

Ah, cool. Thanks for the insight. I have an (apologetically) Django-centric knowledge of python web development, since Django and Flask have met all of my needs.

it looks like you're just piggybacking onto the cache namespaces of Django and Flask

Yup. The current behavior is absolutely non-ideal in that regard. It's nice from the perspective that if your Django or Flask app is already working with Celery, you have no additional configuration, though.

I think it would be cleaner if you had something like a 'CachingApi' object that would get instantiated on setup

Ultimately, I think that is the best idea. By default, I do think we should follow the advice of #10 and piggy-back off of the CELERY_RESULT_BACKEND, but we should allow users to pass jobtastic configurations to the Celery app. Something like:

# your_app.py
from celery import Celery

app = Celery()
app.config_from_object('celeryconfig')

and then

# celeryconfig.py

CELERY_ENABLE_UTC = True
CELERY_TIMEZONE = 'Europe/London'
# Available options mirror the CELERY_RESULT_BACKEND option
JOBTASTIC_CACHE = 'cache+memcached://127.0.0.1:11211/'

Then, inside JobtasticTask, maybe we piggyback on the Celery app object to store a jobtastic_cache object using celery.app.app_or_default and their backend loading. Something like:

from celery.backends import get_backend_by_url

app = celery.app.app_or_default()
if not hasattr(app, 'jobtastic_result_backend'):
    # Initialize the cache object using Celery's code
    jobtastic_cache = get_backend_by_url(app.conf.JOBTASTIC_CACHE)
    app.jobtastic_cache = jobtastic_cache

Later, when we need to use what is now the cache object, like in this case:

app.jobtastic_cache.set('foo', 'bar')

Thoughts?

[jvanasco]

i like the idea of custom configurations via the jobtastic cache. i didn't like the idea at first, as we use postgres for the result backends and I'd rather use memcached/redis for this... but the custom config solves that.

[winhamwr]

we use postgres for the result backends and I'd rather use memcached/redis for this

Yup. I bet that is a common situation. #35 is another similar problem this would help. I think configuration like this might even make #2 easier.

The earliest time I would have to start working on this would be next week, but that's not very likely. I am very excited about this idea, though and thanks for walking through it with me. Pull requests are certainly welcome :)

-Wes

[rhunwicks]

@winhamwr I need to get JobTastic to use a separate cache from the Django default one because we want to switch our default cache from shared redis to local redis for performance reasons. This seems like the best approach to fixing it. Are you still interested in a pull request?

[rhunwicks]

One problem with using the Celery Result Backend(s) as the cache is that the Result Backend doesn't expose the expiry/timeout as a parameter to set() - it is set for the Backend instance as a whole and read from self.expires, and you can't use the underlying client very easily because of the different syntax that the Memcache and Redis clients use.

[winhamwr]

Are you still interested in a pull request?

Very much so, yes.

Does the jobtastic_cache API I proposed seem like it would meet your use case?

[rhunwicks]

Yes, perfectly - if I can work out how to use the Celery Backends with a custom expiry on each set() command.

[rhunwicks]

After examining the Celery Backends code, I have concluded that:

1. We can't use the Celery Backend without changing the Celery to allow the expiry to be passed to set() as a parameter
2. Even if we could get that change accepted upstream, it would make JobTastic dependent on very recent Celery, and would probably inconvenience a lot of existing users.
3. Therefore reusing the Celery backends, however attractive, is not going to work.

I think our next best alternative is to create a jobtastic.backends.BaseBackend that defines an API suitable for use by JobTastic (get/set/delete/lock (as a context manager)) and some specific implementations (Django Cache or Flask with Redis or Memcache) and a way to set the correct one using the Celery app.conf.

That way, we can deliver the existing functionality with a better API to allow users to provide a custom cache class if necessary.

[rhunwicks]

@winhamwr given that we can't use the Celery backends directly because we can't control the timeout on the set(), I have create a new cache module. It contains get_cache which attempts to find a cache from the Celery app.conf - using a JOBTASTIC_CACHE setting if it exists but falling back to the Django default cache if Django is installed, or the default Celery backend if Werkzeug is installed and the Celery default backend is using Memcached or Redis. This is roughly the same as the existing functionality (except that Werkzeug + Redis should work).

Because the JOBTASTIC_CACHE setting can be a subclass of jobtastic.cache.BaseCache it should be easy for users to add new cache providers.

There is also a jobtastic.cache.WrappedCache which should be capable of wrapping any cache provider that supports get/set (with timeout)/delete and either an atomic add or a distributed lock.

As part of this I have incorporated the locking mechanism described in #6.

I also think it would help with #59, although if Task-level cache busting is required then the solution there is preferable and could also be incorporated.

My initial code is at kimetrica@8fe5133 - please can you take a look and give some feedback. I'll look at tests, etc. once I know you are happy with the general approach.

[winhamwr]

@rhunwicks your approach sounds perfect. A great combination of backwards-compatibility and forward flexibility. I also like the movement of the locking logic to the cache module. 👍 from me