Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,11 @@ These let you tweak the default behavior.
Most often, you'll just be setting the `cache_duration`
to enable result caching.

#### always_start_new_herd

Skip herd avoidance check if True, but still trigger avoidance for other tasks.
Defaults to False.

#### cache_duration

If you want your results cached,
Expand Down
13 changes: 8 additions & 5 deletions jobtastic/task.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ class JobtasticTask(Task):
front-end code so that users know what to expect.
"""
abstract = True
always_start_new_herd = False

#: The shared cache used for locking and thundering herd protection
_cache = None
Expand Down Expand Up @@ -218,11 +219,13 @@ def apply_async(self, args, kwargs, **options):
'Found existing cached and completed task: %s', task_id)
return self.AsyncResult(task_id)

# Check for an in-progress equivalent task to avoid duplicating work
task_id = self.cache.get('herd:%s' % cache_key)
if task_id:
logging.info('Found existing in-progress task: %s', task_id)
return self.AsyncResult(task_id)
if not self.always_start_new_herd:
# Check for an in-progress equivalent task to avoid
# duplicating work
task_id = self.cache.get('herd:%s' % cache_key)
if task_id:
logging.info('Found existing in-progress task: %s', task_id)
return self.AsyncResult(task_id)

# It's not cached and it's not already running. Use an atomic lock to
# start the task, ensuring there isn't a race condition that could
Expand Down