ConcurrentModificationError with multiple celeryds

[yannzz365]

Hello,

I'm starting several celeryds having sqlakombu as a broker backend :
$ celeryd-multi start 3 -c 3
When a message is put in the queue, it seems that the consumers stomp on each other's feet when committing the 'visible' flag update :

File "XXX/lib/python2.6/site-packages/kombu_sqlalchemy-0.9.0-py2.6.egg/sqlakombu/transport.py", line 68, in _get
   self.session.commit()
[...]
File "XXX/lib/python2.6/site-packages/SQLAlchemy-0.5.8-py2.6.egg/sqlalchemy/orm/mapper.py", line 1423, in _save_obj
    raise exc.ConcurrentModificationError("Updated rowcount %d does not match number of objects updated %d" % (rows, len(update)))

What should be done ? Catch a ConcurrentModificationError and rollback, if we assume that another worker has consumed the message ?

Thanks,
Yann

[ask]

Interesting! The SQLAlchemy backend is still young, and I don't have much experience with SQLA. Your guess is good as mine, and catching the exception may be the solution, but maybe we should do a little research into this before deciding.