Two Approaches to Concurrent-Write Safety in Django

Sometimes when dealing with Django models accessed by multiple people, you want a way to make sure two requests don’t perform writes at the same time.

For example, lets say you run a poker site, and you have 3 users playing a poker game together in the browser.

poker_game

You want only one active player to be able to perform game actions, and you want only one action accepted at a time.  Imagine the catastrophe if your active player could open the game in 2 tabs, fold his hand in one, and bet in the other simultaneously!

Here are two easy solutions to create concurrent-write-safe code (to prevent your hypothetical poker site from losing millions of dollars to devious 2-tab players):

The first solution is commonly used when doing lock-free programming, but is difficult to implement properly.  The second solution is simpler to implement, but can cause deadlocks if your code throws an exception and you leave models locked.  Here I illustrate how to do both.  The code examples use Django, but these principles are generalizable to almost any web backend that mutates shared data.

Solution 1: Atomic Transactions with Check-Before-Write

def save(self, *args, **kwargs):
    if self.id:
        on_disk = Game.objects.get(pk=self.id)
        if on_disk.modified > self.modified:
            raise StaleWriteError('Tried to save outdated Game')
    super(Game, self).save(*args, **kwargs)

However, THIS CODE IS NOT CORRECT, this naive solution is not good enough to guarantee concurrent-write safety.  Look over the code and think about it for a minute before reading on to find out why.

 

The reason is: a race condition can occur between the timestamp if-check and the SQL update query.  The proper way to do this is make the condition part of the update query (which is atomic).

updated = Game.objects.filter(Q(id=game.id) && Q(version=game.version))\
      .update(field_name=new_field_value, version=game.version + 1)
if not updated:
      raise StaleWriteError('Tried to save outdated Game')

Source: this Stack Overflow question

Alternative implementations use a last-modified timestamp instead of a version number, or even the whole state of the database row (which is slow but avoids requiring a separate version column) to check before writing.  If your database supports fast row-hashing, you can also compare row hashes before writing.

Solution 2: Row Locking With a Redis Lock-Table

import redis

from django.db import models


lock_table = redis.StrictRedis(host='localhost', port=6379)


class ConcurrentModificationError(ValueError):
    """Base error class for write concurrency errors"""
    pass


class StaleWriteError(ConcurrentModificationError):
    """Tried to write a version of a model that is older than the current version in the database"""
    pass


class AlreadyLockedError(ConcurrentModificationError):
    """Tried to aquire a lock on a row that is already locked"""
    pass


class WriteWithoutLockError(ConcurrentModificationError):
    """Tried to save a lock-required model row without locking it first"""
    pass


class LockedModel:
    """Add row-level locking backed by redis, set lock_required=True to require a lock on .save()"""

    lock_required = False  # whether a lock is required to call .save() on this model

    @property
    def _lock_key(self):
        model_name = self.__class__.__name__
        return '{0}__locked:{1}'.format(model_name, self.id)

    def is_locked(self):
        return lock_table.get(self._lock_key) == b'1'

    def lock(self):
        if self.is_locked():
            raise AlreadyLockedError('Tried to lock an already-locked row.')
        lock_table.set(self._lock_key, b'1')

    def unlock(self):
        lock_table.set(self._lock_key, b'0')

    def save(self, *args, **kwargs):
        if self.lock_required and not self.is_locked():
            raise WriteWithoutLockError('Tried to save a lock-required model row without locking it first')
        super(LockedModel, self).save(*args, **kwargs)


# example usage to require locking on a model when calling .save():
class Game(models.Model, LockedModel):
    lock_required = True

    players = models.ManyToManyField(Player)

Locking is a generic pattern than can be used for more complex operations than just single-row locking.  You can always manually use the lock_table to create and hold locks over whole blocks of code that aren’t linked to a specific DB row.

Here’s how you’d use the above row-locking feature in a real use case, (e.g. calling perform_game_action from inside a view).

from django.db import IntegrityError, transaction
from .models import Game, Player

def perform_game_action(game: Game, new_player: Player):
    # acquire redis write-lock on db objects
    game.lock()
    try:
        with transaction.atomic():
            # modify your database object here
            game.players.add(new_player)
            # save all modified state to database
            game.save()
    except ConcurrentModificationError, IntegrityError:
        # handle write integrity errors/lock contention cases here
        print('Game transaction failed!')
    finally:
        # release redis write-lock on table object
        game.unlock()

Alternative Solution 2: Use Django’s built-in locking

I’ve recently been informed of a third solution by the kind people of the internet and this Stack Overflow question.  This is the canonical “Django Solution”, but my version above with Redis locking gives a little bit more exception granularity, allowing you to handle different contention cases separately.  The Django method just throws a generic DatabaseError regardless of the situation.

Quoting the Django Docs, select_or_update():

Returns a queryset that will lock rows until the end of the transaction, generating a SELECT ... FOR UPDATE SQL statement on supported databases.

games = Game.objects.select_for_update().filter(active=True)

All matched games will be locked until the end of the transaction block, meaning that other transactions will be prevented from changing or acquiring locks on them.

Usually, if another transaction has already acquired a lock on one of the selected rows, the query will block until the lock is released. If this is not the behavior you want, call select_for_update(nowait=True). This will make the call non-blocking. If a conflicting lock is already acquired by another transaction, DatabaseError will be raised when the queryset is evaluated.


Which approach you choose is up to you, just remember to test your code with a variety of write conditions and load levels.  Prefer atomic operations over non-atomic, and test every lock contention edge-case before deploying your shiny new “concurrent-write-safe” code to production.  Beware of TOCTTOU bugs!

Remember, a single edge case that happens even 1 out of every 10,000 requests can be exploited by a devious user, and could potentially cost your tiny startup lots of money!  (follow-up post on rate-limiting in Django coming soon)

Please leave comments if you have other solutions to share, or if you find any errors in my implementations!


Related reading:

P.S. Sorry about the ads and lack of syntax highlighting on my blog!  I’ll work on migrating it to Jekyll eventually…

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s