Race condition / non-determinism in /api/v1/experiments endpoint

[hindog]

We are noticing non-determinism when querying the /api/v1/experiments endpoint with a filter defined. When calling the endpoint sequentially, it returns the expected results, but when called concurrently, sometimes it returns the wrong or empty results.

I've written a quick-n-dirty python script to reproduce the behavior. The variables at the top of the script should be modified to work in your environment, including at least two different experiments for the script to query.

The script will invoke the endpoint concurrently with a randomly selected experiment from the list and filter via: filter=experiment_label=<label>. I would expect each response to contain only the experiment we query for, but instead it will sometimes return the wrong experiment, and other times return empty results. The script will print any responses that don't contain the experiment label in the response to show the "failure" cases.

import requests
import random
import time
import threading

baseUrl = 'http://some.wasabi.host:8080'
experiments = ['vp-369-movie-detail-no-ad', 'vp-853-text-me-the-app']
credentials = ('admin', 'admin')

failed = 0

def fetch_experiments(experiment):
    global failed, finished, baseUrl, credentials
    r = requests.get(baseUrl + '/api/v1/experiments?filter=experiment_label%3D' + experiment, auth=credentials)
    if experiment not in r.text:
        print 'query for experiment: ' + experiment + ', returned unexpected response: ' + r.text
        failed += 1

try:
    for i in range(0, 1000):
        e = random.choice(experiments)
        t = threading.Thread(target=fetch_experiments, args=[e])
        t.start()

except:
    print "Error: unable to start thread"

time.sleep(1)

print ""
print "failures: " + str(failed)