Feature flags do not support halting a staged rollout

[sneak]

if i’m reading the current implementation correctly, if we set the flag’s probability to 10%, then change it to 50%, some of those 10% will lose access. that shouldn’t happen.

also, if we set it to 50%, and 100 people use it, 50 being enabled, and we later change it to 10%, slowing the rollout, 40 of those 50 people will lose access, which shouldn’t happen.

whenever a feature flag is read for a user, it should be stored to a persistent datastore. i think an SQL table is appropriate for this because we will need to batch update sets of users who got false-valued feature flags set probabilistically when we roll things out to wider audiences.

[jnordberg]

if i’m reading the current implementation correctly, if we set the flag’s probability to 10%, then change it to 50%, some of those 10% will lose access. that shouldn’t happen.

That's not correct, an user will never loose access to a flag during a rollout increase. I've documented the behaviour in the README

also, if we set it to 50%, and 100 people use it, 50 being enabled, and we later change it to 10%, slowing the rollout, 40 of those 50 people will lose access, which shouldn’t happen.

That is correct but I'm unsure what you mean by slowing the rollout

[sneak]

let’s say we have a database that grows a gigabyte per user who uses it. we roll out to 1% of users. it goes well. we roll out to 50% of users, but site traffic spikes that hour, and the per minute active users are higher than during the 1% trial, and instead of seeing 50x the db loaf we get 200x due to a traffic spike that causes 50% of those users to be bigger than we wanted. we set the dial to 20% - the users that already got the feature during the 50% window should not lose it. only the users reading the feature flag for the first time should get the 20% probability.

[jnordberg]

Ok I understand. Is this a requirement now? Original design goal was using s3 as storage for flags, will have to go back to the drawing board if we want to change this.

Just to be clear, current implementation supports staged rollout but not slowing it during traffic spikes.

[relativityboy]

Keep the condenser's api the same plz. We want to start using what's already built for notifications.

[jnordberg]

API will stay the same @relativityboy

[sneak]

hmmm. what sort of storage should we use? must be something supported by AWS as justin is already too busy and we don’t have more sysadmin bandwidth yet.

i think this is a perfect fit for a simple SQL table, and conveyor is gonna need an RDS anyway.