Release Management - releasing to a subset of user

Question

I read somewhere (sorry don't exactly remember the source) that facebook has release Tuesdays. They release the new features to their internal employees first, then to a small set of external users and then to the whole world. I believe google also does something similar

I work primarily with Microsoft stack (TFS for source control, IIS, asp.net, sql server with huge data). Public facing sites of course, so they have to be up 24x7x365. Though I can envision releasing my api/dll only on one of the servers (in the webfarm) and testing it out, how would I do this if there are DB (stored proc signatures, table schema changes )? Presently we are versioning SPs (the new ones will be mySPNameV2, where as the old one will be mySPNameV1 - both taking different set of parameters hence the renaming) and the new APIs would use SP-V2 where as the old API would continue with SP-V1.

I see some design smell, but is there a better way to do it?

Edit: we release the new code to only one server and test it, what is difficult is how would you abstract (may be abstract is not the right word, but you get the idea) db schema changes from multiple concurrent versions of the application

Answer 1

I was at QCon last year, when a guy from Facebook's Web Team was talking about it this approach. You are 100% correct that there will be code smells in order to implement that. But it takes a lot more than just a code smells (actually they call those code smells Gate Keepers :).

First of all - their way of representing and storing data is a lot more sophisticated, they are not using Foreign keys, or any other "advanced" features of the databases at all :), as far as I remember, their data is not that "relational" as we all want to keep ours :).

What you can do is add gate keepers (if (user is from New Zeland) { use then new version with the new classes } else { stick ot the old one }).

You can probably imagine to what kind of code duplication will this lead in case of structured model (e.g. you will need to have 2 Users, 2 Orders, 2 Order Details). Leaving that aside, I think a lot of regression will also take place if you don't be extremelly careful.

As far as I remember - they are releasing a lot more often than just Tuesdays. I think they have a continous deployment and code goes live every day, just they clean old Gate Keepers / schema changes Tuesdays as they switched all the users to the new functionality by then.

So basically:

1. You add gate keepers everywhere a new functionallity is taking place.
2. You maintain 2 versions of the Schema.
3. You add more and more users to switch to the new one.
4. You clean the old one.
5. You go to point 1 (if another team isn't already there :).

I realize this answer may not be full enough to qualify for an answer, but I heard this from their guy and thought it is worth sharing.

Answer 2

Maybe I'm simplifying here but why not just instance the database further?

When you are doing a subset of users to "test" be it internal, subset of external, or the world aren't you in fact performing something like a beta test? Wouldn't this call for a different instance of the database that has the beta version of the stored procs vs. the current version/instance? Just a matter of pointing connection strings in web configs at that point isn't it?

Maybe the constraint here is cost of servers and "huge data" being on them, which I'll admit I'm glancing over or is the question really about versioning the stored prococedures, etc.? Can't they just be source controlled as well as the schema changes?

I've probably asked more than answered.

Answer 3

Option 1:

A (sub)domain for the (semi)(public?) devsite. Certain users are accepted on the devsite based on a manual set cookie (employees etc) < private testing

The main domain which sets cookie for certain users (cookie set when time is between time X and time Y) < depending on your traffic. If you get 1000 (unique)visitors every hour and you want 10% on the dev domain you make sure the delta time is 6 minutes. You can simply delete the cookies when you want users to be redirected to the normal site.(make sure you redirect the whole incoming url to prevent broken bookmarks.

Option 2:

Loadbalancing a certain percentage of the traffic to servers running new app.

Database

1:Interact with live database while developing < regular backups + regular skilled devs = safe

2: look at master slave replication to create a "live" shadow copy of your DB

Answer 4

I really thought this question would have received a lot more attention. Considering what you said and the existing answers, I think your existing solution is the most straight forward and easiest to manage. As you said, there is some "design smell" (I like that phrase) but it makes the most sense.

Perhaps going one step further and combining some slight modifications of the suggestions with your own:

• Keep your existing database version convention
• Release specific candidate versions to either a sub-domain candidate01.yoururl.com or a specific server if you have a server farm
• Use a flag on your user/member table to indicate which production server or sub-domain the user should be directed to
  • Provides the ability to direct some users to the release candidate server(s)
  • Doesn't require the amount of code the per feature coding option mentioned in DK's answer would (you could go either way though depending on how targeted you want to be but I think the least complicated route would be best here and would direct users to specific versions of the app rather than try to turn individual features on/off per user)

Other than that, great question! Oh yes, and when that is all ready you just flip the fluger switch to engage the double buffer mechanism and you are all set.

Answer 5

The simplest approach IMO would be to separate features by sets of pages that use security to determine which page a user gets. Beyond security and other means to ensure a user cannot get to a page directly to which they do not have access (site maps etc.), you could store in the database the list of features and which users or roles have access to that feature:

Create Table Features
    (
    Code varchar(10) not null Primary Key
    , StartPage nvarchar(max) not null
    , Description nvarchar(max) not null
    )

Create Table UserFeatures 
    ( 
    UserId ... not null
    , FeatureCode varchar(10) References Features ( Code )
    )

First, the reason I would use a text code for the feature primary key and not a surrogate key like an IDENTITY column or guid is that only the system will be querying for features. Users will never have the ability to arbitrarily add a feature. Thus, it makes your code much cleaner and easier to read to query for ...Where FeatureCode = 'AdvancedEntry' than ...Where FeatureId = 13.

Second, in this approach, the code of the pages themselves would determine which procedure to call. However, that may mean quite a bit of duplication if the features simply involve additional fields of information.

Thus, if the features are tightly integrated into the existing code base and presentation layer (which by the way is why versioning is so difficult), an alternate approach would be to store the name of the stored procedure that should be used in the Features table. Your code would query join the above tables and return the stored procedure name that should be used. For the arguments, you could store a parameterized call in the database (e.g. exec Schema.Foo @bar, @gamma, @beta) and when executing the query simply check whether that string contains a given parameter and if it does, add that parameter value:

if ( ProcTemplate.Contains( "@bar")
    commandInstance.Parameters.AddWithValue( "@bar", barValue );
if ( ProcTemplate.Contains( "@gamma")
    commandInstance.Parameters.AddWithValue( "@gamma", gammaValue );
...

If you map features to user roles or groups, then you will need to devise "trumping" rules which determine which feature should be used in the case of multiple being returned. In this approach, you would leave existing stored procedures alone barring a change to the schema which required a change to the stored procedure (e.g., the removal of a column). A bonus result of this approach is that you could add a date to the Features table to determine when a new feature should go online.

Answer 6

If I understood you correctly, what you want is to have two mechanisms that use the same live data, but different API versions. Now, assuming you're already working with a double buffer mechanism, I guess that your actual problem is to use live tables during that transition.

The solution is for your tables to include both V1 and V2 columns (e.g., one users table that will include fields from both APIs). Note: All non-common fields must have default values.

In order for this to work seamlessly, you should create views for V1 and V2, exposing only the relevant fields for each API version, and you should develop for views instead of tables (similar to the concept of developing for interfaces instead of implementation).

The same goes for stored procedures - all non-common parameters must have default values, etc...