Two Week Code Pushes: Fast or Slow?
I just finished reading a TechCrunch interview with Andrew Anker of Six Apart about their excellent new service, Vox. It’s a nicely conducted interview about a well-designed product that I think will be very successful, but this line struck me as a bit odd:
“Very early on in Vox’s development, we created a two week rapid iteration cycle where we made sure to push code religiously every two weeks. By doing that, we made sure that we were building a design cycle that was always two weeks away from fixing any problem.”
Do bi-weekly code pushes qualify as “rapid” these days? Even at Disney, we generally didn’t take much longer than that, and at Newsvine we push every single day. Multiple times even. We like the rapid pushes so much, in fact, that we even set up a machine to triumphantly speak out the filepath whenever someone pushes out code: click for a sample.
Maybe it’s because I’m impatient about site improvement and problem solving, but Andrew’s statement — if it were applied to me — would advocate a deliberate slowing down of the push cycle (perhaps for the better).
Does anyone else agree? With web technologies the way they are today, do you consider religious code pushes every two weeks a fast thing or a slow thing? Are scheduled infrequent pushes (meaning, much less than once a day) preferable to your development habits?
Seems slow-ish to me. At World Online, our programmers push code when there’s code ready to be pushed — which is usually several times per day, and several times per week at the very slowest.
Almost sounds like Microsoft and their Patch Tuesday. Its not the first time I have heard Six Apart being compared to Microsoft.
I’d consider it fast. Sure, in the realm of personal blogging sites it might be a bit slow, or companies run by people such as yourself, it might be a bit slow. But I’m very used to seeing websites retain bugs for months, let alone weeks. This is especially visible with large sites and ones that aren’t tech related.
Flickr, for one, pushes code multiple times per day.
I guess I’m still stuck in the software mindset – monthly sprints at best. So, pushing new code live every day, let alone every 2 weeks, seems a little over the top. Why not build it up a bit and release in bigger chunks? What’s the rush all about, unless something is urgent?
I think it used to be considered fast, but in today’s age I’d have to consider 2 weeks to be “slow-ish” as Jeff said.
Waiting two weeks to publish code seems like an eternity.
Granted, there might be performance/load reasons for updating code infrequently. But if that’s not an issue, my personal preference is to update code as soon as it’s tested and ready to go.
To me, something doesn’t feel “done” until it’s actually live on the site, and I like to have things done as soon as possible so I can move onto the next task. The fewer outstanding tasks to juggle in the brain, the better.
at datek, we pushed code whenever necessary. when ameritrade swallowed us, they instituted a monthly push with a two-week “maintenance” release — supposedly to help the QA process.
i guess it was a different beast, being that we were dealing with authenticated accounts and people’s money. then again, the Tech group didn’t bend to get non-secure stuff out. it actually made a lot of us get gray hair — too much pressure, ’cause if we missed the push, we’d be waiting at least two more weeks.
at espn.com there were pushes by multiple people multiple times a day… it was chaos, but things seemed to work out, and the site is clearly a living, breathing thing.
at foxsports.com, we pushed once every week or two, which killed me. if you’re building social software, i think it’s important for the the users to know that the software is dynamic and growing, like the community. if broken things stay broken for too long, and enhancements aren’t make quickly enough, i wonder why i should care about the system / community when it’s clear that the developers aren’t as passionate about it as they should be.
kareem
We should push every hour. We try to push things every hour, even if there isn’t any code to push, we will write some and push it anyway. It works, trust me, I used to use ‘the google’
I think two week time boxed releases are just right for us but in my experience the “right” frequency is highly dependant on the company and domain.
We used to push code live as soon as it was ready which was a good thing early on but as our code base and team grew we realized we weren’t embracing refactoring and our code base was stagnating because it always had to be in a highly “stable” state so projects could be moved live at any moment.
Moving to 2 week release cycles turned out to be the right balance of encouraging refactoring (which inevitably brings a system into a less stable state even with high test coverage) while at the same time keeping the amount of change per release at a very manageable level.
To me the bottom line is that the release frequency is highly dependant on the company and the domain. While an early phase Web startup should be pushing changes very often, an online web brokerage most certainly should not. Once a company has more than 2 developers it’s my opinion that priority driven time boxed release cycles work best which I’ve written about here:
Time boxed versus Feature boxed release cycles
and the discussion that ensued on the server-side is here:
http://www.theserverside.com/news/thread.tss?thread_id=38287
For where I work, it’s always been “it goes live when it’s done”. We’ve been burned on missing QA before, and so we’re a little more cautious. That isn’t to say we don’t push a lot though. There’s something going out every day except Friday, which is for emergencies only.
In response to your “annoucement system” Mike, over here releases get “theme songs” which echo throughout the office. I believe our invitation system was given Airwolf.
It all depends on how big the client is and how much sign off’s we require. Sure the dev fix may be on avg 1-4hrs to fix a bug but then it has to be tested on staging servers… and then client sign of and go thorugh client managers etc etc.. So for a payment gateway bug perhaps as low as a day, for a trivial issue, 2-4 weeks depending on workload.
As always, it depends on the domain and the scope of the change. For our flagship application, we are publishing content every work-day, for several months of the year, with seasonal slowdowns to 30hr/wk, maybe. Publishing content is easy.
On the other hand, the application platform (and therefore the code base) is very “mature” (read: several million lines of code, much of it more than 5 years old), the client is very concerned with quality of releases, and we have developers working across three countries, for about a half-dozen companies, committing to the head development branch.
QA for platform releases is non-trivial. Rapid-response issues are often posted the next-day, but those are not where most work occurs. Our ideal incremental versioned release cycle was suppposed to be three weeks, last I heard. Reality is that the client dictates the schedule completely. What are ya gonna do? :)
Now for our other, smaller projects, say the stuff I lead, we operate without branches in the code, and rely on continuous-integration tools to help us enforce a policy of always being stable for a release, and we set milestones to meet the needs of the client.
we sometimes do daily pushes but try to restrain prod to once or twice a week only for QA (not just web work). The two week release cycle is actually what my shop in a larger mothership is moving towards, but 2 weeks for new feature dev, 2 weeks qa, 2 weeks design, etc.
This question completely depends on the content and application that will be pushed. I work for a company who creates very sophisticated web applications that are used by very non-technical individuals. Bug fixes get pushed immediately, features get rolled out over a longer period of time to allow for a user to prepare him/herself for the change.
Tough call on this one without knowing the specifics, Mike.
As a developer: Two weeks is slow. Again, it depends on dependencies etc., but my feeling is: slow.
As a user: Changing my application too often makes it seem either buggy or unstable. I’ll wonder why so many changes, and at some point wish it could just be left alone, I think.
Personally: We released two changes last week to our CMS, which ~40 people use at one company to manage ~80 websites. The changes were announced via an e-mail and a modal window, but I still received bug reports about the changes in the interface.
So as usual, this depends on the project.
People can’t seem to agree what “code” means. I find it hard to believe that a web company does not push code out to the live site EVERY DAY. But a lot of that is background code to keep the site running smoothly.
Pushing out a client facing Feature every 2 weeks is a different story. And even if you are pushing out a feature you could be doing it in small steps. Adding functionality as the feature turns out to be more popular.
In general I think when some people say “code” they mean little code fixes and when other people say code they mean “features”.
seems like a rather arbitrary goal. why not push code when it’s ready?
Albeit, I feel that code-pushing is heavily dependent on what the application/site/project at hand is… I would have to say that once every two weeks for an up-and-coming deal such as Vox, leans towards the slower side.
Pushing out code every two weeks seems like an eternity, given the immediacy of the web. Sounds like what happens when companies move from a “small startup” to the “big corporation” mentality. Maybe it’s a side-effect of trying to appeal to corporate customers- “Hey, you can trust us, we move just as slow as you do.”
I’m working on a project for a large wireless carrier, and their idea of innovating is creating a separate area of one building which has funky chairs and cubicles, with lots of open spaces for collaboration. Problem is, everyone is still bound by big corporation policies- limited internet access, limited user profiles, must use Windows…, etc.
A “rapid” two-week code cycle is just par for the corporate course.
I just finished a contract at a recipe site with a *lot* of users and a dev team of ~8 people. They pushed once a week for non-emergency stuff, and as needed for high priority stuff.
:Mike Papageorge – amen.
I think most of the commenters here probably confuse about ‘design codes’ and ‘maintenance / new features codes’.
Design codes are written at the early phase of a product development. It involves a certain number of iterations, from schema design, UML mapping and writting constructors and classes for the base frameworks to building a frontend engine. Sometimes regression testing might also be carried out at such early stage. So as the quote says ‘Very early on in Vox’s development…’, i think that makes much sense.
Mostly after releasing the product to beta testers, that’s when you will likely to see a more frequent codes pushing practice. Bugs fixing codes obviously have to be pushed out right away. New features, if they dont require any rewrites or changing the core frameworks (this can be avoided by having a robust, well-designed ‘design codes’), can also be pushed out daily.
But i wont be suprise if there are companies out there that actually push ‘design codes’ out every hours. But in general, I would think it’s more of a team preference than being caused by any sort of technical limitations and whatnots.
One factor to consider, for those who may be speaking without actually being a user of Vox is that their updates have pretty frequently involved not just new features but changes to existing features, and even rearranging the interface. Doing that too frequently is definitely just going to confuse people, as Mike P. mentioned above.
Where I’m at now, two weeks is a lightining quick respone, considering that even a small change needs to go through a monolithic approval process with consensus from all 5200 people on the committee, before it’s sent over to branding for review. Then IT needs to approve it and post it…
(Okay, so it’s a bit of an overstatement, but we’re still not adjusted to rapidly doing anything when it comes to online stuff… Part of it’s lack of understanding interactive, in general, and fear of the unknown. At least in my division.)
Push? What does this word mean? I update my pages where they sit on the web server. The only thing I do is “Pull” stuff over for a backup copy.
:)
Large web sites are developed on a Development Server and then the files are pushed over to the live site. We use subversion to accomplish this.
At a previous job, I posted to a test server all day and pushed live once a day. I can’t imagine dragging that out to two weeks, especially if you have multiple developers as I’m sure Six Apart does. They must have an excellent testing and tracking system — I can’t imagine going over 14 days worth of work to fix any issues that arose when something went live. I would just cry.
Heh, here at work we push code whenever there’s code that needs to be pushed; yet we do set deadlines for pushing it, or wait on pushing some (not so important) code when we’re prepping out some more new code.
Love the idea of having your machine speak whenever someone pushes code on it. Love the “thanks [user name]” even more :P
I think 2 weeks is a bit slow when its a full time gig and you’ve got a team to work on it. My business partner and I are building a web app using 2-week iterations right now, but we’re in college and have little free time to build the thing.
I’d call it moderately fast in the world of web development. A lot of institutional and smaller storefront apps don’t get updated for months at a time, so while not the fastest a two-week cycle is still fast.
I’m a bit charmed by all the backseat driving on what’s essentially a dissection of one or two sentences out of what was probably an hour-long conversation. So, is a two week schedule slow (as some have said above) or fast (as others have said above)? Probably both.
Some constraints that might help people judge the release schedule more effectively:
I can (and have!) go on ad nauseum, but to those who are interested in this admittedly compelling topic, it’s always handy to have more information. And one last bit of love for Adam:
“Its not the first time I have heard Six Apart being compared to Microsoft.”
Is that in the “has millions of users” sense, or in the “is trying to make a real business that gets people to use new technologies” sense? :) Because surely contributing huge amounts of open source code, engaging in honest conversations with people all over the blogosphere, and having a sense of humor about ourselves aren’t usually traits I’ve seen ascribed to the good people at MS.
Thanks for sharing Anil, it’s always nice to hear a bit about the behind the doors of a web-app… I was wondering if documentation and language issues were part of the reason for two weeks. Building a multi-language app can be orders of magnitude harder then a single language version…
Anil: Thanks for chiming in. I’m interested in it from a “rhythm” perspective rather than a speed perspective. Certainly didn’t mean to imply that two-week code pushes are a *bad* thing… rather just not something I would consider particularly “rapid” when it comes to code rolling. But, to your statement, you said that code is actually rolled a lot more frequently than every two weeks so I guess that renders my ponderings with relation to Vox moot.
The root of my question is this: for whatever project someone is working on, is it generally a positive or a negative to use discrete time intervals for code pushes? Some people apparently think it’s good and some people would rather just push code whenever it’s ready. We’re just too impatient to wait and I think a lot of others are as well, but that doesn’t mean it’s right.
Two weeks in a large company as SixApart is rather amazing. I work in a company with 150 people, and on my project there are something like 60: database, middleware, front-end, QA, deployment – if we manage to achieve two weeks = good days.
Quick pushes like you mention are possible in smaller teams. I can’t possibly imagine daily *stable* pushes in any project involving 10+ people.
I would say that the speed of deployment has to be based on what you are deploying. As an example, you couldn’t push an interrnet banking application often as it requires downtime to do so. If you’re pushing a site which doesn’t have a lot of user post data, then pushing more frequently would be more acceptable.
The other thing I think plays or should play, a huge role in release cycles is your testing setup. For instance if you’re deploying a legacy application which sprawling code, then publishing regularly is going to burn you sooner rather than later. If on the other hand you have a well designed application with unit testing at each layer and good code coverage; then by all means release frequently.
Al.
I think it’s about average. Best to push small changes often than huge chunks at once. Where I work now the words agile and the like are akin to ‘cowboy programming’.
Saving up all your changes and pushing every 6 months is a huge undertaking and a recipie for disaster. Trust me, they do it where I work now.
I’ve got to find a new job.
Beef