I’ve written this post because I have a pretty unique perspective on the deployment process. I sit on the outside of development teams, both at Red Gate and with customers, and I peer in. I speak to lots of people every day with the same issues. A surprising number of them think they are alone. They really aren’t.
Open disclosure: I am part of the Deployment Manager team… I am the sales guy.
It’s with this in mind that I’ve taken the time to share one of the most difficult (and common) conversations that I have with customers. It may sound familiar:
Ring Ring…
>> Hello?
>> Hi, is that Joe Bloggs?
>> Yeah, who’s this?
>> My name’s Alex, I’m calling from Red Gate because you downloaded SQL Compare last week. How are you getting on?
>> Hi, yeah. It looks good. But we aren’t going to buy it. We have our own application for doing deployments. It’s been working fine for a couple of years now so we aren’t interested.
>> OK, thanks for letting me know. May I ask why you downloaded SQL Compare?
This is the point where Joe tells me that something went wrong. Joe downloaded the SQL Compare trial for a quick fix. It’s OK now. It’s sorted.
The problem with home-made deployment processes
Joe and his team have a custom application and process for deploying their software that some developer put together this one time and it works OK (almost) all the time. It was “free” because they did it in-house (I’m biting my tongue). OK, only that one developer knows how to use it (it’s hardly ‘ingeniously simple’), but it works, and it suits their particular peculiarities. This application may or may not include the database (Who cares about the database?). The application is fairly restrictive, but that is OK because Joe and his team can change the way they work so that they won’t break the release. And besides, modifying the release application is too scary (what if it goes wrong?) and too much work to bother.
But things do go wrong. And the first time things go wrong it comes as a shock. I’m sure Joe (or Joe’s manager) is fully aware of the cost of a bad release. It’s more than the cost of Joe and his teams’ overtime or the spike in the company’s coffee bill. If this gets anywhere near customers they could lose confidence. The company could even lose business. It could damage the brand. It will also delay the next bit of work. And most importantly, Joe had plans for the weekend!
If this isn’t scary enough, what happens if that developer who built Joe’s release process is on holiday or off sick? What if they leave the company? When that happens I tend to have very different conversations with clients.
The trouble is, Joe is a busy person. He doesn’t have time to try something new. “If it ain’t broke don’t fix it.” (But it is broke!?) There are other priorities. Joe wants to get back to the important stuff. Because as far as developers are concerned, the important stuff is anything but deployment, right? That is, until deployments go wrong.
Besides, that developer who built the deployment app swears by it. It’s their baby. Not many other people really understand it but they are happy to trust it and get on with the day job. So Joe will wait until it goes wrong again, and then will most likely have to buy a tool designed for the job anyway.
This may or may not sound familiar to you. Some people do have a great system set up. For example Annette Allen and Dave Green at FirstDataBank have detailed their Continuous Integration and code promotion model for us and Ben Adderson, who manages all of our releases at Red Gate, discusses how he improved his deployment process below:
Why automate your deployments?
By automating manual processes both examples demonstrate that less stuff goes wrong and more stuff gets done more quickly. Also, by using software that is supported full time you know that you won’t out-grow your release application and it will support the work you do rather than restrict it. By adopting an approach that makes your development/release process visible and maintainable by many people the business is no longer tied to the schedule of a single person. It will probably also be cheaper to set up.
Whether or not you decide to use Red Gate tools to manage and deploy code, please take it from someone who speaks to dozens of like-minded people a day:
- Developing (and maintaining) a tool that you build in house will be much more expensive than you expect. Especially when you take into account the on-going support costs. Besides, you can probably write a list of things that you would prefer to be doing than fixing a deployment process.
- Using tools that are maintained by a team of developers full time will be more secure than trusting something that was built in house by a developer as a side-project. Also, the tools will be far more likely to stay up to date and improve over time, supporting your development efforts, rather than restricting them, for years to come.
- Putting yourself in a position where there is only one person (perhaps it is you?) who understands the release process is unwise. The longer you leave it, the more it becomes an issue. (Besides, that one person who you are leaning on may well get frustrated at being called on their day off, in the middle of the night, or when they are trying to work on something else.)
Does this sound familiar to you?
You raise an interesting point. We have automated way more than most companies ever consider. But I’m the one who built it. And it was a side project.
In most companies this would be a big issue. However, being an agile agency, we have a lot of information sharing.
Now on to the reason I built the tool. There was nothing else really available. More to the point, there was no complete solution which was flexible (configurable) enough, easy to set up, easy to use, suited our needs, AND was low impact to our current processes.
I have high hopes for DM. If it can be seamlessly integrated with Team City, with automatic deployment after Team City builds, then that would give us our current functionality in a more standard tool that looks prettier that something that would be developed in house.
Anthony, Anthony and Mattias: Thank you all for your feedback.
In response to Anthony (Dang), we have now published a couple of articles that provide instructions to use the command line to call on Deployment Manager. This means that you can automate Deployment Manager from within TeamCity:
• Command line syntax: http://www.red-gate.com/supportcenter/Content/Deployment_Manager/help/1.0/dm_cl_syntax
• Examples using the command line: http://www.red-gate.com/supportcenter/Content/Deployment_Manager/help/1.0/dm_cl_examples
Would you let me know if this resolves the aspirations you have for Deployment Manager that you mentioned in your final paragraph?
We are looking at implementing Deployment Manager for exactly the reasons you describe above.
From a management point of view, I want deployments to be as smooth and effortless as possible, with minimum downtime and zero errors. Our manual process at the moment is none of these and guarantees loss of confidence. And we’re relying on one person, our “point of failure” as we like to call him!
I’m glad to hear we’re not the only ones with these sorts of issues and there is light at the end of the tunnel!
We’ve contemplated moving away from a predominantly manual, and therefore very error-prone, deployment process for quite some time. After reading about this new product from Red Gate called Deployment Server, we decided to evaluate it.
After familiarizing ourselves with the product and realizing its potential, we decided to go ahead and buy it. Currently, we are in the process of applying it to our deployment routine. (At the same time, we’ve also started using TeamCity for automatic builds).
We’ve “started slow” by using it to deploy updates to our test servers. Smooth sailing! Now it’s almost fun to deploy applications (and bearable to deploy database changes)!
Soon we’ll let DM take over the responsibility for the production environment as well.
However, yesterday we had to install an urgent bug fix on that production server! Under these circumstances, just concentrating on the bug fix, we decided to use the old deployment process.
And boy, was it a return to the bad old days – cumbersome, stressful – and with some familiar “oops!” moments along the way, such as: “But this runs on my development box – what’s wrong with the freaking server?! Oh, did I miss THAT step in the routine?”
Must I add that, at this moment, I really missed the new deployment routine?!
After this return to the horrors of yore, we’ll finish moving to the new deployment routine, with DM as backbone, as soon as possible!
No more shaky deployments!