At this point, I’m sure you have heard about DevOps. Your C level guys might even be talking about it. They might even be telling everyone that the company should be doing DevOps. Of course, you are busy keeping the systems afloat, so you might file this under “Someone Else’s Problem” and tune it out. What you should understand is that not only is this kind IT culture not going away, it actually can make your life as a data pro a LOT easier.
The DevOps Foundation
There’s a lot going around these days about what DevOps is and how you should do it. Like every other tech cliche, it seems everyone has a slightly different definition. They all vary in length and detail, but when you boil it down all these definitions share a common element.
Let’s get Development and Operations working together.
That’s it. Really. To build out a DevOps culture, really all you need to do is get your developers and operations guys talking to each other and working to deploy code. Of course, anyone who has worked in the tech industry probably already knows this is a pretty wide gap to bridge. There are large divides built around “They keep writing crappy code” and “I can’t get anything done because they won’t let me” that make working together hard.
This is especially true when it comes to databases. As data professionals, we’re used to dealing with the code written by a .Net dev who thinks “SQL is easy”. Maybe, though, they’re using an Object Relational Mapper (ORM) to generate code using generalized rules that struggle getting anything right. Regardless of the cause, we are usually on the whip end of poor SQL, rewriting queries and adding indexes to get performance right. The whole time we are grumbling under our breath about “those idiot developers”.
This is where we need to step back a moment. The reality is most developers are not dumb or idiots, just ignorant. When I have approached developers in the past, most of them actually WANT to write good SQL code, they just do not understand the nuance. These folks want to work with me and are happy to have the help. Our challenge is getting inserted into an earlier phase of the development loop. To accomplish this, we need to reach out to our developers and find out how we can help them, opening that channel of communication.
Day 12: The Developers Have Accepted Me As One Of Their Own
This is where it starts. By working with developers, helping them write code, and participating in their sprint planning meetings, we can help them write better code. Better code in development means less problems in production. The idea is to deal with problems as early as possible, so we can avoid scrambling to add a fix in production.
This is a challenge at the beginning, because we are still tasked with keeping the lights on and systems up and running. You will need management buy in to support you in these efforts. Again, most management teams I’ve talked to will embrace this because of the long term benefits. The selling point is that, over the long term, the more you work with development teams on their database code, the less fires and problems you will have to put out. It is an investment that can really pay off.
Habits of Highly DevOps DBAs
Once you are communicating with your developers, it is time to start improving your processes. After all, we only have so much time in the day to talk to people and do our jobs, so we need to start building good habits to maintain our databases in a DevOps fashion. There are two habits you should focus on to buy more of that precious time resource that use for building better databases.
The first habit is source control for our databases. I am sure you have seen it before: your dev environment, stage environment, and production databases are all different. And not just a little different, a LOT different. Indexes are in one place, but not in another. Performance tuning work gets overwritten by the next release because it did not make it back into the scripts. This is all because we lack a golden source of truth of our databases.
How do we get this truth? Source control is the answer. There are several ways to do this. Personally, I use SQL Server Data Tools and store my databases as a Visual Studio project. However you do it, what you need is a source that you can use to build a clean copy of your database. Only then will you know what it is really supposed to look like, allowing you to make sure that what you are releasing to production is actually what you tested in stage.
The second habit to get into is smaller changes. This one should be a no-brainer, though I have been in many shops where we release massive amounts of code every few months and spend hectic nights trying to make the release “work”. It’s not only stressful, but wasteful and creates down time.
Think of it from a troubleshooting perspective: when you are working on solving a problem, you change one thing and then test. Then you change another thing, then test. Code releases should be the same. If you are updating database code, keep your changes minimal so you can release them and make sure they are working. This makes it faster to release code and and easier to identify when a release does not work. The crazy bit here is that, by making smaller releases, we can complete those releases faster and open up the pipeline to do more frequent releases.
Tools for the Job
Once we are communicating and start building out our habits, it is time to consider the tools we would use for the job. The first two parts of creating a DevOps culture can mostly be done without any special tooling, it is just a matter of changing your processes. However, to accelerate you need to think about putting some tools around your process. Not only will this help you get tasks done quicker, you will also make them more consistent.
When thinking of tools, you will need to think about your process and where the tools are needed. The first we have already talked about: source control. There are a lot of options to choose from, what is important is that you use something. Without a tool to properly manage your database state, you will spend more time managing your environment drift and not be able to see any benefits.
The second area of tooling is often called the build manager. While we can check code into and out of source control, eventually we have to make something useful out of it. A build manager, such as a Team City or Jenkins, can take our checked in artifacts and make something useful out of them. This will usually take the form of scripts, but these kind of tools take the guesswork out of comparing what we keep in source control and what has been deployed out.
Build managers are very important, especially when bringing DevOps to databases. The reason is that databases are stateful. We can’t simply drop and replace them, as we can with .Net dlls and web pages. This means we need a tool that can take the version of the database in source control and compare it to where we will deploy it. Doing this manually can be a headache and, without any sort of consistent tooling, lead to mistakes. Mistakes mean we are spending time in the wrong place, looking backwards at old code instead forward at the new. Making sure you have a good tool for building your release is vital for your success.
The final piece to automate is optional for getting started with DevOps, but a good idea nevertheless. It also will only succeed if we have the other two tools. Once we have source control and a build manager, it is a good idea to automate your deployments to some environments. This kind of tooling will take away the tedium of running the scripts and, if you already have these scripts generated, it only makes sense to automate their execution.
It is a matter of opinion where you want to automate your deployments, but the more you automate this, the more time you have to work on your work with the development teams. Every environment you add this tooling to is one less batch of time you have to spend. Additionally, it means development teams do not have to wait on you to test and release changes. You are not just making your life easier, you are making their lives easier as well.
A Journey, Not A Destination
The real strength here is that instituting DevOps can make your life easier, not just give you a different set of hoops to jump through. I get it, that there is a bit of gloss or shine that comes with the latest en vogue IT movement. We here big promises that typically fail to deliver. As someone who has worked in a variety of shops, I can tell you that DevOps will help you when done right.
This is the hurdle. With all the competing visions of what DevOps is and how you get there, you can easily get bogged down with poorly designed processes. The key is to keep it simple and to focus on these three concepts:
- Communicate with your developers (or DBAs, if you are on the dev side).
- Practice good habits of source control and release size.
- Get the right tools to maintain and automate your process.
Do not try and make it more than this, at least to start. Sure, it can be a long road, but focus on what is in front of you. Start small, build out, and you will be amazed at the stability that comes from working together.