Production Change Control Process

How do you make a change to production?

Here is an overview of the process I like to use:

1. Sync your test environment to production

2. Apply your changes to your test environment

3. Run your test plan in the test environment

4. Plan with your team for the production change

5. Update production

6. Communicate

7. Celebrate

8. Follow-up

9. Archive your plans for the next time

Sync Your Test Environment To Production

The first step is to make sure your test system is exactly the same as production.

Tip: If for some reason you don’t have a test system, get one. It is a valuable tool for ensuring you can successfully roll out changes to a production environment.

There is nothing more stressful than finding out during a production deploy window that things are not the same as the environment you tested in. At a minimum, verify the following:

• Same software version/patches

• Same config files

• Same environment (operating system)

There are some great tools out there for identifying changes to a directory of files. One of my favorites is a free tool called WinMerge. I can quickly use this tool to identify the changes between two different environments. It gives me that assurance that I have completed step 1 correctly.

Before proceeding to step 2, verify you have a test plan with known good results. If you don’t, take the time to make one and run it against your test environment. Record the results so that you can use them as a baseline.

Sync Your Test Environment To Production

The first step is to make sure your test system is exactly the same as production.

Tip: If for some reason you don’t have a test system, get one. It is a valuable tool for ensuring you can successfully roll out changes to a production environment.

There is nothing more stressful than finding out during a production deploy window that things are not the same as the environment you tested in. At a minimum, verify the following:

• Same software version/patches

• Same config files

• Same environment (operating system)

There are some great tools out there for identifying changes to a directory of files. One of my favorites is a free tool called WinMerge. I can quickly use this tool to identify the changes between two different environments. It gives me that assurance that I have completed step 1 correctly.

Before proceeding to step 2, verify you have a test plan with known good results. If you don’t, take the time to make one and run it against your test environment. Record the results so that you can use them as a baseline.

Apply Your Changes To The Test Environment

There is a reason you are going through this effort, it may be to modify a product, it may be to adopt a new version of a 3rd party tool. Now is the time to make that change to your test environment. This is a good time to start a draft of your deploy script. Take some notes of the steps you are going through to apply your changes.

Run Your Test Plan In The Test Environment

With your changes made, it is now time to run your test plan. You will want to record your results and compare them to your baseline results. The next step is to interpret the results. Sometimes you want to see that there was no change in the results, other times you are looking for a positive change. Interpret the results and hopefully you conclude they are desirable. If not, feel good that you caught something before it went to production. Take measures to fix things and start back on step 1 when the new candidate is ready.

Production Change Planning

You’ve ran your tests, you are convinced this is a positive change to make. It is now time to involve the rest of the organization in your efforts and put the change in motion. Work with your operations team to identify a suitable window of time in the future to perform the upgrade. Communicate to your organization about the change you plan to make and highlight the benefits for them. Prepare them for any changes that will affect the way they work. Give them a suitable amount of time to process the information you gave them and raise any objections to the change.

While giving them time to review the change, finalize your production deploy script. Put all the steps in a list that you can check off when they are complete. I put all steps in the list whether they are easy or hard steps. The list is a reassurance that not only did I perform the intended steps, but if I get interrupted, I have a clear place to go back to and successfully finish the process. All good deploy scripts end with a smoke test. The smoke test(s) should validate that your change is effective, and that it didn’t have any unexpected side effects.

The other script that you should create is your rollback plan. Pretend that you have to rollback the change that you are introducing, what are the steps you expect to follow. Creating this script is also a good validation to your deploy steps. For example, if your rollback script includes restoring a database, then you better make sure your deploy steps included making a backup of the database.

Update Production

You have done all the planning and invested many hours leading up to this moment. The hard work should be behind you and the real update window should be a mechanical process.

The first step, is to validate that nothing has changed since the day you synced your test environment to production. If it has, identify the change and reschedule the deployment until you can mitigate that new risk. If there have been no changes, take a deep breath, find a quiet place to do your work and follow your scripted steps. Take notes of anything that is interesting. Run your smoke test and validate that things have gone as planned.

Communicate The Change

Let everyone know that your change is complete and give control of the system back to production. Be redundant and restate the reason for the change and the positive effect it will have. Communicate how you want to be reached if someone has questions or experiences something they feel is unexpected and related to the update.

Follow-up

You will know an appropriate time to solicit feedback from the people affected by your production change. Maybe it is only a half hour later, maybe you have to wait a day. In either case, go out of your way to get confirmation that the change is performing well for those who were affected by it. Some of the more satisfying nuggets for future changes can be learned by following up on previous changes.

Archive Your Plans

By having a change process and following it, you have just demonstrated some skills that will almost ensure that you will get to do it again. Archive your work and the next time around will be streamlined from those efforts. By documenting the process from your test plan, results, communication, deploy scripts, etc… you have now enabled another team member to perform this task too.

If you have delays during any step, communicate them early. Nobody likes to hear of a delay, but if they understand it is because you are working towards a flawless production update, it lightens the blow.

What are your best production change process tips?