Build & Release Challenges : Manual DB Updates

The first problem that I'm gonna discuss is manual db updates. In our current application we do have automated DB updates execution in the production environment, but not in the rest of environments i.e dev, qa, stage, performance test ... etc.

The process that we use for automated scripts execution in production environment  is that we create a release folder, this release folder contains all the sql scripts for the release along with a meta file. The release meta file contains the list of all the scripts that needs to be executed, the current system reads this meta file & executes all the scripts of release. This process is fair enough for production system since the release is deployed only once on production system. In production systems we don't have to track whether a script got executed or not i.e all the scripts execution is treated as atomic that is either all the scripts are executed or none is executed.

The drawback of atomic execution is the reason because of which this approach can not be applied to rest of the environments, since the db updates will always be incremental in rest of environments. In case of all other environments apart from production environment the release will be deployed multiple number of times, with each release new db scripts can be added  to the system and only those new scripts needs to be executed.

The new system that I'm trying to develop will have incremental db update capability. The system that I'm planning to deveplop will have following characterstics :

• It should be able to keep track of script name for later reference.
• It should store the release mapping to which this script belongs.
• The sequence of the script to enforce the order of execution.
• The system should also maintain whether the script is already or not.
• The system should be able to handle error scenario i.e if a script execution fails a corrective action should be taken by the system
• It should be extensible enough so that various kind of reports can be generated from it

In the next blog I'll be talking about the actual system how it is built

Prev
Next

Build & Release Challenges : Problems

So here is the consolidated list of the problems that current system have, I've categorized all the issues in different categories so that they can be managed properly

• CI Builds
• Code stability builds are not in place
• Code quality builds are not in place
• Code deployment builds for non-prod environments not in place
• A lot of manual steps in prod deployments
• All the projects are performance critical so builds to do profiling of projects
• Database updates is manual at all the stages of deployment
• Automated smoke testing of all the applications
• Integration of bug tracking system with version control system
• A release server needs to be set-upped so that release can be managed properly
• No documentation at all :)
• Version Control System : We are using git as version control system
• Branching strategy has some flaws as a result of which merging takes a lot of time
• We don't have a GUI to manage git server

Prev                                                                                                                  Next

Build & Release Challenges

I was planning to write this blog 2-3 days back in fact not a blog but a blog series & this blog will be the starting of this blog series. This blog will only give you an overview what I'll be discussing in the coming blogs, first the reason why I'm writing this blog :) well the reason is I've changed the job :) and I'll be working as a Build & Release manager. The challenge that I've have is that right now their is very little or no streamlined processes defined for build and release and obvious very less automation so I've to work on these things. This means that in the coming blogs I'll be discussing various problems that I'll be facing & how to overcome that.

Next 

Automated Database Update Or Rollback

One of the important step during release is doing database update and rollback in case something goes wrong, usually people perform this operation manually. In this blog I'll talk about how we can automate this process by following some convention.

Here I'm taking mysql database as an example we can have same conventions for other databases also

Convention to manage rollback/updates of a release

• Each project codebase at it's root will have a folder database_scripts
• The database_scripts folder will contain folder for each release i.e Release1_1, Release2_0...
• The database scripts release folder will in turn contains two folders update & rollback which will contain updates & rollbacks scripts for a release.

Automating the rollback/update

• The update folder will have a source input file FileSequencer.txt. This file will point to all the update scripts in correct order that needs to be executed for the release
• In the similar manner rollback folder will have a source input file FileSequencer.txt. This file will point to all the rollback scripts in correct order that needs to be executed for the release
• At last we will have a utility shell script, this script will take db details and execute all the scripts referred in FileSequencer.txt using mysql command

Release Strategy for Java Web based projects

In this post I'll be discussing about the 2 strategies that  we can follow for releasing a Java based web project.

A project can be primarily released in two ways
    Incremental Release
    Full Release

Incremental Release is done in big projects which has multiple modules & usually few modules gets updated between two releases. It makes sense to include only updated modules in release archive and during deployment only update those modules in application server.

Full release is usually done in small projects where the release archive contains all the components and then this release archive can be deployed to the application server as a whole

Both incremental & full release strategy has their pros & cons, where full release strategy scores in simple release archive generation & deployment incremental release has upper hand in space usage by only having modified components in it, although it brings overhead when doing rollback.


Release Steps in Incremental Release Strategy: If you are following incremental strategy in general you need to perform following steps

1.) Checkout the latest code for the release
2.) Generate the list of components which needs to be deloyed for the release
3.) Generate the release archive based on the list of components
4.) Stop the server(If hot deployment of components is not available)
5.) Take the backup of existing application on application server as we may need to do rollback in case of any issues
6.) Replace the components in application server with the components in the release archive
7.) Start the server(If hot deployment of components is not available)


Release steps in Full Release Strategy: As explained earlier Full release strategy is fairly simple, steps involved are:
1.) Checkout the latest code for the release
3.) Generate the release archive for whole application
4.) Stop the server(If hot deployment of components is not available)
6.) Deploy the release archive to the application server
7.) Start the server(If hot deployment of components is not available)

Git : How to fix issues in a merged branch

Sometime back we faced a peculiar problem in our project where we have to fix some issue introduced due to a merged branch. To explain this problem first I've to explain the way we manage branches. As we entered in the maintenance mode of project there was a dire need of fixing issues and working on change requests in isolation. Whenever we need to fix an issue we cut a issue-branch out of main branch fix that issue in isolation and after verifying and making sure the issue is fixed we merge that issue-branch back in main branch. The issue we faced with the approach is that after merging an issue branch with the main branch sometimes we came across some regression bugs due to the issue branch. There were two possible solutions to overcome the problem.

First solution is to fix the issue in issue branch and merge the issue branch again with master. This solution can work but the problem with this approach is that till the time you haven't fixed the issue introduced due to issue branch and merged it back with master branch you can not create a new issue-branch and much bigger problem is you can't release.

The second solution is if we can somehow revert the merge of issue branch with main branch and then fix the issue in issue-branch, after that merging the fixed issue branch with the main branch. This approach seems to be straight forward and more logical. Git comes up with a cool command git-revert which can revert existing commits and even revert merge of another branch. I'll talk about the solution in the next blog ;)

My experience with Git

Few days ago I came to know about git, so to give it a try I created an account on github. To those who doesn't know much about it GIT is a fast version control system :). Since I'm using windows I've to download a git client, so the first google result that came on typing "git window" was "msysgit". Msysgit comes with 2 options "gitbash" a command line utility and git ui so you can use any of them, for initial learning perspective I preferred GitBash.

I'll not explain the concepts of git as there is lot of information available on the net
http://en.wikipedia.org/wiki/Git
http://git-scm.com/

So first I've to create a remote repository. GitHub provides online project hosting using git, they have different plans as according to your needs. I've created a free account so I can create public repostiories, which means anybody can checkout the code and you can allow people to modify the code. If you want to create private repository then you have to create a paid account, they have multiple plans as per your need https://github.com/plans. So my user at github is sandy724.

In next blog I will discuss how I've used GitBash for synching up the code with remote repository