jgit-flow maven plugin to Release Java Application

Introduction

As a DevOps I need a smooth way to release the java application, so I compared two maven plugin that are used to release the java application and in the end I found that Jgit-flow plugin is far better than maven-release plugin on the basis of following points:

• Maven-release plugin creates .backup and release.properties files to your working directory which can be committed mistakenly, when they should not be. jgit-flow maven plugin doesn't create these files or any other file in your working directory.
• Maven-release plugin create two tags.
• Maven-release plugin does a build in the prepare goal and a build in the perform goal causing tests to run 2 times but jgit-flow maven plugin builds project once so tests run only once.
• If something goes wrong during the maven plugin execution, It become very tough to roll it back, on the other hand jgit-flow maven plugin makes all changes into the branch and if you want to roll back just delete that branch.
• jgit-flow maven plugin doesn't run site-deploy
• jgit-flow maven plugin provides option to turn on/off maven deployment
• jgit-flow maven plugin provides option to turn on/off remote pushes/tagging
• jgit-flow maven plugin keeps the master branch always at latest release version.

Now let's see how to integrate Jgit-flow maven plugin and use it

How to use Jgit-flow maven Plugin for Release

Follow the flowing steps 

1. Add the following lines in your pom.xml for source code management access

          
          <scm>
            <connection>scm:git:<Git Repo URL></connection>
            <developerConnection>scm:git:git:<Git Repo URL></developerConnection>
          </scm>
          
    

2. Add these line to resolve the Jgit-flow maven plugin and put the other option that will be required during the build

          
   <build>
       <plugins>
         <plugin>
           <groupId>com.atlassian.maven.plugins</groupId>
           <artifactId>maven-jgitflow-plugin</artifactId>
           <version>1.0-m4.3</version>
           <configuration>
             <pushReleases>true</pushReleases>
             <keepBranch>false</keepBranch>
             <autoVersionSubmodules>true</autoVersionSubmodules>
             <noTag>true</noTag>
             <allowUntracked>true</allowUntracked>
             <pullDevelop>true</pullDevelop>
             <pullMaster>true</pullMaster>
             <allowSnapshots>true</allowSnapshots>
             <flowInitContext>
               <masterBranchName>master-test</masterBranchName>
               <developBranchName>deploy-test</developBranchName>
             </flowInitContext>
           </configuration> 
        </plugin>
     </plugins>
   </build>
            

Above code snippet will perform following steps:
• Maven will resolve the jgitflow plug-in dependency
• In the configuration section, we describe how jgit-flow plug-in will behave.
• pushRelease XML tag to enable and disable jgit-flow from releasing the intermediate branches into the git or not.
• keepBranch XML tag to enable and disable the plug-in for keep the intermediate branch or not.
• noTag XMl tag to enable and disable the plug-in to create the that tag in git.
• allowUntracked XML tag to whether allow untracked file during the checking. 
• flowInitContext XML tag is used to override the default and branch name of the jgit-flow plug-in
• In above code snippet, there is only two branches, master from where that code will be pulled and a intermediate branch that will be used by the jgit-flow plug-in. as I have discussed that jgit-flow plug-in uses the branches to keep it records. so development branch will be created by the plug-in that resides in the local not remotely, to track the release version etc. 
3. To put your your releases into the repository manager add these lines

          
   <distributionManagement>
     <repository>
       <id><auth id></id>
       <url><repo url of repository managers></url>
     </repository>
     <snapshotRepository>
       <id><auth id></id>
       <url><repo url of repository managers></url>
     </snapshotRepository>
   </distributionManagement>
         
    

4. Put the following lines into your m2/settings.xml with your repository manager credentials

           
   <settings>
     <servers>
       <server>
           <id><PUT THE ID OF THE REPOSITORY OR SNAPSHOTS ID HERE></id>
          <username><USERNAME></username>
          <password><PASSWORD></password>
       </server>
     </servers>
   </settings>
         
    

Start Release jgit-flow maven plugin command

To start the new release execute jgitflow:release-start.  

Finish Release jgit-flow maven plugin  command

To finish new release, execute mvn jgitflow:release-finish.

For a example I have created a repository in github.com. for testing and two branch master-test and deploy-test. It is assumed that you have configured maven and git your system.
In the deploy-test branch run following command
$ mvn clean -Dmaven.test.skip=true install jgitflow:release-start


This command will take input from you for release version and create a release branch with release/<version>. then it will push this release branch into github repository for temporarily because we are not saving the intermediate branched

Now At the end run this command
$ mvn -Dmaven.test.skip=true jgitflow:release-finish
after finishing this command it will delete release/<version> from local and remote.

Now you can check the changes in pom file by jgitflow. in the above snapshot, it is master-test branch, you can see in the <version> tag it has removed the snapshot and also increased the version.  It hold the current version of the application.

And in the deploy-test branch it show you new branch on which developers are working on

Opstree SHOA Part 1: Build & Release

At Opstree we have started a new initiative called SHOA, Saturday Hands On Activity. Under this program we pick up a concept, tool or technology and do a hands on activity. At the end of the day whatever we do is followed by a blog or series of blog that we have understood during the day.

Since this is the first Hands On Activity so we are starting with Build & Release

 

What we intend to do 

 

Setup Build & Release for project under git repository https://github.com/OpsTree/ContinuousIntegration.

What all we will be doing to achieve it

• Finalize a SCM tool that we are going to use puppet/chef/ansible.
• Automated setup of Jenkins using SCM tool.
• Automated setup of Nexus/Artifactory/Archiva using SCM tool.
• Automated setup of Sonar using SCM tool.
• Dev Environment setup using SCM tool: Since this is a web app project so our Devw443 environment will have Nginx & tomcat.
• QA Environment setup using SCM tool: Since this is a web app project so our QA environment will have Nginx & tomcat.
• Creation of various build jobs
• Code Stability Job.
• Code Quality Job.
• Code Coverage Job.
• Functional Test Job on dev environment.
• Creation of release Job.
• Creation of deployment job to do deployment on Dev & QA environment.

This activity is open for public as well so if you have any suggestion or you want to attend it you are most welcome

Revert a patch in most awesome way

If you are a Release Engineer, System Admin or Support Engineer you have definitely come across a requirement where you have to apply patches to the target systems be it production or non-production. I'm assuming that you are using some automated system to manage the patches i.e applying them and reverting them. In this blog I would be discussing about the standard way of patch management and how you can have an out of the box solution to revert your patch in most simplistic way and without much fuss. At the end of the blog I would like to see an expression where you will say what the hell it's so awesome yet so simple :).

People usually use some tool to apply patch to a target system which in addition to applying a patch also manage the history the patches so that it can be reverted in case the patch goes wrong. The patch history usually contains below details:

1. The new files that were added in the patch, while reverting the patch those files should be deleted.
2. The files that were deleted by the patch, while reverting the patch the deleted files should be restored back.
3. The files that were modified by the patch, while reverting the patch the modified files should be restored back.

You can definitely create a tool that can revert the patch for you as the use cases are not much, but do you really need to put this much effort if you can have an out of the box solution for this. What if I tell you that we use git for managing our patch history and reverting them. As git comes with a local repository concept so we created a local git repository at our app server codebase location only. Git comes with all the file level tracking we map each patch with one git commit, so at the time of reverting a specific patch you can ask git to simply revert the commit for you.

Extra steps to be done after applying patch:
To make git track the changes done in patch, you just need to perform 2 extra commands

git add . : This command will track all the files that have been modififed, added or deleted in the system.
git commit -m "Applying Patch" : This command actually adds the files information tracked by previous command with a message in the git system


Steps to be done in reverting changes done by a patch:
Once you have all the information tracked in git it will become no-brainer to revert the patches.

To view the details of all the patches: You can use git log command that will provide you the list of all the patches that you have applied or reverts that you have done

sandy@sandy:~/test/app1$ git log
commit f622f1f97fc44f6897f9edc25f9c6aab8e425049
Author: sandy
Date:   Thu Jun 19 15:19:53 2014 +0530

    Patch 1 on release2

commit 9a1dd81c7799c2f83d897eed85914eecef304bf0
Author: sandy
Date:   Thu Jun 19 15:16:52 2014 +0530

    Release 2

commit 135e04c00b3c3d5bc868f7774a5f284c3eb8cb29
Author: sandy
Date:   Thu Jun 19 15:16:28 2014 +0530

    Release 1

Now Reverting a patch is as simple as executing a simple command git revert, with the commit id of the patch

git revert f622f1f97fc44f6897f9edc25f9c6aab8e425049
[master 0ba533f] q Revert "Patch 1 on release2"
 1 file changed, 1 deletion(-)


If you run git log command, you will see the patch revert history as well

sandy@sandy:~/test/app1$ git log
commit 0ba533fda95ed4d7fcf0b7e6b23cd1a5589946a7
Author: sandy
Date:   Thu Jun 19 15:20:24 2014 +0530

    Revert "Patch 1 on release2"
   
    This reverts commit f622f1f97fc44f6897f9edc25f9c6aab8e425049.

commit f622f1f97fc44f6897f9edc25f9c6aab8e425049
Author: sandy
Date:   Thu Jun 19 15:19:53 2014 +0530

    Patch 1 on release2

commit 9a1dd81c7799c2f83d897eed85914eecef304bf0
Author: sandy
Date:   Thu Jun 19 15:16:52 2014 +0530

    Release 2

commit 135e04c00b3c3d5bc868f7774a5f284c3eb8cb29
Author: sandy
Date:   Thu Jun 19 15:16:28 2014 +0530

    Release 1


I hope this blog has given you a very different perspective of managing the patches, let me know your thoughts about this. Also if you have such more ideas do share with me.

A wrapper over linode python API bindings

Recently I've been working on automating the nodes creation on our Linode infrastructure, in the process I came across the Linode API and it's bindings. Though they were powerful but lacks at some places i.e:

1. In case of Linode CLI, while creating a linode you have to enter the root password so you can't achieve full automation. Also I was not able to find an option to add private ip to the linode
2. In case of Linode API python binding you can't straight away create a running linode machine.

Recently I've launched a new GitHub project, this project is a wrapper over existing python bindings of linode and will try to ease out the working with linode api. Currently using this project you can create a linode with 3 lines of code

from linode import Linode
linode=Linode('node_identifier')
linode.create()

You just need to have a property file,/data/linode/linode.properties:

[DEFAULT]
UBUNTU_DIST=Ubuntu 12.04
KERNEL_LABEL=Latest 64 bit
DATACENTER_LABEL=Dallas
PLAN_ID=1024
ROOT_SSH_KEY=
LINODE_API=

 The project is still in development, if someone wants to contribute or have any suggestions you are most welcome.

Puppet module to setup nodejs deployment 2

As I said in the previous blog Puppet module to setup nodejs deployment, the nodejs module was for providing the basic infrastructure for automated node app's deployment & as promised I've released the next module "nodeapp" that can be used to setup a node app on the target server.

First of all I'll talk about what this module will do to facilitate the automated deployment of a nodejs app, as already discussed we are following a convention that all the node app's code will be present at /home/nodejs/ which is referred by startNodeApp.sh script so we create the directory of nodejs app. The deployNodeApp.sh script was using the upstart to manage the nodejs app instance i.e starting/stoppping the nodejs app, the nodeapp module takes care of creating the require upstart configuration at /etc/init/.conf. Also we use monit to monitor the nodejs app's so that we can start/stop the nodejs app's using the web ui of monit & also see various stats such as cpu, memory, load.. consumption of nodejs app.

This nodeapp module is a userdefined type which takes the name of nodeapp as an argument, as a result of which you can setup any number of nodejs app's on a system.
i.e nodeapp{'search-demo': app_name => "search-demo"}
This entry will create below files

/etc/init/search-demo.conf : An upstart configuration file, using which search demo nodejs app can be managed as a service.

#!upstart
description "node.js search-demo server"
author      "sandy"

start on startup
stop on shutdown

script
    export HOME="/home/nodejs"

    echo $$ > /var/run/search-demo.pid
    exec sudo -u nodejs /home/nodejs/startNodeApp.sh search-demo
end script

pre-start script
    # Date format same as (new Date()).toISOString() for consistency
    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/log/search-demo.sys.log
end script

pre-stop script
    rm /var/run/search-demo.pid
    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" >> /var/log/search-demo.sys.log
end script

/etc/monit/conf.d/search-demo.monit : A monit configuration file, using which search-demo nodejs app can be monitored & even automatedly restarted

check process search-demo with pidfile /var/run/search-demo.pid
 stop program = "/sbin/stop search-demo"
 start program = "/sbin/start search-demo"

So using these 2 modules nodejs & nodeapp you can make any system up & running for nodejs app's automated deployment

Automated DB Updater Release First Release

Initial version of Automated DB Updater Release ADU

With this blog I'm releasing the intial version of a python utility to provide automated db updates across various environments for different components.

The code for this utility is hosted on github
https://github.com/sandy724/ADU

You can clone the read only copy of this codebase by url given below
https://github.com/sandy724/ADU.git

To understand the basic idea about this utility go thorugh this blog
http://sandy4blogs.blogspot.in/2013/07/automated-db-updater.html

How to use this utility
    Checkout the code at some directory, add the path of this directory in PYTHONPATH environment variable
    Create a database with a script's metadata table with given below ddl
CREATE TABLE `script_metadata` (
  `name` varchar(100) DEFAULT NOT NULL,
  `version` int(11) DEFAULT NOT NULL,
  `executed` tinyint(1) NOT NULL DEFAULT '0',
  `env` varchar(30) DEFAULT NOT NULL,
  `releas` varchar(30) DEFAULT NOT NULL,
  `component` varchar(30) DEFAULT NOT NULL
)
    Create a database.properties, containing connection properties of each environment database

[common_db]

dbHost=localhost

dbPort=3306

dbUser=root

dbPwd=root

db=test

[env1]

dbHost=localhost

dbPort=3306

dbUser=root

dbPwd=root

db=test

Here common_db represents connection to database which will contain metadata of scripts for monitoring

    Now execute the pythong utility
        Copy the client(updateDB.py) to directory of your choice, make sure that property configration file should also be at this directory
        python updateDB.py -f -r --env

Automated DB Updater

In continuation with my blog series I'm finally introducing a automated db updater tool. You can read about the idea in previous blogs by going to below links

Manual DB Updates challenges
Manual DB Updates challenges-2

The short form of my tool is ADU(Automated DB Updater). Now some details about this tool

Each application will have database_script folder at the root level, this folder will contain folders corresponding to each release i.e release1, release2, release3...

A database release folder will contain

• Meta file :sql_sequence.txt, this file will contain the sequence in which sql files will be executed, only files mentioned in this file will be entertained
• SQL Files : A sql file must have a naming convention like this __.sql/__.sql

Process of automatic execution of scripts on an environment

• Input
• release_name : to figure out the folder from where scripts will be executed
• environment : Environment on which scripts will be executed
• Execution
• sql_sequence.txt file will be read line by line having one sql file name in each line
• The sql file will be verified whether it has been already executed or not
• If the sql file is already executed then two conditions are verified
• A new version of sql should be available
• Undo version of last executed sql should be present
• After execution of undo file the latest version of the sql file will be executed and the info is stored accordingly that it has been executed so that it will not be picked again
• Validations & Boundary Conditions
• All the files mentioned in sql_sequence.txt should exist.
• Undo script should be present for all the versions of a sql file barring the latest version of sql file.
• Undo script will only be executed if next version of script is available.

Very soon I'll share the github url of this project keep waiting :)

Puppet module for setting up Multiple mongo's with replication

In this blog I'll be talking about a puppet module, that can be used to installing multiple mongo's with replication on a single machine. Since I'm very new to puppet so you may find this module very crude, but it works :). Their were couple of puppet module already available but most of them are only for installing a single instance of mongo at a machine & I've a specific requirement of installing multiple instances of mongo having master slave replication between them. As I already said that this module may be quiet crude or basic so please bear with that & my approach may also seem a bit unconventional so please let me know what all can be improved in this module or how things could have been done in a better way.

So let's start with the actual details first of this module is hosted on github(https://github.com/sandy724/Mongo), if you want to look at the source code you can clone it from github. For installing mongo you would be executing the command
puppet apply -e "class {mongo:port => , replSet => ,master => ,master_port => ,}"

Command for installing master
puppet apply -e "class {mongo:port => 27017, replSet => sdrepsetcommon,master => master, master_port => 27017,}"

Command for installing slave
puppet apply -e "class {mongo:port => 27018, replSet => sdrepsetcommon,master => slave,master_port => 27017,}"

Before going into the details what all this module is doing I will share some details of mongo

• You can start mongo by executing mongod command
• You can provide a configration file which contains details such as
• log directory where mongo would be generating the logs
• port at which mongo would be listening for requests
• dbpath where mongo would be storing all the data
• pidfilepath containg process id of mongo instance, that would be used to check whether mongo is running or not
• replSet name of the replicaset
• You need to have a mongo as a service installed in you system to start an instance of mongo
• For replication you need to execute rs.initiate command on the master mongo
• For adding another instance into replication you need to execute rs.add(":")  command on the master mongo

   
Now let's go into more details what all this component does, I'll be listing down all the steps in bulleted points

• As you can figure out this module is expecting few parameters :
• port : port at which mongo would be listening,
• replSet : name of replicaset which would be used for managing replication
• master : A string parameter which would signify whether the mongo setup is for master or slave
• master_port : Port at which master instance of mongo would be listening
• First of all we create a mongo user
• Parent Log directory for the mongo instance is created if it doesn't exists with mongo user as owner.
• Mongo db directory is created under /data/mongo with a naming convention replSet_port, i.e if replSet parameter is sdrepsetcommon & port is 27017 then the data directory for this mongo instance would be  /data/mongo/sdrepsetcommon_27017. This directory would be owned by mongo user.
• A mongo service would be installed if not already their.
• A mongo restart shell script is also placed at the mongo db directory
• A file is also placed under the mongo db directory that have a mongo command to setup replication, this file is created conditionally depending on whether we are setting up a master or slave instance.
• Finally the replication command is executed on mongo server & restart script is also executed

This concludes the setting up of a mongo instance on a machine.

Just for more details to start mongo we are using mongod -f command, this configuration file is saved as a template & the mongo modules processes the template with the values passed & creates the desired mongod.conf. In our case we are evaluating following properties of mongod.conf : logpath, port, dbpath, pidfilepath, replSet

Initial thoughts for an automation testing framework/utility

My first exposure to selenium was in 2010/2011 & I was quiet impressed with it, the way you can use selenium for the testing of web application was totally awesome. At that time I was working with xebia,  our team was working for website revamp of a dutch travel company. We were using selenium for all the regression & functional testing of website, 80% of the website testing was done only by Selenium.

One of the challenge with selenium is that for each test scenario you have to write code for that & if you don't manage your test scenario/cases effectively, the management of selenium test cases becomes a task in itself. At that point of time also we tried to make maximum use of Java to make selenium test cases as structured & Object oriented possible so that they can be extended & managed easily. I always had a desire to do some improvement in that area so that the selenium test cases management should be made more easier.

In my current company most of the testing is done manually, since I had a prior experience of Selenium & experienced the power it brings to your testing. I was pretty determined to bring the selenium advantage in our company. Off-late an automation testing was set-upped in our company as well which was working on leveraging the power of selenium in testing, but again it was same problem you have to write a lot of code. The other challenge that automation team was facing, the UI of the site was changing very frequently so whatever work they used to do was becoming back to zero after few iterations.

Last week along with my team I've started doing some head banging that let's see if can do something out of the box, in a normal discussion with my team members one idea stuck to us. The manual QA team of our company is very strong & they have complete in & out of idea of whole application, but they have so much work assigned to them that they can't spend their time in the selenium. We wanted to club the knowledge of our manual testing team & the power of selenium.

As a POC we buit a very simple utility that will read a meta information file and executes the commands listed in that meta file. As an example if they want to open a page one of the line of meta file will contain a command "open url", similarly if they have to click a button the command will be something like "click . This utility was doing exactly what we wanted to do. We are still in the POC phase where we are trying to include as much commands as possible


Let me know about your thoughts for this approach, suggestions are most welcome.

Automation tips and tricks February 2013

As promised I'm back with the summary of cool stuff that I've done with my team in Build & Release domain to help us deal with day to day problems in efficient & effective way. As I said this month was about creating tools/utilities that sounds very simple but overall their impact in productivity & agility of build release teams and tech verticals was awesome :).

Automated deployment of Artifacts : If you have ever worked with a set of maven based projects that are interdependent on each other, one of the major problem that you will face in such a setup is to have the latest dependencies in your local system. Here I'm assuming two things you would be using a Maven Repo to host the artifacts & the dependencies would be SNAPSHOT dependencies if their is active development going on dependencies as well. Now the manual way of making sure that maven repo will always have the latest SNAPSHOT version is that every-time somebody does change in the code-base he/she manually deploy that artifact to maven repo. What we have done is that for each & every project we have created a Jenkins job that check if code is checked in for a specific component & if so that component's SNAPSHOT version get's deployed to maven repo. The impact of these utilities jobs was huge as now all the developers doesn't have to focus on deploying their code to maven repo & also keeping track of who last committed the code was also not needed.

Log Parser Utility : We have done further improvement in our event based log analyzer utility. Now we also have a simple log parser utility through which we can parse the logs of a specific component & segregate the logs as per ERROR/WARN/INFO. Most importantly it is integrated with jenkins so you can go to jenkins select a component whose log needs to be analyzed, once analysis is finished the logs are segregated as per our configuration(in our case it is ERROR/WARN/INFO) after that in the left bar these segregations are shown with all the various instances of these categories and user can click on those links to go exactly at the location where that information is present in logs

Auto Code Merge : As I already told we have a team of around 100+ developers & a sprint cycle of 10 days and two sprints overlap each other for 5 days i.e first 5 days for development after tat code freeze is enforced and next 5 days are for bug fixing which means that at a particular point of time there are 3 parallel branches on which work is under progress one branch which is currently deployed in production second branch on which testing is happening and third branch on which active development is happening. You can easily imagine that merging these branches is a task in itself. What we have done is to create an automated code merge utility that tries to merge branches in a per-defined sequence if automatic merge is successful the merge proceeds for next set of branches otherwise a mail is sent to respective developers whose files are in conflict mode

Hope you will get motivated by these set of utilities & come up with new suggestions or point of improvements

Initial thoughts for a patch framework for a java based web project

Although this blog was not in the pipeline of feb month but I got a requirement to build a patch framework for a java based web project, so along with building this framework I thought of writing this blog as well so that I'll get idea from other people as well.

First of all I will talk about what can be patched using this patch framework, majorly it will be three resources/things that can be patched

• Class Files
• JSP's
• Static Resources such as images, css, js ...

I'm thinking of adding few other features in this patch framework as well

• Sequence of patches should be maintained : Since we have a big team around 80 developers working on same code-base, their may be a scenario that we can have two or more patches which needs to be applied to a target system. Their may be a fair chance that those patches have to be executed in a sequence or you can say their could be dependency among patches.
• Validation while applying patches : One of the validation that I can think of is that the resources that have to be patched will be either new or existing one & in case of existing resources the system should verify the location at which resources are patched should already have those resources
• Rollback : The patch framework should have rollback capability

I'm planning to build this patch framework using

• Shell scripting : For programming aspect
• Git : As a version control system for storing patches
• Jenkins : Provide a front-end to user for applying patches
• Mysql : Not so sure about it yet but I've to store few information such as what all patches are applied, sequence of patches.... I can use file systems as well for storing this information

Let me know your thought about  this framework or any other feature that you can think of

Automation tips and tricks January 2013

I'm starting a new blog series in which I'll be talking about various cool things or automations that I along with my team done in a month and what are my plans for next month.

Talking about January 2013, I've done following things

1.) Streamlining of environments : The big step in streamlining the environments is to change the owners of the application from root user to tomcat user & making ports of all the application consistent across environments i.e dev, qa, pt & staging. This will help me in my long term goal of introducing a server configuration tool most preferably puppet.
2.) Log Analyzer Utility : One of the major challenge that teams face is to get real time notifications of any exceptions that occur in the server logs, to overcome this problem we have written a log analyzer utility that will scan a log file backed by a meta file, this meta file have the information about who all should be notified for an exception. This utility is written in shell script and integrated with Jenkins CI server so that we can schedule the execution of this utility as per convenience, currently jenkins is executing this utility after every 15 minutes.
3.) System monitor : Off late we were facing challenge of servers getting disk out of space & when whole system goes down then only we were able to figure out the issue is due to disk space outage due to huge log files, to overcome this problem we have built a small shell utility that scans couple of folder's recursively and provide a list of top 10 files whose size is greater then a specified threshold. In our case we have set this threshold as 1 GB, also all these variables can be provided as input to this utility such as folder's to scan regular expression of files which needs to be considered the threshold value

This is what we have achieved in the month of January 2013 although these utilities seems to be but obvious and simple one but the effect they have in the productivity of the team is considerable.


Now plans for the month of February 2013, usually I choose those things which we are doing manually, this month we will be working on following things
1.) Utility which can perform automated merge if possible
2.) Utility that can automatically upload the artifacts to a central server(artifactory in our case)
3.) Integration of git common operations with Jenkins

Efficiently handling Code merge in Version Control System

One of the painful & mundane task that release engineers have to perform is to merge changes of one branch into another branch & in case of code conflicts the release engineer has to co-ordinate with all the developers to resolve those merge conflicts.

In our current setup the problem is more critical as development of two releases overlap with each other . We have a sprint cycle of 10 days where we have 5 days of active development after that code freeze is implemented & rest 5 days are only for big fixes. The next sprint starts just after the code freeze date of previous release. In ideal scenario this setup should work well but the biggest assumption behind successful execution of the process is their should be minimum code check-ins after code freeze & usually that doesn't happens. This results in parallel development in 2 branches & therefore while merging two branches their are lot of code conflicts.

The real problem starts when we start merging code, as currently their are close to 100 developers working on the same code-base which means a huge list of files in conflict mode & you have to chase down each & every person to resolve those conflicts. To overcome the above-said problem we are planning to do 2 things.

First one is instead of doing merge after a long duration we are planning to increase the frequency of merge from once in 5 day to twice a day which would help us to reduce the list of conflicting files.

As I always strive to automate things as much as possible, the second part is to at-least create an automated tool that will perform a dummy merge of two branches and list out all the files that would result in conflict mode along with listing the last user's who have modified the files in respective branch.

We are expecting 60-70% efficiency in code merge process, let's see how things goes. Feel free to drop any ideas if you have or in case of any concerns :).

Although I tried to be as generic as possible, but just to let you know we are using Git as version control system.

Build & Release Challenges : Manual DB Updates Part 2

Previos

This blog was supposed to be about the new system, I thought of building to solve the problem that I discussed in my previous blog. Well for your disappointment this blog will be not about that, the reason is scope of the problem changed. In this blog I'll be discussing about the new scope and how discussion moved forward about it and what is the current state, which means that I'm still not able to solve this problem & suggestions are welcome :).

I'll again state the problem which was very simple enough, "database updates were not automated in non prod environments as same db scripts were modified during development". You need to refer to previous blog for more details about this problem. To solve this problem I came up with incremental db update approach, as per this approach all new modifications will be done as a new sql update which means that let's say you had a file 1.sql, if you need to do any modification a new file 1'.sql should be committed. In this way the system don't have to track the files, it just have to maintain what all files got executed, the new files which needs to be executed and execute the new files only. This solution can work in a normal setup very well, in fact in my last assignment I was using this approach only to have automated db updates across all environments.

The incremental db updates can't be run in current setup, the reason for that is we have very huge database order of 100GB, you can easily imagine that we can't afford to run same script with slight modifications i.e first script adding a column of size 20 then another script to change it's size to 40 finally renaming it to some other name. Instead of that a single script should be created after consolidating all these scripts.

The first solution that came to my mind after this new issue emerged, during non prod deployments we should already have database dump of previous release, more preferably cold dump. During deployment 3 steps would be performed first load previous release db dump, run all the consolidated scripts which will be consolidated & do the code deployment. Initially this solution looked fine enough but QA team raised concern as loading previous release dump meant that all the test data  they have created on the QA server would be lost and I was at the beginning of square :).

Another solution that could be implemented  was to have rollback script for each & every script committed. This convention will have an advantage of supporting incremental update i.e whenever a script will be updated first it's corresponding rollback script will be executed & then the script can be executed. This solution has it's own challenges the first challenge was it's really difficult to write rollback script of each & every script, another issues was you have to carefully manage the script files so that there will be no tight coupling between them as execution of rollback of one script will impact another script. Third issue although less significant is that you have to deal with data loss

We could also have used a hybrid approach that is combination of incremental & full db updates. Till QA phase we can use incremental db update mechanism in which all new script modifications are done as a new script and then they can be executed incrementally but for staging & production deployment db updates will be done as a full update which requires a human intervention i.e consolidation of scripts. This approach had 2 challenges the first & foremost was that it had manual intervention & second major issue was that we were duplicating the db scripts.

So these were the few approaches that we thought of & none was able to solve our problem completely, so we are still struggling with fully automating the db update process. Again any suggestions are most welcome :)

Build & Release Challenges : Manual DB Updates

The first problem that I'm gonna discuss is manual db updates. In our current application we do have automated DB updates execution in the production environment, but not in the rest of environments i.e dev, qa, stage, performance test ... etc.

The process that we use for automated scripts execution in production environment  is that we create a release folder, this release folder contains all the sql scripts for the release along with a meta file. The release meta file contains the list of all the scripts that needs to be executed, the current system reads this meta file & executes all the scripts of release. This process is fair enough for production system since the release is deployed only once on production system. In production systems we don't have to track whether a script got executed or not i.e all the scripts execution is treated as atomic that is either all the scripts are executed or none is executed.

The drawback of atomic execution is the reason because of which this approach can not be applied to rest of the environments, since the db updates will always be incremental in rest of environments. In case of all other environments apart from production environment the release will be deployed multiple number of times, with each release new db scripts can be added  to the system and only those new scripts needs to be executed.

The new system that I'm trying to develop will have incremental db update capability. The system that I'm planning to deveplop will have following characterstics :

• It should be able to keep track of script name for later reference.
• It should store the release mapping to which this script belongs.
• The sequence of the script to enforce the order of execution.
• The system should also maintain whether the script is already or not.
• The system should be able to handle error scenario i.e if a script execution fails a corrective action should be taken by the system
• It should be extensible enough so that various kind of reports can be generated from it

In the next blog I'll be talking about the actual system how it is built

Prev
Next

Build & Release Challenges : Problems

So here is the consolidated list of the problems that current system have, I've categorized all the issues in different categories so that they can be managed properly

• CI Builds
• Code stability builds are not in place
• Code quality builds are not in place
• Code deployment builds for non-prod environments not in place
• A lot of manual steps in prod deployments
• All the projects are performance critical so builds to do profiling of projects
• Database updates is manual at all the stages of deployment
• Automated smoke testing of all the applications
• Integration of bug tracking system with version control system
• A release server needs to be set-upped so that release can be managed properly
• No documentation at all :)
• Version Control System : We are using git as version control system
• Branching strategy has some flaws as a result of which merging takes a lot of time
• We don't have a GUI to manage git server

Prev                                                                                                                  Next