Attach a new volume to EC2 Instance

This blog will talk about how to mount a new volume to an existing EC2 instance, though it is very straightforward & simple, but it's good to have a checklist ready with you so that you can do things in one go instead of searching here and there. The most important thing to take note of in this blog is that you have to do couple of manual operations apart from mounting the volume through AWS Web UI.

1. Go to the AWS Volumes screen, create a new volume if not created already.
2. Select Attach Volume in Actions button
   
   
   
   
   
3. Choose the instance, to which this volume needs to be mounted
   
   
   
4. Confirm the volume state changes from available to in-use
5. Go to the AWS Instances screen, select the EC2 instance to which volume was attached
6. Check the Block Devices in the details section you can see the new volume details their. Let’s say it is mounted at /dev/sdf.
   
   
   
   
   
7. Now log in to the EC2 instance machine, you can’t see the mounted volume yet(it is like an external un-formatted hdd that is connected to a linux box)
8. To make it usable execute below commands

sudo su -                              [Switch to superuser]

mkfs -t ext3 /dev/xvdf                      [Format the drive if it is a new volume]

mkdir /home/mettl/mongo                      [Simply create a new directory]

mount /dev/xvdf /home/mettl/mongo   [Mount the drive on newly created directory]

Make sure to change permissions according to how you use it.

9. To mount EBS volumes automatically on startup add an entry in /etc/fstab

/dev/xvdf    /home/mettl/mongo    ext3    defaults,nobootwait,comment=cloudconfig    0    0

Hope you will find this blog useful, rest assured this is the starting point of a new series I would be talking about couple of other best practices such as why do you need to have this kind of setup, how you will upgrade volume in case of a running ec2-instance..

How to create an extra swap space using file system

Sometimes you feel constrained due to the the RAM limit of your system especially when you are running heavy duty software’s, in this blog I'll talk about how you can overcome this problem by hav‌ing an extra swap space to give you extra computing power

First of all you can execute swapon command to check how much swap space you already have in your system
$ swapon -s
Filename                Type        Size    Used    Priority
/dev/sda5                               partition    8130556    44732    -1

The above output gives you an indication that you already have a swap space at partition /dev/sda5. The numbers under "Size" and "Used" are in kilobytes. Though I have considerable amount of swap space configured on my system :), let's continue and try to create a new swap using file system. Before starting with creation of swap space let's make sure that I've enough disk space available in my system

$df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       448G  123G  303G  29% /
udev            1.9G  4.0K  1.9G   1% /dev
tmpfs           767M   40M  727M   6% /run
none            5.0M     0  5.0M   0% /run/lock
none            1.9G  804K  1.9G   1% /run/shm

So I've a powerful system with 303G of disk space still available, that means I have a liberty of creating a swap space of my liking. I'll user the data dump(dd) command to my supplementary swap file, make sure that you would be running this command using root user.
$dd if=/dev/zero of=/home/sandy/extraswap bs=1M count=512
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 2.41354 s, 222 MB/s

Now we have created a file /home/sandy/extraswap of size 512M which we will be using as a swap partition. Swap can be created by issuing mkswap command
$mkswap /home/sandy/extraswap
Setting up swapspace version 1, size = 524284 KiB
no label, UUID=685ac04a-ad31-48a8-83df-9ffa3dbc6982

Finally we have to run swapon command on our newly created swap partition to bring it into the game
$swapon -s
Filename                Type        Size    Used    Priority
/dev/sda5                               partition    8130556    46248    -1
$swapon /home/sandy/extraswap
$swapon -s
Filename                Type        Size    Used    Priority
/dev/sda5                               partition    8130556    46248    -1
/home/sandy/extraswap                   file        524284    0    -2

As you can notice when we first executed the swapon -s command at that time swap partition was not in the picture, once we executed the command swapon /home/sandy/extraswap  the swap partition was available.

One last thing that we have to do is to add the entry of this swap partition in our /etc/fstab file as with the next system boot the swap partition will not be active by default we have to do the entry of this swap in our /etc/fstab file.

How to securely access your private app on cloud

One of the suggested practices in cloud administration is to always host your applications on a Virtual Private Cloud. Also, you should have a public subnet hosting the public facing apps, and a private subnet which hosts the private apps (like a database or a back-end service/app). To know more about why you need such kind of a setup, please read more about VPC.

This blog will talk about a scenario where you have multiple Virtual Private Clouds (hereafter referred to as VPC), and you need to access a private app hosted in one VPC from another VPC. An example of this scenario could be that you have a VPC for your staging environment and another VPC for production environment, then you'd like to sync the database from of production environment from the staging environment. In this case, it might not be straight forward to do this, as you might not be able to access the production database from outside the production VPC.

One of the solutions for this problem would be to first take a dump of the production database on one of the public facing machines in the production VPC, and then copy that dump to a public facing machine in the Staging VPC and finally applying this dump to the private database of Staging environment. This approach will work, but it would not be a perfect solution, as you have to copy the db dump between VPC's.

A much better approach would be if you could directly connect to the production database from the Staging VPC & execute the dump & restore command, for that you need direct access of production database from staging environment. This approach is called port-forwarding. We configure port-forwarding at one of the public facing machines(NAT is the preferred one) in the production VPC in such a manner that if a request comes on this machine at port x it will be forwarded to port y on a private facing machine in the production VPC which is the database production in this case.

In the next blog I will talk about other alternate approaches that can be used to solve this problem.

Puppet module to setup nodejs deployment 2

As I said in the previous blog Puppet module to setup nodejs deployment, the nodejs module was for providing the basic infrastructure for automated node app's deployment & as promised I've released the next module "nodeapp" that can be used to setup a node app on the target server.

First of all I'll talk about what this module will do to facilitate the automated deployment of a nodejs app, as already discussed we are following a convention that all the node app's code will be present at /home/nodejs/ which is referred by startNodeApp.sh script so we create the directory of nodejs app. The deployNodeApp.sh script was using the upstart to manage the nodejs app instance i.e starting/stoppping the nodejs app, the nodeapp module takes care of creating the require upstart configuration at /etc/init/.conf. Also we use monit to monitor the nodejs app's so that we can start/stop the nodejs app's using the web ui of monit & also see various stats such as cpu, memory, load.. consumption of nodejs app.

This nodeapp module is a userdefined type which takes the name of nodeapp as an argument, as a result of which you can setup any number of nodejs app's on a system.
i.e nodeapp{'search-demo': app_name => "search-demo"}
This entry will create below files

/etc/init/search-demo.conf : An upstart configuration file, using which search demo nodejs app can be managed as a service.

#!upstart
description "node.js search-demo server"
author      "sandy"

start on startup
stop on shutdown

script
    export HOME="/home/nodejs"

    echo $$ > /var/run/search-demo.pid
    exec sudo -u nodejs /home/nodejs/startNodeApp.sh search-demo
end script

pre-start script
    # Date format same as (new Date()).toISOString() for consistency
    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/log/search-demo.sys.log
end script

pre-stop script
    rm /var/run/search-demo.pid
    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" >> /var/log/search-demo.sys.log
end script

/etc/monit/conf.d/search-demo.monit : A monit configuration file, using which search-demo nodejs app can be monitored & even automatedly restarted

check process search-demo with pidfile /var/run/search-demo.pid
 stop program = "/sbin/stop search-demo"
 start program = "/sbin/start search-demo"

So using these 2 modules nodejs & nodeapp you can make any system up & running for nodejs app's automated deployment

Puppet module to setup nodejs deployment

I would like to share my puppet module to setup nodejs deployment infrastructure on a linux box. This module performs the basic setup required to facilitate the automated deployment of a nodejs app. Very soon I'll be introducing another generic puppet module that will run on top of this module & provide a full fledged automatic deployment of any node app. To view the source code of this module you can refer my github repository.

Let's talk about what this module actually does. First of all we create a nodejs user which we will use for all deployment related activities of all the node app's, as a convention we have created a folder /home/nodejs/nodeapps this folder will contain all the code of our node applications.

This modules adds 2 scripts as well the first one is deployNodeApp.sh, deployNodeApp.sh is a generic script that assumes that node app code will be present in tar form at /home/nodejs it will clean existing code of nodeapp at /home/nodejs/nodeapps untar the code at corresponding directory of node app & restart the node app. As another convention we are using upstart for managing the node app i.e starting & stopping the node app I'll talk about the upstart configuration in my next blog where I'll talk about generic puppet module for a node app. Another script startNodeApp.sh will take care of starting the node app after doing some per-processing such as loading some environment specific properties of node app which we don't want to commit in the codebase i.e want to separate it out from deployment process choosing a specific version of node.

This module also takes cares of installing nvm for nodejs user so that nodejs version can be managed locally for this user or app.

Though we already have a puppet module for nodejs, but I had some specific requirements which I wanted to handle that's why I've created this module.

Let me know if you have some points of improvement in this module, one thing that I wanted to add in this module is to add npm installation but it had some other dependencies also I had some doubts whether I should have npm as part of nodejs module or not.

System Monitoring

One of the main task of a system administrator is system monitoring, system monitoring usually involves monitoring the ram & disk space usage of the system .... In this blog I'll be talking about my experience as a system admin & how I do it.

Usually system monitoring is divided into 2 parts Continuous system monitoring and troubleshooting system issues when system crosses a threshold value & you have to figure out the issue & try to resolve it.

In continuous system monitoring a system is put under continuous monitoring i.e the system ram usage is within defined limit or not, the disk space occupancy should not cross a predefined threshold .... To achieve continuous monitoring you can use couple of tools available in market such as nagios, omd we are primarily using these tools their would be other tools available also for this purpose.


Continuous system monitoring serves one purpose where they notify about any deviation from the expected state of the system, the next step is to troubleshoot this issue & resolve it accordingly. As a first step I usually execute top command, top is a very powerful command apart from just viewing the processes activity in real you can do a lot of things i.e

• If you want to add/remove fields : press f & then you can choose the fields to add/remove
• If you want to change ordering of  fields : press O & then you can move fields
• If you want to change the sort order : press F or O

there are lot of other options available as well, if you want to explore them pressing h will provide you a list of all the options.

You can also read about htop, htop is an advanced form of top where you can view some graphs as well though I'ven't used htop so much but I'm planning to :)

One thing to note sometimes you are not able ot run top command due to high resource utilization, in that case you have to use cat /proc/loadavg to view the load on the system & cat /proc/meminfo to view current memory state of the system.

One of the useful command if top doesn't work
ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -5
This command will give the top 5 processes by memory usage.

Also there are couple of other commands that you can use
free : To view the memory usage of system
df : To view the file system information
du : To view the disk usage

One tip : To increase the memory of system you can create a swap memory & it is always recommended to create a swap on a partition only. Another best practice for swap area is if your system RAM is below 8 gb your swap area should be double of your ram otherwise it should be half of your RAM size

Tip : Setting up Git Jenkins integration on windows box

If you have ever tried setting up git as a version control system in a Jenkins installation on a windows box you would have faced an error message ssh key not available.


The reason behind this issue is that if you are using git with ssh protocol it tries to use your private key to perform git operations over ssh protocol & the location it expects is the .ssh folder at home directory of user. To fix this issue you have to create a HOME environment variable and point to your home directory where your .ssh folder exists after that restart Jenkins & now it should work fine.

Linux Utility to manage login to systems

One of the problem I used to have as build & release engineer is to manage login to huge number of boxes through my Linux system. At the scale of 5-10 machine it's a not a big problem but once you have close to 100+ boxes then it is not humanly possible to remember the ip's of those boxes.
The usual approach for this problem is to maintain a reference file, from where you map machine name with the ip & find the ip of the box from this file, but again after some time this solution seems to be not that efficient. Another solution is to have a DNS server where you can store such mappings & then you can access these machines using their names only, this is the idle solution but what if you don't have DNS server also still you have to execute the ssh command 'ssh user@machine".

I developed a simple solution for this problem, I created a utility script connect.sh, this script takes machine name as an argument & then we have multiple conditions statements which checks which ssh command to be executed for the machine name.

#!/bin/bash
if [ "mc1" == $1 ]; then
    ssh user@
elif [ "mc2" == $1 ]; then
    ssh user@
elif [ "mc3" == $1 ]; then
    ssh user@
.
.
.
fi

This solution worked really well for me as now I'm saved from executing whole ssh command, also for machine name I've followed a convention i.e _ for example the entry for a machine for release environment that is hosting an application catalog the machine name would be release_catalog, similarly dev_catalog, staging_catalog, pt_catalog.. so you don't have to remember machine names as well :).

Automated DB Updater Release First Release

Initial version of Automated DB Updater Release ADU

With this blog I'm releasing the intial version of a python utility to provide automated db updates across various environments for different components.

The code for this utility is hosted on github
https://github.com/sandy724/ADU

You can clone the read only copy of this codebase by url given below
https://github.com/sandy724/ADU.git

To understand the basic idea about this utility go thorugh this blog
http://sandy4blogs.blogspot.in/2013/07/automated-db-updater.html

How to use this utility
    Checkout the code at some directory, add the path of this directory in PYTHONPATH environment variable
    Create a database with a script's metadata table with given below ddl
CREATE TABLE `script_metadata` (
  `name` varchar(100) DEFAULT NOT NULL,
  `version` int(11) DEFAULT NOT NULL,
  `executed` tinyint(1) NOT NULL DEFAULT '0',
  `env` varchar(30) DEFAULT NOT NULL,
  `releas` varchar(30) DEFAULT NOT NULL,
  `component` varchar(30) DEFAULT NOT NULL
)
    Create a database.properties, containing connection properties of each environment database

[common_db]

dbHost=localhost

dbPort=3306

dbUser=root

dbPwd=root

db=test

[env1]

dbHost=localhost

dbPort=3306

dbUser=root

dbPwd=root

db=test

Here common_db represents connection to database which will contain metadata of scripts for monitoring

    Now execute the pythong utility
        Copy the client(updateDB.py) to directory of your choice, make sure that property configration file should also be at this directory
        python updateDB.py -f -r --env

Automated DB Updater

In continuation with my blog series I'm finally introducing a automated db updater tool. You can read about the idea in previous blogs by going to below links

Manual DB Updates challenges
Manual DB Updates challenges-2

The short form of my tool is ADU(Automated DB Updater). Now some details about this tool

Each application will have database_script folder at the root level, this folder will contain folders corresponding to each release i.e release1, release2, release3...

A database release folder will contain

• Meta file :sql_sequence.txt, this file will contain the sequence in which sql files will be executed, only files mentioned in this file will be entertained
• SQL Files : A sql file must have a naming convention like this __.sql/__.sql

Process of automatic execution of scripts on an environment

• Input
• release_name : to figure out the folder from where scripts will be executed
• environment : Environment on which scripts will be executed
• Execution
• sql_sequence.txt file will be read line by line having one sql file name in each line
• The sql file will be verified whether it has been already executed or not
• If the sql file is already executed then two conditions are verified
• A new version of sql should be available
• Undo version of last executed sql should be present
• After execution of undo file the latest version of the sql file will be executed and the info is stored accordingly that it has been executed so that it will not be picked again
• Validations & Boundary Conditions
• All the files mentioned in sql_sequence.txt should exist.
• Undo script should be present for all the versions of a sql file barring the latest version of sql file.
• Undo script will only be executed if next version of script is available.

Very soon I'll share the github url of this project keep waiting :)

Ubuntu Rest Assurer

Before I'll start talking about this utility I would like to talk about the story behind the creation of this utility. Few days back we have a session ERGONOMICS about healthy lifestyle, one of the main thing was that a person should take a break after every 20 minutes. After the session I was having a chit chat about one of my colleague Rahul Narang & we were discussing about this 20:20:20 rule & then we thought about creating a utility that will force a person to leave his/her system after some stipulated time & that's how the idea came about this utility.

So what this utility does
1.) It runs after every half hour or configured amount of time
2.) Prompts a snoozer dialog box which will allow a person to snooze the screen locking for some amount of time
3.) After that a lunching video will run for 10 seconds
4.) Once a lunching video completes the screen gets locked for a configured amount of time
5.) If user try to unlock the system before configured amount of time the utility will lock the screen again

This complete utility is created using shell script only, we have used couple of command to do that
vlc : To run a launcher video
zenity : To prompt a snooze dialogue box
gnome-screensaver-command : To operate on screen lock

You can find the source code of this utility at my github account
https://github.com/sandy724/REAS

Puppet module for setting up Multiple mongo's with replication

In this blog I'll be talking about a puppet module, that can be used to installing multiple mongo's with replication on a single machine. Since I'm very new to puppet so you may find this module very crude, but it works :). Their were couple of puppet module already available but most of them are only for installing a single instance of mongo at a machine & I've a specific requirement of installing multiple instances of mongo having master slave replication between them. As I already said that this module may be quiet crude or basic so please bear with that & my approach may also seem a bit unconventional so please let me know what all can be improved in this module or how things could have been done in a better way.

So let's start with the actual details first of this module is hosted on github(https://github.com/sandy724/Mongo), if you want to look at the source code you can clone it from github. For installing mongo you would be executing the command
puppet apply -e "class {mongo:port => , replSet => ,master => ,master_port => ,}"

Command for installing master
puppet apply -e "class {mongo:port => 27017, replSet => sdrepsetcommon,master => master, master_port => 27017,}"

Command for installing slave
puppet apply -e "class {mongo:port => 27018, replSet => sdrepsetcommon,master => slave,master_port => 27017,}"

Before going into the details what all this module is doing I will share some details of mongo

• You can start mongo by executing mongod command
• You can provide a configration file which contains details such as
• log directory where mongo would be generating the logs
• port at which mongo would be listening for requests
• dbpath where mongo would be storing all the data
• pidfilepath containg process id of mongo instance, that would be used to check whether mongo is running or not
• replSet name of the replicaset
• You need to have a mongo as a service installed in you system to start an instance of mongo
• For replication you need to execute rs.initiate command on the master mongo
• For adding another instance into replication you need to execute rs.add(":")  command on the master mongo

   
Now let's go into more details what all this component does, I'll be listing down all the steps in bulleted points

• As you can figure out this module is expecting few parameters :
• port : port at which mongo would be listening,
• replSet : name of replicaset which would be used for managing replication
• master : A string parameter which would signify whether the mongo setup is for master or slave
• master_port : Port at which master instance of mongo would be listening
• First of all we create a mongo user
• Parent Log directory for the mongo instance is created if it doesn't exists with mongo user as owner.
• Mongo db directory is created under /data/mongo with a naming convention replSet_port, i.e if replSet parameter is sdrepsetcommon & port is 27017 then the data directory for this mongo instance would be  /data/mongo/sdrepsetcommon_27017. This directory would be owned by mongo user.
• A mongo service would be installed if not already their.
• A mongo restart shell script is also placed at the mongo db directory
• A file is also placed under the mongo db directory that have a mongo command to setup replication, this file is created conditionally depending on whether we are setting up a master or slave instance.
• Finally the replication command is executed on mongo server & restart script is also executed

This concludes the setting up of a mongo instance on a machine.

Just for more details to start mongo we are using mongod -f command, this configuration file is saved as a template & the mongo modules processes the template with the values passed & creates the desired mongod.conf. In our case we are evaluating following properties of mongod.conf : logpath, port, dbpath, pidfilepath, replSet

Initial thoughts for an automation testing framework/utility

My first exposure to selenium was in 2010/2011 & I was quiet impressed with it, the way you can use selenium for the testing of web application was totally awesome. At that time I was working with xebia,  our team was working for website revamp of a dutch travel company. We were using selenium for all the regression & functional testing of website, 80% of the website testing was done only by Selenium.

One of the challenge with selenium is that for each test scenario you have to write code for that & if you don't manage your test scenario/cases effectively, the management of selenium test cases becomes a task in itself. At that point of time also we tried to make maximum use of Java to make selenium test cases as structured & Object oriented possible so that they can be extended & managed easily. I always had a desire to do some improvement in that area so that the selenium test cases management should be made more easier.

In my current company most of the testing is done manually, since I had a prior experience of Selenium & experienced the power it brings to your testing. I was pretty determined to bring the selenium advantage in our company. Off-late an automation testing was set-upped in our company as well which was working on leveraging the power of selenium in testing, but again it was same problem you have to write a lot of code. The other challenge that automation team was facing, the UI of the site was changing very frequently so whatever work they used to do was becoming back to zero after few iterations.

Last week along with my team I've started doing some head banging that let's see if can do something out of the box, in a normal discussion with my team members one idea stuck to us. The manual QA team of our company is very strong & they have complete in & out of idea of whole application, but they have so much work assigned to them that they can't spend their time in the selenium. We wanted to club the knowledge of our manual testing team & the power of selenium.

As a POC we buit a very simple utility that will read a meta information file and executes the commands listed in that meta file. As an example if they want to open a page one of the line of meta file will contain a command "open url", similarly if they have to click a button the command will be something like "click . This utility was doing exactly what we wanted to do. We are still in the POC phase where we are trying to include as much commands as possible


Let me know about your thoughts for this approach, suggestions are most welcome.

Automation tips and tricks February 2013

As promised I'm back with the summary of cool stuff that I've done with my team in Build & Release domain to help us deal with day to day problems in efficient & effective way. As I said this month was about creating tools/utilities that sounds very simple but overall their impact in productivity & agility of build release teams and tech verticals was awesome :).

Automated deployment of Artifacts : If you have ever worked with a set of maven based projects that are interdependent on each other, one of the major problem that you will face in such a setup is to have the latest dependencies in your local system. Here I'm assuming two things you would be using a Maven Repo to host the artifacts & the dependencies would be SNAPSHOT dependencies if their is active development going on dependencies as well. Now the manual way of making sure that maven repo will always have the latest SNAPSHOT version is that every-time somebody does change in the code-base he/she manually deploy that artifact to maven repo. What we have done is that for each & every project we have created a Jenkins job that check if code is checked in for a specific component & if so that component's SNAPSHOT version get's deployed to maven repo. The impact of these utilities jobs was huge as now all the developers doesn't have to focus on deploying their code to maven repo & also keeping track of who last committed the code was also not needed.

Log Parser Utility : We have done further improvement in our event based log analyzer utility. Now we also have a simple log parser utility through which we can parse the logs of a specific component & segregate the logs as per ERROR/WARN/INFO. Most importantly it is integrated with jenkins so you can go to jenkins select a component whose log needs to be analyzed, once analysis is finished the logs are segregated as per our configuration(in our case it is ERROR/WARN/INFO) after that in the left bar these segregations are shown with all the various instances of these categories and user can click on those links to go exactly at the location where that information is present in logs

Auto Code Merge : As I already told we have a team of around 100+ developers & a sprint cycle of 10 days and two sprints overlap each other for 5 days i.e first 5 days for development after tat code freeze is enforced and next 5 days are for bug fixing which means that at a particular point of time there are 3 parallel branches on which work is under progress one branch which is currently deployed in production second branch on which testing is happening and third branch on which active development is happening. You can easily imagine that merging these branches is a task in itself. What we have done is to create an automated code merge utility that tries to merge branches in a per-defined sequence if automatic merge is successful the merge proceeds for next set of branches otherwise a mail is sent to respective developers whose files are in conflict mode

Hope you will get motivated by these set of utilities & come up with new suggestions or point of improvements

Initial thoughts for a patch framework for a java based web project

Although this blog was not in the pipeline of feb month but I got a requirement to build a patch framework for a java based web project, so along with building this framework I thought of writing this blog as well so that I'll get idea from other people as well.

First of all I will talk about what can be patched using this patch framework, majorly it will be three resources/things that can be patched

• Class Files
• JSP's
• Static Resources such as images, css, js ...

I'm thinking of adding few other features in this patch framework as well

• Sequence of patches should be maintained : Since we have a big team around 80 developers working on same code-base, their may be a scenario that we can have two or more patches which needs to be applied to a target system. Their may be a fair chance that those patches have to be executed in a sequence or you can say their could be dependency among patches.
• Validation while applying patches : One of the validation that I can think of is that the resources that have to be patched will be either new or existing one & in case of existing resources the system should verify the location at which resources are patched should already have those resources
• Rollback : The patch framework should have rollback capability

I'm planning to build this patch framework using

• Shell scripting : For programming aspect
• Git : As a version control system for storing patches
• Jenkins : Provide a front-end to user for applying patches
• Mysql : Not so sure about it yet but I've to store few information such as what all patches are applied, sequence of patches.... I can use file systems as well for storing this information

Let me know your thought about  this framework or any other feature that you can think of

Automation tips and tricks January 2013

I'm starting a new blog series in which I'll be talking about various cool things or automations that I along with my team done in a month and what are my plans for next month.

Talking about January 2013, I've done following things

1.) Streamlining of environments : The big step in streamlining the environments is to change the owners of the application from root user to tomcat user & making ports of all the application consistent across environments i.e dev, qa, pt & staging. This will help me in my long term goal of introducing a server configuration tool most preferably puppet.
2.) Log Analyzer Utility : One of the major challenge that teams face is to get real time notifications of any exceptions that occur in the server logs, to overcome this problem we have written a log analyzer utility that will scan a log file backed by a meta file, this meta file have the information about who all should be notified for an exception. This utility is written in shell script and integrated with Jenkins CI server so that we can schedule the execution of this utility as per convenience, currently jenkins is executing this utility after every 15 minutes.
3.) System monitor : Off late we were facing challenge of servers getting disk out of space & when whole system goes down then only we were able to figure out the issue is due to disk space outage due to huge log files, to overcome this problem we have built a small shell utility that scans couple of folder's recursively and provide a list of top 10 files whose size is greater then a specified threshold. In our case we have set this threshold as 1 GB, also all these variables can be provided as input to this utility such as folder's to scan regular expression of files which needs to be considered the threshold value

This is what we have achieved in the month of January 2013 although these utilities seems to be but obvious and simple one but the effect they have in the productivity of the team is considerable.


Now plans for the month of February 2013, usually I choose those things which we are doing manually, this month we will be working on following things
1.) Utility which can perform automated merge if possible
2.) Utility that can automatically upload the artifacts to a central server(artifactory in our case)
3.) Integration of git common operations with Jenkins