Monday, April 13, 2015

Setting the environment variable

Setting environment variables (taking GRADLE_HOME  as an example)

  1. Download the gradle zip from the gradle distribution website, here
  2. Extract it to a directory.
    1. Right Click My Computer
    2. Click Properties
    3. Click Advanced System Settings
    4. Click Advanced tab
    5. Click Environment Variables
    6. Click New... under User variables for <user>
    7. In Variable Name box type GRADLE_HOME, and in Variable Value box type <Gradle_Extracted_Directory>
    8. Click OK to save this variable
    9. Now scroll the list of User Variables and edit Path variable and at the end add %GRADLE_HOME%/bin by keeping semi colon(;) as a separator
    10. Click OK  and come out of all the windows and your variable is set for GRADLE in Windows machine

    1. To get all the previously available environment variables
      1. $ set
    2. To get a specific value of the variable 
      1. echo "$HOME"
    3. Setting the variable 
      1. export PATH=${PATH}:${HOME}/bin
    4. Using the above command the variable is gonna set for a session 
    5. As soon as the session is out the settings gonna reset
    6. To make things permanent for the user we have to edit the bash profile
    7. For editing the bash profile
      1. vi ~/.bash_proflle
      2. Append the path setting as follows
        1. export PATH=${PATH}:${HOME}/bin
        2. save it and exit (ESC + wq)
Verifying the installation
  1. Once you perform the required steps to install and set the variable you can check the installation using gradle --version. It will show the information something like this:

Mongo DB Sharding Setup

Mongo DB: (from "humongous") is an open-source document database, and the leading NoSQL database. Written in C++
Sharding : Sharding is the process of storing data records across multiple machines and is MongoDB’s approach to meeting the demands of data growth. As the size of the data increases, a single machine may not be sufficient to store the data nor provide an acceptable read and write throughput. Sharding solves the problem with horizontal scaling. 
Here are the steps to do sharding setup on MongoDB. For sharding, now, we are going to have :
  1. 3 config servers
  2. 3 mongo servers 
  3. 1 sharded server
Config Server
  • Config servers are special mongod instances that store the metadata for a sharded cluster. Config servers use a two-phase commit to ensure immediate consistency and reliability. Config servers do not run as replica sets. All config servers must be available to deploy a sharded cluster or to make any changes to cluster metadata.
    A production sharded cluster has exactly three config servers. For testing purposes you may deploy a cluster with a single config server. But to ensure redundancy and safety in production, you should always use three.
  • These are the machines that do not store the data rather they just be used to manage the metadata of the sharded environment
Mongo Server
  • These are the machines that actually have the data, so we need these machines to be powerful enough to hold the data and do the work heavily and that too continuously
  • Also we can have the replica of these machines, the way we configure replica sets in normal scenarios
Sharded Server
  • This machine is going to do pretty heavy work and manipulation.

The environment what we will deploy is going to look like this:
For demo let us consider the IP of the 7 machines are as follows:
EC2 Instance(Size)
Machine A192.168.100.101ReplicaBig Machine
Machine B192.168.100.102                             ReplicaBig Machine
Machine C192.168.100.103ReplicaBig Machine
Machine D192.168.100.104ConfigMedium Size
Machine E192.168.100.105ConfigMedium Size
Machine F          Size
Shard Machine192.168.100.100ShardBig Size
  • Now while installing we have to make sure about the permissions of the  folder to the user of the mongod
  • Also for config server we have to make a different directory other than the data directory say /data/configdb (default path what config server of mongo db searches for)
    • make sure to change the ownership of this directory as well
Now the commands to be followed:
Machine D, E, F :
  • mongod     --configsvr
    • This starts the config server in these machines which is going to be used by the sharded server to connect to
    • by default it starts on 27019, we can specify the port via  script or via command,please see the docs provided by mongo db for that (doc here)
Shard Machine:
  • We have to start a shard server by connecting to these config server(Machine D, E, F), for that we will have to start a command :
    • mongos  --configdb,,  --port  27020
        • this will start a sharded server at port 27020
Now we have to attach clusters to this shard server but before that we have to setup the shard cluster as an individual mongod  instance with shard server parameter, so for that use the following step:
Machine A, B, C :
  • mongod  --shardsvr
    • this will start the shard enabled cluster on the default port 27018
Adding the shard enable cluster in the shard machine:
  • For this connect via mongo client to the sharded machine in the admin mode as
    • mongo
      • This will connect you to the shard  server
After this we have to add the shard enabled clusters in the shard server (Machine A, B, C)using the:
  • sh.addShard(""); 
  • sh.addShard(""); 
  • sh.addShard(""); 
    • These commands will add the shard enabled cluster to our shard machine
If our shard enabled servers are replica sets in that case we just have to add one machine out of that replica set and that will going to take care of other replica set machines:
  • sh.addShard("<replica_set_name>/");
  • Before version 2.0.3, it is required to specify all the replica set members to be added to the sharded cluster like
    • sh.addShard("<replica_set_name>/,,"); info here
If everything goes well you can see the status of the shard environment using:
  •  sh.status()
After the setup we have to decide which database to shard under which, which collection to shard under which, which is going to be the shard key and what type of shard key it will be like normal/sharded:
  • db.runCommand({enableSharding:"testDB"})
  • sh.shardCollection("testDB.testCollection",{"testKeyname":1});
    • 1 denotes the normal
    • this is for making the collection named testCollection under testDB as a sharded collection with testKeyname as the key
    • if we want to  make a group of keys as sharded key then we can't have as hashed key(here)
      • Use the form {field: "hashed"} to create a hashed shard key. Hashed shard keys may not be compound indexes.
  • You have to configure the sharding again for your databases in case you deleted the databases.
  • Also, when you do that be sure you are connected to admin database.(use admin)
  • db.runCommand({enableSharding:"<database_name>"})
  • sh.shardCollection("<database_name>.<collection_name>",{"token":"hashed"});
  • Also for tuning the mongodb shard setup some of the points to take care of:
    • sudo vi /etc/security/limits.conf
    • add this at the end of the file in the format given:
      • root - nofile 64000
        * - nofile 64000
        root - nproc  64000
        * - nproc 64000
        root - stack 16384
        * - stack 16384
      • save and close the vi
      • then reset the limits for the current session
      • ulimit -n 64000
        ulimit -u 64000
        ulimit -s 16384

Installing a plugin in a Linux Machine

Hello Friends lets take a small example of installing the plugin in a Linux Machine.
  1. Checking whether the plugin is installed or not.
  2. sudo yum list installed | grep <plugin_name>
         This will give you the list of plugins that are currently installed in your machine with the name you gave. This search would be local, i.e the search would be based on the plugins that are installed in the machine not the one present in cloud repository of the Linux
  3. Checking the repository for the specified plugin.
    sudo yum list | grep <plugin_name>
    This will list out all the plugins that are available for installations. Now pick the specific plugin and type
  4. Install the plugin
    sudo yum -y install <plugin_name_came_in_last_command>
  5. -y option is for auto yes when ask for confirmation

Wednesday, April 8, 2015

Installing Redis 3.0 in a Linux machine

Hi Friends, lets see the way to install Redis 3.0 in Linux machines.

Basic Installation:
  1. Firstly download the package in Linux machine
  2. Extract it to your favorite directory 
  3. Browse to the directory and pass the command make and press enter
  4. If make command version is perfect, it will start logging some output and you will be able to run the command successfully
  5. After that type make test  and wait for some time till the test gets completed
Advanced Installation (Cluster Mode):

Mode 1 : Installing the cluster setup in one machine

In this mode we will look in to mode of installing the full setup (3 Master, 3 Salves) in one machine, this is non production setup, never try to run production from one machine.
Also the setup process we are discussing here is basically useful for Linux based machines, and not Windows. 

Here are the steps.

  1. Create a folder /root/redis    $ mkdir /root/redis/
        $ cd /root/redis 
  2. Downloading the build from Redis Website
        You can download the build from here, (if not available please visit
        $ wget 
  3. Extract to a folder of your choice, for this doc lets consider we had extracted the redis.tgz  in /root/redis/
        $ tar -xvf redis-3.0.0.tar.gz
    this command will extract the files and create a folder named redis-3.0.0 inside the folder /root/redis, if you have downloaded the file in/root/redis, and extracted inside of it.
  4. Go to that newly created folder
        $ cd redis-3.0.0
  5. Now issue the command make inside of the folder
        $ make
  6. Testing the build
        $ make test
  7. Clean the build
        $ make distclean
  8. Now the Redis is ready to run in cluster or in a normal mode, depending upon the mode you selected.
  9. Now lets make a prerequisites for the cluster setup of Redis in one machine, for this setup let us consider that we are going to work with 6 instances of Redis(3 Masters, 3 Slaves)
  10. Since we are running in 1 instance we will create 6 directories and each directory will have a different configuration
  11. So lets suppose we are going to run the Redis in 7000-7005 port of machine, so create 6 directories naming 7000, 7001, 7002, 7003, 7004, 7005under /root/redis/cluster-test
  12. Here is the configuration template of basic cluster setup, for port make sure to keep perfect number, same of the directory, and cluster-config-file as the nodes0.conf and so on (attach with the last number of the port), and keep it in specific directory.

  1. Redis Config Template
    port 7000
    cluster-enabled yes
    cluster-config-file nodes0.conf
    cluster-node-timeout 5000
    appendonly yes
    So now the directory structure would look something like this,
        --- redis.conf
        --- redis.conf
        --- redis.conf
        --- redis.conf
        --- redis.conf
        --- redis.conf
  2. Now its time to start all the instances of Redis, for that lets start by starting the redis-server with these configurations
       $ /root/redis/redis-3.0.0/src/redis-server /root/redis/cluster-test/7000/redis.conf &
       $ /root/redis/redis-3.0.0/src/redis-server /root/redis/cluster-test/7001/redis.conf &
       $ /root/redis/redis-3.0.0/src/redis-server /root/redis/cluster-test/7002/redis.conf &
       $ /root/redis/redis-3.0.0/src/redis-server /root/redis/cluster-test/7003/redis.conf &
       $ /root/redis/redis-3.0.0/src/redis-server /root/redis/cluster-test/7004/redis.conf &
       $ /root/redis/redis-3.0.0/src/redis-server /root/redis/cluster-test/7005/redis.conf &
  3. If everything goes well it will show some log messages along with message telling "listening port on 7000" and so on.
        [82462] 26 Nov 11:56:55.329 * No cluster configuration found, I'm 97a3a64667477371c4479**32083***8db5858b1`
    This may be different for different machines.
  4. Now we have to create our cluster setup, till now we have started the Redis instances but didn't attached those in our cluster. So for that we will use the redis utility named redis-trib.rb   $ /root/redis/redis-3.0.0/src/redis-trib.rb create --replicas 1
    This command is telling to create a cluster with 1 replica each, and rest are the <ip:port> configurations to tell redis , what machines to use for the cluster.
  5. After this Redis will ask you for the confirmation, say yes and wait for redis to complete the setup, if everything goes well, you will get a log something like this:
    [OK] All 16384 slots covered
  6. Now we will test the setup:
       $ redis-cli -c -p 7000
          redis> set foo bar
          -> Redirected to slot [12182] located at
          redis> set hello world
          -> Redirected to slot [866] located at
          redis> get foo
          -> Redirected to slot [12182] located at
          redis> get hello
          -> Redirected to slot [866] located at
FYI: There may be chance that you will get some different log messages since its up-to redis, how it actually handles the data and pushes it into the cluster.
The parameters used in template are as follows
  1. port
    Tells the redis on which port to start to
  2. cluster-enabled     Tells redis to start on cluster mode
  3. cluster-config-file     Tells redis to use which file to look up for saving the info of the instance. This is not an user editable configuration file, but the file where a Redis Cluster node automatically persists the cluster configuration (the state, basically) every time there is a change, in order to be able to re-read it at start up.
  4. cluster-node-timeout     The maximum amount of time a Redis Cluster node can be unavailable, without it being considered as failing.
  5. appendonly
    This tells redis to append the log which is fully-durable strategy for Redis. So if your computer running Redis stops, your power line fails, or you accidentally kill -9 your instance, the latest data written on Redis will get lost. While this may not be a big deal for some applications, there are use cases for full durability, and in these cases Redis was not a viable option.
Mode 2 : Installing the cluster setup in different machines
  1. Make sure you have installed the redis in all the machines that are to be used as a part of the cluster.
  2. Make sure the port is open in firewall of AWS(security group) as well as local firewall for all the other machines of the cluster to be able to talk to.
  3. Also we have to open the <port number> + 10000, since the higher port is used for the Cluster bus, that is a node-to-node communication channel using a binary protocol. The Cluster bus is used by nodes for failure detection, configuration update, fail-over authorization and so forth. Clients should never try to communicate with the cluster bus port, but always with the normal Redis command port, however make sure you open both ports in your firewall, otherwise Redis cluster nodes will be not able to communicate.
  4. All the other part remains same as of the steps mentioned in above process.
Some common errors that you might face while installation
  1. make[3]: gcc: Command not found
    1. This basically comes when there is no compiler (gcc) for the make command to use
    1. If your machine supports yum installation you have to check for the latest version of gcc : $ sudo yum list gcc
    2. Then install the latest gcc: $ sudo yum -y install  <version that you want to install, came from last command>
  2. zmalloc.h:50:31: fatal error: jemalloc/jemalloc.h: No such file or directory
    1. While downloading and then building the redis some files gets half written and may start problems which we clean it and then restart the make process again.
    1. Before make, please use make distclean: $  make distclean
  3. You need tcl 8.5 or newer in order to run the Redis test
    1. Internally while testing the build by  make test , it internally uses the tcl package to complete the process 
    1. Firstly check the latest version of tcl available: $ sudo yum list tcl
    2. Then install the version using : $ sudo yum -y install  <version that you want to install, came from last command>
  4. /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- redis (LoadError)
    1. Internally while starting the cluster mode Redis uses ruby, rubygems package so you have to make sure that these are installed beforehand.
    1. Firstly check the latest version of ruby available: $ sudo yum list ruby
    2. Then install ruby, follow this
    3. Check the availability of rubygems: $ sudo yum list rubygems
    4. Then install rubygems, follow this
    5. install the redis gem  : gem install redis
  5. Here is the list of commands that can be used directly to install the redis.
    sudo -i
    sudo yum -y install gcc.noarch
    sudo yum -y install tcl.x86_64
    sudo yum -y install ruby ruby-devel rubygems rpm-build
    gem install redis
    gem install  bundler
    mkdir /root/redis
    cd /root/redis
    tar -xvf redis-3.0.0.tar.gz
    cd redis-3.0.0
    make distclean
    make test
    cd ..
  6. vi redis.conf
    keep the following content in it

    port 7000
    cluster-enabled yes
    cluster-config-file nodes.conf
    cluster-node-timeout 5000
    appendonly yes
    appendfsync everysec
    no-appendfsync-on-rewrite yes
    daemonize yes
    logfile /root/redis/logs/redislog.log
    loglevel notice
  7. Open /etc/sysctl.conf
    vi /etc/sysctl.conf 
    and add this in to that file
    vm.overcommit_memory = 1
    sysctl -w net.core.somaxconn = 65535

    This is useful to make the settings available if the machine restarts but we have to make this settings 
    effective right now:
    sysctl vm.overcommit_memory=1
    sysctl -w net.core.somaxconn=65535
  8. Open /etc/rc.local
    vi /etc/rc.local
    and add this in to file
    # Redis config
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
  9. To start the service:/root/redis/redis-3.0.0/src/redis-server redis.conf &
  10. To create the cluster:
    /root/redis/redis-3.0.0/src/redis-trib.rb create --replicas 1 Master1:IP Master2:IP