What is Difference between worker and prefork

Apache (HTPD) is  very popular and widely deployed web server arround the world. A-Patchy server comes with multiple modules. The term MPM is used for multiprocessing module. We can check for default mpm by running this command “ httpd -l ”

Apache 2 is available with following 2 MPM modules.

PREFORK
WORKER

(mpm_winnt This Multi-Processing Module is optimized for Windows NT.)
(mpm_netware Multi-Processing Module implementing an exclusively threaded web server optimized for Novell NetWare)

A) Prefork MPM

A prefork mpm handles http requests just like older Apache 1.3. As the name suggests it will pre-fork necessary child process while starting Apache. It is suitable for all those websites which don’t want threading for compatibility. i.e for non-thread-safe libraries . It is also known as the best MPM for isolating each incoming http request.

How it works: – A single control (master) process is responsible for launching multiple child processes which serves incoming http requests. Apache always tries to maintain several spare (not-in-use) server processes, which stand ready to serve incoming requests. In this way, clients do not need to wait for a new child processes to be forked before their requests can be served.
We can adjust this spare process through the Apache configuration. Default settings are usually enough for small amount of traffic. One can always tune those Directives / Values as per their requirements.

Pre-Fork is the default module given by Apache.

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

B) Worker MPM

A worker mpm is an Multi-Processing Module (MPM) which implements a hybrid multi-process multi-threaded server. By using threads to serve requests, it is able to serve a large number of requests with fewer system resources than a process-based server.

The most important directives used to control this MPM are ThreadsPerChild, which controls the number of threads deployed by each child process and MaxClients, which controls the maximum total number of threads that may be launched.

Strength : Memory usage and performance wise its better than prefork
Weakness : worker will not work properly with languages like php

How it works : – A single control process (the parent) is responsible for launching child processes. Each child process creates a fixed number of server threads as specified in the ThreadsPerChild directive, as well as a listener thread which listens for connections and passes them to a server thread for processing when they arrive.

Apache always tries to maintain a group of spare or idle server threads, which stand ready to serve incoming requests. In this way, clients do not need to wait for a new threads or processes to be created before their requests can be served. The number of processes that will initially launched is set by the StartServers directive. During operation, Apache assesses the total number of idle threads in all processes, and forks or kills processes to keep this number within the boundaries specified by MinSpareThreads and MaxSpareThreads. Since this process is very self-regulating, it is rarely necessary to modify these directives from their default values. The maximum number of clients that may be served simultaneously (i.e., the maximum total number of threads in all processes) is determined by the MaxClients directive. The maximum number of active child processes is determined by the MaxClients directive divided by the ThreadsPerChild directive

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         4
MaxClients         300
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

Apache / HTTPD : No space left on device: Cannot create SSLMutex

It is true that life teaches you new lesson every day… Yesterday for first time I came across the server where I was unable to restart apache / httpd service on server. It looked bit strange but after checking error.log if found following errors ..

Apache: No space left on device: Cannot create SSLMutex

After searching on web I found that Apache is leaving a bunch of stray semaphore sets lying around after an attempted restart of httpd / apache. In lay man’s term “semaphore” is a dead object in memory or locked process in operation… huh !!!! Don’t worry, there is a way out for this .. we need to list and grep those processes (dead processes)  and terminate all such locked instances of apache. Use following command to list those processes.

ipcs -s | grep apache

Most likely you will see a fairly large list here. You need too, and it is safe too, have these deleted. The following command will again do the trick:

ipcs -s | grep apache | awk ' { print $2 } ' | xargs -n 1 ipcrm -s 

Note: If your apache is running as nobody or another user, be sure to substitute that other user in place of  apache above.
————————————————————————————————————-

;Cannot create SSLMutex solution

;
At the heart of the problem, is most likely a poorly configured Apache server. By default, SSMutex is configured to the default setting, as it was on this one server of ours. If you read the Apache.org pages for mod_ssl configuration, they have this to say about the default setting:

;

The following Mutex types are available:

none | no

This is the default where no Mutex is used at all. Use it at your own risk. But because currently the Mutex is mainly used for synchronizing write access to the SSL Session Cache you can live without it as long as you accept a sometimes garbled Session Cache. So it’s not recommended to leave this the default. Instead configure a real Mutex.

;

There are of course optional configuration settings. At the very least, it is suggested that you set SSLMutex to sem, which will let Apache choose which SSLMutex type to use.

You will most likely find this setting in the ssl.conf file located at /etc/httpd/conf.d.

Automate execution of shell scripts owned by non-root users at boot

Hi all, This is really going to be useful. many times we need to execute some commands or shell scripts as non-root user at the time of booting. say for example you need to mount samba share to be mounted as non-root user at time of system boot. Many people do argue that why you need to mount share as non-root user … A good answer for that argument is, not all applications are running as root user on server. For security reasons it is a good practice to have different non-root users as owner of different applications.

Follow these steps to execute a single command as non-root user at time of bootinig in unix/linux.

  1. edit your rc.local script as this is the script which will be execute immediately after booting. usually it is available under /etc/ directory. type following line of command to mount a external samba share as non-root user.su – {userid} -c {COMMAND}If you think you have more arguments in your command line or you have more then 1 command to be executed then put them alltogether in one shell script and then usesu – {userid} -s {shell-script}

    e.g. you need to mount two mounts on single linux servers and those two mounts are on different servers then your shell-script called myscript will become something like. also it is good idea to keep that script in users home directory or in a directory which is accessible by that non-root user other wise it will not work.


    #!/bin/bash
    /usr/bin/smbmount //{first server name OR IP address}/{share name} {first local path to mount} -o username,password rw
    /usr/bin/smbmount //{second server name OR IP address}/{share name} {second local path to mount} -o username,password rw
    exit;
    And for this case your entry in /etc/rc.local will be

    su – {non-root-user} -s {/home/non-root-user/myscript}

Make sure that non root user have valid shell available other wise this will not work

UMASK for sftp users / connections – Linux / centos / fedora / ubuntu

As Internet is now growing like anything, various requirements / demands come across. recently i was setting up a web development server where multiple developers were required to add / edit / update files on same directory ….. hmmm .. Initially i thought i will create a group and add those developers in that group and everything will be done .. but as default UMASK on LINUX server is set to 0022, it dosen’t grant write permissions to group by default.. Hence I need to change UMASK to 0002 for those sftp users… and here you go ..There are multiple ways to achieve this .. first way to set up a shell script which will start sftp sub system with umask 0002 …

Create following shell script:

#!/bin/bash

umask 0002

# The path to your sftp-server binary may differ
exec /usr/libexec/openssh/sftp-server

Then I pointed the Subsystem directive in the sshd_config file to my script:

Subsystem       sftp    /opt/sftp-server-script.sh

A quick restart/reload of the sshd configuration and I was in business. Both users could see and edit each others files. Email or comment with questions.

—Second option :
Or even simpler still as @Gilles pointed out in the comments you can do away with the wrapper script entirely and simply change the Subsystem line in your sshd_config to this:

Subsystem sftp /bin/sh -c ‘umask 0002; /usr/libexec/openssh/sftp-server’

Thanks Mate… much appreciated.

—Third Option
There is a new flag for the sftp-server, ‘-u’, that allows you to directly set the umask, overriding the user umask. So to use it, just do this:

Subsystem sftp /usr/libexec/openssh/sftp-server -u 0002

Configure Sendmail to log “Subject” line for each email sent.

Hi Friends,
In this article we will learn how to configure sendmail so that you can log “Subject” in /var/log/maillog as by default sendmail does not log Subject to maillog file.

This is really interesting. Business people many times are interested in getting mail log files analyzed. To analyse mail logs they need various field to appera in mail logs. e.g. “From”“To”“Subject” etc from the sent email. By default sendmail logs From and To fields but it does not log Subject field. In this article you will learn how to enable sendmail to log “Subject”.

  • First of all take backup of your “sendmail.mc” and “sendmail.cf” files. Default location for those files is /etc/mail.
  • Now open “sendmail.mc” in your favourite editor and add following lines in it and save it. I usually prefere to add it at boottom of file so you can easily identify your modifications.

LOCAL_CONFIG
Klog syslog
HSubject: $>+CheckSubject

LOCAL_RULESETS
SCheckSubject
R$* Press TAB Key $: $(log Subject: $1 $) $1

This last line is very crusial. after R$* press tab key as suggested. Then after $: press space key.
After log there is a space key, after Subject: there is a space key after $1 there is a space key. After closing bracket there is a space key before $1.

  1. Now you need to regenerate sendmail.cf file. use m4 macro to do so. 

    #m4 sendmail.mc > sendmail.cf

  • Now restart sendmail and you verify your maillog file. You will see Subject line for each mail which is being sent from or relayed from your email server

What is shell ?

Shell is a program which allows users to interact with computers.

Shell is an command language interpreter that executes commands read from the standard input device (keyboard) or from a file. Several shells available fo Linux and UNIX operating systems are :

SH (Bourne Shell) Old Unix Shell.
BASH (Bourne-Again Shell) GNU
CSH (C Shell) BSD
TCSH (Popular extension of C Shell)
KSH (Korn Shell) Bell Labs
ZSH (Popular Extension of Korn Shell)
RSH (Remote Shell) TCP/IP

User’s use keyboard to send commands to system. The interface they are using is called CLI (Command Line Interface). If you are normal user (non administrator) prompt is called
“$” prompt (dollar prompt). If you are administrator (super user) then prompt is called
“#” prompt (pound /hash).

If you want to know how many shells are supported by your system then use following command.
$cat /etc/shells
OR
#cat /etc/shells
Note : Above command will display content of file “/etc/shells” on your screen.

How do I find out what shell I’m using?

As We mentioned earlier that shell is a program which allows users to interact with system, we can find out which shell you are using right now using “ps” command with -p switch.
ps -p $$
So what is $ argument passed to -p option? Remember $ returns the PID (process identification number) of the current process, and the current process is your shell. So running a ps on that number displays a process status listing of your shell. In that listing you will find the name of your shell (look for CMD column) .

nilesh@gnulinux:~$ ps -p $$
  PID TTY          TIME CMD
 3301 pts/0    00:00:00 bash

What is shell ? Unix / Linux / centos / redhat / suse / fefora / bsd / solaris / Aix
How to find out which shell I am using ?
what is shell prompt ? what is difference between $ (dollar) and #(pound)(hash) prompt?
About shell, command interpreter program.
What is shell script?

How to setup redundant NFS Servers. Heartbeat-DRBD-NFS. centos/redhat/ubuntu/debian

Hi all, hope you are enjoying series of articles published on this website for free.. totally free.

In this article you will learn how to setup redundant NFS servers using DRBD (Distributed-Replicated-Block-Devices) Technology. Complete step-by-step procedure is listed below. It works for me so hopefully it will work for your also. No gurantees are given.

1. Two servers with similar storage disk (harddrive) setup (To create a redundant nfs server)
2. One client system/server where the nfs share will be mounted.
3. Static IPs for all servers.

First step is to install CentOS on both machines. During the install process, create a separate blank partition on both machines to be used as your nfs mount. Make sure you creat exact same size partitions on both servers. Set the mount point to /nfsdata during installation.

From this point on i’m going to be referring to both nfs servers by their IPs and hostnames.
server1 will be nfs1 with ip 10.10.10.1 and server2 will be nfs2 with ip 10.10.10.2. Your may use different range of private IPs, so make sure to put in the correct IPs where necessary within this how-to.

Do the following on nfs1(10.10.10.1) and nfs2(10.10.10.2):

To view mount points on your system:

vi /etc/fstab

Search for the /nfsdata mount point and comment it out to prevent it form automatically being mounted on boot. Take note of the device for the /nfsdata mount point. Here is what my fstab looks like.

LABEL=/boot /boot ext3 defaults 1 2
#/dev/VolGroup00/LogVol04 /nfsdata ext3 defaults 1 2
……

Note:

If you are going to use external storage as NFS partition then you can set that up after base os installation. It is not mandatory to create nfs partition during OS installation. Also it is best to use LVM for partitioning instead of fix size device. LVM gives greater flexibility.

Now unmount /nfsdata partition if it is mounted as heartbeat will take care of mounting that.

umount /nfsdata

Make sure that ntp and ntpdate are installed on both of the nfs servers.

yum install ntp ntpdate

The time on both servers must be identical. Edit your /etc/ntp.conf file and verify settings.

Now lets check and make sure that the nfs service is not running on startup and that selinux is also turned off.

setup

If you have not installed full operating systems or all administration tools then you can use

system-config-securitylevel-tui

and disable firewall and selinux.

Now We will install DRBD and DRBD-Kernel module

Note: If you are installing DRBD and DRBD-kernel on physical system then you will need to install:
drbd-8.0.16-5.el5.centos.x86_64.rpm AND kmod-drbd-8.0.16-5.el5_3.x86_64.rpm
And if you are using DRBD and DRBD kernel module on virtual machine (VM) then you need to install:
drbd-8.0.16-5.el5.centos.x86_64.rpm AND kmod-drbd-xen-8.0.16-5.el5_3.x86_64.rpm
You can choose version of package according to your choice.
Along with DRBD and DRBD-kernel module, we will install following rpm’s also.

yum install Perl-TimeDate net-snmp-libs-x86_64

rpm -Uvh heartbeat-pils-2.1.4-2.1.x86_64.rpm

rpm -Uvh heartbeat-stonith-2.1.4-2.1.x86_64.rpm

rpm -Uvh heartbeat-2.1.4-2.1.x86_64.rpm

Now its time to edit drbd.conf file. /etc/drbd.conf is default location for that config file.

common { syncer { rate 100M; al-extents 257; } }

resource r0 {
protocol C;
handlers { pri-on-incon-degr “halt -f”; }
disk { on-io-error detach; }
startup { degr-wfc-timeout 60; wfc-timeout 60; }

on nfs01 {
address 10.10.10.1:7789;
device /dev/drbd0;
disk /dev/LogVol04/nfsdrbd;
meta-disk internal;
}
on nfs02 {
address 10.10.10.2:7789;
device /dev/drbd0;
disk /dev/LogVol04/nfsdrbd;
meta-disk internal;
}
}

Let me give you little bit more information about parameters used in above config file.
So lets start from the top.

  • Protocol – This is the method that drbd will use to sync both of the nfs servers. There are 3 available options here, Protocol A, Protocol B and Protocol C.Protocol A is an asynchronous replication protocol. The drbd.org manual states, “local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has been placed in the local TCP send buffer. In the event of forced fail-over, data loss may occur. The data on the standby node is consistent after fail-over, however, the most recent updates performed prior to the crash could be lost.”Protocol B is a memory synchronous (semi-synchronous) replication protocol. The drbd.org manual states, “local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has reached the peer node. Normally, no writes are lost in case of forced fail-over. However, in the event of simultaneous power failure on both nodes and concurrent, irreversible destruction of the primary’s data store, the most recent writes completed on the primary may be lost.”

Protocol C is a synchronous replication protocol. The drbd.org manual states, “local write operations on the primary node are considered completed only after both the local and the remote disk write have been confirmed. As a result, loss of a single node is guaranteed not to lead to any data loss. Data loss is, of course, inevitable even with this replication protocol if both nodes (or their storage subsystems) are irreversibly destroyed at the same time.

You may choose your desired protocol but Protocol C is the most commonly used one and it is the safest method.

  • rate – The rate is the maximum speed at which data will be sent from one nfs server to the other while syncing. This should be about a third of your maximum write speed. In my case, I have only a single disk that can write about 45mb/sec so a third of that would be 15mb. This number will usually be much higher for people with raid setups. In some large raid setups, the bottleneck would be the network and not the disks so set the rate accordingly.
  • al-extent – This data on the disk are cut up into slices for synchronization purposes. For each slice there is an al-extent that is used to indicate any changes to that slice. Larger al-extent values make synchronization slower but benefit from less writes to the metadata partition. In my case, I’m using an internal metadata which means the drbd metadata is written to the same parition that my nfs data is on. It would benefit me to have less metadata writes to prevent the disk arm from constantly moving back and forth and degrading performance. If you are using a raid setup and a separate partition for the metadata then set this number lower to benefit from faster synchronization. This number MUST be a prime to gain the most possible performance because it is used in specific hashes that benefit from prime number sized structures.
  • pri-on-incon-degr – The “halt -f” command is executed if the node is primary, degraded and if the data is inconsistent. I use this to make sure drbd is halted when there is some sort of data inconsistency to prevent a major mess from occuring.
  • on-io-error – This allows you to handle low level I/O errors. The method I use is the “detach” method. This is the recommended option by drbd.org. On the occurrence of a lower-level I/O error, the node drops its backing device, and continues in diskless mode.
  • degr-wfc-timeout – This is the amount of time in seconds that is allowed before a connection is timed out. In case a degraded cluster (cluster with only one node left) is rebooted, this timeout value is used instead of wfc-timeout, because the peer is less likely to show up in time, if it had been dead before.

The rest of the config is pretty self explanatory. Replace nfs1 and nfs2 with the hostnames of your nfs servers. To get the hostnames use the following command on both servers:

uname -n

Then replace the disk value with the device name from your fstab file that you commented out. Enter the IP address of each server and use port 7789. The last part is the meta-disk. I used an internal meta-disk because I only have one hard disk in the server and it would not give me any benefit to create a separate partition for the metadata. If you have a raid setup or a separate disk from your data partition that you can use for the meta data than go ahead and create a 150mb partition. Replace the word “internal” in the config file with your device name that you used for the meta data partition.

Now that we finally have our drbd.conf file ready we can move on. Lets go ahead and enable the drbd kernel module.

modprobe drbd

Now that the kernel module is enabled lets start up drbd.

drbdadm up all

This will start drbd, now lets check its status.

cat /proc/drbd

You can always use the above command to check the status of drbd. The above command should show you something like this.

0: cs:Connected st:Secondary/Secondary ld:Inconsistent
ns:0 nr:0 dw:0 dr:0 al:0 bm:1548 lo:0 pe:0 ua:0 ap:0
1: cs:Unconfigured

You should get some more data before it but the above part is what we are interested in. If you notice it shows that drbd is connected and both nodes are in secondary mode. This is because we have not assigned which node is going to be the primary yet. It also says the data is inconsistent because we have not done the initial sync yet.

I am going to set nfs1 to be my primary node and nfs2 to be my secondary node. If nfs1 fails, nfs2 will takeover but if nfs1 comes back online then all the data from nfs2 will be synced back to nfs1 and nfs1 will take over again.

First of all lets go ahead and delete any data that was created on the /data partition that we setup during our intial OS installation. Be very careful with the command below. Make sure to use the appropriate device because all data on that device will be lost.

dd if=/dev/zero bs=1M count=1 of=/dev/VolGroup00/LogVol04; sync

Instead of “/dev/VolGroup00/LogVol04″, replace it with your device for the /data parition. Now that the partition is completely erased on both servers, lets create the meta data.

drbdadm create-md r0

Do the following ONLY on nfs1(10.10.10.1)

Now that the metadata is created, we can move onto assigning a primary node and conducting the initial sync. It is absolutely important that you only execute the following command on the primary node. It doesn’t matter which node you choose to be the primary since they should be identical. In my case, I decided to use nfs1 as the primary.

drbdadm — –overwrite-data-of-peer primary r0

Ok, now we just have to sit back and wait for the initial sync to finish. This is going to take some time to finish even though there is no data on each device, drbd has to sync every single block on /data partition from nfs1 to nfs2. You can check the status by using the following command.

cat /proc/drbd

Do the following on nfs1(10.10.10.1) and nfs2(10.10.10.2):

After the initial sync is finished, “cat /proc/drbd” should show something like this.

0: cs:Connected st:Primary/Secondary ld:Consistent
ns:12125 nr:0 dw:0 dr:49035 al:0 bm:6 lo:0 pe:0 ua:0 ap:0
1: cs:Unconfigured

If you notice, we are still connected and have a primary and secondary node with consistent data.

Do the following ONLY on nfs1(10.10.10.1) :

Now lets make an ext3 file system on our drbd device and mount it. Since drbd is running, the ext3 file system will also be created on the secondary node.

mkfs.ext3 /dev/drbd0

The above command will create an ext3 file system on the drbd device. Now lets go ahead and mount it.

mount -t ext3 /dev/drbd0 /data

We know that NFS stores important information in /var/lib/nfs by default which is required to function correctly. In order to preserve file locks and other important information, we need to have that data stored on the drbd device so that if the primary node failes, NFS on the secondary node will continue from right where the primary node left off.

mv /var/lib/nfs/ /data/
ln -s /data/nfs/ /var/lib/nfs
mkdir /data/export
umount /data

So lets go over what we just did.

  • We have now moved the nfs folder from /var/lib to /data.
  • We created a symbolic link from /var/lib/nfs to /data/nfs since the operating system is still going to look for /var/lib/nfs when nfs is running.
  • We created an export directory in /data to store all the actual data that we are going to use for our nfs share.
  • Finally, we un-mounted the /data partition since we finished what we were doing.

Do the following ONLY on nfs2(10.10.10.2):
Since we moved the nfs folder to /data, that was synced over to the secondary node as well. We just need to create the symbolic link so that when the /data partition is mounted on nfs2 we have a link to the nfs data.

rm -rf /var/lib/nfs/
ln -s /data/nfs/ /var/lib/nfs

So we removed the nfs folder and created a symbolic link from /var/lib/nfs to /data/nfs. The symbolic link will be broken since the /data parition is not mounted. Don’t worry about that because in the event of a failover that partiton will be mounted and everything will be fine.

Now we need to configure heartbeat on both nfs servers ns1 and nfs2. we have already installed required softwares.
Create /etc/ha.d/ha.cf on both nfs servers with following contents in it

keepalive 2
deadtime 30
bcast eth0
node ukibinfs01 ukibinfs02

Replace names for “node” as per your hostnames. to find out your hostname use uname -n.

Now we need to create “/etc/ha.d/haresources” configuration file on both nfs servers with following configuration

nfs1 IPaddr::10.10.10.3/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfslock nfs
IP address used in haresources config file is floating IP. Out of both NFS server which ever is primary, will have that IP configured on eth0:0.

Hi all, hope you are enjoying series of articles published on this website for free.. totally free.

In this article you will learn how to setup redundant NFS servers using DRBD (Distributed-Replicated-Block-Devices) Technology. Complete step-by-step procedure is listed below. It works for me so hopefully it will work for your also. No gurantees are given.

1. Two servers with similar storage disk (harddrive) setup (To create a redundant nfs server)
2. One client system/server where the nfs share will be mounted.
3. Static IPs for all servers.

First step is to install CentOS on both machines. During the install process, create a separate blank partition on both machines to be used as your nfs mount. Make sure you creat exact same size partitions on both servers. Set the mount point to /nfsdata during installation.

From this point on i’m going to be referring to both nfs servers by their IPs and hostnames.
server1 will be nfs1 with ip 10.10.10.1 and server2 will be nfs2 with ip 10.10.10.2. Your may use different range of private IPs, so make sure to put in the correct IPs where necessary within this how-to.

Do the following on nfs1(10.10.10.1) and nfs2(10.10.10.2):

To view mount points on your system:

vi /etc/fstab

Search for the /nfsdata mount point and comment it out to prevent it form automatically being mounted on boot. Take note of the device for the /nfsdata mount point. Here is what my fstab looks like.

LABEL=/boot /boot ext3 defaults 1 2
#/dev/VolGroup00/LogVol04 /nfsdata ext3 defaults 1 2
……

Note:

If you are going to use external storage as NFS partition then you can set that up after base os installation. It is not mandatory to create nfs partition during OS installation. Also it is best to use LVM for partitioning instead of fix size device. LVM gives greater flexibility.

Now unmount /nfsdata partition if it is mounted as heartbeat will take care of mounting that.

umount /nfsdata

Make sure that ntp and ntpdate are installed on both of the nfs servers.

yum install ntp ntpdate

The time on both servers must be identical. Edit your /etc/ntp.conf file and verify settings.

Now lets check and make sure that the nfs service is not running on startup and that selinux is also turned off.

setup

If you have not installed full operating systems or all administration tools then you can use

system-config-securitylevel-tui

and disable firewall and selinux.

Now We will install DRBD and DRBD-Kernel module

Note: If you are installing DRBD and DRBD-kernel on physical system then you will need to install:
drbd-8.0.16-5.el5.centos.x86_64.rpm AND kmod-drbd-8.0.16-5.el5_3.x86_64.rpm
And if you are using DRBD and DRBD kernel module on virtual machine (VM) then you need to install:
drbd-8.0.16-5.el5.centos.x86_64.rpm AND kmod-drbd-xen-8.0.16-5.el5_3.x86_64.rpm
You can choose version of package according to your choice.
Along with DRBD and DRBD-kernel module, we will install following rpm’s also.

yum install Perl-TimeDate net-snmp-libs-x86_64

rpm -Uvh heartbeat-pils-2.1.4-2.1.x86_64.rpm

rpm -Uvh heartbeat-stonith-2.1.4-2.1.x86_64.rpm

rpm -Uvh heartbeat-2.1.4-2.1.x86_64.rpm

Now its time to edit drbd.conf file. /etc/drbd.conf is default location for that config file.

common { syncer { rate 100M; al-extents 257; } }

resource r0 {
protocol C;
handlers { pri-on-incon-degr “halt -f”; }
disk { on-io-error detach; }
startup { degr-wfc-timeout 60; wfc-timeout 60; }

on nfs01 {
address 10.10.10.1:7789;
device /dev/drbd0;
disk /dev/LogVol04/nfsdrbd;
meta-disk internal;
}
on nfs02 {
address 10.10.10.2:7789;
device /dev/drbd0;
disk /dev/LogVol04/nfsdrbd;
meta-disk internal;
}
}

Let me give you little bit more information about parameters used in above config file.
So lets start from the top.

  • Protocol – This is the method that drbd will use to sync both of the nfs servers. There are 3 available options here, Protocol A, Protocol B and Protocol C.Protocol A is an asynchronous replication protocol. The drbd.org manual states, “local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has been placed in the local TCP send buffer. In the event of forced fail-over, data loss may occur. The data on the standby node is consistent after fail-over, however, the most recent updates performed prior to the crash could be lost.”Protocol B is a memory synchronous (semi-synchronous) replication protocol. The drbd.org manual states, “local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has reached the peer node. Normally, no writes are lost in case of forced fail-over. However, in the event of simultaneous power failure on both nodes and concurrent, irreversible destruction of the primary’s data store, the most recent writes completed on the primary may be lost.”

Protocol C is a synchronous replication protocol. The drbd.org manual states, “local write operations on the primary node are considered completed only after both the local and the remote disk write have been confirmed. As a result, loss of a single node is guaranteed not to lead to any data loss. Data loss is, of course, inevitable even with this replication protocol if both nodes (or their storage subsystems) are irreversibly destroyed at the same time.

You may choose your desired protocol but Protocol C is the most commonly used one and it is the safest method.

  • rate – The rate is the maximum speed at which data will be sent from one nfs server to the other while syncing. This should be about a third of your maximum write speed. In my case, I have only a single disk that can write about 45mb/sec so a third of that would be 15mb. This number will usually be much higher for people with raid setups. In some large raid setups, the bottleneck would be the network and not the disks so set the rate accordingly.
  • al-extent – This data on the disk are cut up into slices for synchronization purposes. For each slice there is an al-extent that is used to indicate any changes to that slice. Larger al-extent values make synchronization slower but benefit from less writes to the metadata partition. In my case, I’m using an internal metadata which means the drbd metadata is written to the same parition that my nfs data is on. It would benefit me to have less metadata writes to prevent the disk arm from constantly moving back and forth and degrading performance. If you are using a raid setup and a separate partition for the metadata then set this number lower to benefit from faster synchronization. This number MUST be a prime to gain the most possible performance because it is used in specific hashes that benefit from prime number sized structures.
  • pri-on-incon-degr – The “halt -f” command is executed if the node is primary, degraded and if the data is inconsistent. I use this to make sure drbd is halted when there is some sort of data inconsistency to prevent a major mess from occuring.
  • on-io-error – This allows you to handle low level I/O errors. The method I use is the “detach” method. This is the recommended option by drbd.org. On the occurrence of a lower-level I/O error, the node drops its backing device, and continues in diskless mode.
  • degr-wfc-timeout – This is the amount of time in seconds that is allowed before a connection is timed out. In case a degraded cluster (cluster with only one node left) is rebooted, this timeout value is used instead of wfc-timeout, because the peer is less likely to show up in time, if it had been dead before.

The rest of the config is pretty self explanatory. Replace nfs1 and nfs2 with the hostnames of your nfs servers. To get the hostnames use the following command on both servers:

uname -n

Then replace the disk value with the device name from your fstab file that you commented out. Enter the IP address of each server and use port 7789. The last part is the meta-disk. I used an internal meta-disk because I only have one hard disk in the server and it would not give me any benefit to create a separate partition for the metadata. If you have a raid setup or a separate disk from your data partition that you can use for the meta data than go ahead and create a 150mb partition. Replace the word “internal” in the config file with your device name that you used for the meta data partition.

Now that we finally have our drbd.conf file ready we can move on. Lets go ahead and enable the drbd kernel module.

modprobe drbd

Now that the kernel module is enabled lets start up drbd.

drbdadm up all

This will start drbd, now lets check its status.

cat /proc/drbd

You can always use the above command to check the status of drbd. The above command should show you something like this.

0: cs:Connected st:Secondary/Secondary ld:Inconsistent
ns:0 nr:0 dw:0 dr:0 al:0 bm:1548 lo:0 pe:0 ua:0 ap:0
1: cs:Unconfigured

You should get some more data before it but the above part is what we are interested in. If you notice it shows that drbd is connected and both nodes are in secondary mode. This is because we have not assigned which node is going to be the primary yet. It also says the data is inconsistent because we have not done the initial sync yet.

I am going to set nfs1 to be my primary node and nfs2 to be my secondary node. If nfs1 fails, nfs2 will takeover but if nfs1 comes back online then all the data from nfs2 will be synced back to nfs1 and nfs1 will take over again.

First of all lets go ahead and delete any data that was created on the /data partition that we setup during our intial OS installation. Be very careful with the command below. Make sure to use the appropriate device because all data on that device will be lost.

dd if=/dev/zero bs=1M count=1 of=/dev/VolGroup00/LogVol04; sync

Instead of “/dev/VolGroup00/LogVol04″, replace it with your device for the /data parition. Now that the partition is completely erased on both servers, lets create the meta data.

drbdadm create-md r0

Do the following ONLY on nfs1(10.10.10.1)

Now that the metadata is created, we can move onto assigning a primary node and conducting the initial sync. It is absolutely important that you only execute the following command on the primary node. It doesn’t matter which node you choose to be the primary since they should be identical. In my case, I decided to use nfs1 as the primary.

drbdadm — –overwrite-data-of-peer primary r0

Ok, now we just have to sit back and wait for the initial sync to finish. This is going to take some time to finish even though there is no data on each device, drbd has to sync every single block on /data partition from nfs1 to nfs2. You can check the status by using the following command.

cat /proc/drbd

Do the following on nfs1(10.10.10.1) and nfs2(10.10.10.2):

After the initial sync is finished, “cat /proc/drbd” should show something like this.

0: cs:Connected st:Primary/Secondary ld:Consistent
ns:12125 nr:0 dw:0 dr:49035 al:0 bm:6 lo:0 pe:0 ua:0 ap:0
1: cs:Unconfigured

If you notice, we are still connected and have a primary and secondary node with consistent data.

Do the following ONLY on nfs1(10.10.10.1) :

Now lets make an ext3 file system on our drbd device and mount it. Since drbd is running, the ext3 file system will also be created on the secondary node.

mkfs.ext3 /dev/drbd0

The above command will create an ext3 file system on the drbd device. Now lets go ahead and mount it.

mount -t ext3 /dev/drbd0 /data

We know that NFS stores important information in /var/lib/nfs by default which is required to function correctly. In order to preserve file locks and other important information, we need to have that data stored on the drbd device so that if the primary node failes, NFS on the secondary node will continue from right where the primary node left off.

mv /var/lib/nfs/ /data/
ln -s /data/nfs/ /var/lib/nfs
mkdir /data/export
umount /data

So lets go over what we just did.

  • We have now moved the nfs folder from /var/lib to /data.
  • We created a symbolic link from /var/lib/nfs to /data/nfs since the operating system is still going to look for /var/lib/nfs when nfs is running.
  • We created an export directory in /data to store all the actual data that we are going to use for our nfs share.
  • Finally, we un-mounted the /data partition since we finished what we were doing.

Do the following ONLY on nfs2(10.10.10.2):
Since we moved the nfs folder to /data, that was synced over to the secondary node as well. We just need to create the symbolic link so that when the /data partition is mounted on nfs2 we have a link to the nfs data.

rm -rf /var/lib/nfs/
ln -s /data/nfs/ /var/lib/nfs

So we removed the nfs folder and created a symbolic link from /var/lib/nfs to /data/nfs. The symbolic link will be broken since the /data parition is not mounted. Don’t worry about that because in the event of a failover that partiton will be mounted and everything will be fine.

Now we need to configure heartbeat on both nfs servers ns1 and nfs2. we have already installed required softwares.
Create /etc/ha.d/ha.cf on both nfs servers with following contents in it

keepalive 2
deadtime 30
bcast eth0
node ukibinfs01 ukibinfs02

Replace names for “node” as per your hostnames. to find out your hostname use uname -n.

Now we need to create “/etc/ha.d/haresources” configuration file on both nfs servers with following configuration

nfs1 IPaddr::10.10.10.3/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfslock nfs
IP address used in haresources config file is floating IP. Out of both NFS server which ever is primary, will have that IP configured on eth0:0.

Quick How To on Logical Volume Manager (LVM)

How to create / mange partitions using logical-volume-manager (LVM).

In computer storage, logical volume management or LVM provides a method of allocating space on mass-storage devices that is more flexible than conventional partitioning schemes. In particular, a volume manager can concatenate, stripe together or otherwise combine partitions into larger virtual ones that administrators can re-size or move, potentially without interrupting system use.
Volume management represents just one of many forms of storage virtualization; its implementation takes place in a layer in the device-driver stack of an OS (as opposed to within storage devices or in a network).
Important terms :
PV : Physical Voume : hard disks, harddisk partitions or LUN (Logical Unit Numbers).
VG : Volume Group : One physical volume or multiple physical volumes can make one volume group.
LV : Logical Volume : one volume group or multiple volume groups can make one logical volume.

Scenario 1 : You have some free space (unallocated) on a hard drive and you want to create different logical volumes after OS installation. Also, you have already created partitions using LVM during OS installation. Centos / RedHat and Fedora does create LVM by default during installation.
Please refer following screen shoot. I have used 3 commands. pvscan , vgscan , lvscan.

pvcreate, vgcreate, lvcreate, pvscan, vgscan, lvscan,

 

 

 

 

 

 

I am using centos 5.7 for this tutorial. By default system has created physical volume called /dev/vda2pvscan command is used to list current physical volumes in linux.
Then you can see that 1 volume group VolGroup00 is created.vgscan command is used to list current volume groups on your linux (GNU/Linux) system.
The actual partition we can mount is called logical volume. lvscan command is used to list actual partitions or logical disks.  /dev/VolGroup00/LogVol00 is one of the logical volume on my system.

Steps to create new logical volume:
e.g you have added a new disk to your server and it is listed as /dev/vdb block device under /dev directory on your system. vd is notation for virtual disk. it could be xvdb or xvdc or xvdd if you are using citrix virtualization. It could be /dev/sdb or /dev/sdc or /dev/sdd if the disk is physical disk or VMWare virtualization.

1. pvcreate /dev/vdb (To create physical volume from newly added disk). this step is mandatory even if you are planning to extend your existing volume group.
2. vgcreate <volume group name> /dev/sdb
E.g vgcreate volgroup100 /dev/sdb
3. lvcreate -L+100G -n <logical volume name> <volume group name in step 2>
E.g lvcreate -L+100G logvol100 volgroup100
4. now use lvscan command to list all logical volume. you will able to see newly added LV.
5. now we need to format newly created logical volume so that we can mount it on system.
mkfs.ext4 /dev/volgroup100/logvol100

MySQL database backup script?

How to backup multiple mysql databases ?
A very powerful shell script to backup your mysql databases totally free.. enjoy

#!/bin/bash
NOW=`date +"%Y-%m"`;
BACKUPDIR="location/of/your/backup/dir/$NOW";

### Server Setup ###
#* MySQL login user name *#
MUSER="user";

#* MySQL login PASSWORD name *#
MPASS="pass";

#* MySQL login HOST name *#
MHOST="your-mysql-ip";
MPORT="your-mysql-port";

# DO NOT BACKUP these databases
IGNOREDB="
information_schema
mysql
test
"

#* MySQL binaries *#
MYSQL=`which mysql`;
MYSQLDUMP=`which mysqldump`;
GZIP=`which gzip`;

# assuming that /nas is mounted via /etc/fstab
if [ ! -d $BACKUPDIR ]; then
mkdir -p $BACKUPDIR
else
:
fi

# get all database listing
DBS="$(mysql -u $MUSER -p$MPASS -h $MHOST -P $MPORT -Bse 'show databases')"

# SET DATE AND TIME FOR THE FILE
NOW=`date +"d%dh%Hm%Ms%S"`; # day-hour-minute-sec format
# start to dump database one by one
for db in $DBS
do
DUMP="yes";
if [ "$IGNOREDB" != "" ]; then
for i in $IGNOREDB # Store all value of $IGNOREDB ON i
do
if [ "$db" == "$i" ]; then # If result of $DBS(db) is equal to $IGNOREDB(i) then
DUMP="NO";         # SET value of DUMP to "no"
#echo "$i database is being ignored!";
fi
done
fi

if [ "$DUMP" == "yes" ]; then # If value of DUMP is "yes" then backup database
FILE="$BACKUPDIR/$NOW-$db.gz";
echo "BACKING UP $db";
$MYSQLDUMP --add-drop-database --opt --lock-all-tables -u $MUSER -p$MPASS -h $MHOST -P $MPORT $db | gzip > $FILE
fi
done

The vi, vim editor in linux

In this quick tutorial I will share some vim basics. Vim is a powerful text editor used in CLI (command line interface). As there are lots of configuration files in gnu/linux, which are all in clear text format, you’ll often need to edit them using a text editor. vim (in short vi) is a great tool to use.
One of the first things to know about vi is that it typically functions in three different modes:

  1. Command mode
  2. Insert mode
  3. Last line mode

vi command mode

When you first start editing a file with the vi editor you will be in vi command mode. In this mode you can issue many vi commands, including commands like insert, append, and delete, and other search and navigation commands that let you move around your file.

Possibly the most important thing to know is that when you’re in command mode you can’t insert text immediately. You first need to issue an insert, append, or open command to insert text. These commands are actually fairly simple, and I’ve documented them in this

Lets talk about other vim modes.

vi insert mode

Once you issue a vi insert, append, or open command, you will be in vi insert mode. If you’re working with a modern vi or vim implementation, your vi editor is typically configured to show the current mode of operation, so when you go into insert mode, you’ll see a text string like this on the last line of your vi editor window:

-- INSERT --

At this point you can (a) type text into your file and (b) use the arrow keys to navigate around your file just as you would do with any other text editor.

A very important concept to know is that when you’re in vi insert mode, but you want to switch back to vi command mode, you easily move back to command mode by pressing the [Esc] key. This command is so important, I’ll show it again:

[Esc]

This command is very common, and I often see expert vi users press the [Esc] key several times in a row. They usually do this (a) to be sure they hit the key and they’re really back in command mode, and (b) to hear the beep from the computer, which happens when you press the [Esc] key when you’re already in vi command mode. This seems to serve as a form of feedback which assures them that they’re in command mode.

vi last line mode

The last vi mode is known as vi last line mode. You can only get to last line mode from command mode, and you get into last line mode by pressing the colon key, like this:

:

After pressing this key, you’ll see a colon character appear at the beginning of the last line of your vi editor window, and your cursor will be moved to that position. This indicates that vi is ready for you to type in a “last line command”.

From this vi command prompt you can do all sorts of really amazing things. You can do simple things, like quitting your vi session, like this:

:q

or this:

:q!

or this:

:wq

From last line more you can also perform some amazing vi search commands or vim search and replace commands. Another cool thing is that you can issue Linux or Unix commands from within your vi editor session, like this simple ls command:

:!ls

It’s really handy sometimes to be able to stay in your vi editing session but still be able to run Unix or Linux commands.

And finally, you can also issue many vi configuration commands, such as this command that tells vi to show lines numbers in your current editor window:

:set shownumber

There is a ton of power in this vi last line mode, and I’ve tried to share pieces of this power in a variety of different vi tutorials. (Just search this blog for “vi” or “vi editor” and you’ll find a wealth of vi tutorials.)

One last note about the vi last line mode: If you’re in last line mode, and you want to switch back to command mode, there are several different ways to do this. For consistency, one way to do this is to press the [Esc] key twice, like this:

[Esc][Esc]

(This is consistent with the method of moving from insert mode back to command mode, except you have to press the [Esc] key twice.)

A second way is to press the [Backspace] key until anything you typed and the initial “:” character are gone. At this point you’ll be back in command mode.

Finally, if you haven’t typed anything at all, and you’re just looking at the “:” prompt on the last line, you can just press [Enter], and you’ll be placed back in vi command mode.

After opening a new OR existing file in vim editor with vim you can try modes listed here.

Insert mod : lets you insert text in a document. shotcurt : “i” (insert where the cursor is) or “o” (insert at the beginning of the following line).

Visual mod : permits to select the text like you would do with a mouse but using the keyboard instead of the mouse. Useful to copy several lines for example. shotcurt : V

Let’s now speak about the command mode.

A command begins with the symbol “:”.

When you are in another mod you can use the escape key (sometimes you’ll need to hit it twice) to come back to command mod at any time.

save : :w
save and exit : :wq
exit : :q
force : ! (example :w! :q!)
vertical split : open a document and then type :vsplit /path-to-document/document and this will open the specified document and split the screen so you can see both documents.
copy : y
copy a line : yy
paste : p
cut : d
cut a line : dd

I repeat these are very basic commands for vim, but they are very useful, and I hope this will help you configuring your Linux.