TECHIES WORLD

For Techs.... Techniques.... Technologies....

Linux

How to configure Lsync (file replication) in Linux servers

admin

lsyncd stands for live syncing daemon, and it works the way that you configure it to watch a directory using event monitoring interfaces such as inotify or fsevents, it queues up these events in a queue and every x second (Can be defined) it will execute these events and copy/create the files in another directory either locally or remotely.

This article explains the steps to configure lsynd in Linux servers.

Enable passwordless authentication

On the master server, run the following command:

#ssh-keygen -t rsa

Two new files are created in ~/.ssh/ id_rsa and id_rsa.pub directories.

Use SFTP to access the master server, copy the id_rsa.pub file to the /root/.ssh/ folder of each of your subordinate servers, and change the name of each copied file to master.pub.

On your subordinate servers, run the following command:

#cat ~/.ssh/master.pub >> ~/.ssh/authorized_keys

From the master server, SSH to log in to each subordinate server and choose Yes to accept the generated key.

Now you can log in to the subordinate servers without a password.

Install Package Dependencies on the Master Server

Run the following command for your OS to install all needed dependencies:

For CentOS:

#yum -y install lua lua-devel pkgconfig gcc asciidoc

For Ubuntu:

#apt-get update && apt-get install -y lua5.1 liblua5.1-dev pkg-config rsync asciidoc

Install Lsyncd

Important: Be sure to manually check the destination directory on the subordinate server. If you have NFS or CloudFuse mount your Cloud Files, these could be deleted by the Lsyncd process.

Note: Lsyncd version 2.0.4 has a known issue with permissions. Ensure that you install either 2.0.7 (stable) or 2.1.5+ (latest). This procedure installs 2.1.5.

To install the version of Lsyncd required for this procedure, you need to compile the code from source. If you are unfamiliar with this process, you can copy and paste the following commands:

#cd /usr/local/src
#wget http://lsyncd.googlecode.com/files/lsyncd-2.1.5.tar.gz
#tar xzvf lsyncd-2.1.5.tar.gz
#cd lsyncd-2.1.5
#./configure && make && make install

When the make install command finishes running, the Lsyncd service is installed on the master server. The compressed Lsyncd package does not come with startup scripts, so you must install your own.

Set Up the startup scripts

Perform the following steps for your OS.

For Ubuntu

Create the following file at /etc/init/lsyncd.conf:

description "lsyncd file synchronizer"

start on (starting network-interface
or starting network-manager
or starting networking)

stop on runlevel [!2345]

expect fork

respawn
respawn limit 10 5

exec /usr/local/bin/lsyncd /etc/lsyncd.lua

Create a symbolic link to preserve functionality when called as a deprecated init script.

#ln -s /lib/init/upstart-job /etc/init.d/lsyncd

If you want to disable Lsyncd from starting automatically at startup (for example, if the customer is scaled down to one server), you can create the following file at /etc/init/lsyncd.override:

/etc/init/lsyncd.override
manual

For Centos

Create the following file at /etc/init.d/lsyncd. (Skip this step if you ran the yum install lsyncd command earlier.)

#!/bin/bash
#
# lsyncd: Starts the lsync Daemon
#
# chkconfig: 345 99 90
# description: Lsyncd uses rsync to synchronize local directories with a remote
# machine running rsyncd. Lsyncd watches multiple directories
# trees through inotify. The first step after adding the watches
# is to, rsync all directories with the remote host, and then sync
# single file buy collecting the inotify events.
# processname: lsyncd

. /etc/rc.d/init.d/functions

config="/etc/lsyncd.lua"
lsyncd="/usr/local/bin/lsyncd"
lockfile="/var/lock/subsys/lsyncd"
pidfile="/var/run/lsyncd.pid"
prog="lsyncd"
RETVAL=0

start() {
if [ -f $lockfile ]; then
echo -n $"$prog is already running: "
echo
else
echo -n $"Starting $prog: "
daemon $lsyncd -pidfile $pidfile $config
RETVAL=$?
echo
[ $RETVAL = 0 ] && touch $lockfile
return $RETVAL
fi
}
stop() {
echo -n $"Stopping $prog: "
killproc $lsyncd
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f $lockfile
return $RETVAL
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
status)
status $lsyncd
;;
*)
echo "Usage: lsyncd {start|stop|restart|status}"
exit 1
esac

exit $?

Configure Lsyncd to start at boot:

#chkconfig lsyncd on

Configure log rotation

After Lsyncd has an assigned log area, it will create a log event if an error occurs during replication. Log files can become large and unruly when not properly monitored, causing a hard disk to fill up. To ensure this does not happen, add Lsyncd to logrotate.

Start by creating a directory to store the log files.

#mkdir /var/log/lsyncd

Create a file at /etc/logrotate.d/lsyncd with the following shell script:

#cat << EOF > /etc/logrotate.d/lsyncd
/var/log/lsyncd/*log {
missingok
notifempty
sharedscripts
postrotate
if [ -f /var/lock/lsyncd ]; then
/sbin/service lsyncd restart > /dev/null 2>/dev/null || true
fi
endscript
}
EOF

With this script enabled, the logs will be rotated instead of accumulating until they are manually deleted.

Create the configuration file

On the master server, create a file at /etc/lsyncd.lua and insert the following data with a text editor.

Note: Replace the target parameter with the private IP address of the subordinate server. You can find the IP address in the Rackspace Cloud Control Panel.

settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd-status.log",
statusInterval = 20
}
sync {
default.rsync,
source="/var/www/",
target="subordinatePrivateIpAddress:/var/www/",
rsync = {
compress = true,
acls = true,
verbose = true,
rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no" }
}

Take special note of the block of code beginning with sync. If you require more web servers, this block of code will must be replicated with the IP address of each additional subordinate server.

With configuration complete, start the lsyncd service.

#service lsyncd start
Configuration Options

settings {
logfile = "/var/log/lsyncd.log", -- Sets the log file
statusFile = "/var/log/lsyncd-status.log" -- Sets the status log file
}
sync{
default.rsyncssh, -- uses the rsyncssh defaults Take a look here: https://github.com/axkibe/lsyncd/blob/master/default-rsyncssh.lua
delete = false, -- Doesn't delete files on the remote host eventho they're deleted at the source. This might be beneficial for some not for others
source="/var/www/domain.com", -- Your source directory to watch
host="remote.host", -- The remote host (use hostname or IP)
targetdir="/var/www/domain.com", -- the target dir on remote host, keep in mind this is absolute path
rsync = {
archive = true, -- use the archive flag in rsync
perms = true, -- Keep the permissions
owner = true, -- Keep the owner
_extra = {"-a"}, -- Sometimes permissions and owners isn't copied correctly so the _extra can be used for any flag in rsync
},
delay = 5, -- We want to delay the syncing for 5 seconds so we queue up the events
maxProcesses = 4, -- We only want to use a maximum of 4 rsync processes at same time
ssh = {
port = 8213 -- This section allows configuration for ssh specific stuff such as a different port
}
}

Leave a Reply