Synchronization with lftp

This page talk about folder synchronization with lftp.


Install lftp package.


There is no specific configuration to do but bookmarks can simplify the use of lftp.

Usage: bookmark [SUBCMD]
bookmark command controls bookmarks

The following subcommands are recognized:
  add <name> [<loc>] - add current place or given location to bookmarks
                       and bind to given name
  del <name>         - remove bookmark with the name
  edit               - start editor on bookmarks file
  import <type>      - import foreign bookmarks
  list               - list bookmarks (default)

Example :

lftp :/> bookmark add
lftp :/> bookmark list              
lftp :/>

It's possible to use bookmarks with the “connect” command or directly on the command line :


Manual synchronization

Before automatic synchronization, it's advised to do some tests manually.
:!: Be carreful, there is a risk to delete source file by mistake.

Synchronization is done with “mirror” :

Usage: mirror [OPTS] [remote [local]]

Mirror specified remote directory to local directory

 -c, --continue         continue a mirror job if possible
 -e, --delete           delete files not present at remote site
     --delete-first     delete old files before transferring new ones
 -s, --allow-suid       set suid/sgid bits according to remote site
     --allow-chown      try to set owner and group on files
     --ignore-time      ignore time when deciding whether to download
 -n, --only-newer       download only newer files (-c won't work)
 -r, --no-recursion     don't go to subdirectories
 -p, --no-perms         don't set file permissions
     --no-umask         don't apply umask to file modes
 -R, --reverse          reverse mirror (put files)
 -L, --dereference      download symbolic links as files
 -N, --newer-than=SPEC  download only files newer than specified time
 -P, --parallel[=N]     download N files in parallel
 -i RX, --include RX    include matching files
 -x RX, --exclude RX    exclude matching files
                        RX is extended regular expression
 -v, --verbose[=N]      verbose operation
     --log=FILE         write lftp commands being executed to FILE
     --script=FILE      write lftp commands to FILE, but don't execute them
     --just-print, --dry-run    same as --script=-

When using -R, the first directory is local and the second is remote.
If the second directory is omitted, basename of first directory is used.
If both directories are omitted, current local and remote directories are used.

The option –dry-run is usefull to make some tests without risk.

Some example of synchronisation (download) :

mirror /Data/Photos /data/transfert/photos -v --log /var/log/lftp/download_nas --parallel=10 --no-perms -e
  • /Data/Photos is the source folder located on the remote server
  • /data/transfert/photos is the local destination folder
  • -v activate the verbose mode
  • –log option set the file to use for logs, this file is overrided each time and is not used with –dry-run
  • –parallel activate parralel transferts, usefull with small files
  • –no-perms disable permission synchronization
  • -e delete files on the destination folder that are no more present on the source

Another example (upload) :

mirror -R /space/Videos/ /data/videos/ --ignore-time --no-perms -e -v
  • -R activate the reverse mode (upload)
  • /space/Videos/ is the local source
  • /data/videos/ is the destination folder on the remote server
  • –ignore-time disable date check used to determined if the file need to be synchronized (only the file size is used)
  • –no-perms disable permission synchronization
  • -e delete files on the destination folder that are no more present on the source
  • -v activate the verbose mode

When uploading, it is not possible to set the date/time on the files uploaded, that's why –ignore-time is needed.

When the complete command is defined, it's possible to use it inside an lftp script.

Example :

connect nas
mirror /Data/Photos /data/transfert/photos -v --log /var/log/lftp/download_nas --parallel=10 --no-perms -e

To use the script, launch lftp with -f option.

Automatic synchronization

To run automaticaly the synchronization, please find below a small batch that will launch lftp.
lftp is started in background like a daemon.

To use the script, you need to :

  1. copy it inside a new file
  2. change NAME variable as you like
  3. specify the link to your lftp script inside DAEMON_OPTS
  4. make the script executable

The script has the following options :

  • start : start the synchronization
  • stop : stop the daemon
  • restart : restart the daemin
  • check : indicate if the synchronization is running or not

Script :


# lftp synchronization script by Matthieu Bouthors

# Change NAME and DAEMON_OPTS before use
set -e

# Must be a valid filename
#This is the command to be run, give the full pathname
DAEMON_OPTS="-f /chemin/vers/cmd.lftp"

export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"

# check if lftp is running

if [ -e $PIDFILE ]
        currentpid=`cat $PIDFILE`
        if `ps -p $currentpid > /dev/null`
                rm -f $PIDFILE

case "$1" in
        if $RUNNING
                started=`date -r $PIDFILE`
                echo "daemon $NAME already started since $started, you need to stop it first"
                echo -n "Starting daemon: "$NAME
                start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $DAEMON -- $DAEMON_OPTS
                echo "."
        echo -n "Stopping daemon: "$NAME
        start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
        rm -f $PIDFILE
        echo "."
                echo -n "Restarting daemon: "$NAME
        if $RUNNING
                start-stop-daemon --stop --quiet --oknodo --retry 30 --pidfile $PIDFILE
                rm -f $PIDFILE
        start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $DAEMON -- $DAEMON_OPTS
        echo "."
        if $RUNNING
                echo "$NAME is running"
                echo "$NAME is not running"

        echo "Usage: "$1" {start|stop|restart|check}"
        exit 1

exit 0
en/linux/synchro_lftp.txt · Last modified: 2012/01/06 18:35 by matthieu
Recent changes RSS feed Debian Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki