Reflectiv's blog

Script Shell de backup de dossier via FTP

J'ai optimisé le précédent script de backup de dossier en y ajoutant une sauvegarde via FTP.

J'ai mis du temps à mettre en place cette technique car cela ne me convenait pas pour la simple et bonne raison que le transfert de l'archive générée peut-être longue (et lourde), surtout si on sauvegarde une quantité conséquente de fichiers !

Mais après en avoir parlé à Pierre, il s'est avéré que le temps de transfert, même pour de gros fichier, se faisait de manière très rapide, surtout entre serveurs.

Donc à défaut d'avoir les moyens de se payer deux serveurs et de les mettres en mirroring (ca ne saurait tarder) (les deux serveurs .... pas l'argent ...), je suis bien obligé d'utiliser une sauvegarde via FTP.

Voici donc le fameux script que j'utilise :

#!/bin/bash
#
# Shell script (BASH) used in cron to made a backup of the indicated directory

# In order to run this script, you must have following tools installed:
# - gpg
# - ncftp

# And we will use theses :
# - tar
# - rm
# - chown
# - chmod
#
#
# Installation
# Customize the script according to your need. You need to setup :
# - A list of folders, separated by space, and finished with the last /
# - A GPG passphrase
# - The file where to log the messages                         (default /var/log/backup.log)
# - The number of days the backups are stored         (default 5)
# - The FTP credentials
# - The Folders in that FTP
# - The Email to send reports when error occurs
#
# --------------------------------------------------------------------
# This is a free shell script under GNU GPL version 3.0 or above
# Copyright (C) 2005 ReFlectiv project.
# Feedback/comment/suggestions : http://www.reflectiv.net/
# -------------------------------------------------------------------------

# List of folders to save
LIST_F0LDERS="/etc/ /home/ /var/lib/mysql/";

# The password for the archive
GPG_PASS="my_passwd";

# File name where to log the messages
LOG_FILE="/var/log/backup.log";

# Number of days the archives are keeped
KEEPING_DAYS=5;

# FTP Credentials
FTP_SERVER="ftp.server.tld"
FTP_USER="user"
FTP_PASS="password"

# Backup directory on the FTP :
FTP_DIR="/backups/files"

# Email to send when error occurs
USER_EMAIL="email"

# Only change if your UNIX stores bin in diffrent location
GPG="/usr/bin/gpg";
NCFTP="/usr/bin/ncftp"

#######################################################################
# Do not change anything below
#######################################################################

# We mades some vars
UNIX_DATE=`date +"%Y_%m_%d"`;
CURRENT_DATE=`date +"%d/%m/%Y - %H:%M:%S"`;
TBZ_FILE="/tmp/"$UNIX_DATE".tbz";
GPG_FILE=$TBZ_FILE".gpg";

# Adding new clear entry to the log file
echo "" >> $LOG_FILE."tmp";
echo "----------------------------------------------------------------------------------------------------" >> $LOG_FILE."tmp";
echo "" >> $LOG_FILE."tmp";

# Function called to log info into the file
function log {
    echo "["$CURRENT_DATE"] - "$1 >> $LOG_FILE".tmp";

    if [ ! $2 == "" ]; then
                mail -s "[$(hostname)] - Remote backup error" $USER_EMAIL > $LOG_FILE;
                rm $LOG_FILE".tmp";
        exit 1;
    fi
}

log "Starting remote backup";

# Preliminary tests
[ -x $GPG ] || log "gpg not found" 1;
[ -x $NCFTP ] || log "ncftp not found" 1;

for FOLDER in $LIST_FOLDERS
do
        # We test if the original folder exists
    [ -d $FOLDER ] || log "The folder "$FOLDER" does not exists !" 1;
done

# We make the archive
tar -cjf $TBZ_FILE $LIST_F0LDERS
# If an error occured ...
[ $? -eq 0 ] || log "An error occured with the command tar" 1;

# We encrypt the archive
$GPG --yes -c --passphrase $GPG_PASS --s2k-cipher-algo RIJNDAEL256 $TBZ_FILE;
# If an error occured ...
[ $? -eq 0 ] || log "An error occured with the command gpg" 1;

# We delete the archive
rm $TBZ_FILE;
# If an error occured ...
[ $? -eq 0 ] || log "An error occured with the command rm" 1;

# We modify the rights for the file
chmod 400 $GPG_FILE;
# If an error occured ...
[ $? -eq 0 ] || log "An error occured with the command chmod" 1;

# We modify the owner and group of the file
chown $UID $GPG_FILE;
# If an error occured ...
[ $? -eq 0 ] || log "An error occured with the command chown" 1;


# We build the name of the archive from $KEEPING_DAYS days ago
OLD_ARCHIVE=`date --date $KEEPING_DAYS' days ago' "+%Y_%m_%d"`".tbz.gpg";

log "Starting update to FTP server "$FTP_SERVER;

$NCFTP -u$FTP_USER -p$FTP_PASS $FTP_SERVER > $LOG_FILE;
rm $LOG_FILE".tmp";

exit 0;

Le script effectue une sauvegarde des répertoires dans une archive tar.bz. Il chiffre ensuite cette archive avec gpg en utilisant un algorithme synchrone, puis envoie l'archive au serveur distant. Il prends aussi soin de bien supprimer les éléments qui ne sont plus utilisés afin d'éviter toute tentative de lecture (droits et accès aux fichier tbz et gpg puis suppression de tous les éléments).

Beaucoup de paramètres sont modifiable, comme vous pouvez le voir, et ce afin que vous puissiez l'adapter au mieux.

Le seul problème restant est que je n'ai pas réussi à faire taire ncftp ! Si quelqu'un connait une solution (l'argument -v quiet semble fonctionner pour les anciennes versions et n'y est vraisemblablement plus présent :s), je suis preneur !

Bon backup à vous :)

Filed under // Development Scripts Unix/Linux backup bash dossier fichier file folder ftp script shell

Posted by Cyril Nicodème

Astuce : Supprimer récursivement les répertoires .svn

Une petite commande bien utile donnée par CrainBrandy qui permet de supprimer récursivement tous les répertoires .svn en commençant par un dossier.

Je suis sûr que vous avez déjà été confrontés à ce genre de problème ! :p

rm -rf `find . -type d -name .svn`

ou vous pouvez aussi faire :

find . -type d -name .svn -exec rm -rf {} \;

à vous de choisir votre méthode préférée.

Filed under // Development Scripts Unix/Linux bash commande dossier linux repertoire récursif récursivement shell supprimer svn

Posted by Cyril Nicodème

Script Shell de backup de dossier

Voici un script shell qui permet de sauvegarder un certain nombre de répertoires dans un autre répertoire, en archivé, chiffré et daté.

Vous n'avez qu'à paramétrer :

Une liste de répertoire séparés par des espaces
Le dossier de destination
La passphrase pour GPG
Le fichier ou logguer les actions (default: /var/log/backup.log)
Le nombre de jours où l'on garde les archives (defaut: 5)
L'UID de l'utilisateur responsable des archives (défaut: root)
Le GID de l'utilisateur responsable des archives (défaut: root)
Le chmod pour l'accès au répertoire (défaut: 400)

Voici le code :

#!/bin/bash
#
# Shell script (BASH) used in cron to made a backup of the given directory in $1
# to the directory in $2

# In order to run this script, you must have following tools installed:
# - tar
# - gpg
# - rm
# - chown
# - chmod
#
#
# Installation
# Customize the script according to your need. You need to setup :
# - A list of folders, separated by space, and finished with the last /
# - The folder where the backup goes
# - A GPG passphrase
# - The file where to log the messages                         (default /var/log/backup.log)
# - The number of days the backups are stored         (default 5)
# - The owner uid                                                 (default root)
# - The owner gid                                                 (default root)
# - The owner rights                                                 (default 400)
#
# --------------------------------------------------------------------
# This is a free shell script under GNU GPL version 2.0 or above
# Copyright (C) 2005 ReFlectiv project.
# Feedback/comment/suggestions : http://www.reflectiv.net/
# -------------------------------------------------------------------------

# List of folders to save
LIST_F0LDERS=`/var/www/site1/ /var/www/site2/ /var/www/site3/`;

# Folder where the backups goes
BACKUP_FOLDER="/root/snapshoots/";

# The password for the archive
GPG_PASS="passphrase";

# File name where to log the messages
LOG_FILE="/var/log/backup.log";

# Number of days the archives are keeped
KEEPING_DAYS=5;

# UID & GID of the archived file
BU_UID="root";
BU_GID="root";

# Access rights to the archived file
BU_CHMOD=400;

# Only change if your UNIX stores bin in diffrent location
TAR="/bin/tar";
GPG="/usr/bin/gpg";
RM="/bin/rm";
CHOWN="/bin/chown";
CHMOD="/bin/chmod";
MKDIR="/bin/mkdir";

#######################################################################
# Do not change anything below
#######################################################################

# We mades the vars
UNIX_DATE=`date +"%Y_%m_%d"`;
CURRENT_DATE=`date +"%d/%m/%Y - %H:%M:%S"`;
TGZ_FILE=$UNIX_DATE".tgz";
GPG_FILE=$TGZ_FILE".gpg";

# Function called to log info into the file
function log {
        echo "["$CURRENT_DATE"] - "$1 >> $LOG_FILE;

        if [ ! $2 == "" ]; then
                exit 1;
        fi
}

log "Starting backup";

# Preliminary tests
[ -x $TAR ] || log "tar not find" 1;
[ -x $GPG ] || log "gpg not find" 1;
[ -x $RM ] || log "rm not find" 1;
[ -x $CHOWN ] || log "chown not find" 1;
[ -x $CHMOD ] || log "chmod not find" 1;
[ -x $MKDIR ] || log "mkdir not find" 1;

# We test if the file is correctly called
#[ $# -eq 2 ] || log "Invalid number of parameters" 1;

# We test if the destination folder exists
[ -d $BACKUP_FOLDER ] ||
log "The backup destination folder "$BACKUP_FOLDER" seems to not exists." 1;

for FOLDER in $LIST_F0LDERS
do
        # We test if the original folder exists
        [ -d $FOLDER ] || log "The folder "$FOLDER" does not exists !" 1;

        # We mades sample vars
        BACKUP_SUBFOLDER=`basename $FOLDER`;

        if [ ! -d $BACKUP_FOLDER$BACKUP_SUBFOLDER"/" ]; then
                # We create the folder
                $MKDIR $BACKUP_FOLDER$BACKUP_SUBFOLDER"/";

                # We set the same rights and acces as the archived/encrypted file
                $CHOWN $BU_UID:$BU_GID $BACKUP_FOLDER$BACKUP_SUBFOLDER"/";
                $CHMOD $BU_CHMOD $BACKUP_FOLDER$BACKUP_SUBFOLDER"/";
        fi

        # We goes to the backup folder
        cd $FOLDER;

        # We make the archive
        $TAR -czf $BACKUP_FOLDER$BACKUP_SUBFOLDER"/"$TGZ_FILE .
        # If an error occured ...
        [ $? -eq 0 ] || log "An error occured with the command tar" 1;

        # We goes to the backup directory
        cd $BACKUP_FOLDER$BACKUP_SUBFOLDER;

        # We encrypt the archive
        $GPG --yes -c --passphrase $GPG_PASS --s2k-cipher-algo RIJNDAEL256 $TGZ_FILE;
        # If an error occured ...
        [ $? -eq 0 ] || log "An error occured with the command gpg" 1;

        # We delete the archive
        $RM $TGZ_FILE;
        # If an error occured ...
        [ $? -eq 0 ] || log "An error occured with the command rm" 1;

        # We modify the user/groups for the file
        $CHOWN $BU_UID:$BU_GID $GPG_FILE;
        # If an error occured ...
        [ $? -eq 0 ] || log "An error occured with the command rm" 1;

        # We modify the chmod for the file
        $CHMOD $BU_CHMOD $GPG_FILE;
        # If an error occured ...
        [ $? -eq 0 ] || log "An error occured with the command rm" 1;

        OLD_ARCHIVE=`date --date $KEEPING_DAYS' days ago' "+%Y_%m_%d"`".tgz.gpg";
        if [ -f $OLD_ARCHIVE ]; then
                log "Deleting old archive "$OLD_ARCHIVE;
                rm $OLD_ARCHIVE;
        fi

        log $FOLDER" folder saved";
done

log "Backup finished successfully";
exit 0;

Filed under // Development Scripts Unix/Linux backup bash dossier folder shell

Posted by Cyril Nicodème

About Me

My Other Sites

Contributors

Tags

Archive

dossier

Script Shell de backup de dossier via FTP

Astuce : Supprimer récursivement les répertoires .svn

Script Shell de backup de dossier