wiki.ipfire.org

The community-maintained documentation platform of IPFire

User Tools

Site Tools


addons:dirvish:start

Dirvish

By usage of IPFire as a home server with the Samba-Addon there might be comes a question up, how to backup all the user data. Also the digital photos, which are so beautifully in the Webgallery world-wide attainable, should become absolutely properly secured, finally there are no more negatives in the cabinet.

The following describes, how the data can be secured by using Dirvish with a simple, yet effective backup to a USB-hard drive.

Dirvish is a Perl-script, which uses rsync to create a weekly/daily/hourly complete backup. Therefor, so called hard links will be used.

Every file in a Linux-filesystem are delineate over a hard link. If a second hard link points to the same file, the file is still only once. The file will be deleted tough only until you delete the last hard link. This characteristic makes Dirvish for itself usable. It writes on every backup a full copy of the files which are to be saved as a hard link. If nothing changes on the filesystem, only the hard links will be written, which require only a little disk space. Thus, it can be produce every day a complete image (snapshot), without the need of a huge backup media.

Install the rsync-Addon

As usual you can install rsync over Pakfire .

Download Dirvish

The actual version of Dirvish is downloadable in here . You can copy the Dirvish package with Putty or WinSCP to /root/dirvish onto the IPFire. Apply the directory “dervish” first with “F7” or over the command “mkdir” .

For the next steps, logon as root with Putty onto the IPFire and type the following commands into it:

cd /root/dirvish
tar -xzf dirvish-xyz.tgz
./install.sh

Now the system variable will be prompted:

perl to use (/usr/bin/perl)  -> Return it with Enter
What Installation prefix should be used?() -> /var/ipfire/dirvish
/var/ipfire/dirvish doesn't exist, create it? (n) -> y

All the next requests should be entered, respectively return it with yes. Dirvish will be installed to /var/ipfire/dirvish . On this place all the Addons from IPFire are located, and it should be the same with this one.

Install the missing Perl-moduls

Dirvish requires additional Perl modules, which are not part of IPFire. Since CPAN is also not a part of IPFire, the missing CPAN modules can simply loaded down.

The following modules are required:

Time::JulianDay
Time::ParseDate
Time::Period
Time::Timezone

This *.pm files could be copied with WinSCP onto the IPFire into the path:

/usr/lib/perl5/5.8.8/Time/...

Define the backup-media

As backup-media a external USB-hard drive will be used here but it is also possible to use a second internal hard drive or you save the backup onto a remote Linux-computer. Under no circumstances you should write the backup to the same hard disk on which the data already exists because in a case of a hard drive crash all backup data are lost too.

The further description is based on a scenario with a USB-hard drive. This should be formatted with FAT32, so you can easily read-in the backup even on Windows machines.

On the webinterface of IPFire the external hard drive will be shown under “services→ExtraHD” as dev/sd.. . There you can choose a convenient location in this example it is /mnt/usb .

Setup Dirvish

You can find also a pretty good how-to for Dervish in here . For simplicity in here the only absolutely necessary steps will be explained.

The principle of Dirvish is to define so-called “Vault´s” (safe) in which the backup is stored. A “Vault” is a directory in the backup-media. There are a parent-config file and per “Vault” there is also a special one. For the first we will edit the parent-config file called master.conf .

cd /var/ipfire/dirvish/etc/
touch master.conf

The file needs to have the following content:

File: /etc/master.conf
  ### /var/ipfire/dirvish/etc/dirvish/master.conf
  #First configure the backup-media
  bank:
          /mnt/usb
  # xdev: 1/0 - follows/follows not the symbolical links
  xdev:   1
  index:  gzip
  image-default:  %Y%m%d-%H:%M
  exclude:
          lost+found/
          core
          .nfs*
          *.tmp
          *.TMP
          .recycle/
  # Definition of the secured vaults (safes), the names do not have to
  # be identical to the Actual names of the directories!
  #
  Runall:
          vault1
          vault2
  #                
  # Do not forget:
  # to initialize a new vault (safe) with the command 'dirvish --vault safename --init'
  # how long are repealed by default all backups?
  expire-default: +30 days
  # differs from the standard (see above) …
  expire-rule:
  # Minute
  #       Stunde
  #               Day of month
  #                       Month
  #                               Day of week (Sonntag ist Nr. 1!)
  #                                       Zeit bevor Loeschung
  # Min   Hr      Dom     Mon     Dow     STRFTime_FMT
    *      *       1       *       *       +3 months

“Expire-default” defines for every backup a durability of 30 days. Deviating therefrom with “expire-rule” the durability will be set from the first day of the month to 3 month´s. Under “Runall” every “Vaults” which are exist, will be registered. Let´s define these “Vaults”:

cd /mnt/usb
mkdir vault1

In every “Vault” it have to be a directory called “dervish”, in which Dirvish writes his log-files. In addition, the configuration for this vault file comes into that directory.

cd /mnt/usb/vault1
mkdir dirvish
cd dirvish
touch default.conf

The default.conf needs to have the following content:

File: /mnt/usb/vault1/dirvish/default.conf
  ## /mnt/usb/vault1/dirvish/default.conf
  # the config file for Dirvish for the safe (vault) vault1
  client:         server #here are the hostname be placed
  tree:           /var/ipfire/samba/user1
  exclude:        *.bak
                  *~
                  .*
                  *.tmp
                  lost+found/
  #		here you can define other durabilities for this vault, which are different
  #       from /var/ipfire/dirvish/etc/dirvish/master.conf, 
  # expire-default:       +2 days
  # expire-rule:
  # Min   Hr      Dom     Mon     Dow     STRFTIME_FMT

The first line tells Dirvish, to secure in the specified path of the second row on his own computer. The second row needs to be adapted respectively. If other paths should also be saved, you need to create a new vault.

Initialise Dirvish

Dirvish looks before each backup if there is an old backup, which should be deleted according to the “expire-rule”. Because on the first run there are no backups, the initial backup will be created first:

/var/ipfire/dirvish/bin/dirvish --vault vault1 --init

This takes a longer while, depending on the amount of data which should be saved. After a successful working progress the secured data are located under

/mnt/usb/vault1/...

in a directory with a name which indicates the start time of the backup. The command is required to repeat for every “Vault”.

Automate Dirvish

To start the backup automatically, a cronjob needs to set up. First, you need to generate a script which contains the necessary commands:

cd /etc/fcron.daily
touch dirvish.cron
chmod 700 dirvish.cron

The content of the file dirvish.cron looks like the following lines:

File: /etc/fcron.daily/dirvish.cron
  /var/ipfire/dirvish/bin/dirvish-expire
  /var/ipfire/dirvish/bin/dirvish-runall

If the script dirvish-runall produces no backup or during a manual call the responds with done If the script dirvish-runall produces no backup or during a manual call a back respond with done comes too quickly, it may help to set a symbolic link:

ln -s /var/ipfire/dirvish/bin/dirvish /usr/local/bin/dirvish

Thanks to Enrico, who put this solution in the forum: http://forum.ipfire.org/index.php/topic,5332.0.html

Observe Dirvish

Now that everything works and is setup, you can make the next step.

A simple way of monitoring whether dirvish has done his job properly, offers this dirvish-status.sh|script , which reads out the status of Dirvish and sent it by e-mail. It will be copy to /var/ipfire/dirvish/bin and needs to be made executable with a:

chmod 700 /var/ipfire/dirvish/bin/dirvish-status.sh

Then, we will edit it yet as follows:

In the first row, the path to the “Vault” need to be set (in this example: /mnt/usb ). The entry of the mail recipient should be out-commented.

The last lines should be adapted as it follows:

File:/var/ipfire/dirvish/bin/dirvish-status.sh
  ...
  echo "Disk usage:"
  /bin/df -h
  #echo
  #cat $tmpfile
  #) | mail -s "${WARNSUB}Dirvish status (${YESTERDAY}@`hostname`)" $MAILTO  
  )
  sendEmail -f sender@web.de -u "${WARNSUB}Dirvish status (${YESTERDAY}@`hostname`)" -o message-
  file=$tmpfile -t recipient@web.de -s smtp.web.de -xu sender-username -xp sender-password
  rm  $tmpfile

The example is based on an e-mail account of the sender at web.de and needs naturally to be completed with the individual Data. Sender and recipient can be the same.

The script assumes that dirvish not noted the time in the directory name. Therefore, the master.conf needs to be re-edited:

File:/var/ipfire/dirvish/etc/master.conf
  #/var/ipfire/dirvish/etc/master.conf
  ...
  #image-default:  %Y%m%d-%H:%M
  image-default:  %Y%m%d
  ...

But now only one backup on the day is possible, but the trial period is over so ;-) .

So that it all works, the application “sendEmail” needs to be installed. Download it simply from here and copy it with WinSCP to the directory /root onto the IPFire and type in the following commands to untar, copy and set the right file permissions to it:

cd /root
tar -xzvf sendEmail-v1.55.tar.gz
cp sendEMail-v1.55/sendEmail /bin/sendEmail
chmod 700 /bin/sendEmail

Now add the script into the cronjob:

File:/etc/fcron.daily/dirvish.cron
  #/etc/fcron.daily/dirvish.cron
  /var/ipfire/dirvish/bin/dirvish-expire
  /var/ipfire/dirvish/bin/dirvish-runall
  /var/ipfire/dirvish/bin/dirvish-status.sh

Backup with a remote computer

Dirvish provides the ability to access remote machines through SSH and create backups over the network or the internet. In order to get additional protection against hardware failures and still provide protection against theft or fire. Therefor the following lines needs to be adapted for every default.conf :

File:/mnt/usb/vault1/dirvish/default.conf
  ##default.conf from vault1
  client:         user@domain.remote.machine
  rsh:            ssh -i /path/to/private/key
  tree:           /path/to/backup/on/remote/machine
  ...

It is naturally required, that

  • the user have SSH-access and also read-access to the remote path
  • the remote machine have rsync installed
  • the public key was read-in on the remote machine under /home/user/.ssh/authorized_keys .

A precise description how to generate and integrate the key´s, can be found in the above called Dirvish-how-to.

If you use as an external USB-hard disk as backup media you should according to the amount of data generate the initial backup directly on the remote computer (therefor of course Dirvish have to be installed there, but now we know how to do this :-) and then transmit only the changes each day via the Internet.

addons/dirvish/start.txt · Last modified: 2018/09/19 02:53 by Jon