How To automate Linux backups to a remote pc

updated 2018-05-03

passwordless ssh:

automated Linux shell backups to another computer

ssh remote login without a password

Accessing a remote site through ssh without a password

How to do unattended backups to a remote linux computer

Quite often, you want a shell script to ssh into a remote server (your web site?) unattended and do a (regularly scheduled, cron job?) backup, for instance. SSH allows this in a very secure way using public key authentication.

The implication is that you have been logging across, with ssh, to other computers, you just have to enter a password every time, etc.

Set up SSH Key Gen

On Fedora 24,25,26,27,…
1. go to a terminal window, command line, login as root
and confirm that sshd is enabled and running (if not, activate it)
# systemctl status sshd
# systemctl start sshd
# systemctl enable sshd

2. Log in as root, and use an editor, emacs, gedit, or etc.
3. gedit /etc/ssh/sshd_config
(40-some lines down)
  RSAAuthentication yes
  PubkeyAuthentication yes

and restart ssh:
[root@computer ssh]# systemctl restart sshd

in Fedora 16+ check your .ssh folder
ls ~/.ssh
for the files id_rsa and id_rsa.pub (see below). If they exist, “ssh-keygen” is already done. Go to the next step, “Set up SSH Remote Login”

if not. is your ~/.ssh folder there? if not, create it and set its permissions to 755
then, run ssh-keygen:

[you1@computer1]$ ssh-keygen

After entering “ssh-keygen” you will be prompted for a different file name (which defeats the purpose of not doing it as a parameter, and you will be asked for a passphrase (twice). Simply hit Enter on all these prompts, the whole point, here, is to not need these.

You only need to do this once, per computer, (and per user … if more than one user on a computer is involved, like you and root, perhaps) and the 2 files created are id_rsa and id_rsa.pub :

  -rw------- 1 you1 you1 1675 Sep 22 20:04 id_rsa
  -rw-r--r-- 1 you1 you1 418 Sep 22 20:04 id_rsa.pub

Keep in mind that from this computer, you can execute commands on both machines.

Set up SSH Remote Login

Then, as user “you1” your userid on the computer(1) that will contain and run the unattended script(s). (the server if it will be running unattended backups, sending the backup-files off to another computer)

next:
[you1@computer1]$ ssh-copy-id you2@computer2
and or
[you1@computer1]$ ssh-copy-id -i ~/.ssh/id_rsa.pub root@computer2

You will be prompted for the password one last time.
Either of these commands will append the public key (id_rsa.pub) into the remote user’s ~/.ssh/authorized_keys file.

you should get a reaction like this:

Now try logging into the machine, with "ssh you2@computer2", and check in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.


Do try logging across and confirm that you no longer are prompted for a password.

If you don’t use the ssh-copy-id command and, instead, copy id_rsa.pub over with something like
  scp id_rsa.pub root@www.xxx.yyy.zzz:/root/
then you will have to log over there and append or move it into ~/.ssh/authorized_keys and confirm that the permission on the file is correct.
Also:
SSH can use either “RSA” (Rivest-Shamir-Adleman) or “DSA” (“Digital Signature Algorithm”) keys. Both of these were considered state-of-the-art algorithms when SSH was invented, but DSA has come to be seen as less secure in recent years. RSA is has been the only recommended choice for new keys, and has been and is the default.

send a command to the remote computer and see the output on your screen:
ssh you2@computer2 ls -al /home/you2/Desktop/*
or, to find backups under /home/you2/ (this command requires quotes around it)
ssh you2@computer2 'find /home/you2/ -type f -iname "*.bak" -exec ls -l {} \;'
after seeing the list, you may want to delete them:
ssh you2@computer2 'find /home/you2/ -type f -iname "*.bak" -exec rm -f {} \;'

Remote, passwordless, unattended, Scripting-Programming

Now that you have logged in without a password, you can execute the backup commands (or whatever) and see that you are not prompted for a password. You can put your commands in a text file (make the last 3 characters of the file name .sh to follow the convention and make it easy to recognize, make the first line in the file #!/etc/bash, make the file executable (chmod 755). On a command line you can execute it with ./yourfilename.sh or, from another shell script you can call it with sh /full/path/yourfilename.sh (even without making it executable with “chmod 755” etc)

You do not have to use full path names, but if you do, you can move the script to any folder and you can call it from anywhere — like a crontab entry — often important for unattended scripts. unattended scripts, backups etc, can be made to co-process politely with interactive users using the “nice” function: nice sh /full/path/yourfilename.sh

For me, I wanted it to execute with an event, not a time. I had a php program which was the last in a series of programs which wrote new records to 2 databases on an event. I added 1 line to the bottom of that “final” php program:
exec('echo > /full/path/backup-latest-event.flag'); (for example)
I could have used
exec('nice sh /full/path/backup-latest-happening-1.sh');
to, directly, call a shell program/script — provided your server, Apache, has authority to all the files, (no databases) that you want to backup – whatever.
Otherwise, do as I did and just update the timestamp on a file (“flag”, for example) then, have a script which does no more than check / compare that time stamp to a second one – one set by your backup programs/scripts. (by only checking the date-time of the file, and not opening/reading the file, you avoid a lot of (all) disk access) … for example: If the event flag is newer than the backup flag then, call your
“backup-latest-happening-1.sh”
(In the code below, I also put in the condition that a backup had not occurred within the past 30 minutes All the code below is to give you ideas – to change all of it to meet your situation – if you use any of it at all.)
For example:
I have a crontab entry calling my sentinel script every 5 minutes:

Unattended-Batch communication
with a Remote Server

If you write an unattended – batch (cron) job [an .sh shell program] you will want the program to check and confirm that the remote computer [server-client-…] is connected.
Ping the remote system. . . . -c1 = only ping once; -w3 = only wait 3 seconds [2 or 3 sec. is enough]
Example – another computer [192.168.0.100] on your lan:

#!/bin/bash
#
if ! ping -c1 -w3 192.168.0.100 > /dev/null
then
echo "sorry, not connected"
else
; all your commands involving the remote system.
fi


the -w3 is not needed if you get a response . . . you will most likely get it in mili-seconds.
If there is no response, ping will wait 10 sec. or more.

Backup to a Remote Server

Rsync

Rsync is a backup-copy command that can copy only the files that have changed.
from one folder to another on the same computer:
example: rsync -auv /var/www/html/* /home/[user]/backups/
this will copy everything under html/ into backups/ that was not there already or that is there but it has been changed, is newer, in html. (given that you have authority over all the files involved)

To a remote computer:
nice rsync -auv -e ssh /var/www/html/* root@123.234.222.60:/home/[user]/public_html/
First, you will be prompted for the root password on the remote server unless you have set up ssh-keygen for yourself to the remote root user. “nice” makes sure visitors to your site are not impacted with slow response times because of your file transfer running.

Note: Always use rsync over ssh
Since rsync does not provide any security while transferring data it is recommended that you use rsync over an ssh session. (as in the above example “-e ssh”) This allows a secure remote connection.

Note: there are many good tutorials on rsync and on Unix shell scripting, just do a google search on either one.

*/5 * * * * /full/path/backup.sh >> /full/path/backup.log


#!/bin/bash

if ! ping -c1 -w2 [your remote server]
then
echo " Our backup server may be down. Backup failed." `date +' %F %r %Z'` | mail -s " backup failed " you@youremail.com
else
rsync -avz --delete --exclude 'cache' -e ssh /home/[user]/public_html/* root@[your-backup-server]:/var/www/html/
ssh root@[your-backup-server] 'mysql -u [user] -p[password] your_database < /var/www/html/your_database.sql ' fi

redirecting to 2>> instead of >> will redirect only the error messages "stderr"

to Suppress stdout but Redirect stderr to file
{command} > /dev/null 2>> /tmp/all


to Suppress both stdout and stderr:
{command} > /dev/null 2>&1


to put the date on an output (log) file:
{command} >> /full/path/to/filename-$(date +%m-%d).log