6. Configure Automatic Backup

  1. Start the virtual machine and log in:

    ssh -p 8015 hjc@dynamicshjc.case.edu
    
  2. Check for and install system updates on the virtual machine:

    sudo apt-get update
    sudo apt-get dist-upgrade
    sudo apt-get autoremove
    
  3. Download and install the backup script:

    sudo wget -O /usr/local/sbin/backup-wiki https://neurowiki-docs.readthedocs.io/en/latest/_downloads/backup-wiki
    

    Set the MySQL password inside the script:

    read -s -r -p "MySQL password: " DBPASS && sudo sed -i '/^SQLPASS=/s|=.*|='$DBPASS'|' /usr/local/sbin/backup-wiki; DBPASS= ; echo
    

    Protect the password:

    sudo chown root:www-data /usr/local/sbin/backup-wiki
    sudo chmod ug=rwx,o= /usr/local/sbin/backup-wiki
    

    If you are curious about the contents of the script, you can view it here:

    backup-wiki

    Direct link

    #!/bin/bash
    
    # Copyright (c) 2016, Kendrick Shaw, Jeffrey Gill, Hillel Chiel
    # All rights reserved.
    #
    # Redistribution and use in source and binary forms, with or without
    # modification, are permitted provided that the following conditions are met:
    #
    #  * Redistributions of source code must retain the above copyright notice,
    #    this list of conditions and the following disclaimer.
    #  * Redistributions in binary form must reproduce the above copyright notice,
    #    this list of conditions and the following disclaimer in the documentation
    #    and/or other materials provided with the distribution.
    #
    # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
    # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
    # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
    # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
    # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
    # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
    # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
    # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
    # POSSIBILITY OF SUCH DAMAGE.
    
    
    DEFAULTNAME=$(hostname)-backup-$(date +'%Y%m%dT%H%M%S%z') # ISO-8601 timestamp
    
    SQLUSER=root
    SQLPASS=<MySQL password>
    WIKIDB=wikidb
    DJANGODB=djangodb
    
    WWWDIR=/var/www
    WIKIDIR=$WWWDIR/mediawiki
    DJANGODIR=$WWWDIR/django
    
    
    function usage {
        echo "Usage:"
        echo "  backup-wiki backup  [-q] [backup_filename]"
        echo "  backup-wiki restore [-q]  backup_filename"
        echo
        echo "Options:"
        echo "  -q  Quiet mode, print only errors"
        exit
    }
    
    
    # read in options
    QUIET=false
    while getopts ":q" opt; do
        case $opt in
            q)
                # the flag -q was set
                QUIET=true
                ;;
            \?)
                # an unrecognized flag was set
                echo "Invalid option: -$OPTARG" >&2
                usage
                ;;
        esac
    done
    shift $((OPTIND-1))
    
    
    # read in backup_file argument
    if [ -z "$2" ]; then
        # second argument not provided -- use default file name
        BACKUPNAME=$DEFAULTNAME
    else
        BACKUPNAME=$(dirname $2)/$(basename $2 .tar.bz2)
    fi
    
    
    ##### BACKUP MODE #####
    
    if [ "backup" == "$1" ]; then
    
        if [ $QUIET = false ]; then
            echo "Starting backup to archive $BACKUPNAME.tar.bz2 ..."
        fi
    
        # create a temporary directory
        if [ $QUIET = false ]; then
            echo "Creating temporary directory ..."
        fi
        TEMPDIR=$(mktemp -d)
    
        # copy the MediaWiki, Django, and JSNeuroSim source code
        if [ $QUIET = false ]; then
            echo "Copying files ..."
        fi
        cp -R $WWWDIR $TEMPDIR
    
        # dump the wiki database in sql format
        if [ $QUIET = false ]; then
            echo "Exporting the wiki database in SQL format ..."
        fi
        mysqldump --user=$SQLUSER --password=$SQLPASS $WIKIDB --complete-insert --result-file $TEMPDIR/wikidb.sql 2>&1 | grep -v "\[Warning\] Using a password"
    
        # dump the django database in sql format
        if [ $QUIET = false ]; then
            echo "Exporting the Django database in SQL format ..."
        fi
        mysqldump --user=$SQLUSER --password=$SQLPASS $DJANGODB --complete-insert --result-file $TEMPDIR/djangodb.sql 2>&1 | grep -v "\[Warning\] Using a password"
    
        # dump the django database in json format (a more portable backup of the content)
        if [ $QUIET = false ]; then
            echo "Exporting the Django database in JSON format ..."
        fi
        python $DJANGODIR/manage.py dumpdata > $TEMPDIR/djangodb.json
    
        # compress everything into a single file
        if [ $QUIET = false ]; then
            echo "Compressing directory into an archive file ..."
        fi
        mkdir -p $(dirname $BACKUPNAME)
        tar -cjf $BACKUPNAME.tar.bz2 -C $TEMPDIR .
        chmod o= $BACKUPNAME.tar.bz2
    
        # delete the temporary directory
        if [ $QUIET = false ]; then
            echo "Deleting temporary directory ..."
        fi
        rm -rf $TEMPDIR
    
        if [ $QUIET = false ]; then
            echo "Done!"
            echo
            echo "NOTE: The backup you just created contains sensitive student information, quiz"
            echo "answers, and passwords. Keep this file in a safe place!"
            echo
        fi
    
    
    ##### RESTORE MODE #####
    
    elif [ "restore" == "$1" ]; then
    
        if [ -z "$2" ]; then
    
            echo "Missing argument: backup_filename" >&2
            usage
    
        elif [ -e "$BACKUPNAME.tar.bz2" ]; then
    
            if [ "$(whoami)" != "root" ]; then
    
                echo "Aborting: superuser privileges needed" >&2
                usage
    
            else
    
                if [ $QUIET = false ]; then
                    echo "Starting restoration from archive $BACKUPNAME.tar.bz2 ..."
                fi
    
                # extract the files
                if [ $QUIET = false ]; then
                    echo "Unpacking archive ..."
                fi
                TEMPDIR=$(mktemp -d)
                tar -xjf $BACKUPNAME.tar.bz2 -C $TEMPDIR
    
                # stop the server
                if [ $QUIET = false ]; then
                    echo "Stopping the web server ..."
                fi
                apache2ctl stop
    
                # copy the files, fixing permissions and owners
                if [ $QUIET = false ]; then
                    echo "Copying files ..."
                fi
                rm -rf $WWWDIR
                cp -R $TEMPDIR/$(basename $WWWDIR) $WWWDIR
                chown -R www-data:www-data $WWWDIR
                chmod -R ug+rw $WWWDIR
                find $WWWDIR -type d -exec chmod g+s {} \;
    
                # restore the wiki database
                if [ $QUIET = false ]; then
                    echo "Restoring the wiki database ..."
                fi
                echo "DROP DATABASE IF EXISTS $WIKIDB;" | mysql --user=$SQLUSER --password=$SQLPASS 2>&1 | grep -v "\[Warning\] Using a password"
                echo "CREATE DATABASE $WIKIDB;" | mysql --user=$SQLUSER --password=$SQLPASS 2>&1 | grep -v "\[Warning\] Using a password"
                mysql --user=$SQLUSER --password=$SQLPASS $WIKIDB < $TEMPDIR/wikidb.sql 2>&1 | grep -v "\[Warning\] Using a password"
    
                # restore the django database
                if [ $QUIET = false ]; then
                    echo "Restoring the Django database ..."
                fi
                echo "DROP DATABASE IF EXISTS $DJANGODB;" | mysql --user=$SQLUSER --password=$SQLPASS 2>&1 | grep -v "\[Warning\] Using a password"
                echo "CREATE DATABASE $DJANGODB;" | mysql --user=$SQLUSER --password=$SQLPASS 2>&1 | grep -v "\[Warning\] Using a password"
                mysql --user=$SQLUSER --password=$SQLPASS $DJANGODB < $TEMPDIR/djangodb.sql 2>&1 | grep -v "\[Warning\] Using a password"
    
                # restart the server and clean-up
                if [ $QUIET = false ]; then
                    echo "Restarting the web server ..."
                fi
                apache2ctl start
                rm -rf $TEMPDIR
    
                if [ $QUIET = false ]; then
                    echo "Done!"
                fi
    
            fi
    
        else
    
            echo "Bad argument: file $BACKUPNAME not found" >&2
            usage
    
        fi
    
    
    else
        usage
    fi
    
  4. Todo

    Make the backup script that lives on DynamicsHJC and the cron job list downloadable.

    In a different Terminal window, log into the VirtualBox Machines (vbox) account on DynamicsHJC:

    ssh vbox@dynamicshjc.case.edu
    

    Create a script that remotely executes the backup script and moves the archive to the backup drive. Create the file

    mkdir -p /Volumes/CHIELWIKI/backups/neurowiki/2016
    vim /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh
    

    and fill it with the following:

    #!/bin/bash
    
    #REMOTESSH="ssh hjc@neurowiki.case.edu"
    REMOTESSH="ssh -p 8015 hjc@dynamicshjc.case.edu"
    
    #REMOTESCP="scp -q hjc@neurowiki.case.edu"
    REMOTESCP="scp -q -P 8015 hjc@dynamicshjc.case.edu"
    
    REMOTEFILE=neurowiki-backup-`date +'%Y%m%dT%H%M%S%z'`.tar.bz2
    if [ -z "$1" ]; then
        LOCALFILE=/Volumes/CHIELWIKI/backups/neurowiki/2016/$REMOTEFILE
    else
        LOCALFILE=/Volumes/CHIELWIKI/backups/neurowiki/2016/$1
    fi
    
    $REMOTESSH backup-wiki -q backup $REMOTEFILE
    $REMOTESCP:$REMOTEFILE $LOCALFILE
    chmod go= $LOCALFILE
    $REMOTESSH rm $REMOTEFILE
    
  5. Make the script executable:

    chmod u+x /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh
    
  6. Copy the public SSH key from the vbox account on DynamicsHJC to the virtual machine to allow automatic authentication (you will be asked to accept the unrecognized fingerprint of the virtual machine — this is expected — and you will be asked for your password to the virtual machine twice):

    ssh-keygen -R [dynamicshjc.case.edu]:8015
    scp -P 8015 ~/.ssh/id_rsa.pub hjc@dynamicshjc.case.edu:
    ssh -p 8015 hjc@dynamicshjc.case.edu "mkdir -p .ssh && cat id_rsa.pub >> .ssh/authorized_keys && rm id_rsa.pub"
    

    Test whether automatic authentication is working by trying to log into the virtual machine from the vbox account on DynamicsHJC — you should NOT need to enter your password:

    ssh -p 8015 hjc@dynamicshjc.case.edu
    

    If this works without you needing to enter a password, automatic authentication is properly configured. You should logout to return to the vbox account on DynamicsHJC.

  7. In the vbox account on DynamicsHJC, create a backup schedule. Create the file

    vim /Volumes/CHIELWIKI/backups/neurowiki/2016/crontab
    

    and fill it with the following:

    ################################################################################
    #                                  NEUROWIKI                                   #
    ################################################################################
    # Make hourly backups on the odd-numbered hours (except 1 AM and 3 AM)
    0 5,9,13,17,21  * * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh hourA.tar.bz2
    0 7,11,15,19,23 * * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh hourB.tar.bz2
    # Make daily backups at 1 AM
    0 1 * * 0 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh sunday.tar.bz2
    0 1 * * 1 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh monday.tar.bz2
    0 1 * * 2 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh tuesday.tar.bz2
    0 1 * * 3 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh wednesday.tar.bz2
    0 1 * * 4 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh thursday.tar.bz2
    0 1 * * 5 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh friday.tar.bz2
    0 1 * * 6 /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh saturday.tar.bz2
    # Make weekly backups at 3 AM on the 1st, 8th, 15th, and 22nd of the month
    0 3 1  * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh `date +'\%Y-\%m'`.tar.bz2
    0 3 8  * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh 8th.tar.bz2
    0 3 15 * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh 15th.tar.bz2
    0 3 22 * * /Volumes/CHIELWIKI/backups/neurowiki/2016/backup-neurowiki.sh 22nd.tar.bz2
    
  8. Schedule the backups. In the vbox account on DynamicsHJC, dump the existing scheduled jobs to a temporary file:

    crontab -l > /tmp/crontab.old
    

    Edit the temporary file, and delete the backup jobs for last year’s NeuroWiki (leave the jobs for NeuroWikiDev for now). You can use Shift+v in Vim to enter Visual Line mode, the up and down arrow keys to select a block of lines, and d to delete them all at once.

    vim /tmp/crontab.old
    

    Now append the new jobs to the old and schedule them:

    cat {/tmp/crontab.old,/Volumes/CHIELWIKI/backups/neurowiki/2016/crontab} | crontab
    

    Verify that the backup jobs for this year’s NeuroWiki are properly scheduled, and that backup jobs for last year’s NeuroWiki are absent:

    crontab -l
    
  9. Log out of the vbox account:

    logout
    

    You can now return to your original Terminal window.

  10. Shut down the virtual machine:

    sudo shutdown -h now
    
  11. Using VirtualBox, take a snapshot of the current state of the virtual machine. Name it “Automatic backups configured”.