I was asked by an academic client to install Boinc Server on FreeBSD. BOINC is a software platform for volunteer computing and desktop Grid computing. FreeBSD has a boinc-client port, but not a port for the server, so it must be compiled manually. Following the instructions located on the Boinc wiki for a FreeBSD installation would result in some errors.

Here is how to get started in FreeBSD:

Follow the normal prerequisite / prep instructions found here: http://boinc.berkeley.edu/trac/wiki/ServerIntro. All the prerequisites can be found in FreeBSD ports, install them including Python and /usr/ports/databases/py-MySQLdb . I installed this in a Jail, stripped down Apache to the bare minimum and set this up for my client so that Apache would automatically include his httpd.project.conf files after running make_project.

The following are some of the things I did differently compared to the Linux install:

svn co http://boinc.berkeley.edu/svn/branches/server_stable boinc
cd boinc
./_autosetup

Without editing the configure script you’ll get the following error while making:

/usr/bin/ld: cannot find -ldl

To fix this, edit the ./configure script:

vi configure

and remove all occurances of “-ldl “:

:%s/-ldl //g

-ldl is not required in FreeBSD, it is available in libc.

Run:

./configure --disable-client --disable-manager --with-boinc-platform=x86_64-pc-freebsd

You may need to replace x86_64-pc-freebsd depending on your hardware. Check the list here: http://boinc.berkeley.edu/trac/wiki/BoincPlatforms

If you ran `make` instead of GNU make`gmake` you’d get these errors:

“Makefile”, line 19: Missing dependency operator
“Makefile”, line 23: Need an operator
“Makefile”, line 27: Need an operator
make: fatal errors encountered — cannot continue

Run:

gmake

That’s it! Follow the rest of the instructions on the Boinc wiki.

The solution to the age old problem of locking SFTP users into their home directory is setting up a chroot environment. This normally requires that you copy the necessary binaries and libraries so that your jailed users can make use of the allowed tools for file transfer.

As of OpenSSH 4.9p1, things have gotten a bit easier. OpenSSH has two features that make the task of locking users into their home directories a piece of cake. They are:

1. A built in SFTP subsystem.
   With a built in SFTP subsystem, you no longer need binaries and the required libraries to provide the services necessary in a chroot environment. OpenSSH provides an internal SFTP subsystem.
2. The Match keyword.
   This allows you to target specific users or groups in the sshd_config file and specify settings particular to them, like a chroot option and ForceCommand internal-sftp.

Getting it working is simple.

1. Add a group called sftponly and add the users who you’d like to lock into their home directories to that group.
2. Edit your sshd_config file (/etc/ssh/sshd_config if you’re on FreeBSD) and add the following to the bottom:

   Match Group sftponly X11Forwarding no AllowTcpForwarding no ForceCommand internal-sftp ChrootDirectory %h

3. That’s it, HUP sshd (/etc/rc.d/sshd restart if you’re running FreeBSD) and test it out.

If you’re running a local mail client to send mail from your web application, you’ve probably already spent hours upon hours wondering why mail always ends up in the recipients spam/junk folder. No matter what combination of custom headers you pass to PHP’s mail function- yahoo, hotmail and maybe even gmail still marks it as spam. Have no fear. The solution is quite simple.

There are two options that are very similar. They are DKIM and DomainKeys. When I was first experimenting, I had installed DomainKeys only to realize it worked with gmail and yahoo, but not hotmail. After some reading I found that DKIMProxy implemented both DKIM and DomainKeys, so I ditched DomainKeys and went with DKIMProxy. This covers all the major email services and ISPs. Switching was easy, because they work exactly the same with only a few minor differences in syntax.

Read more about how those protocols work here:
http://en.wikipedia.org/wiki/DomainKeys_Identified_Mail

Essentially, DKIM is a solution for email providers to verify that mail being sent from a particular domain is authorized. The way it works is that the domain owner inserts a TXT record into DNS for that domain. This record contains a public key. Each e-mail that gets sent must be signed using the private key, the signature gets placed into the header of the email. Email providers can then verify the authenticity of each email that is sent using classic pgp-like signature techniques. DKIMProxy is a service that runs on your server, works with your already existing mail server (sendmail, postfix, qmail, etc..) and can automatically sign your mail pieces.

Below I highlight the steps for setting up PostFix + DKIMProxy on FreeBSD to send mail from the local machine ONLY. This will be setup on an application server whose sole purpose is sending mail through the local relay, not receiving or relaying email from other machines. If you’ve already setup PostFix for wider purposes, you’re probably already pretty comfortable with it; just use the same techniques to specify the content_filter for any other listening service and skip my PostFix write-up. There will be some differences if you’re running this instance of FreeBSD in a jail (as I do), however I will document those differences and the security precautions you’ll need to take as well.

If you’re not using FreeBSD, you’ll have to substitute my use of ports with your distribution’s package management command. The location of your config files may also vary, as well as the start-up scripts; otherwise, it’s all the same.

Installing / Configuring Postfix:

cd /usr/ports/mail/postfix
make & make install

Choose yes to modify mailer.conf

Stop sendmail if it’s running:

cd /etc/mail
make stop

Edit your rc.conf file to turn off sendmail and enable postfix. Add the following:

sendmail_enable="NONE"
postfix_enable="YES"

If we only want to send local mail we can comment out the following line in /usr/local/etc/postfix/master.cf :

#smtp      inet  n       -       n       -       -       smtpd

This is beneficial if we are running postfix in a jail that doesn’t have access to a loopback device. If you were running in a jail, but wanted to run an SMTP server as well, you could specify an IP address to listen on by prepending the line with the ip address as follows:

10.0.0.10:smtp      inet  n       -       n       -       -       smtpd

It’s fine to comment out though.

Edit your /usr/local/etc/main.cf file according to your needs. Since I am just using this to send mail from my domain, but am using Google Apps to receive mail I do the following in main.cf :

myhostname = mydomain.com
inet_interfaces = loopback-only
mydestination = localhost
local_recipient_maps=

The default main.cf file has those specific parameters documented for your reference (RTFM).

Now start PostFix :

/usr/local/etc/rc.d/postfix start

Check /var/log/maillog for errors. If there is an error about reading /etc/aliases.db, you probably just need to generate it. Run the `newaliases` command and it should solve your problems.

Test sending mail from the local machine to make sure PostFix works. You can either use a PHP script or run sendmail manually. If it doesn’t show in your inbox, check the spam folder and /var/log/maillog .

Installing DKIMProxy:
You need to install the following Perl modules… CPAN doesn’t work well from a jail console, and if you’re like me and don’t want to install SSH or Tmux to fix your tty issues in jails, the following commands will install the needed Perl modules from the command line and works no matter what distribution you use:

perl -MCPAN -e'CPAN::Shell->install("Crypt::OpenSSL::RSA")'
perl -MCPAN -e'CPAN::Shell->install("Digest::SHA")'
perl -MCPAN -e'CPAN::Shell->install("Digest::SHA1")'
perl -MCPAN -e'CPAN::Shell->install("Mail::Address")'
perl -MCPAN -e'CPAN::Shell->install("MIME::Base64")'
perl -MCPAN -e'CPAN::Shell->install("Net::DNS")'
perl -MCPAN -e'CPAN::Shell->install("Net::Server")'
perl -MCPAN -e'CPAN::Shell->install("Test::More")'
perl -MCPAN -e'CPAN::Shell->install("Error")'
perl -MCPAN -e'CPAN::Shell->install("Text::Wrap")'
perl -MCPAN -e'CPAN::Shell->install("Mail::Address")'
perl -MCPAN -e'CPAN::Shell->install("Mail::DomainKeys")'
perl -MCPAN -e'CPAN::Shell->install("Mail::DKIM")'

cd /usr/ports/mail/dkimproxy
make && make install

Add the following to your rc.conf :

dkimproxy_out_enable="YES"

We won’t configure “in” because we are only interested in sending mail out right now.

Generate some keys:

cd /usr/local/etc/
openssl genrsa -out privatedkim.key 1024
openssl rsa -in privatedkim.key -pubout -out publicdkim.key

My /usr/local/etc/dkimproxy_out.conf file looks like this:

# specify what address/port DKIMproxy should listen on
listen    10.0.0.10:10027
 
# specify what address/port DKIMproxy forwards mail to
relay     10.0.0.10:10028
 
# specify what domains DKIMproxy can sign for (comma-separated, no spaces)
domain    mydomain.com
 
# specify what signatures to add
signature dkim(c=relaxed)
signature domainkeys(c=nofws)
 
# specify location of the private key
keyfile   /usr/local/etc/privatedkim.key
 
# specify the selector (i.e. the name of the key record put in DNS)
selector  selector1

You can just copy /usr/local/etc/dkimproxy_out.conf.example to /usr/local/etc/dkimproxy_out.conf and make the appropriate changes. The selector parameter is what we’ll specify later in DNS.

Quick Explanation
dkimproxy is going to listen on the ip address / port (10027) we’ve specified for mail and it’s going to proxy smtp traffic to the ip address / port (10028) we’ve specified, with the appropriate headers attached.

Security concern: If you’re using a jail and/or specified a public facing IP address to listen on, you’ve just opened up relaying for the world even if relaying is turned off in postfix. The reason is because you’ve opened a proxy that will listen publicly and forward internally to postfix. SOOO… If you’ve specified a public facing IP address, either change it to an internal-only or firewall it off.

Back to PostFix Configuration:

Now we need to configure postfix to use the DKIM proxy as a content_filter (sending mail out to it on port 10027) and then receiving it back on 10028.

We’ll be editing /usr/local/etc/postfix/master.cf and since we are only concerned with sending local mail, you can edit the pickup line to reflect the following:

pickup    fifo  n       -       n       60      1       pickup
        -o      content_filter=dksign:[10.0.0.10]:10027

and add the following somewhere below:

dksign  unix    -       -       n       -       10      smtp
    -o smtp_send_xforward_command=yes
    -o smtp_discard_ehlo_keywords=8bitmime
 
10.0.0.10:10028 inet  n  -      n       -       10      smtpd
    -o content_filter=
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
    -o smtpd_helo_restrictions=
    -o smtpd_client_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o mynetworks=10.0.0.10
    -o smtpd_authorized_xforward_hosts=10.0.0.10,mydomain.com,localhost

Now restart postfix and start dkimproxy:

/usr/local/etc/rc.d/postfix reload
/usr/local/etc/rc.d/dkimproxy_out start

Check netstat -na to make sure you’ve got services listening on both 10027 and 10028 . If not, check /var/log/maillog for errors.

Your DNS Records:

You’ll need to add three DNS records:

@       3600    IN      TXT     "v=spf1 a mx ~all"
_domainkey      3600    IN      TXT     "t=y; o=~;"
selector1._domainkey    3600    IN      TXT     "g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDhrfYr2KLFiU7Zo6H06LlhFBEpif/Tb7oBJvKdEIm1uED9FqJump/q6RSt3Yw1iuM3iBaQcPohGbdoGiuaJGOUWMOblsSXkAOWxl4lbI5UQ6zCTBpVdLVDVWJ0E3UW1YJs1crSBdmG9G3WghrvIRkHzxfDMqndIV5gliYt+nmqXQIDAQAB;"

Once your tests are complete you can remove t=y; More info

Where you see p=MIGfMA…, you need to concatenate the publicdkim.key file into one line without the header/footer. So for example my publicdkim.key file originally looks like this:

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDhrfYr2KLFiU7Zo6H06LlhFBEp
if/Tb7oBJvKdEIm1uED9FqJump/q6RSt3Yw1iuM3iBaQcPohGbdoGiuaJGOUWMOb
lsSXkAOWxl4lbI5UQ6zCTBpVdLVDVWJ0E3UW1YJs1crSBdmG9G3WghrvIRkHzxfD
MqndIV5gliYt+nmqXQIDAQAB
-----END PUBLIC KEY-----

You can see how my p=… is related.

If you are using a GUI DNS editor tool, you’ll have to figure out how to place these into DNS using that tool.

Now to test it out:

Assuming DNS has propagated, you can try sending a couple e-mails. tail -f /var/log/maillog while you do and you should be able to see the proxy in action. If you send mail to hotmail or google, you can view the message source and see the DKIM and DomainKeys header information as well.

You can also use the following tools which have been extraordinarily helpful to me when setting this up:

• Send an e-mail from your server to: check-auth@verifier.port25.com

It will respond with some helpful diagnostic information, but make sure you can receive an e-mail response to the reply-to field.

Submit your SPF record to evil Microsoft
If everything passes, you should submit your spf record to Microsoft. They are sort of like the really strong dumb kid on the block who can steal your candy, but can’t perform a DNS lookup on their own. Without submitting to Microsoft your mail will end up in the hotmail / live.com SPAM folder with a senderid temperror. You can sign up for a hotmail account and view full message source to see where it fails, then burn your monitor, keyboard and clothes for having signed up for a Microsoft product.

DomainKeys DNS Testing Tools:

• A DomainKey Policy Record Tester.
• A DomainKey Selector Record Tester.

That should get you on your way. I’ve pretty much covered every hurdle I’ve come across when setting this up. You’ll find plenty others who have similar, but slightly different setups by doing a cursory Google search.

This isn’t going to be an AWK how-to, but an AWK-‘why’. If you want a quick AWK tutorial check out Grymoires’ site.

System Administrators, DevOps or whatever you want to call them these days often need to parse large amounts of data in log files in order to extract relevant data. In fact, this is an often asked interview quiz question for many ‘nix sys admin jobs.

“Here’s a log file and a ‘nix shell. Write a script that tells me x, y and z.”

Often, smart, sys admins opt for PERL, and there is nothing wrong with that. However, did you know AWK was designed specifically for generating reports of this kind? The principles and techniques will be the same as in PERL, but AWK gives you a neat framework for generating them.

Here is a sample AWK script that parses an Apache log file, and spits out a list of IP addresses that have generated 1000 or more hits and how many hits they’ve generated sorted in descending order.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

#!/usr/bin/awk -f
BEGIN {
    OFS="tt"                 #Set the output field separator
    print "IP Address", "Hits"    
}
/^[1-9]/ {                     #Parse lines that start with a number (IPv4).
    iphash[$1]++               #Increment IP in our associative array.
}
END {
    sort="sort -k2 -nr"        #The sort command we'll use. Parameters may
                               #vary depending on your flavor of 'nix.
                               #You may need to replace -k2 with +2.
 
    for (i in iphash) {
        if (iphash[i] >= 1000) print i, iphash[i] | sort 
                               #AWK's output buffer benefits us here
    }
    close (sort)               #close the sort pipe to flush output buffer
    print "TOTAL", NR          #print total number of records (hits).
}

Save it as ‘something.awk’, chmod it to executable and run `./something.awk /var/log/your.log`

In my prior post I made the case against a rotating password policy and suggested two-factor authentication as a password policy that worked. Two-factor authentication requires both a password that is memorized and an item you have to verify that you are who you say you are.

Two-factor authentication doesn’t have to be expensive. In fact, Google has developed a solution that makes use of the smart phone to generate time-based, one-time verification codes that are unique to each individual user and the machine they are logging in to. It’s called Google Authenticator.

Installing it on FreeBSD is simple, but it probably won’t behave as you intended out of the box. I’ll walk you through installing it and tailoring it to your needs.

Step 1: Install libqrencode. This is optional, but you’ll be happy you did when you see the QR code in your SSH terminal to be scanned by your smartphone. This doesn’t have to be installed first, in fact, you can install it after Google Authenticator is installed as well.

cd /usr/ports/graphics/libqrencode
make && make install

Step 2: Install Google Authenticator.

cd /usr/ports/security/pam_google_authenticator
make && make install

Step 3: Download the Google Authenticator app to your smart phone. Sorry, I can’t help you with this step. Search for it in the market place, it’s free.

Step 4: As the user you’d like to generate a key for, run the program (on your FreeBSD machine):

google-authenticator

Choose ‘y’ for yes to update your .google_authenticator file and scan the QR code into your smartphone’s Google Authenticator app.
Step 5: Configure your OpenPam config file. Lets set it up to work for SSH… Using your favorite editor add the following line to the bottom of your /etc/pam.d/sshd file:

auth     optional     /usr/local/lib/pam_google_authenticator.so

You’re “done!” If you’ve setup Google Authenticator for a particular account it will prompt you for the verification code that your smartphone now generates.

Problems:

You’ve noticed that we used “optional” in the pam config file. This means the use of it is, well, optional. Not only can users login who have not setup Google Authenticator yet, but users who have set it up can get away without using it by leaving the verification code blank after being prompted for it.

If you want to require Google Authenticator for your users you can change that to “required.” However, you then face another problem: users who havn’t set it up yet will not be able to login.

The desired effect should be that users who have set it up are required to use it, but users who have not set it up can get away without it (at least during early deployment).

If you used LinuxPam, this would be easy. LinuxPam allows for conditional statements within the pam config file that upon matching, the specified number of following modules could be skipped. OpenPam, used by FreeBSD, to my knowledge doesn’t have that feature. So, to get the intended functionality we can apply a simple patch to pam_google_authenticator.c.

Warning: I suspect Google didn’t do it this way because with this patch, by setting the control-flag in the pam configuration file to “sufficient” you will essentially open a gapping security hole that allows anyone to log in without a password or a verification code to an account where google-authenticator is not setup. You have been warned. With this patch, only use the “required” control-flag in the pam configuration.

Modify pam_google_authenticator.c by adding the following code just above the ‘// Clean up’ comment:

  if ((rc != PAM_SUCCESS) &&
      (secret_filename != NULL) &&
      (access(secret_filename, F_OK) == -1) &&
      (errno == ENOENT)) {
 
     log_message(LOG_ERR, pamh, "No config file found, skipping authentication");
     rc = PAM_SUCCESS;
  }

It’s pretty straight forward. If the secret_filename (by default .google_authenticator) doesn’t exist in the users home directory, then return the PAM_SUCCESS return code, thus satisfying the two-factor authentication requirement for users who have not set it up.

Enjoy. If you have trouble re-compiling with the changes, post a comment.

Policies that require users to change their password every couple of months do nothing to increase security. I’ll try to make the case that a rotating password policy does nothing to protect against these attacks, but instead encourages users to write down their passwords.

Lets look at a couple of ways in which passwords are often compromised.

1. A hardware or software based key-logger / virus / spyware is harvesting passwords.
   In this case, the employee or user types a password on a compromised machine, either at home, at a hotel, Internet cafe or on a friend/family members computer.
2. Password is obtained by someone sniffing network traffic.
   This occurs when only making use of a plaintext protocol.
   
3. The server’s password list is compromised.
   You had an SQL injection vulnerability in your web app or your password file was compromised and someone managed to get the hashed list of passwords. If some of the passwords were simple and a basic hash function without salt was used, then some of the passwords could be obtained by a hash-table dictionary or brute force. 
4. Password was so simple that someone guessed it or that it was brute forced.
   This is rare, but most people who don’t understand security think this is hacking thanks to improper portrayal of hackers in movies.

There are plenty of other ways in which a password can be compromised, but these will suffice for now.

The first and only question that needs to be asked in order to debunk the rotating password policy is this:

Once someone’s passwords is compromised, how does changing it six months later stop the attacker from using it today, tomorrow or next week? Even if the attacker is selling password lists on the black market, every criminal knows a list older than a couple days is useless.

A second point demonstrates the ridiculous nature of forcing people to change their passwords:

When you force anyone to change their password every couple months and demand that it be some complicated combination of alpha-numeric characters, you’re forcing them to write it down, and most uneducated users will leave their written passwords in a cubicle.

A better solution

Two factor authentication is the way to go if you don’t mind inconveniencing people and want to enforce a serious password policy. Two factor authentication requires two things: either a memorized password / private key and a physical item that either generates a time-sensitive token unique to the user or verifies the user via SMS. Usually this is accomplished with a little device that can go on your keychain, but now smart phone apps are capable of providing the same thing. I installed Google Authenticator on one of my servers.

With two factor authentication, even if someone obtains my password, they won’t be able to login without the addition of the physical device I carry around in my pocket. Can two factor authentication be broken? Yes, someone can use one of the methods above to steal my password, then hit me over the head with a hammer and take my device. Someone could also obtain the private key used to generate my one-time tokens, or break the algorithm or even obtain physical access to the server!

Conclusion

Security isn’t about making it impossible to break in. At the end of the day we can concoct a zillion scenarios where even the pentagon could be overtaken. Security is about plausibility and probability. The plausible and probable methods of compromising a machine are not protected by a rotating password policy. They are protected by two factor authentication.

An added note: real security comes from educating users.

There may be situations when you want to throttle the amount of requests a specific user or IP address can make to your website. This works great if you are using Apache as a reverse proxy for security, availability or performance reasons.  Back in the Apache 1.x days there was a module called mod_dosevasive that did just the trick.  Unfortunately it did not work as well in Apache 2.x.

A much better solution is to use a module called mod_security.  mod_security allows you to write sophisticated, stateful rules and take action based on particular conditions. Using mod_security you can do a lot more than DOS evasive maneuvers. You can filter for XSS, SQL Injection, Mail Header Injection and lots more. It uses Perl regular expressions for the win.

In preliminary tests the filter does not block search engine spiders (at least not the ones that count).

If you are using FreeBSD ports, you’ll also need to change the default:

SecRuleEngine DetectionOnly

to:

SecRuleEngine On

and add:

SecDataDir /tmp

in /usr/local/etc/apache22/Includes/mod_security2/modsecurity_crs_10_config.conf

The following can be used in a VirtualHost directive, or included directly in httpd.conf. It can be easily tailored to suit your needs:

# Ignoring media files, count requests made in past 10 seconds.
SecRule REQUEST_BASENAME "!(css|doc|flv|gif|ico|jpg|js|png|swf|pdf)$" 
            "phase:1,nolog,pass,initcol:ip=%{REMOTE_ADDR},setvar:ip.requests=+1"
 
# This is where every other example online goes wrong.  We want the var to expire and leave it
# alone. If we combine this with the increments in the rule above, the timer never expires unless
# there are absolutely no requests for 10 seconds. 
SecRule ip:requests "@le 2" "phase:1,nolog,expirevar:ip.requests=10"
 
# if there were more than 20 requests in 10 seconds for this IP
# set var block to 1 (expires in 30 seconds) and increase var blocks by one (expires in 5 minutes)
SecRule ip:requests "@ge 20" "phase:1,pass,nolog,setvar:ip.block=1,expirevar:ip.block=30,setvar:ip.blocks=+1,setvar:ip.requests=0,expirevar:ip.blocks=300"
 
# If user was blocked more than 5 times (var blocks>5), log and return http 403
SecRule ip:blocks "@ge 5" "phase:1,deny,log,logdata:'req/sec: %{ip.requests}, blocks: %{ip.blocks}',status:403"
 
# if user is blocked (var block=1), log and return http 403
SecRule ip:block "@eq 1" "phase:1,deny,log,logdata:'req/sec: %{ip.requests}, blocks: %{ip.blocks}',status:403"
 
# 403 is some static page or message
ErrorDocument 403 "<html><body><h2>Too many requests.</h2></body></html>"

The above blocks users who send more than 20 requests in a 10 second period. They will be blocked for 30seconds unless this has be a frequent occurrence. If they were blocked more than five times within five minutes they will be blocked for five minutes.

In a previous post, “DNS and Offsite Failover“, I documented the implementation of an automated, offsite, fail-over using DNS. For static websites there is nothing left to do except perhaps use `rsync` to keep the files up to date.

Unfortunately there is a lot more to think about for dynamic websites and applications that make use of a database.

1. The databases need to be sync’d realtime.
2. Keeping track of state is important, each server needs to know when it and its counter-parts are up, down, active or standby.
3. In the event of a failover, they need to coordinate cleanup, merge changes and start replicating again.

With the level of complexity involved small businesses typically accept defeat and embrace the potential downtime. Some even plan for a manual failover in the event of a disaster.

There are two automated solutions that can be weighed in the balances.

The Reduced Functionality Offsite Failover
The easiest solution by far is to implement what I call the “Read-Only” or reduced functionality offsite failover. In this configuration we’ll keep another database sync’d offsite. In the event of downtime the offsite backup takes over but in a reduced functionality mode. If the site supports user login or performs transactions, they can be disabled temporarily. Interactive sites effectively become “read-only” for the time being.

This poses a problem for commercial sites whose business is to perform transactions. Potential sales are lost, but at least new and old customers can still learn about products and get information. Nobody receives an ugly “Server not found” browser error; in-fact a custom message can be crafted explaining that full functionality will return shortly. This would be great for university sites, magazines, journals or even popular blogs where the primary purpose of the site is to get information.

This makes the life of the system administrator easy because the primary server doesn’t need to keep track of state, never needs to merge changes later on and can continue as it was when and if the network connection is restored.

In fact, if it’s acceptable for the offsite server to be a day behind the primary, implementation can be as simple as our prior DNS solution and a nightly cronjob that looks something like this:

1
2
3
4
5
6
7
8
9
10
11

#!/bin/sh
ssh USERNAME@primary.location.edu 'mysqldump --single-transaction -uUSERNAME -pPASSWORD -h DBSERVER DATABASE' | /usr/local/bin/mysql -uLOCALUSERNAME -pLOCALPASSWORD LOCALDATABASE
MYSQL='/usr/local/bin/mysql -uLOCALUSERNAME -pLOCALPASSWORD'
$MYSQL <<EOF
use LOCALDATABASE;
delete from shared_sessions;
delete from main_cache_page;
delete from groups_cache_page;
insert into main_access (mask, type, status) values ('%', 'user', 0);
insert into groups_access (mask, type, status) values ('%', 'user', 0);
EOF

The above is an example of a Drupal site being placed into “read-only” mode. The ENTIRE database is pulled from the primary location via SSH (shared key is used instead of a password). The session table and cache is cleared and Drupal’s access table is modified to block all but the admin user from logging in. If the database is large it’s best to avoid this method and stick to replication. Replication will also give you real-time updates. We’ll cover that in the “fully functional implementation” in Part II. You can borrow elements from both while developing your own custom solution.

You can even set server variables in Apache:

setenv readonly yes

And in the theme layer detect the server variable ‘readonly’ to know when to hide the login box or display your custom reduced functionality nofication so that two separate code bases do not need to be maintained. A Drupal example:

66
67
68

  if ($region == 'login_slide' && !$_SERVER['readonly']) {    // Don't display the login field if the server is in read-only mode.
    drupal_set_content('login_slide', pframe_login_slide());
  }

Fully Functional Offsite Failover
There are few sites that can afford to deny customer transactions. Reduced functionality mode is the easiest solution to implement but isn’t very practical for most business models. It’s harder to implement a fully functional offsite failover. As stated prior this requires both machines to keep track of state and resolve data conflicts when the primary changes from down to up. The logic is increasingly more complicated when you take into account the transition back from secondary to primary. The delays introduced by DNS caching and ISPs not respecting TTLs can potentially lead visitors to both primary and secondary locations at the same time and throw our databases out of sync. This method requires a lot more thought.

We’ll address the implementation of the fully functional offsite failover in Part II.