Load Balancing Magento Enterprise with HAProxy – Part Three

Part 1
Part 2

the meat of the plan

Section 3: Deployment – the Meat of the Plan

We have five servers now and all servers are running CentOS 5, so the Linux commands that I list below are all based on a CentOS command line. You can find the Ubuntu equivalents pretty easily, I’m sure. More apt-get and sudos and less yum with Ubuntu, but similar commands overall.

Here’s what our deployment plan was:

  1. Before doing anything, if you are pointing your domain to your server using nameservers, you should change that to using solely an A record that points to the dedicated IP address of your current live server. So make that switch and wait however long it takes for the DNS to propagate around the web. DNS propagation takes anywhere from 30 minutes to 48 hours.
  2. Once you’ve confirmed that the A record method is in use instead of nameservers, connect to your current web server (the one holding the production site right now) via SSH as root. We’re eventually going to make it the new load balancer server.

    If you’re not familiar with SSH, then to connect to the server, open up Putty (download it if you don’t have it. It’s free) and enter root@111.111.111.11 (where 111.111.111.11 is the server’s IP Address and root is your server’s root user) into the Host Name field. Leave the port as 22. Next, click the Open button. It will prompt you for a password.

    Something that is not immediately obvious is that you can right-click to paste text from your clipboard into Putty. You won’t be able to see it since SSH (and thus Putty) hides all passwords from display on-screen, but it works, trust me. Copy the password off the assets section above into your computer’s clipboard and then right-click in the Putty window to paste it into the screen. Click Enter and you’ll be logged into the server. Most likely the /root/ folder of the server.

    I’m not going to get into command line too deep, but if you want to view the files in the current folder, type the command:

    ls

    And if you want to move around, use:

    cd

    Examples: Move up a folder:

    cd ..

    Open a folder called “etc”:
    cd etc

  3. At the time of this writing, HAProxy 1.5.3 is the latest stable version of HAProxy, so we’ll be using the installation detalis for that.

    a) Install Linux Development Tools via Yum:

    yum grouplist -v "development" | grep tools

    yum install @development

    b) Install OpenSSL Development Tools via Yum:
    yum install openssl-devel

    c) Download HAProxy 1.5 (go to http://www.haproxy.org/download/ to find the URL of the most recent build):
    wget http://www.haproxy.org/download/1.5/src/haproxy-1.5.3.tar.gz

    d) Extract HAProxy tar:
    tar xzf haproxy-1.5.3.tar.gz

    e) Go into untarred directory:
    cd haproxy-1.5.3.tar.gz

    f) Compile HAPRoxy 1.5:
    make TARGET=linux26 USE_OPENSSL=1 USE_ZLIB=1 ADDLIB=-lz

    g) After the compile is done, make sure the binary is linked to the openssl library by typing the following command:

    ldd haproxy | grep ssl

    You should see something like:

    libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007fb0485e5000)

    h) Install the binary:

    make PREFIX=/usr/local/haproxy install

  4. If you don’t have an SSL certificate at the front of your application, you can skip this step and steps 5 and 6. If you want to have a similar setup to ours then connect to Current Web Server’s Plesk Control Panel (or wherever you upload your SSL certificate to your site):

    a) Copy the CRT and Private Key to Notepad or download them to your computer if that is an option. They should end with .crt and .key, respectively. If you can’t download them, copy and paste each into a separate notepad file and save them as two files. One could be certificate.crt and the other private.key. If you have a CA certificate, put it directly after your main certificate in certificate.crt. In other words, where it looks like this:

    ------BEGIN CERTIFICATE---------
    ALL YOUR CERTIFICATE JUNK HERE
    ------END CERTIFICATE------
    ------BEGIN CERTIFICATE---------
    ALL YOUR CA CERTIFICATE JUNK HERE
    ------END CERTIFICATE------

    b) Confirm that they are both in PEM Format (they were for me). In other words: If they begin with —–BEGIN and you can read them in a text editor (they use base64, which is readable in ASCII, not binary format), they are in PEM format.

    If the file is in binary, for the server.crt, you would use the command:
    openssl x509 -inform DER -outform PEM -in server.crt -out server.crt.pem

    For server.key, use openssl rsa in place of openssl x509.

    The server.key is likely your private key, and the .crt file is the returned, signed, x509 certificate.

  5. Connect to the load balancer server via SFTP as root:

    a) Upload the .crt and .key where you want the .pem file to be. For example, you might want the SSL certificates to be stored at /etc/ssl/certs/ – and that is as good a place as any.

  6. Back in SSH for the load balancer server:a) Navigate to the location you put them at in 2a and combine them into a single file (certificate, then private key) called certificate-key.pem (or elsewhere, just make sure that the config file below has the right path). Combination command is:

    cat server.crt server.key > certificate-key.pem.

    You could, of course, run the cat command anywhere and just type out the full locations of the files in question. But as you know by now, I like to keep things as simple as possible.

  7. Back to SFTP as root:

    a) Navigate to /etc/ (or it might have installed to /etc/haproxy/) and download haproxy.cfg to your local computer. Or, if you are so inclined, you can edit it right there on the server using vi. I’m not so inclined.

  8. In your local development IDE (Dreamweaver or Notepad++ or whatever):

    a) Open haproxy.cfg and replace all with the following:

    global
    log 127.0.0.1 local0
    maxconn 4096
    daemon
    # drop privileges after port binding
    user nobody
    # store pid of process in the file
    pidfile /var/run/haproxy.pid
    # create this socket for stats
    stats socket /var/run/socket-haproxy

    defaults
    log global
    mode http
    option httplog
    option dontlognull
    retries 3
    option redispatch
    maxconn 32000
    # I think these next three should probably always be the same. Fiddle with them if you have complaints of 504's.
    # Time in ms to wait.
    contimeout 90000
    clitimeout 90000
    srvtimeout 90000

    #VERY IMPORTANT OPTION: Analyze each request individually, evaluate acls for each request. No tunnel mode.
    option http-server-close
    # redispatch the request in case primary server based on session stickyness is down
    option redispatch

    #5s is a little steep for timeoutconn. This is typically supposed to be 1-2 seconds and can even be set to ms
    timeout connect 5s

    #These next two should be the same.
    timeout server 90s
    timeout client 90s

    # timeout for keep alive
    timeout http-keep-alive 180s

    # maximum time to wait for client to send full request. Keep it around 5s for get DoS protection
    timeout http-request 10s

    # to have stats enabled, uncomment out the following four lines add your own password after the colon and your username before the colon.
    # YOU WANT TO HAVE STATS ENABLED!
    #stats enable
    #stats auth haproxy_admin:fFgG!12341234
    #stats refresh 10s
    #stats uri /stats
    #With the above, the URL for the Stats Interface would be https://siteurl.com/stats/

    #I think the https_frontend <name} can be any name. This one came straight from a tutorial.
    frontend https_frontend
    #If you're not doing HTTPS, then skip this next line
    bind *:443 ssl crt /etc/ssl/certificate-key.pem
    bind *:80
    mode http
    option httpclose
    option http-server-close
    option forwardfor
    redirect scheme https code 301 if !{ ssl_fc }
    default_backend web_server

    backend web_server
    #this next command may not be required in your system. It definitely is for Magento.
    reqadd X-Forwarded-Proto:\ https
    mode http
    #default option for balance is roundrobin, but if you have spiking problems, then leastconn is the way to go.
    balance leastconn
    cookie SERVERID insert indirect nocache

    # There are a lot of options available in this next section. The ones shown here are nearly the exact same ones used in our system.
    # Even though the servers are identical in spec, server 1 runs the worst of the three so we give it a lesser weight so it's not used as often.
    # If you don't have that problem, assign them all an equal weight of 1.
    # We've had good luck with a maxqueue of 25, but I'm always wondering whether it should be higher.
    server s1 333.333.333.33:80 check cookie s1 weight 1 maxqueue 25 rise 2 fall 1 maxconn 250
    server s2 444.444.444.44:80 check cookie s2 weight 2 maxqueue 25 rise 2 fall 1 maxconn 250
    server s3 555.555.555.55:80 check cookie s2 weight 2 maxqueue 25 rise 2 fall 1 maxconn 250

  9. Save the haproxy.cfg file and then go back to SFTP as root:

    a) Upload haproxy.cfg into place. Most likely goes into /etc/

    b) Check to make sure the timestamp updated. If it didn’t, you might have a problem with your permission settings, so check the FTP log to confirm that the file was actually uploaded OK without errors.

  10. Back to SSH of the Load Balancer server as the root user:

    a) Check to make sure config file is OK by running command (if your haproxy installed to /etc/haproxy/, then adjust the command):

    /usr/local/haproxy/sbin/haproxy -c -f /etc/haproxy.cfg

    b) Fix any reported problems, if necessary.

    c) Yay! HAProxy is set up and ready to go. Now we need to add the domain to our new web servers.

  11. In Web Browser:

    Repeat the following steps for each new web server (i.e. the servers besides your database server and load balancer server). If at all possible, try to follow the EXACT same methodology when doing the steps on each server. This is so that you don’t end up with weird site folders:

    a) Log into Plesk Control Panel (or whatever you use) for Web Server.

    b) If necessary, Add a new domain – your live production domain.

    c) Copy down the FTP details into the client file and into the assets section above with note of subscription ID’s and Server Identifier (i.e. Server 1). This is different than the root SFTP info. It is the FTP info for the domain, not the server.

    Note: I should probably add that it’s a bit more technical, but you can do all this by editing the virtual hosts (vhosts) file too if you know what you’re doing.

  12. Back to SSH on Load Balancer Server as root:

    a) Navigate to folder location that is above site root after confirming that the folder you think is the site root contains the site. In other words, if /var/www/vhosts/yourdomain/httpdocs is the site root, then you’d want to be sitting in /var/www/vhosts/yourdomain before you run the below command.

    b) You will need to know the location of the site root on each of your web servers. Assuming that on Web Server 1, the IP address is 333.333.333.33 and the domain folder is located at /var/www/vhosts/yourdomain/httpdocs, then to Rsync the entire codebase to Web Server 1, you’d do:

    rsync -rvzh httpdocs/ root@333.333.333.33:/var/www/vhosts/yourdomain/httpdocs/

    * Note the closing slash after the httpdocs. It’s important if you want to get everything.

    c) Enter the prompted password

    d) Log into standard FTP of Server 1. This is the domain FTP or SFTP, not the root SFTP

    e) Confirm that files made it and that they have an appropriate group and user. A key thing to look for is a zero at the front. That’s indicative of root and not the web user.

    f) If they do not have an appropriate user/group, SSH as root to 333.333.333.33 and navigate to httpdocs and run the following command:

    chown -R USERHERE:GROUPHERE httpdocs/

    * Note the closing slash after httpdocs.

    * If you don’t know the proper user and group, then go back to regular FTP instance and upload a new file and note what user and group the file you uploaded is granted.

    * Example command:

    chown -R 10001:504 httpdocs/

  13. If you are doing a Magento store, now is the time to download and edit the file at /app/etc/local.xml and change [file] to [db] for sessions. If you had database set to localhost, you’ll need to change it to point to the database server now. Additionally, to ensure Magento is always in https, you should go into index.php of the main store and any child stores and add these lines:
    if( isset($_SERVER['HTTP_X_FORWARDED_PROTO']) ) {
    $_SERVER['HTTPS'] = 'on';
    $_SERVER['SERVER_PORT'] = 443;
    }

    That code should be added right before the Mage::run command near the bottom of the file.

  14. If you had a Magento multi-store set up with sub-folders, then now is time to re-add all of your symbolic links.

    Navigate to the store’s sub-folder (which should have come over with rsync even though the symbolic links won’t) and then run the following commands:

    ln -s ../app ./app
    ln -s ../errors ./errors
    ln -s ../includes ./includes
    ln -s ../js ./js
    ln -s ../lib ./lib
    ln -s ../media ./media
    ln -s ../skin ./skin
    ln -s ../var ./var

  15. Repeat steps 12-14 for each child web server. Web Server 2 and Web Server 3, in our example.
  16. If you edited your server config files previously, and you like the settings, then copy them over to the three new web servers. Examples of config files you might have edited and want to copy to the new servers are:
    • httpd.conf
    • fcgid.conf
    • php.ini
    • my.cnf
    • Any other files with abnormally recent timestamps in relation to the rest of the files in /etc/ and its subfolders
  17. Restart the affected services on each server if you updated their config files. You can usually do this from the server control panel. In MyHosting, you do this from Virtuozzo, which you can access from the VPS Management tab. You can also do this at the command line through Putty. Examples abound on the web for how to do so, but the general syntax is:

    service [theServiceName] restart

    where [theServiceName] is replaced by the service you want to restart. In some hosting environments, you’d need to add sudo to the beginning of the command or it won’t work.

  18. Alright, if you’ve reached this point, we’ve done everything but throw the switch. Get up, move around a bit. Get yourself some caffeine or something. Back? OK, here we go.

    First things, first. Open up a web browser and point it to the home page of the website. It should be hitting the dedicated IP address of the Load balancer server.

    We can’t have Apache running at the same time as the load balancer. So we need to shut down Apache and immediately start the Load Balancer right after. So back to SSH of the Load Balancer Server as root:

    a) Stop the Apache server with:

    service httpd stop

    b) Start HAProxy:

    /usr/local/haproxy/sbin/haproxy -f /etc/haproxy.cfg

  19. In Web Browser:

    a) Refresh home page of website. It should load the home page. If it immediately 404’s or if anything else goes wrong:

    1. In Load Balancer SSH session, Stop HAProxy: /etc/init.d/haproxy stop (or) service haproxy stop (or) /usr/local/haproxy/sbin/haproxy stop
    2. Start HTTP server (Apache): service httpd start
    3. Figure out what’s wrong and try again (I didn’t have any issues).

Results

It’s been over six months since we went live with this solution and we’ve made key adjustments along the way that helped with performance. The information in this article contains the most up-to-date recommended configuration settings based on our experience. If you’re dealing with Magento Enterprise, I also strongly recommend that you spend time optimizing your database configuration for the database server. We saw page load times drop by over half from an average of 8 seconds to an average of 3-4 seconds by making changes to the database configuration.

The site that we set this up for currently averages 700,000 HAProxy sessions a day. In an average month, it does 1TB of outgoing bandwidth and 250GB of incoming bandwidth.

The server usage and load has consistently stayed in the 0-50% range of optimal, even during social media “special deals” traffic spikes. We have had one server outage in the last 40 days, and it was down for 35 seconds, which basically means that either the hosts rebooted or it rebooted itself. During those 30 seconds, the other two servers picked up the slack nicely. But it was only down for 35 seconds.

We recently updated the design to be responsive, and in doing so cut down the number of JavaScript files called on a vanilla uncached new visit by over half. Doing so dropped the average round trip application event (i.e. time in PHP and time in database) to 1.5 seconds from around 3 seconds.

We are currently reviewing Redis as an option for storing sessions over file-based sessions and are looking at CDN options for content delivery of images, CSS and JavaScript files, so I hope with those implementations to knock that number below 800ms by the end of 2014.

Comments are closed.