Case Study: Load Balancing Magento Enterprise with HAProxy

Overview

This is a case study of our first experience load balancing Magento Enterprise. It now serves an average of 700,000 sessions a day with an average response time (time in database and PHP) of 1500ms or less.

Starting Point with Client

Pre-Launch and Initial Launch

The client came to us with the requirement of building a Magento Enterprise eCommerce store. After hearing the needs of the client and their expected traffic, we suggested three VPS instances, each with the following specs:

  1. 200GB hard drive
  2. 8GB of memory
  3. 16 virtual processors
  4. 2 Dedicated IP Addresses
  5. 3.6TB of bandwidth per month
  6. Parallels Plesk for the server control panel and Virtuozzo for the VPS control panel

We considered some combination of low-end dedicated servers first, but ultimately liked the flexibility of virtual servers and that we could get multiple higher-end VPS’ for the same price as a low-end dedicated server.

We were able to find these at MyHosting.com (by the way, that link will get you 20% off the first year if you use it) at the relatively cheap cost of between $80 and $90 a month.

We split our servers into the following tasks: One for a database server, one for a web server, and one additional web server to be a development server. We mentioned in the initial plan that going with VPS’ would be good if their traffic got out of hand because we could slap a load balancer in place and build additional servers to handle the traffic.

The general consensus in the room at presentation time was that the current plan should be more than sufficient to handle their expected traffic.

We used the “Business VPS – Centos + Plesk” plans from MyHosting. This gets us Linux CentOS servers with Plesk Parallels and Virtuozzo installed. I like Plesk for its email, database, domain, and DNS management tools. You can do all of those things through another UI or through the command line, but I try to keep it simple for both myself and the client.

load-balancer-2

We did all of the pre-launch requirements with the client, including Magento setup, theme redesign, and product entry.

Then came launch day.
Launch Day
Everything was working great, the client and I were happy. Tired, but happy. And just so glad to get to launch and let the money start rolling in.

Oops

But oops! Right after we launched we discovered that they had severely understimated the amount of traffic they were going to have in the store. Like … by a lot. A lot lot.

Server CPU usage was spiking to 70, 80, 90 percent up to 100 percent at various points throughout the day on the main web server. As I’m sure you can imagine, performance was grinding to a halt and the client was asking some rather pointed questions about how to fix the problems.

Don’t panic. Analyze.

To determine what was going on, we first decided to install New Relic’s server and application monitoring programs on the server. Oh … my … god. Excuse my french, but New Relic is the shit. They have a lite version that is free. If you care about your web application, install it now. You’ll love it. I promise. And then sign up for the paid version. It’s totally worth it and your clients will easily pay for it with an extra billable hour because you’re making their applications run so much better and faster.

We optimized the application based on New Relic’s feedback, but wouldn’t you know it? Every bit of extra performance that we were able to wring out of the server and application, new users rushed to fill the hole and push the CPU and server load to red-line.

Time for a new plan

Realizing at that this point that this one-server business simply wasn’t going to work for this client, I told the client that we needed to put together a deployment plan for a load balancer immediately and get it installed. This was on a Thursday and as a glutton for punishment, I set the goal of deploying the new system on Monday. We made it, but barely.

Part 2 – Choosing the load balancer and building the plan

Comments are closed.