Web Performance Watch

How Do I Compare Thee? Linode vs Bluehost Web Host Performance Shootout

If you are evaluating or switching hosting companies for your Web site, don't just ask for an SLA guarantee, but do a site performance shootout before you cut the check. Recently I wrote a post on Beta Program, a site that I created to highlight how businesses can use Web technologies to run their operations better, for everything from using Web-based accounting software to building a better, faster website. The article, Mirror Mirror On The Wall, Who’s The Fastest Web Host Of Them All, demonstrated the performance gains I experienced when moving a website from Web hosting company Bluehost to Linode. For that article, I had focused on the overall user experience, concluding that the same site performance fluctuated between 1-2 seconds for Linode, compared to between 2-4 seconds for Bluehost.

A commentator asked "Vik– Just curious, when you ran your monitoring experiment comparing Linode to Bluehost, did you notice any trends in the performance details? Like significant differences in DNS lookup time, versus time to first byte, versus content components. Just wondering if the data reveals any specific “soft spots” with Bluehost?" I thought it would be useful to share my findings from this experiment on the Keynote Web performance blog. 

Methodology. I took a website that was built on the LAMP stack - Linux, Apache, MySQL, and PHP - and duplicated it on both Bluehost and Linode Web hosts. In both cases, I used the lowest plan that was available, and with Linode this gave me a Virtual Private Server (VPS). I made very minor modifications to the site to make it work right on the Linode environment, along the lines of changing the value of some variables that referred to the root URL. Then, I used Keynote's IE browser monitoring agent to run measurements every 5 minutes from the US-8, a group of 8 cities in the US. You could use other products as well, including the free WebPageTest service, which I used to create the video of the two sites.These were all high-speed, high-bandwidth connections, and I was able to ensure that I had a clean lab of performance monitoring agents to conduct this test. My assertions here are made after observing over 2000 datapoints per day on each website, from Mar 6 until today, for about 3 weeks. For all datapoints, I used Arithmetic Mean (though I could have used 95th percentile or median or geometric mean if I so chose, and if I remembered high school math better).

Visually Comparing Site Speed. Using WebPageTest, I first ran a site speed comparison. This measurement was taken from a server in Dulles, VA, and the video shows how long it takes for the two sites to load in the same browser. It's a great first step to help you understand the site visitor's experience. If you don't see the widget below, watch it on Youtube.

 

 

Measuring the User Experience Time. UX time is the time elapsed from when the browser started navigating to the page until the browser finished loading the page contents. This includes DNS lookup time (the time it takes for the browser to translate the URL you typed in to a host address), time to process all JavaScript on the page, to load the base HTML page, and all objects on the page such as images and JavaScript files. Here is how the two hosts stacked up over 3 weeks:

The site performance on Bluehost fluctuated between 2-3 seconds, and on LInode between 1.5-2 seconds. To answer the question on whether there were "soft spots" in Bluehost, I calculated the arithmetic mean on each of these performance metrics as well:

As you can tell, the big gains are in the network and server infrastructure. For example, it took Bluehost 448ms, almost half a second, to return the first byte of the Web page, whereas for Linode it was 39ms. If you look at all the content downloaded, there isn't much of a difference between the hosting companies - Linode is only 4% faster (though, at almost 1.5s, there is room for improvement with speed optimization on the website content itself). 

Linode recently wrote about its network upgrades in an initiative called Linode Nextgen. It appears that their work is paying off, and if you host a website, you and your users would be well served by Linode.

Super Disappointing

The performance community has grown quite a bit over the past few years. Web Performance meetup groups now can be found in almost every large metro in the U.S. and around the world. The Velocity conference continues to grow with another venue added this year. And vendors have been providing better technology to help site owners monitor and improve performance. The result: many of the top websites we visit regularly deliver a pretty solid user experience.

It’s heartening to see that web performance seemingly matters to web businesses, and that the value of high performance has become understood.

 

And then the Super Bowl happens.

Each year’s edition of the big game represents a huge opportunity for advertisers to capitalize on their significant investment in air time by engaging consumers beyond the television—on their laptops, tablets and smartphones. Advertisers know that this produces huge surges in traffic, and without proper planning, can create a very poor customer experience.

This year, we closely monitored the advertisers across all 3 screens to see how well they prepared to engage with consumers. Here are the results, and some observations which should serve as reminders of how web performance considerations are just as important today as they were 5 years ago:

1. Mobile performance is tragically misunderstood

Over in our sister blog Mobility, we’ve talked about the differences between traditional desktop Web performance, and mobile performance. What seems clear is that most companies still do not understand how forcing the desktop experience down onto a smartphone impacts the end user experience. With no other Super Bowl advertiser is this more clear than Century 21 Real Estate. Their promotional site was not optimized for the smartphone, sending a staggering 94 objects to the browser. Because our smartphone testing agents connect to wireless carrier networks, the average response time during the game was over 54 seconds, compared to the 2.97 second average measured by our Internet Explorer agents. Such a huge difference illustrates how important it is to treat the mobile experience uniquely—even for tablet users.

On the other end of the spectrum, Axe presented a nicely optimized site for smartphone browsers. Coming in it nearly 7 seconds, Axe maintained a low page weight of 270KB. In contrast, one image on the Century 21 page was over 1,000KB. Axe also decided to provide their tablet visitors with the optimized smartphone site, instead of the fuller featured desktop site—emphasizing their focus on a high performing experience.  

2. 40 million viewers… you should probably run some web load tests

While the 49ers repeated failures on defense and special teams were shocking to many fans, the most surprising crash was that of the Coke website.

You can see how the site availability deteriorates rapidly once the ads aired.

Fans of the iconic brand took to Twitter to vent their frustrations:

Coke reported that its campaign was a success by many measures. However, their public statements also reveal how significantly the brand probably under-estimated capacity for the website component of the campaign. Keynote worked with other Super Bowl advertisers to test their sites in advance of the game by generating real traffic from the Internet—a practice known as Web Load Testing. Those advertisers planned for 1.5 – 3.0 million site visitors. Coke’s estimate was less than 1 million. Doing a bit of ‘Monday morning quarterbacking’ here, this low estimate likely explains much of why Coke’s sites performed so poorly during the Super Bowl.

By contrast, the online retailers (having weathered many a Cyber Monday) such as Best Buy performed flawlessly with 100% availability. Best Buy also delivered some of the fastest response times across all 3 screens.

3. The (good) old rules still apply

It was surprising to find that many the advertising sites did not make use of CDNs and common optimization techniques. It’s clear many site owners can benefit from taking a fresh look at their pages. So revisit those web performance best practices and take an inventory of how well your site is delivering end user experience.

 

Cyber Monday Breaks Records, But Could It Have Been Even Better?

Oh, what a difference a day makes!

By many accounts, Cyber Monday was another record setting day for online retailers. Some estimates have suggested that total sales reached nearly $2 billion! At the handmade & vintage goods marketplace Etsy, more sellers completed deals on Cyber Monday than at any time in their 7-year history. And IBM analytics reported that 18% of all site traffic came from mobile devices.

This is great news for e-commerce companies. But for many, Cyber Monday was not the smooth sales day we observed in our retail indices for Web and mobile performance over the Thanksgiving/Black Friday weekend.

The average Total User Experience Time for home pages in our Top Retailers index was 2.98 seconds, 7% slower than the previous day. As the amount of traffic surged in the afternoon and evening hours, many sites began to struggle. In the graph below, we’ve overlaid the real time sales volume that IBM tracked throughout the day against the hourly average of Total User Time for the home pages in our index.

Although the number of retailers represented in the index is relatively low, there does appear to be a correlation between increased load (sales volume) and performance. We’ve observed a similar correlation in the past.

Overall site availability was high, however, some retailers experienced significant issues. Third party content failures appeared in many measurements, and back-end problems appeared in others. The Saks Fifth Avenue website was unable to successfully complete transactions for nearly 2 hours from 1:39pm ET to 3:26pm ET.

Their Twitter feed confirmed the site’s difficulties:

Keynote tracks the performance of 22 different retail transactions which allows us to monitor the user experience of someone visiting a site, searching for a product and then adding it to the shopping cart. The data from these transactions inclusive of Cyber Monday will be available next week.

Our Mobile Commerce index also observed that mobile website performance slowed across the board for home pages accessed on smartphones. At a couple points during the day, average page load time exceeded 18 seconds, 2X the previous week’s average and well below user expectations.

Mobile sales nearly doubled this year to 13% of all receipts online. But given the functionality issues our recent study of several major retailer mobile sites revealed, and the slow response times of the sites in our index, mobile commerce still has a long way to go. This may be why bounce rate for mobile is 20% higher than the desktop bounce rate (39.42% vs. 32.77%).

Overall, retailers performed well this Cyber Monday—especially for their desktop customers. Record setting traffic and sales volume is the ultimate testament to the preparation and execution of web operations professionals across the retail industry. But of course the holiday shopping season has only just begun!

Retailers Zip Through Black Friday

U.S. online retailers once again set record sales for the long Thanksgiving weekend and we’re already receiving reports that today—Cyber Monday—is setting up to be massive as well. Looking at the performance of major e-commerce sites reveals that most sites were well prepared for the heavy traffic and transactions of the past four days.

The average Total User Experience Time across the Keynote Top Retailers index was 2.77 seconds and Time to Interactive Page coming in at 2.27 seconds. Average Time to First Paint—the point at which a new user first sees content in their browser—clocked in at 789 milliseconds.

While retailers across the board should improve their initial impression with first-time customers (250 milliseconds is recommended best practice), most home pages were complete or perceptibly usable within the recommended tolerance of 3 seconds.

A few retailers, however, did experience some performance issues. Keynote’s Retail Apparel transaction index showed that starting at 8:05 a.m. PT on Black Friday until 8:40 a.m., the Foot Locker site struggled with internal server errors. Transactions during this period could not be successfully completed. You may have been able to get to the home page, but performing an item search or adding it to the shopping cart would not have worked.

But while Foot Locker’s site may not have been completely prepared for the onslaught of Black Friday traffic, most of the other 47 sites in the Top Retailers index experienced no major outages during the weekend. The number of Black Friday shoppers grew 18% over last year and the preparations of online retailers appear to have paid off—to the tune of a more than $1 billion!

Keynote’s monitoring of retail mobile sites also showed that most sites performed consistently through Black Friday. As with the desktop, these sites optimized for smartphones and tablets delivered comparable performance to non-peak periods. However as we had previously reported, the performance of most retail sites on smartphones across major U.S. cellular carriers was generally slow, relative to user expectations. Our Mobile Commerce Index results for the week will be published shortly in Internet Retailer with more detail.

We’ll wrap up our online retailer performance analysis for Cyber Monday tomorrow.

From Network Performance to User Experience

Earlier this year, Keynote announced the addition of User Experience metrics to our web monitoring products.  These metrics allow you to broaden the conversation about web performance.  In addition to seeing the performance of the "bits and bytes" going over the wire, you can now also understand how quickly your end users get to the "clicks and pics" on your site.

We thought it would be fun to take a look at how some of these metrics compare across sites and industries.  Did you know that it takes about 1.5 seconds longer to interact with most of the content on news sites than on social networking sites?  Or that it takes over twice as long to load the home page on Foursquare than Twitter?  These are some of the factoids included in the following infographic:

Monitoring the 340 Trillion Trillion Trillion: Keynote and IPv6

As a kid, were you fascinated by really big numbers?  I was and remain so to this day.  While the number of addresses that IPv6 can theoretically support is not quite a googol, it is a really big number: 2¹²⁸, or more than 340 trillion trillion trillion.

Speaking candidly, it would thrill me to no end to see Keynote monitor that many sites one day!  For now, let's focus on monitoring just one site: Google's IPv6 test site, ipv6test.google.com.  As I sit here typing this on my laptop, I'm on a workaday vanilla IPv4 network, so when I go to this site I see the following:

Now, Keynote has a set of monitoring agents running real Internet Explorer and Firefox browsers that are connected to IPv6 networks.  But rather than taking someone else's word that these agents can monitor any of those 340 trillion trillion trillion addresses, I wanted to test this out myself!

Using Keynote Internet Testing Environment (KITE), I recorded a test script that would explicitly fail unless it were being played back over an IPv6 network.  I did this by adding a simple error text validation condition:

So when I play this back in KITE on my own laptop, I get an error because the site says "You don't have IPv6".  But when I upload this script to our monitoring agents, it works like a charm.  In fact, I can drill into my monitoring data to see the screenshots the agents have captured:

And I can easily see that this is a different message from what I get in my own browser.  What's more, I can get a breakdown of the load times of each of the resources on the page, including those resources that are loaded from an IPv6 address:

So there we have it: the future of the Internet in its full 128-bit glory!

Guidelines for 3-screen Performance Management

Recently, the web performance team at Walmart joined us for a webcast on site performance in a 3-screen world. The emergence of both smartphones and now tablets as primary vehicles for driving online interaction including e-commerce is well documented. What’s still not as well understood are the implications to site design that impact performance across the desktop, smartphone and tablet environments.

While I encourage you to watch the webcast, here are a few launch-pad performance benchmarks and tips to think about for your Web and mobile sites:

1. 3-screen is more than 3-screen
   On the desktop, multiple browsers add some complexity to understanding performance. But the multiple permutations of OS/hardware/versions representing today’s smartphone and tablet environments exacerbate the complexity of the user experience online.
2. You can’t take it with you
   Performance management concepts for one screen do not translate to the others. A study by Yankee Group found that Apple’s website ran nearly 200% faster on desktop than Amazon’s site, but ran 3% slower on tablet.
3. Starting-point benchmarks
   
   
   
4. Minimize, and gracefully enhance
   Develop with the 3G connection in mind first, and add to the experience from there. For smartphone and tablets, consider the following practices:
   

   >> Limit element count to 10 or fewer new HTTP requests/page

   >> Avoid redirects

   >> Reduce the number of DNS lookups/page

   >> Always use HTTP Keep Alive

   >> Audit image content for appropriate resolution, quality settings and compression

5. One size does not fit all!
   Commit to accurately and consistently measure performance and optimize based on where you have issues/latency and within your technical constraints. Your unique situation will impact how you approach improvement, e.g.: front end versus back end, CDN versus network, 3^rd party versus CMS, etc. The benchmarks above are only as good as what is normal for you, your industry and your competitors. Find other benchmarks, and network with your peers.

Test Your Site on IE 9 and Measure User Experience

Last month, we announced that the Keynote Global Network was being updated with Internet Explorer 9. As a result, our real browser monitoring service, Transaction Perspective™, is now measuring the performance of Web applications and sites using Microsoft’s latest Web browser. This makes Keynote the first on-demand monitoring service built on IE 9, which is pretty cool. But what’s even cooler is the ability that IE 9 gives us to measure a new class of performance metrics we call user experience metrics.

As we’ve discussed here previously, IE is still the big kid on the block when it comes to browser usage. With the demise of IE 6 in the United States, and the rise of Firefox and Chrome, it’s clear that users are quickly leaving “old” browsers for “modern” ones like IE 9. With high performance and broad support for open Web standards, browsers like IE 9 make it easier for companies to create a rich and snappy experience for consumers. In response, 34% of the top Internet sites now use HTML 5, and the use of JavaScript continues to rise. Transaction Perspective built on IE 9 allows customers to get a more precise view of their sites’ performance, especially those leveraging new Web standards.

Our new Live Beta preview of MyKeynote 11 with Transaction Perspective lets you see performance in very important ways:

• Time to First Paint – This new metric tells you when a user begins to see your site render in the browser.
• Time to Interactive Page – Tells you when the Document Object Model (DOM) begins to process user events for the document.
• Total User Experience Time – With User Experience Time, you know how long a page (or series of pages) took to render and become usable for a real user.  It is the ultimate measure of a page’s speed, factoring not only the time it took for data to be downloaded, but also rendered and made interactive.

These key moments are just a handful of the browser events we capture. If you’re a performance expert, you’ll appreciate that we measure all the Browser Navigation Timing events and can graph them individually over time (multiple measurements), as well as display them in a timeline view for an individual measurement.

KITE (Keynote Internet Testing Environment) lets you to test for free your site’s performance from 5 cities on the Keynote Global Network, on demand. But if you’d like to test drive User Experience monitoring, click the “24x7 Monitoring” button in KITE.

Once you’ve activated your trial, click the “Try Beta Version” link in MyKeynote. There, you’ll be able view User Experience metrics for everything monitored during your trial.

Soon, you’ll be able to see these new browser event metrics from your desktop in KITE, as well. Your copy of KITE should automatically update itself to version 5, or you can manually check for the new version. 

Let us know what you think of these new features!

Up on the Roof Top… Click, Click, Click

Hi. My name is Ben, and I plan to visit your website next year a lot! I’m a casual user that likes to browse the Internet, and like everyone else I know, I hate to wait. Although your website is really cool, and I love your products and services, the Internet is full of interesting places and I’m easily distracted. Oh, and did I mention that I love using my Android smartphone and iPad to play, connect, shop, bank and book travel when I’m not at work? All I want for the holidays is a speedy web and mobile site.

Here are my top 10 wishes for you and your site in 2012:

1. Please test your code/site on IE and not just Firefox before you launch it. I am one of the 50% that will continue using IE, even after they start auto-upgrades!
2. Understand the difference between browser execution and network/back-end performance. Most pages have both and you need to know which is which to optimize the page/site. One way is to monitor using a real browser.
3. Understand how your page renders. Focus on reducing upfront (pre-render) delay. It’s the one me and your other users feel the most!
4. Please make sure your third party tags (analytics and others) are below your visual content. (Did I mention rendering delays and blocks are aggravating?)
5. Please combine your external JS and CSS files. I’ve been saying this for years, but very few sites seem to follow this recommendation. Do it, and I’ll see a major improvement in the speed of your site.
6. Understand the quality your Content Delivery Network is providing. Every website is unique, and not all CDN providers are created equal.
7. Don’t worry so much about overall page size but instead focus on individual file/resource sizes. Keep them under 100K and you will limit the impact of slower connections. (Did I mention I love my smartphone?)
8. Don’t just push your desktop website to mobile. You will fail.
9. Test your mobile site… please!
10. Read Keynote’s Page Construction Guidelines. They’re chocked full of goodies to help you optimize the performance your Web pages and keep visitors like me happily clicking through them, instead of away to your competitor’s site.

What’s on your wish list for better web and mobile performance in 2012? Let me know!

Cyber-Awesome: Killer Performance Helps Drive Record Revenue

For years, Keynote has been monitoring the ‘net’s best websites and reporting on notable outages during the peak Black Friday and Cyber Monday online shopping period. Typically, we find that many online retail sites—both pure Internet and those with brick and mortar stores—experience some issues. But the trend has been towards better performance and higher availability. This year marks yet more progress toward the goal of a trouble-free and virtually instant shopping experience. And the numbers suggest the payoff for retailers has been huge: a record setting $1.25 billion in online sales for Cyber Monday, 2011.

The Keynote Top Retailers Index tracks 47 online shopping destinations. This year, the average website response time was 3.05 seconds. According to a study by IBM, online sales peaked at about mid-day, and spiked again later in the evening. Average response times across the Top Retailer Index reflected this pattern as peaking demand slowed sites. But the absolute impact even at its worst was only about 1 second of delay. That’s great performance across the industry as a whole.

Additionally, the availability of these websites remained extremely high throughout the period. Out of the 47 sites tracked, only 10 (21%) experienced any downtime at all and their average availability was around 99.0%.

So how did online retailers get to this pinnacle? We see that customers are maturing the technology, processes and culture of their Web Operations around a passion for performance and deep preparation. One example of this is Keynote customer Karmaloop.com who rang up their biggest day in company history and over 100% more sales than the same period last year. Though careful planning and web load testing, they were able prepare their site for the demands of such a dramatic increase in traffic and order transactions.

The improving trend in website performance throughout the increasingly demanding Black Friday and Cyber Monday period is encouraging. But it also means that if your site experiences failure or less than stellar performance, your competitor is in a much better position to attract and retain your abandoning customers. Congratulations to the online retail industry for hitting a remarkable milestone over the past six days. Keep pushing performance higher through December!