Web Performance Watch

How Do I Compare Thee? Linode vs Bluehost Web Host Performance Shootout

If you are evaluating or switching hosting companies for your Web site, don't just ask for an SLA guarantee, but do a site performance shootout before you cut the check. Recently I wrote a post on Beta Program, a site that I created to highlight how businesses can use Web technologies to run their operations better, for everything from using Web-based accounting software to building a better, faster website. The article, Mirror Mirror On The Wall, Who’s The Fastest Web Host Of Them All, demonstrated the performance gains I experienced when moving a website from Web hosting company Bluehost to Linode. For that article, I had focused on the overall user experience, concluding that the same site performance fluctuated between 1-2 seconds for Linode, compared to between 2-4 seconds for Bluehost.

A commentator asked "Vik– Just curious, when you ran your monitoring experiment comparing Linode to Bluehost, did you notice any trends in the performance details? Like significant differences in DNS lookup time, versus time to first byte, versus content components. Just wondering if the data reveals any specific “soft spots” with Bluehost?" I thought it would be useful to share my findings from this experiment on the Keynote Web performance blog. 

Methodology. I took a website that was built on the LAMP stack - Linux, Apache, MySQL, and PHP - and duplicated it on both Bluehost and Linode Web hosts. In both cases, I used the lowest plan that was available, and with Linode this gave me a Virtual Private Server (VPS). I made very minor modifications to the site to make it work right on the Linode environment, along the lines of changing the value of some variables that referred to the root URL. Then, I used Keynote's IE browser monitoring agent to run measurements every 5 minutes from the US-8, a group of 8 cities in the US. You could use other products as well, including the free WebPageTest service, which I used to create the video of the two sites.These were all high-speed, high-bandwidth connections, and I was able to ensure that I had a clean lab of performance monitoring agents to conduct this test. My assertions here are made after observing over 2000 datapoints per day on each website, from Mar 6 until today, for about 3 weeks. For all datapoints, I used Arithmetic Mean (though I could have used 95th percentile or median or geometric mean if I so chose, and if I remembered high school math better).

Visually Comparing Site Speed. Using WebPageTest, I first ran a site speed comparison. This measurement was taken from a server in Dulles, VA, and the video shows how long it takes for the two sites to load in the same browser. It's a great first step to help you understand the site visitor's experience. If you don't see the widget below, watch it on Youtube.

 

 

Measuring the User Experience Time. UX time is the time elapsed from when the browser started navigating to the page until the browser finished loading the page contents. This includes DNS lookup time (the time it takes for the browser to translate the URL you typed in to a host address), time to process all JavaScript on the page, to load the base HTML page, and all objects on the page such as images and JavaScript files. Here is how the two hosts stacked up over 3 weeks:

The site performance on Bluehost fluctuated between 2-3 seconds, and on LInode between 1.5-2 seconds. To answer the question on whether there were "soft spots" in Bluehost, I calculated the arithmetic mean on each of these performance metrics as well:

As you can tell, the big gains are in the network and server infrastructure. For example, it took Bluehost 448ms, almost half a second, to return the first byte of the Web page, whereas for Linode it was 39ms. If you look at all the content downloaded, there isn't much of a difference between the hosting companies - Linode is only 4% faster (though, at almost 1.5s, there is room for improvement with speed optimization on the website content itself). 

Linode recently wrote about its network upgrades in an initiative called Linode Nextgen. It appears that their work is paying off, and if you host a website, you and your users would be well served by Linode.

Cyber Monday Breaks Records, But Could It Have Been Even Better?

Oh, what a difference a day makes!

By many accounts, Cyber Monday was another record setting day for online retailers. Some estimates have suggested that total sales reached nearly $2 billion! At the handmade & vintage goods marketplace Etsy, more sellers completed deals on Cyber Monday than at any time in their 7-year history. And IBM analytics reported that 18% of all site traffic came from mobile devices.

This is great news for e-commerce companies. But for many, Cyber Monday was not the smooth sales day we observed in our retail indices for Web and mobile performance over the Thanksgiving/Black Friday weekend.

The average Total User Experience Time for home pages in our Top Retailers index was 2.98 seconds, 7% slower than the previous day. As the amount of traffic surged in the afternoon and evening hours, many sites began to struggle. In the graph below, we’ve overlaid the real time sales volume that IBM tracked throughout the day against the hourly average of Total User Time for the home pages in our index.

Although the number of retailers represented in the index is relatively low, there does appear to be a correlation between increased load (sales volume) and performance. We’ve observed a similar correlation in the past.

Overall site availability was high, however, some retailers experienced significant issues. Third party content failures appeared in many measurements, and back-end problems appeared in others. The Saks Fifth Avenue website was unable to successfully complete transactions for nearly 2 hours from 1:39pm ET to 3:26pm ET.

Their Twitter feed confirmed the site’s difficulties:

Keynote tracks the performance of 22 different retail transactions which allows us to monitor the user experience of someone visiting a site, searching for a product and then adding it to the shopping cart. The data from these transactions inclusive of Cyber Monday will be available next week.

Our Mobile Commerce index also observed that mobile website performance slowed across the board for home pages accessed on smartphones. At a couple points during the day, average page load time exceeded 18 seconds, 2X the previous week’s average and well below user expectations.

Mobile sales nearly doubled this year to 13% of all receipts online. But given the functionality issues our recent study of several major retailer mobile sites revealed, and the slow response times of the sites in our index, mobile commerce still has a long way to go. This may be why bounce rate for mobile is 20% higher than the desktop bounce rate (39.42% vs. 32.77%).

Overall, retailers performed well this Cyber Monday—especially for their desktop customers. Record setting traffic and sales volume is the ultimate testament to the preparation and execution of web operations professionals across the retail industry. But of course the holiday shopping season has only just begun!

From Network Performance to User Experience

Earlier this year, Keynote announced the addition of User Experience metrics to our web monitoring products.  These metrics allow you to broaden the conversation about web performance.  In addition to seeing the performance of the "bits and bytes" going over the wire, you can now also understand how quickly your end users get to the "clicks and pics" on your site.

We thought it would be fun to take a look at how some of these metrics compare across sites and industries.  Did you know that it takes about 1.5 seconds longer to interact with most of the content on news sites than on social networking sites?  Or that it takes over twice as long to load the home page on Foursquare than Twitter?  These are some of the factoids included in the following infographic:

Filtering Out Web Performance Monitoring Traffic From Google Analytics

If you're a webmaster, site owner, or digital business manager, picture this scenario. You open your Google Analytics dashboard and you see a big spike in traffic. At first, you're really excited - wow, look at all those site visitors! You run off to tell your boss that she can give you that bonus she promised if you got that online marketing campaign to work. And then... you learn that someone in the Site Ops team added web performance monitors, and so it's all so-called robots - synthetic traffic generated for the purpose of keeping tabs on your site's response time and availability. 



You think, no problem, I could filter that out from Google Analytics, and then you learn you can't. It's always going to be there, like, forever. You go crazy, emailing the GA team, posting on forums, and then slowly resign yourself to forever dealing with that spike. It will disappear from your default view, maybe after a month, but that's a long month to wait. And heaven forbid if your execs ask you for a site traffic report for the past year, try explaining why you can't filter out that spike - what, you weren't thinking ahead? What kind of guy did we hire to run our online business, anyway?

Don't let this happen to your career. Plan ahead and learn how to use Google Analytics to filter out all web performance monitors from your site analytics reports. Here's the recipe:

STEP 1: Find Out the User Agent String for your Web Performance Monitors

Keynote's monitors, like all other web performance monitors, insert a special marker in the browser, called an user agent string. So all you have to know are the user agent strings that Keynote adds to the browser. Keynote has several performance monitors - using real IE and Firefox browsers, or mobile browsers. Each performance monitor comes with its own special marker, so you have to construct a regular expression to filter all of these out.

Here are the browser markers that you have to use to filter out web performance monitors:

 Keynote Systems - Use "KHTE" (for Application Perspective monitors), "KTXN" (for the real browser Transaction Perspective monitors), and "Keynote" for Test Perspective load testing agents.  
 Compuware Gomez - "GomezAgent"  
 Smartbear AlertSite - "AlertSite"  
 Pingdom - "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"  
 Yottaa - "YottaaMonitor"  

If you are using a web performance monitor not listed above, Google "<insert your monitoring vendor> user agent string" and you will definitely find the user agent string you need to know.

STEP 2: Change Google Analytics Tag to Ignore Web Performance Monitors

What you do next is to put a conditional wrapper around your Google Analytics tag to never log visits from testing companies. This requires you to edit the JavaScript code on your site. An example Google analytics tag is shown below – the actual Google code depends on your implementation. Note that the Google Analytics account ID below is an example – you should replace it with your own account ID.

   <!-- Google Analytics tracking code that eliminates Keynote robots -->  
   <script type="text/javascript">  
     // These are the strings that Keynote adds to the User Agent string:  
     var gk_KEYNOTE_TXP_MONITOR = "KTXN";   // Transaction Perspective agents  
     var gk_KEYNOTE_APP_MONITOR = "KHTE";   // Application Perspective agents  
     var gk_KEYNOTE_TSP_MONITOR = "Keynote";  // Test Perspective agents  
   
    // if the User Agent string indicates Keynote monitors, then don't track the visit in Google Analytics  
    if (navigator.userAgent.indexOf(gk_KEYNOTE_TXP_MONITOR, 0) == -1 &&   
      navigator.userAgent.indexOf(gk_KEYNOTE_APP_MONITOR, 0) == -1) &&  
      navigator.userAgent.indexOf(gk_KEYNOTE_TSP_MONITOR, 0) == -1) {  
     // Ok, this is not a Keynote monitoring agent, so go ahead and log the visit with Google:  
    var _gaq = _gaq || [];  
    _gaq.push(['_setAccount', 'UA-12345678-9']);  
    _gaq.push(['_setCustomVar',1, 'User Type','Public',2]);  
    _gaq.push(['_trackPageview']);  
    (function () {  
       var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;  
       ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';  
       var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);  
    })();  
    }  
   </script>  
 <!-- Source code formatted using http://codeformatter.blogspot.com -->  

That's it. At this point, Google Analytics will not log your robot traffic (at least for Keynote web monitoring traffic, using the above example) and you can safely keep your job. The code snippet can be easily extended to filter out Gomez, AlertSite, Yottaa, as well.

Next, we will create an Advanced Segment in Google Analytics to filter out any synthetic traffic that has gotten through the cracks.

STEP 3: Create An Advanced Segment to Filter Out Additional Web Performance Monitors

Some monitoring tools don't modify the User Agent string, and instead use other techniques. Ensuring that you are in the MySite tab, click on Advanced Segments.

In the bottom right area of the Advanced Segments dialog box, click on this button:    and name it "Real People". We will now create a segment that filters out the synthetic traffic from web performance monitors. Here's the Advanced Segments dialog box:

Now, here's the tricky part - writing the regular expression that the Advanced Segment requires. Here is the regexp that filters out both the Keynote and Gomez monitors. Be careful to use the string exactly as shown, with the periods and asterisks: .*(KHTE|KTXN|GomezAgent).*

It's critical that you use a regular expression correctly, and getting it wrong is why I suspect many people believe that Google Analytics can't filter out this traffic. Once an advanced segment is created, then all traffic AFTER today will be filtered in the reports, but this will not apply to traffic that was generated prior to your creating this advanced segment. That's what I believe, from trial and error, though Google Analytics help says that you can filter out historical traffic. In any case, it's important you setup these monitors anyway, because you have no control over someone else setting up web performance monitors - even if your company didn't create performance monitors, your competitors could be monitoring your site's performance and creating all this traffic to your site - it is the world wide web, after all.

STEP 4: Select the Real People Advanced Segment When Viewing Data

Drop down the Advanced Segments dialog box, and select "All Visits" on the left hand side, and the advanced segment that you created, "Real People" on the right hand side.

Click on Apply, and view your data:

Now, it would be nice if I could choose "Real People" as the default segment to apply to all my reports, but I can't do that in Google Analytics yet. Nevertheless, you now have a handy way to view all the traffic and exclude web performance monitors, including Keynote.

Monitoring the 340 Trillion Trillion Trillion: Keynote and IPv6

As a kid, were you fascinated by really big numbers?  I was and remain so to this day.  While the number of addresses that IPv6 can theoretically support is not quite a googol, it is a really big number: 2¹²⁸, or more than 340 trillion trillion trillion.

Speaking candidly, it would thrill me to no end to see Keynote monitor that many sites one day!  For now, let's focus on monitoring just one site: Google's IPv6 test site, ipv6test.google.com.  As I sit here typing this on my laptop, I'm on a workaday vanilla IPv4 network, so when I go to this site I see the following:

Now, Keynote has a set of monitoring agents running real Internet Explorer and Firefox browsers that are connected to IPv6 networks.  But rather than taking someone else's word that these agents can monitor any of those 340 trillion trillion trillion addresses, I wanted to test this out myself!

Using Keynote Internet Testing Environment (KITE), I recorded a test script that would explicitly fail unless it were being played back over an IPv6 network.  I did this by adding a simple error text validation condition:

So when I play this back in KITE on my own laptop, I get an error because the site says "You don't have IPv6".  But when I upload this script to our monitoring agents, it works like a charm.  In fact, I can drill into my monitoring data to see the screenshots the agents have captured:

And I can easily see that this is a different message from what I get in my own browser.  What's more, I can get a breakdown of the load times of each of the resources on the page, including those resources that are loaded from an IPv6 address:

So there we have it: the future of the Internet in its full 128-bit glory!

Building a Stronger Web With Automation

Last week at Velocity, we had the pleasure of connecting with lots of old friends, and thousands of new ones. As a long-time sponsor, we value being a part of the Web Performance and Operations community’s most focused gathering. Sure, it gave us an opportunity to sales-pitch you. (Well, at least up until the fire alarm went off.) But more importantly, we got to learn how you in the web ops community continue to innovate and hear how our tools can help keep your websites awesome.

One of the hot topics of conversation around our booth was our new RESTful API. The beauty of any modern API is the ease with which you can take a service and adapt it to solve very unique problems. The Keynote API is currently being used by customers to quickly and easily integrate their Keynote data with dashboard applications and other monitoring tool streams to do some pretty interesting stuff. We recently spoke with Velocity attendee Christian Jorgensen about his use of Keynote API. Christian is responsible for ensuring high availability for a portfolio of very large websites and services. The work he’s doing in monitoring and alerting is cutting-edge.

Question:  What are some of the challenges you’re dealing with in managing and monitoring your environment?

Answer:  “Our primary KPI is Mean Time to Mitigate, or Mean Time to Resolve as most are familiar with it. We require a large amount of monitoring and the monitoring has to be sensitive enough so that we can either be predictive or quickly responsive to live site impacting outages or customer impacting events. (But) we are an extremely noisy team from an incident perspective. I think last month, we had over 8,000 incident tickets that were logged inside of our team. So, that's a real challenge, right?”

“Every packet, every bit of information, every HTTP status code is logged into a system and is presented in real time as an available signal. There is so much of this volume that we obviously can't use it and it's difficult to aggregate. We've been using new approaches like complex event processing, which allows us to aggregate data and actually look at patterns and themes, but still it's too much volume.”

“One of our largest sources (of incidents) today are those generated by our CDN due to files that are recently uploaded and may be invalidated. A really stubborn one is ads and analytics. So, you'll have a large number of time outs and 404's and whatnot due to third party advertisers. These are scenarios that do not require a human response.”

“What Keynote has allowed us to do with the new API is to take in the external signals, compare them to our internal sources and corroborate the two to identify the impact of an incident and, of course, whether that incident is real.”

Question: When you do find a real incident, how does the API help you deal with it?

Answer: “With the graphing API, for example, we have the capability to go in and not only look at, ‘Okay, we've met a failure condition in our testing,’ but we know exactly from the waterfall what the problem was and we can build automation that can capture the correct teams or products that are responsible for those failures and we can quickly through automation route those tickets to the correct teams so that they can quickly engage. It's basically taking the human component of analyzing a signal that Keynote provides and making a decision off that. Well, that's something that you can train a robot to do.”

Question: You mentioned advertising being a stubborn originator of incidents; could you elaborate on what you do with those issues?

Answer: “Oh, yeah. Absolutely. It's a huge blind spot about the impact to page performance and overall advertiser accountability, especially in third party scenarios, which, is a pretty complex scenario.”

“What we've been able to do is use the data to actually demonstrate very clearly not only what the impact is from a performance perspective, but we're also able to actually specifically call out which ads and which advertisers are generating the most noise. And those are ones that we actually have monthly rhythm of business now where we meet regularly with the advertising leadership. We're basically the data provider to measure improvement in that space as well. That's probably why we're not so popular. We have an automated bug system that will write bugs to them every time we encounter a large number of, we call them advertising defects.”

“Now with the new API we're able to look at the data in real time and assign it not just for attention not just from an improve perspective but as an actual incident response. There's a dedicated advertising team that fields those incidents and corrects campaigns or errors with ads in real time. And we were never able to provide a signal for that and they had never had one (previously).”

These are just some of the ways that Christian has innovated his Web operations through automation with tools like Keynote. You too can also take advantage of Keynote API, even if you’re not a customer. Check out our recent post about accessing the Keynote Business 40 index programmatically. This is a great way to build third-party, objective performance references against your monitoring data and corroborate availability issues (externally) in real time. The data is free, so let us know how you use it!

Did You Fall When Facebook Stumbled?

Thursday's outage of Facebook was notable for the ripple it created across the Web. Many online businesses were impacted indirectly as a result of failing content integrated into their websites. The nearly ubiquitous "Like" button and Facebook's other social plugins became a drag on the performance and availability of some websites. We saw notable failures across media, travel and retail sites. Yet, other some appeared to have avoided significant impact.

The websites of CNN, USA Today and Expedia--members of the Keynote Business 40 Index--demonstrate how different approaches to integrating the Facebook social plugin can be the difference between disaster and only a minor setback when 3rd party content fails.

Here are graphs of their performance and availability during the outage (along with Facebook's for reference):

Notice USA Today's performance (the blue line). Their home page continued to happily build despite the lack of content availability from Facebook.

Incidents like this are good reminders of the importance of using page construction best practices that accomodate third party content failures. If you feature ads, widgets or plugins on your site, do they represent a potential single point of failure if the service becomes unavailable?

Even if you take care with integrating third party content into your pages, it's important to continuously monitor each source indepently if your site changes frequently. Keynote helps companies uniquely monitor 3rd party content such as ads, social widgets and plugins so you can quickly take action to mitigate performance issues and focus business improvement with actionable data.

So how did your site perform Thursday evening?

Making IT Matter

Yesterday, I read an article reporting about a company's potential plans to upgrade one of its key business applications. The article featured perspectives from the application's end users, management and of course, the IT organization that developed and now supports it. The dynamic at issue was the app's lack of compatibility across browsers.

When asked about the likelihood of a change, the company's IT director responded with the following:

"[My organization has] been shying away from upgrading a major version of [biz app] because every time we’ve done it, it’s been tremendously difficult." (emphasis added)

Super. And once again, we have another reason why 
"IT doesn't matter."

While the end users could not understand why the application wouldn't run reliably on Safari and Chrome in this case, IT was not convinced it was necessary to change the app to support them. Of course, cross-browser compatibility is a thorny issue. We've explored the reasons why site owners need to consider multi-browser strategies, and considerations for getting there. In this instance, there may no real viable solution for improving this application's end-user experience with regard to browser support.

But saying that it won't be done because "it’s been tremendously difficult" exemplifies the challenges that IT continues to have with finding relevance in a cloudy, mobile, and increasingly consumerized IT landscape. Today, the end user matters more than ever. IT organizations that recognize this, inculcate a user-first approach into how they manage their portfolio as well as tool choices. They adopt standards that support user oriented approaches. And the good news for IT is that these standards and tools exist.

It's possible that the choice of words of this IT manager in this instance belies other realities. But on the surface, the sentiment provides exactly the motivation that will drive this IT organization's customers and business stakeholders to a competitive SaaS application.

The end-user makes IT matter

Making IT to matter means placing the user at the forefront of the priority matrix, from choices in platform to its ongoing monitoring. Your users are changing... rapidly. When an architecture cannot support the expectations today's user, it will not survive.

 

Cyber-Awesome: Killer Performance Helps Drive Record Revenue

For years, Keynote has been monitoring the ‘net’s best websites and reporting on notable outages during the peak Black Friday and Cyber Monday online shopping period. Typically, we find that many online retail sites—both pure Internet and those with brick and mortar stores—experience some issues. But the trend has been towards better performance and higher availability. This year marks yet more progress toward the goal of a trouble-free and virtually instant shopping experience. And the numbers suggest the payoff for retailers has been huge: a record setting $1.25 billion in online sales for Cyber Monday, 2011.

The Keynote Top Retailers Index tracks 47 online shopping destinations. This year, the average website response time was 3.05 seconds. According to a study by IBM, online sales peaked at about mid-day, and spiked again later in the evening. Average response times across the Top Retailer Index reflected this pattern as peaking demand slowed sites. But the absolute impact even at its worst was only about 1 second of delay. That’s great performance across the industry as a whole.

Additionally, the availability of these websites remained extremely high throughout the period. Out of the 47 sites tracked, only 10 (21%) experienced any downtime at all and their average availability was around 99.0%.

So how did online retailers get to this pinnacle? We see that customers are maturing the technology, processes and culture of their Web Operations around a passion for performance and deep preparation. One example of this is Keynote customer Karmaloop.com who rang up their biggest day in company history and over 100% more sales than the same period last year. Though careful planning and web load testing, they were able prepare their site for the demands of such a dramatic increase in traffic and order transactions.

The improving trend in website performance throughout the increasingly demanding Black Friday and Cyber Monday period is encouraging. But it also means that if your site experiences failure or less than stellar performance, your competitor is in a much better position to attract and retain your abandoning customers. Congratulations to the online retail industry for hitting a remarkable milestone over the past six days. Keep pushing performance higher through December!

New MyKeynote Version: Hot Fun In The Summertime!

Keynote's crack Engineering and Operations teams released version 10.4 of the MyKeynote portal on Wednesday night.  We've made some great improvements in the user-friendliness of our graphs, bringing all the controls you need to tweak your visualization directly to the forefront.

 
We've also added the option to include only error datapoints in scatter plot graphs, clustered 3D bar graphs, page-level trending in long-term graphs, and much, much more!