Web Performance Watch

Contrarian View: Amazon Outage Proves the Promise of Cloud Computing

By Ian Withrow

Those who know me well can tell you that I’m hardly a frothy fan boy, indeed I’m a died in the wool skeptic. So it may come as a surprise to you to hear that I view the fallout of the recent Amazon Web Services (AWS) outage as a very positive sign for Cloud Computing. Sure some sites got taken down, including one of my personal favorites Quora. However, another favorite site of mine managed to survive the incident with comparatively minor hiccups: Netflix. This is the bright spot I want to highlight. I just happened to have a performance measurement for Netflix in my Keynote account. On the east coast starting at 12am April 21^st, Netflix’s performance for successful transactions stayed a consistent couple of seconds and was available 96% of the time. Granted this isn’t flawless execution, note that the 27 failed data points are all timeouts resulting in just a red screen. However, compared to what happened to many sites, this is outstanding. (Y-axis details obscured)

It’s not dumb luck that got Netflix off this easy. It’s the product of hard work and engineering time invested in building their Amazon Web Services deployment the right way. As Netflix has been touting in various cloud conferences this year, they’ve been forced to fully embrace AWS due to their tremendous growth. Basically they only run credit card transactions in their private network. To ensure they always have enough capacity (and incidentally are highly available) they have turned provisioning decisions over to their operational systems.  Whenever an Amazon instance is poorly performing they terminate it and get a new one.  Likewise if there is an availability zone acting up (like what happened) then they automatically switch over to another.

This is how real high availability has always been done in networking: ensure that you can automatically failover to logically, physically, and geographically separate resources.  Any real engineer will tell you that problems and failures will happen.  Your availability track record is not based on how frequently this occurs but how gracefully you recover from them.

Herein is the promise of Cloud Computing: namely the favoreable relationship between cost and failover capabilities. In a private network world you would have to build and pay for a lot of stuff yourself: multiple data centers, double the hardware, internet access connections on opposite sides of the building, etc.  Very quickly the cost of high availability gets prohibitive, locking out all but the deepest of pockets.  Netflix explicitly said at Cloud Connect they came to the conclusion that they, even with all their growth, just weren’t big enough to justify building their own network of redundant data centers.

Enter Cloud Computing.  Now having access to redundant data centers is just a matter of purchasing the right performance monitoring tools and the engineering time in programming your applications and operational systems to take full advantage of on demand resources.  In the end you only pay for what you use of the infrastructure, not what you might need as is the case when doing it yourself. That’s what the real shame and promise highlighted by this outage is, young companies like Quora and Foursquare could easily have done just what Netflix has done.  The barrier to entry here isn’t a huge budget but the knowledge and priorities to do the work. The next step of course after fully leveraging Amazon is to be able to failover to different cloud providers, I’d bet you $100 Netflix is working on exactly this right now.

In a way this drives home a point we’ve known all along.  Cloud Computing is not outsourcing, this implies a transfer of risk and responsibility. You, not Amazon or Microsoft or Google etc., are responsible for the performance of your applications whether they are in the cloud or not.  Cloud Computing is a powerful tool to increase performance and availability many fold while reducing costs, if it’s used correctly. If you don’t use the tool properly then an outage isn’t Amazon’s fault, it’s yours.  I'll leave you with this thought, Amazon seems to agree: according to Gartner Analyst Lidya Leong this isn’t an outage that generates service credits. (Quote at very end of article)

How To: Setup First Mile Monitoring with Cloud Application Perspective

By Ian Withrow

One of the most straightforward uses for our new Cloud Application Perspective (CApP) product is monitoring the performance of a website or API up to the edge of your private network. To be sure other use cases, which I plan to talk about soon, focus more purely on internal monitoring. However, this is a good one to start with if you are already using or familiar with Keynote services for monitoring user experience from outside your firewall. In fact if you already have Keynote then this is a great complement for your existing monitoring via our Transaction or Application Perspective products. At a high level you will learn the following: how to request a new CApP agent, where to place it, how to create and deploy a measurement, and of course how you can make use of the resulting data.

Ok but first, why should you care? Well there is a direct and an indirect benefit here. By monitoring first mile performance you can quickly identify if performance problems are inside your data center or outside of it. This need not be limited to consumer facing applications, you could easily monitor the first mile performance of B2B applications or even internal/partner applications. In fact, these scenarios might be more compelling as they are often associated with SLAs or clear expectations of performance. Moreover, I talk about monitoring a private data center here but it could just as easily be a private or public cloud. This brings us to the second benefit: proof. In short, gathering first mile performance data with CApP gives you evidence of where a problem is located. Who doesn’t want to avoid finger pointing?

Hopefully by now you are asking how you can get one of these. Well for starters you have to be a Keynote customer, but assuming you are you can easily request one online in the Keynote Service Center. First, select the “Agent” tab. If you don’t have this tab in your portal then you haven’t been setup for CApP, we’ll address that in a second. Second, click on “Request CApP Agent”. Finally, fill in and submit the resulting form. The only input that involves some research is supplying the public IP address range from which the agent will contact our servers. A full Class A is ok though. If you don’t know the public IP address of a given device then a number of free websites can answer that question, like IP Chicken.

So what if you have Keynote service but haven’t signed up for CApP? Well in the “Add Measurement” screen you may have noticed that you now have a CApP section. You can select this option and choose “Getting Started.” This takes you to a form that will nudge your sales team to get in touch with you to get your contract amended. I know not very SaaS’y, but we wanted to get version 1.0 out quickly. In the future, look for this process to be further automated.

 

After going through all of this you’ll get an email from us with a download link and a license key for your brand spanking new CApP agent. So where to put it? Typically you’d deploy CApP for this use case behind your firewall or possibly in the DMZ, this would provide visibility up to the edge of your network. You can then compare this data with measurements from Keynote’s global test and measurement network. Since CApP is designed to operate on end user machines for remote testing, it doesn’t need much in the way of horse power and can easily be run on shared or virtualized systems. Fair warning, it is Windows only right now. The setup itself is a breeze; you basically speed click through a Windows installer. You know, where you quickly click “yes” and “next” while you unwittingly consign your first born over to Keynote? After that there is a simple dialogue window where you enter the license key we provided you. From here on out CApP runs as a Windows service behind the scenes and only requires outbound Port 443 access, so it can communicate with our SaaS portal.

So speaking of instructions, how do you create a measurement for CApP? CApP, if you didn’t already guess, generates synthetic measurements based on predefined instructions we call scripts. You use KITE to create an Application Perspective (ApP) script. In fact if you already have ApP scripts you can skip this step and use them with CApP. For in depth KITE tutorials or to download this free tool, then check out its web page here. What I’ll show you here is how to create a basic script using the point and click functionality of KITE.

Launch KITE and choose the “Record” drop down menu, not the big red button tempting though it is.  Then select “Record Simulated Browser Script”.

KITE will ask you for a URL to start out with, I chose Keynote’s website for this demo.

Next you’ll be presented with the KITE web browser, which loads the URL you provided. Now simply navigate through this browser session as normal and click “stop” once you’ve completed the transaction you wish to record.

Once you are done, KITE will test out the script you’ve defined right away. Afterwards you can add advanced steps, logic, rules, or edit any errors that are present. If you’ve purchased CApP you should know that you also have a support contract with Keynote, this allows you to get script help from us among other things.

 

For now let’s keep things simple and just save and exit. With this we are ready to provision a measurement in the Keynote Service Center. CApP is provisioned like other Keynote measurements, except that it has its own product box, shown below. The only other important difference is that measurements can be as frequent as 1 minute or infrequent as 60 minutes.

With a measurement, like this one, setup you can make use of the data in all the current ways that are possible in MyKeynote, whether it’s the dashboard, graphs, or alarms. Below I copied a graph that highlights the value of CApP for this particular use case: a graph that compares inside (You) and outside (Me) measurements.

Just to wrap up, you’ve seen at a high level how to setup CApP from scratch in order to get improved visibility into how your website or API performs up to the edge of your private network or cloud. This is a great addition for any customer who is already monitoring their properties externally.

IE No Longer #1 - Are You Keeping Pace?

Today the pace of change in how your customers are accessing your Web site is breathtaking.  The massive rate of change in the past 12 months in browser market share is staggering. Most notably, Firefox 3.5 is now used in Europe more than any other single browser version, and usage of Internet Explorer worldwide is dropping faster than Middle East autocracies.

Why is this important? Browsers matter when it comes to web performance. Different browsers display different behaviors, especially with regard to the AJAX and Flash applications used to create rich Internet experiences. These applications are dependent on client side scripting and the initialization of plug-ins which requires understanding how the browser behaves as it interacts with a Web site, not just the performance of the Web site itself. And although application behavior can be tested across browsers within your development and QA environment, performance issues often manifest once these applications are moved into production. Additionally, network latency and third-party content interactions at the “last mile” can contribute to an unacceptable user experience.

Keynote customer LinkedIn recognizes this and monitors its performance across both IE and Firefox using Keynote Transaction Perspective. Keeping a vigilant watch on the performance of their services from the end user’s perspective is a priority so they can quickly identify and resolve issues.

Is it also yours? I’d love to know if and how you are monitoring your rich Internet applications across multiple browsers.

Best in Class for Web & Mobile Performance Monitoring

These are heady times in the performance business. It's not the stuff that grabs headlines, but those who understand how the cloud works, know that performance can be its achilles heel.  Just today we hosted a Webcast with Cars.com, on the impact of 3rd party content on Website performance. The registration #s were through the roof.  There are a lot of site owners and Web ops folks who get it as evidenced by the questions we were getting. They are concerned by what all the ads and widgets are doing to performance.  There is tacit acknowledgement that what makes the cloud great - interconnectness - also sows problems.  Knowing when critical Web servcies and APIs are down is more important for a deeply networked application then one that merely relies on inhouse resources.  (This Webcast will be available on-demand shortly. You can download a white paper on the subject now.) 

So it gives us great pride to be acknowledged for what we are doing when it comes to cloud performance monitoring.  What started off as just Website monitoring when Keynote debuted in 1997 has gone on to encompass so much more. Specifically, what mobile has brought is ubiquitous broadband on the go and sites such as Facebook and eBay are seeing such a high percentage of traffic and dollars coming from mobile, they've invested heavily to keep up with the demand.  We've responded with a suite of products to monitor and test everything from mobile content and applictions to networks and even sports stadiums.  

With on-demand services for both Web and mobile, we believe we have the most complete offering for cloud performance monitoring in the industry.  EMA, a tech analyst firm, agrees, hailing our offerings as "best in class".  (Read the full report.) 

 

What do AdSense, Friend Connect, and Tribal Fusion have in common?

All three of these services were demonstrated by Google researchers to impact Website performance by double-digit percentages. They are not alone, of course, but representative of the kinds of things that can create bloat on a Website and degrade user experience.  Of course 1 person's bloat is another's person's manna so the best thing to do is 1) find out what 3rd party content is featured on your site and 2) keep watch on how these 3rd party services are performing.  As Web operations teams well know, knowing is half the battle and in no case do you want to be caught flat-footed, without knowledge of what's happening on the Website you are responsible for.

We've made the job of tracking performance for 3rd party content easier for your Website.  It's called Virtual Pages and it works just like the other Web performance monitoring services you've come to rely on from us. Our customers love it and so it gives us satisfaction knowing we're helping them tame the 3rd party content tiger.

The Website Sausage Factory and Impact on Performance – a TechCrunch Case Study

By Ian Withrow

As we’ve discussed in various blog posts, websites are like a sausage.  Ok maybe not so directly, but like a sausage they are made of ingredients than come from many sources even though they are presented in one tidy package to the user.  Today I’m going to break apart the sausage that is the TechCrunch blog, show how this can be easily done for any website using KITE, and as a special bonus show how a sausage maker can monitor all the pieces of their links using a Keynote technology called Virtual Pages. 

TechCrunch Composition

First a fair warning: just like with sausage making, finding out what is inside your favorite website is not always pretty.  If you feel you are of a squeamish disposition then you have been warned.  Second, note that all details in the post are from the time of writing and the balance of ingredients is likely to change overtime.

TechCrunch is a behemoth of a site, weighing in at just over 4 MB of data, 329 page elements, and a whopping 65 domains.  About half of this comes from them directly or really via Wordpress who is evidently the platform they use to power their blog.  The rest comes from over 20 3^rd parties.  You read that correctly, 65 domains in total and half the content originates from someone else.  After direct content the next biggest category is from social sites; Facebook, Twitter, and tools related to these properties total about 1 MB.  Google is about a 500 KB and the ‘Misc.’ category of various 3^rd party tools that TechCrunch uses to improve user experience is about 200 KB.  Ads and ad related content are about 180 KB.  Below is a snapshot from KITE breaking it out for you.  (Note you need to complete the process in the next section to actually get this view)

Now it’s time for a few fun observations.  The amount of content from Facebook and Twitter is huge!  Each alone is bigger than most websites are in total.  Digging into this is unfortunately off topic for this post but it is definitely something on my radar screen for the future.  Another interesting area is the level of user tracking that goes on.  I could identify at least 6 different 3^rd parties that were tracking TechCrunch visitors, not including Google and Facebook.  TechCrunch knows what you are, if not who.

Finally, while these stats make it seem like TechCrunch is hardly advertising, understand that TechCrunch is a very, very, very long page (vertically) and all the ad content is at the top where the user is most likely to see it.  They aren’t dummies giving away their yummy sausage for free.

Scripting Sites for 3^rd Party Monitoring & Analysis

To make sense of this mess I used Keynote’s KITE product.  There are a lot of other great, free products out there that one can use to view all the content and domains of a page.  However, KITE has the ability to permanently parcel out these domains into what we call Virtual Pages for ongoing monitoring and analysis.  Note this section won’t be a detailed how-to; I’m going to focus on highlighting what is possible with the tool.  After which you should be prepared to experiment or watch this training video depending on your learning style.

After downloading TechCrunch in KITE I organized the content by domain as shown below.

This lets me easily see the composition and breakout of a page in a manual fashion.  If I just want to see the domains I can simply collapse the domain groupings.  There are tons of options that I can add to this view like content size and various time breakouts based on my interest.  Here is a complete list:

With just this you can see that I can casually learn a lot about the page.  However, if I’m serious about how TechCrunch and its 3^rd parties perform then I need ongoing data points.  If I’m going to gather a lot of data then I don’t want to do this parsing and analysis manually, it just won’t scale.  The solution is to organize this content into permanent logical pieces.  For example, in a simple scenario I’d carve out a Virtual Page for my advertising so I could monitor and analyze the performance of that content separately from my content with Keynote.  As you can imagine the more complicated your site becomes, the more important this exercise is.  True you can always pick through a waterfall manually to see who did it in the event of the problem but if you want to have ongoing data about 3^rd party performance or be proactive with alerts then you’ll need something like Virtual Pages.  The nice thing is once you’ve designated content into a Virtual Page you can monitor and analyze it like a regular page.

Let’s discuss how I broke-up and organized TechCrunch.  Please note I’m not holding this up as the standard for the best or only way to use Virtual Pages.  One thing we need to keep in mind is cost.  Each page (virtual or otherwise) adds to the cost of the measurement and so in the real world we probably can’t go hog wild with these.  Given an unlimited budget I’d define a Virtual Page for each 3^rd party, possibly even one for each domain if I was especially crazy for detailed data.  My guess is you live in the real world and even if your site isn’t as complicated as this one you’ll need to create some buckets.  Most likely you’d start with prior experience, defining Virtual Pages where you knew or suspected there was a problem.  Here I simply broke the site into the following logical categories:

• TechCrunch direct content plus AOL
• Google (but not Google owned advertising)
• Facebook
• Twitter (and related tools like Postup)
• Wordpress (even though this is the core of the site, I want to evaluate my vendor here)
• Misc. Tools and Widgets for the users
• Analytics and User Tracking
• Ads and Ad related content

Why no CDN category? We certainly encounter CDN’s here but each is tied to a specific 3^rd party.  Facebook has its own CDN, the ad platforms have CDNs and so forth.  So instead I left the CDN’s with their respective masters.

Here is a brief teaser for how this is done in KITE

Step 1) Pick the URL you want to virtualize and run the page once (we did this already)

Step 2) Right click on the page in question and select ‘Insert Virtual Action’

Step 3) We now have a new Virtual Page at the bottom of your script.  Right click ‘Match Page Elements’ and select ‘Add URL Match’.  Here I’ve used the naming convention “vp:TechCrunch” to distinguish Virtual Pages from real pages.  You can name them anything you want in practice though.  There are other options that you can use to construct Virtual Pages, such as content type, that have interesting possibilities but to address 3^rd party content, URL seems ideal to me.  As you see below I’ve created a list of URL matches that should capture all the differently named TechCrunch domains.

Step 4) In the Script Properties Editor you can create the settings for each URL Match.  Note in my script I used a variety of regular expressions so that I could get away with far fewer rules than the 65 domains and still cover all of the page content.

Note that to do this I never had to write any code or do any advanced scripting.  It was all point, click, and form completion.  Hopefully by now you can see how easy it is to create Virtual Pages in KITE and have an idea of its possibilities.

How can I Benefit from Virtual Pages?

The obvious and immediate answer is you can now isolate and monitor the performance of certain 3^rd parties or subsections of you website.  If Facebook slows down you’ll know immediately and explicitly that this is case regardless of the overall impact on your performance.  Moreover, you can easily track and directly report on the performance of these guys overtime without needing to manually crunch the data and objects yourself.  Another interesting possibility is you could monitor your own additions to your site to see how they fair.  Finally, another angle might be to isolate and monitor all the Javascript that your site utilizes.  There are a countless number of ways that Virtual Pages might be used, and my list probably just scratches the surface.  Have fun with it!

 

Elevate Performance Awareness in your Company by Publishing Keynote Graphs

By Ian Withrow

I’ll warn you right off this isn’t my typical post about a concept or a market trend.  I’m going to straight up show you how to do something cool in MyKeynote that’s available right now: export a saved graph.  Why you ask would anyone want to do this?  It’s the perfect way to raise visibility within your organization about the valuable information that your Keynote measurements generate.

In many of the organizations I talk to, Keynote data is an invaluable tool for IT and operational groups but it’s a shame in my opinion that this information doesn’t get out to groups like product management and sales as readily as it should. So today I’m going to show you how to create a graph in MyKeynote that will provide an ongoing view into the performance of your website or application.  I’ll also suggest some types of graphs different business functions might be interested in.  Finally, I’ll spit ball some ideas on how you might distribute this information.

First Create and Save a Graph

First, before we can become heroes (or villains who dare speak the truth) we need to figure out how to actually share a graph.  The first part of this is a routine MyKeynote process so stick with me or skip down to the next section if you already know how to save a graph.

 To start, we head to the graph section and perform the following steps:

1. Select the measurement we want to graph.  This could be more than one measurement if we wanted to make a comparison.  I’ve gone with the Keynote Business 40, The Keynote Business 40 Internet Performance Index (KB40) measures the average download time for the home pages of 40 important US-based business Web sites.  To find out more go here.
2. Pick the graph type, time history is a great option to show performance overtime.
3. Time period is up to you but its important use to a relative period if you want your graph to have permanent relevance.
4. Finally I’ve elected just to show performance to keep things simple in this post.
5. Ok let’s generate the graph, this graph is a big one with a lot of data and measurements in it so it takes some time to generate.  Keep in mind how long your audience will wait when you setup your graph.

Once generated, save the graph using the menu in the top right hand corner.

The next step is important not to mess up so read carefully.  Name your graph whatever you want and click save.  You’ll get a dialog window like the below.  If you want a ‘relative’ graph you need to choose “Cancel”.

Retrieving the URL for your Graph

Ok now we are done with MyKeynote and for the experienced users out there you may note that we haven’t done anything unusual or new yet.  Now for the trick!  Keynote actually publishes your saved graphs in two heretofore undocumented RSS feeds.  One feed publishes the entire MyKeynote graph page, complete with legend and the other publishes just the graphs themselves.  Those feeds are:

1. For the full graph page: http://my.keynote.com/newmykeynote/mykeynoterss.do
2. For just the graph:  http://my.keynote.com/newmykeynote/mykeynoteembed.do

You’ll find that visiting one of these feeds requires you to login in to your MyKeynote account.  Do so with a browser like IE or a feed reader that can handle authenticated RSS feeds and you’ll see something like this.

The URL behind Keynote Business 40 and any other graph is permanent (as long as the graph is saved) and can be requested by clients without logging in.  Note that you can also get this link from the MyKeynote home page “Saved Graphs” widget.  There you can find an RSS feed button that will provide the same view above.  Below is an example of what the link provides:

Putting your Data to Work

This is a handy feature to be sure but how best to make use of it? Well at the simplest level you could simply distribute the link internally to interested parties.  Better might be to embed this page within another page, such as your companies Intranet.  Perhaps the cleverest idea I’ve encountered so far is to actually serve different graphs based on the organizational function of the user logging into an internal knowledge base site.

Presumably different groups want to know different things.  A product manager or marketing manager needs to know how their key pages or transactions are performing and for a large company there may be multiple product areas with different interests.  Developers may want to track their pages to see the impact of any code changes.  In contrast sales may wish to see the company’s page benchmarked against key competitors.  Any outage in a competitor’s site can provide an immediate piece of FUD to use.  Similarly a problem in their site is a heads up that they may be getting customer complaints.  As you can see timescale, graph type, and level of detail can easily vary based on the user’s interest.

As mentioned earlier it’s important to consider load time.  Generating graphs of measurements from many agents over a long period of time can take a few minutes.  You may want to limit the scope of your graphs to something that can be quickly generated to maximize the amount of attention it gets from viewers.  A drill down graph can easily be referenced to for users who are willing to wait for more details.

This feature is new and we see a lot of areas to make it even better.  Let us know how you make use of this feature and what improvements you think are most important to improving its impact.

Adoption rate of 3rd party web content

A colleague of mine pondered the other day about the prevalence of 3^rd party content in web sites.  This is a great question because the performance 3^rd party content has a substantial impact on user experience, unfortunately it is typically negative.  But to really answer this question accurately you need to define what subset of sites you care about.  Even if you could analyze the 200 million some domains out there this is probably not what most people are interested in.  So I modified the question to: what portion of successful web sites have 3^rd party content? Happily, my colleague was good enough to provided a white paper produced by anti-malware company Dasient that studied the Fortune 500 recently on this very topic.

First, before we look at their findings let’s define 3^rd party content.  That’s basically any content that a user’s client receives that wasn’t provided by your domain.  For example: Ads, APIs/Widgets/etc, and analytics all count but having a separate domain for your images does not.

Back to the Fortune 500, Dasient found that 75% of companies had what they called 3^rd party javascript and 42% had advertising.  Surprisingly even many manufacturing companies had advertising.  See graphics below for the breakout by industry vertical.

Fortune 500 3^rd party javascript take rate by vertical:

Fortune 500 Advertising take rate by vertical:

Here is a link to the survey if you want to read their analysis but one thing struck me.  You could use the different behaviors of verticals to explain the results behind these charts but you could also use traffic.  Verticals that seem like they would have more heavily trafficked sights were far more likely to use 3^rd party content of one form or another.

The investigate further, I decided to do my own little mini-investigation of highly trafficked websites.  To do so, I started with Google’s US top 100 list of most visited web properties.  I then decided that everyone who the list said had advertising was automatically on the ‘3^rd party content present’ list, except for companies actually in the web advertising business like Google.  I also exempted sites that I simply didn't believe had any advertising despite what Google said, like the NIH.  To inspect the remainder quickly and efficiently I needed to make use of web development tool.  KITE has a handy feature whereby you can group content by domain of a particular a page.  Then it’s a simple task of visually inspecting the domains to see if any come from a 3^rd party.  As an aside, KITE has this feature so you can setup Keynote monitoring and alerts for 3^rd party content, but I digress.  Below is a KITE screen shot with Yahoo’s content sorted by domain.

 

 

As you can see, while most content at Yahoo comes from them there is one object that comes from a third party, underlined in red.  In the end I found only two players who didn’t have 3^rd party content: a few Google properties and Wikipedia who is a non-profit and happens not to use a 3^rd party analytics javascript.  Even more traditional organizations in this list, like HP.com or NIH.gov, had 3^rd parties in the mix.  I then expanded to Google’s global 1000 list and while I wasn’t masochistic enough to check everyone, the trend held at the top and the bottom of the list.

So what have we learned?  The vast majority of top companies in the US have 3^rd party content on their websites and pretty much any successful site will have it.  We have to be careful here to get cause and effect straight though.  I hypothesize that the virtues that have helped to make top web properties so successful has led them to embrace 3^rd party content.  It’s certainly not a consumer only thing.  Even sites, like Cisco.com, that couldn’t be more business oriented use 3^rd party content.  This all makes sense.  As the public cloud (internet) get’s bigger and more diverse you are going to want to leverage specialists to deliver rich content, integrate with 3^rd parties like Twitter that can help get the word out, and analyze the visitors to your page carefully.  If you don’t then you are likely to miss out on traffic and repeat visitors.  So if the question is what successful websites use third party content then the answer is Yes.

Monitoring for the Long Haul

As Ian discussed in his excellent recent post on the business value of Web Performance Monitoring, Google demonstrated that even a sub-second response time delay has a lingering effect on user engagement, even weeks after the site was sped up again.  A delay of this length might not be severe enough to warrant a critical alert (see this post for more on alerting), but it has a measurable impact on several metrics including revenue per user and user satisfaction.

Around the same time as Google demonstrated this result, a study conducted by Forrester Research and commissioned by Akamai found that most consumers expect a web page to load in under two seconds—compared to an expectation of under four seconds reported three years earlier.  More than half of respondents said that quick page loading was an important factor for continued site loyalty.

For reasons such as these, I believe it's important not to undervalue medium-term and long-term performance monitoring.  Keynote has several capabilities in this area including long term trending, time history graphs, and service level reports.  Let's take a look at each of these capabilities in more detail.

Keynote retains aggregated performance data and provides long-term trend graphs for up to two years.  For example, I can tell that the performance of the Google home page has been fairly consistent over the last couple of years, although lately performance seems to be hovering closer to 0.5 seconds than the 0.3 second range exhibited in late 2008/early 2009.

For medium-term analysis, Keynote retains raw performance data for six weeks that users can analyze across a variety of dimensions using time history graphs.  In the graph below, I've broken out the response time for the Google home page over the last four weeks into times for DNS lookup, connection, first byte download, and content download.  As one would expect, of these components, the one that takes by far the most time is content download.

Finally, Keynote produces service level reports for up to the last month of performance data, in a format that is designed for communication to management or clients with whom you have a service level agreement in place.  Service level reports allow you to compare the current period with the previous period.  Below is a comparison between this month and last month of the performance of a major financial institution's home page.

In this instance, average performance over the last month was close to the average for the previous month (within 0.02 seconds), although there appears to have been a small spike over the last few days.

Generally, in order to understand the specific impact of performance on your business, you should monitor site performance in conjunction with monitoring business metrics such as customer retention and engagement.  Have you found that that site performance is a leading indicator of business performance?

Web Performance Monitoring Business Value Part 2

By Ian Withrow

In my last post on the business impact of Web Performance Monitoring (WPM) I looked at how WPM helps the bottom-line by letting you know when something breaks.  This time I’m going looking for trouble to see how WPM can be used to proactively improve business performance.  To do this I’m going to bring in another topic I wrote about earlier: distribution of WPM data and how it’s not in fact normal.  Specifically I’m going to look at performance data that in aggregate looks good and see whether there is improvement worth pursuing.  Here’s a spoiler, because it’s not normal then using metrics like mean response time leave plenty of opportunity for improvement on the table.

An immediate objection I’m going to get is that you can create a new set of alarms or metrics after the fact for any specific root problem after you’ve encountered.  Thus the argument continues all cases eventually convert to the analysis I made in part 1 such that operations can just behave reactively.   Operations certainly should create new alarms and monitoring points based on experience.  However, there are some limitations to this thinking.  First, if you watch and alarm on too much, your overhead and false positive rate can get prohibitive.  Second, serious web properties are a complicated and ever changing so the attitude that one could quickly get to a state that they watch everything of consequence strikes me as hubris.  Finally, there is a theoretical problem in that non-normal populations defy prediction so creating alarms to cover every case may not be possible.  That said I still hope you will discover things to do proactively in this post.

So now let’s return to our anonymous Tier 1 web property from the probability distribution post and look at the USA performance data just for their very important splash page over the past two weeks.

Here are a few important stats, the number of data points is 3276 and the mean is 2.029 seconds.  That’s really good performance and I don’t think I’m out on a limb here when I say that number alone probably isn’t throwing up red flags, accordingly let’s assume this is their baseline goal for performance of the splash page in the USA.  Note that it’s a pretty heavy page, not Google.  Still we can tell from the distribution that some users aren’t enjoying 2 seconds.  From my last post, I’ve borrowed my estimate that 2 seconds means a 15% reduction in the value of your website.  Then I’ve made some rough estimates of how bad short and longer delays are, to construct the table below. 

We can see that 23% of users didn’t get the target performance rate and that this translates into about a 2.7% impact on the business value of the web property for this time period.  Depending on how conservative or aggressive you want your estimates to be you should fall between 1% and 5%.  If you are a $100MM business, let alone a billion dollar business like this one, you might not be happy with this.

Even if a few percent alone for two weeks doesn’t get your attention there is another matter we should be concerned about: long term impact.  At last year’s Velocity Google revealed a small but important tidbit from their study of page delay on user value.  Users in their 6 week study who were impaired just 400ms continued to be less valuable after the study concluded. 

 

In other words you can cause lasting damage to your users via poor performance.  Unfortunately we can only speculate if just a single incident can do this to your users or if you must hurt them again and again before you earn their lasting mistrust.  Likely the truth is that it depends, but for the time being let’s see if we are hurting the same users over and over again.  For example, perhaps we are slower for users in particular regions.  Keynote has a premade graph for this:

 

Ruh-roh, Raggy!  Indeed it appears that we aren’t giving Chicago and Boston area users the same quality of service.  Next we would explore why this is and if it’s a simple fix or else look at the business case if the solution requires some money.  Keep in mind I didn’t cherry pick this data population.  I picked my guinea pig weeks ago before I had a clear plan for this post.  This is a real problem for a serious company going on for the last few weeks at least.  What gremlins might be lurking in your data?

The key takeaway here though is not that you should setup alarms for regional performance, although that may be a good idea.  What causes some users to be above the mean will vary from case to case, and certainly even here not all of the 23% can be explained by the Chicago/Boston phenomena.  Moreover, just looking at the mean for say every major metro area will miss other factors affecting a significant portion of your users.  The point is that watching the distribution of your site proactively will indicate problems that alarms based on statistical metrics can’t capture.  And just like compound interest in your bank account just a few 3% bumps in revenue, or whatever metric is near and dear to your heart, can very quickly start to add up.  Finally, it is importantly to pursue these issues in a timely fashion as bad experiences can have a lasting impact on users.

 

Photo by Charlie Brewer