Web Performance Watch

Did You Fall When Facebook Stumbled?

Thursday's outage of Facebook was notable for the ripple it created across the Web. Many online businesses were impacted indirectly as a result of failing content integrated into their websites. The nearly ubiquitous "Like" button and Facebook's other social plugins became a drag on the performance and availability of some websites. We saw notable failures across media, travel and retail sites. Yet, other some appeared to have avoided significant impact.

The websites of CNN, USA Today and Expedia--members of the Keynote Business 40 Index--demonstrate how different approaches to integrating the Facebook social plugin can be the difference between disaster and only a minor setback when 3rd party content fails.

Here are graphs of their performance and availability during the outage (along with Facebook's for reference):

Notice USA Today's performance (the blue line). Their home page continued to happily build despite the lack of content availability from Facebook.

Incidents like this are good reminders of the importance of using page construction best practices that accomodate third party content failures. If you feature ads, widgets or plugins on your site, do they represent a potential single point of failure if the service becomes unavailable?

Even if you take care with integrating third party content into your pages, it's important to continuously monitor each source indepently if your site changes frequently. Keynote helps companies uniquely monitor 3rd party content such as ads, social widgets and plugins so you can quickly take action to mitigate performance issues and focus business improvement with actionable data.

So how did your site perform Thursday evening?

Another Win for Keynote Customers

When Keynote introduced Cloud Application Perspective last year, we believed that it addressed a critical gap in the way companies monitored the performance of their Web applications. As “cloud” technologies have emerged, websites have become more a coupling of components from multiple assembly lines, rather than a product manufactured in a dedicated IT application factory. Monitoring the performance of a website in its end-user state—the sum of the many parts—makes sense. But when issues arise, quickly finding their location in the “supply chain” can be a challenge.

What if you could drop something into that “supply chain”, at the key points of potential failure, giving you insight into your Web app’s performance? What if this was so easy to install, a non-IT person could do it in minutes? And what if the cost was just pennies per measurement?

Last week, the SIIA CODiE awards recognized Keynote Cloud Application Perspective for best Cloud Application/Service, along with Google and Scribe Software. They did so because Cloud Application Perspective offers customers a radically new way to monitor their websites and applications. Unlike traditional APM options, Cloud Application Perspective is uber-portable, monkey-simple to implement and really affordable.

Although we’re honored to have won this award, our customers are the real winners. Taking advantage of the cloud can be complex. Monitoring it should be easy. If you’d like to see how easy it can be with Cloud Application Perspective, get a free trial today!

Cyber-Awesome: Killer Performance Helps Drive Record Revenue

For years, Keynote has been monitoring the ‘net’s best websites and reporting on notable outages during the peak Black Friday and Cyber Monday online shopping period. Typically, we find that many online retail sites—both pure Internet and those with brick and mortar stores—experience some issues. But the trend has been towards better performance and higher availability. This year marks yet more progress toward the goal of a trouble-free and virtually instant shopping experience. And the numbers suggest the payoff for retailers has been huge: a record setting $1.25 billion in online sales for Cyber Monday, 2011.

The Keynote Top Retailers Index tracks 47 online shopping destinations. This year, the average website response time was 3.05 seconds. According to a study by IBM, online sales peaked at about mid-day, and spiked again later in the evening. Average response times across the Top Retailer Index reflected this pattern as peaking demand slowed sites. But the absolute impact even at its worst was only about 1 second of delay. That’s great performance across the industry as a whole.

Additionally, the availability of these websites remained extremely high throughout the period. Out of the 47 sites tracked, only 10 (21%) experienced any downtime at all and their average availability was around 99.0%.

So how did online retailers get to this pinnacle? We see that customers are maturing the technology, processes and culture of their Web Operations around a passion for performance and deep preparation. One example of this is Keynote customer Karmaloop.com who rang up their biggest day in company history and over 100% more sales than the same period last year. Though careful planning and web load testing, they were able prepare their site for the demands of such a dramatic increase in traffic and order transactions.

The improving trend in website performance throughout the increasingly demanding Black Friday and Cyber Monday period is encouraging. But it also means that if your site experiences failure or less than stellar performance, your competitor is in a much better position to attract and retain your abandoning customers. Congratulations to the online retail industry for hitting a remarkable milestone over the past six days. Keep pushing performance higher through December!

Deconstructing the Target.com “Fail Doggie”: A Keynote Perspective

Have you met the Target.com “fail doggie?”

He’s cute, but if you are a Target customer, you don’t really want to encounter him in a browser.  I met the “fail doggie” while using the MyKeynote portal to research the nature and extent of the major outage that occurred on Tuesday, September 13^th 2011.  That was the day that Target began allowing online purchases of “Missoni by Target” items.

A lot at stake

According to one source, the Target site re-build from scratch, launched just weeks before this incident, drew on the talents of over 20 vendors, including many of the biggest names in the e-commerce technology space.

Imagine spending millions of dollars and two years of time to “create a more user-friendly, reliable experience” and then have this happen.  Not fun.  When a major event like this outage happens, it can be difficult to get complete details from any one participant.

Passions are high when a crisis like this occurs.  Where can you go to get an objective vantage point from which to make accurate assessments and analysis?  At Keynote Systems, we’ve long been looked to as a neutral third-party with accurate and actionable Internet and mobile performance data, and that day proved to be no exception.

In this blog post, I’ll explain how I used the tools that every Keynote customer has in MyKeynote along with measurements being run for two of our public web performance indices to determine what users were seeing that day and the following morning and to determine just how extensive the outage was.

Target.com appears in a number of our public index measurements, so I had several to pick from – note that we never publish insights based on a customers’ private data, but only based on publicly available data that we collected without payment by any company. Two were particularly useful for figuring out what was going on and capturing screenshots as the day went on. The first measurement visited their home page only, while the second arrived at the home page and then performed a multiple step transaction, just as a customer would when shopping, placing items in a cart and checking out. 

First a brief bit of background: both scripts were written in the Keynote Internet Testing Environment, known as KITE and both measurements were being run with our real browser product, Keynote Transaction Perspective. I point that out so you’ll know that none of what you will see here was retrieved by a “bot” or other emulation system; we were getting the experience of a user launching an actual Internet Explorer browser to go visit Target.com.

How things went down that day

As word of the outage quickly spread, we started taking calls from various news media companies asking if we had data.  We took a look at the home page monitoring scatter plot below and saw what looked like a brief spike in response times followed by a restoration of reasonable response times shortly thereafter. 

(click image to view full size)

At first glance, it was tempting to say that the outage had been brief and thus was relatively uneventful.  But a deeper look at the chart revealed telltale signs that something was not right.  Notice the distribution of the dots before and after the spike?  See how they are randomly distributed somewhere between two and six seconds before the spike (time on the network is on the left scale) but then they are all tightly packed down below the one-second mark afterword?

Each of those dots represents a full visit with a real browser, so I could click down on any of them to see a listing of what was on each of those pages and how long it took to get to the browser.  I did that eventually, but much like an operations team technician would be at such a moment, I was in triage mode at this point.  The first thing I wanted to do was to look for clues as to why those dots were organized as they were.

First, I hovered my mouse over a datapoint from before the spike in performance.   This would be my “normal” baseline to compare to.  Notice that the page contained 147 elements (separate downloaded objects) and a total size of about 2 MB.  The time on the network to download the page and all elements was 5.156 seconds. 

Next, I took a look at the two red triangles, which represent pages where we know there was an error of some sort or another:

In this case, the element count and page size were lower than the baseline but the time to download was skyrocketing to a number eight times higher. The lower object count and page size were due to timeouts being hit… we couldn’t get all of the objects into the browser before time was up, so the agent running the browser quit trying and reported the error.

What was really interesting was the object count from the very next green dot after the red triangles.

That data point showed only one page element and a very small download size of 640 bytes. 

One element?  I knew that if we had downloaded only one element, that it must be the base page of html, but a page with only 640 bytes couldn’t possibly have much to say.  That was my first clue that visitors were probably getting raw error messages.

I quickly scanned over the remaining “spiky” datapoints after the incident began and found more of the same: just one page element and a size of 639 or 640 bytes.   Here’s the last point caught during that initial spike:

So that was all fairly consistent with a site failure; something went very wrong around 8:00am EDT and the server took a long time to send out a very small page that was probably just an error message. 

What about all of those green dots hugging the bottom line after the spike had subsided?   I hovered over a few and got the same thing each time: 5 page elements and a page size of 31550 bytes. 

This wasn’t some random subset of the real page and no error was being recorded, so clearly these speedy responses were something altogether different. 

Now it was finally time to start drilling in for some details.  I clicked one of those data points and made my way to the page detail waterfall graph:

 

(click image to view full size)

Welcome to our waiting room

I hovered over each of the bars and noticed that each object came from a folder called “spawaitingroom.” The image files consisted of a red stripe, a Target logo and a photo of the Target dog posing next to a tool box.  I had met the “fail doggie” for the first time:

 

(click image to view full size)

So let’s recap what I had learned so far:

1. Initial data points have no page objects… just the base page, and that base page was TINY (640 bytes) in comparison to the pages that preceded it, but the amount of time the server took to return that page was HUGE.
2. The datapoints following the big spike downloaded much faster and had larger base pages but only had five page elements instead of the 140+ found in the normal pages.
3. Drilling in on the five-element page datapoints, I found that all five of the objects came from a folder called “spawaitingroom” – the “fail doggie” was part of a “waiting room” feature that was being served in lieu of the real home page. 

Getting the whole picture: what were people actually seeing?

I was making good progress, but I still had a lot of questions to answer.  I could guess that a Target dog next to a toolbox was probably some variation of an “under construction” page, but I didn’t have the text of the page yet.  I really wanted to know what those pages looked like but all I had were some pieces to the puzzle.

The thing that was complicating my sleuthing was that neither the original 640-byte server error pages nor the fail doggie pages were being sent as an “error” (http status of 4xx or 5xx) – they were being sent as successful pages (http status 200).  That prevented me from getting as much diagnostic information as I otherwise might have.  I could poke around at each scatterplot and piece together what I was seeing, but without an error, I wasn’t going to have the html from the page or any screenshots, both of which MyKeynote stores on a fatal error. Fortunately, all was not lost.  There's a benefit in more than 400,000,000 objects per day stored away inside MyKeynote.

Web Content Trending to the rescue

Here’s where the other measurement that was scripted to go five steps deep into the target.com site came in very handy.  The second monitoring script was set up to do the following:

1. Go to the target.com home page.
2. Perform a search for “lil wayne”
3. Filter the search results to the “music” category.
4. Click on the first album in the resulting list to view the details.
5. Click the “add to cart.” button on the details page and then confirm that “1 item added to cart.” appears on the screen.

Without the real site being served up, there was no way the script could complete the search for an album.  When the Keynote agent piloting the Internet Explorer browser went to find the search box and type “lil wayne” into it, there was no search box. The resulting error provided a steady stream of screenshots of the home page throughout the day.

Let me explain a little more about why I got the screenshots.  With Keynote’s Web Content Trending option turned on, every screen is proactively captured and if an error is detected in subsequent steps, all captured screenshots are saved to the MyKeynote portal to support troubleshooting by our customers.  If there is no error, the proactive screenshots are discarded before ever being sent to the database.  Every time the second step failed, (which was on EVERY visit at this point), the screenshot of the home page taken on arrival was stored.

To get those screenshots, I simply had to drill into the scatterplot chart and I saw this:

(click image to view full size)

I clicked on that “E” which is the error recorded for the second step and saw the page details screen below. I have added callouts so you can see where the links to the screenshot and html are:

(click image to view full size)

Things were about to get a lot clearer in a hurry. 

The wrong kind of "direct communication"

I clicked on the thumbnail of the screen snapshot and sighed as I saw what visitors had seen in the moments when the site became unusable around 7:58-8:00am:

We captured the above screenshot at 8:01am.  It shows what the 640 and 639 byte html pages with no images looked like in the browser.  This is a raw server error that was passed through all the way to users’ browsers (something developers and operations teams work very hard to avoid).

In the minutes that followed, the target.com team took rapid action to replace the cryptic server error with something more friendly.

By 8:14 am, we had captured the first of those friendlier images; the “fail doggie” page made its debut. About 17 minutes later we captured a new version of the page.  We saw additional changes again at 11:30 and 12:38.  It should be noted that these are times when we visited with the transaction-based measurement and that particular measurement was only set to visit ten times per hour.  The point is that these times are when we observed the pages and made captures, not necessarily the exact times that they changed.

Here, then, is the full gallery of “fail doggie” pages we recorded:

8:14 am EDT – “Oh no”

This page requested the user to “please try again” and provided a single link to “Target help”

(click image to view full size)

8:31am EDT – “Hello”

This page let folks know the team was “hard at work making the site better” and dropped the “please try again” in favor of “Sorry for the inconvenience – we’ll be back up and running shortly.  It also featured links to three services that were still online: redcard, weekly ad, and find a store.

(click image to view full size)

11:30 – “Woof” #1

Around 11:30 the message changed to “We are suddenly extremely popular.”  Visitors were also asked to “Please stay here and we’ll try to get you in as soon as we can!”  Finally, this version also explained what the three links that began appearing in the previous version were, saying “We are up and running here” just above the links.

(click image to view full size)

12:38pm EDT – “Woof” #2

This version added a plea to not keep hitting refresh, something that can make bringing a site back up very difficult when a large group is all doing it at the same time: “Please know that there is no need to refresh your browser.  Your request will automatically retry in 30 seconds.”  This was the most well organized page, broken into four separate paragraphs, and adding back in an apology with the sentence, “Thank you and our apologies for the inconvenience.”

(click image to view full size)

Sizing up the impact: just how bad was it, and for how long?

I was starting to put it all together.  Now I knew what the raw server error looked like and I had a play-by-play set of screen captures showing how the “waiting room” page evolved over time that morning. 

What I still wanted to know was just how “unavailable” the site had been.  Were any users getting through to the real home page at any point?  The fail doggie page said it would auto-refresh in 30 seconds and implied that perhaps one might eventually get through.  Was the real home page ever turning up, and if so how often? 

I waited until the next morning to size up the duration and intensity of the outage.  My goal was to confidently establish an “end point” for the incident and then do the numbers.

How can you measure “availability” when the site is displaying “OK” pages, but not the right ones?

The home page only measurement had the highest frequency (about 40 per hour) so I wanted to use that to calculate availability.  There are many different reports and graphs in the MyKeynote portal, and most of them will tell you at a glance what your “availability” is – that is, what percentage of the measurements succeeded versus failed in some way.  The catch here was that server error and “fail doggie” pages were sent to the browser as “OK” (http status 200) pages, not errors. 

Scripts can be easily enhanced with validations that look either for required text that should be there or error text that should not be there.  Either validation option would have marked the server message or fail doggie pages as errors and impacted the built-in availability calculation, but the index measurement I was using didn’t have that validation in place.  In practice, validation is usually used at the end of a multi-page script to be sure the right final page had been reached.  Fortunately the information we needed to answer the questions was readily available anyway (more on that below). We just had to step back and consider the available tools to size it up.

The tool I chose to use was MyKeynote’s Object Trending report.  This report is available for any measurement that has been configured with our Web Content Trending (WCT) option.  WCT stores performance information for every object in the page, not just a roll-up for the page as a whole. Once again, storing all those details day and night was about to become very handy.

The Object Trending report has several options, all of which provide ways to view the performance of page elements over time.  To display it, I chose the Target measurement, selected Object Trending and set the date range to the 24 hour period after the incident had begun:

(click image to view full size)

Here’s what that report looks like by default, which is a separate line for each domain that objects originated from:

(click image to view full size)

The above default view is great for determining if one or more domains are particularly slow or unavailable, but I needed even more detail than that, so I dropped down the menu at the bottom of the graph and changed it to “Object data by object without parameters:”

This option would give me each separate element in the page, ignoring any query-string data that might be appended to the object name.  Think of the home page as a stage with a cast of characters.  The object trending report was about to show me who had appeared in what number of performances. 

I clicked “Generate Graph Now” and scrolled down to the table below the graph. Now I had a listing of the frequency of every object that had appeared on the home page from 8:00am EDT onward:

(click image to view full size)

What I was particularly interested in was the object name on the left and the “Included datapoints” on the right.   Every visit always had retrieved at least the base page (www.target.com). By comparing the count of base pages observed to the count of all the other objects, I could make some meaningful conclusions about how often the “fail doggie” had been turning up. 

I needed to do a little math, so I pulled the results into an Excel spreadsheet with a simple copy and paste, sorted by the datapoints column and added a new column to compare the count of each element to the count of the base page.  I did this a number of times, varying the time period a bit to narrow in on useful takeaways.

For starters, I re-ran the report to look just at the objects observed in the first hour between 8:00am EDT and 9:00am EDT.  The cast of characters is quite small; we either got just the base page of html, or the base page plus the elements of the fail doggie page.  Remember, these aren’t observations of all user traffic, they are the results observed by the visits of our real browser agents. While we were just a drop in the bucket of actual visitors, we did make it through to the site 31 times in that first hour.  Here’s what we saw: The “fail doggie” appeared just 55% of the time and we got nothing but the base page in the remaining 45% of visits.  The hundreds of thousands (or millions) of users attempting to visit during that same period likely saw a similar mix.

(click image to view full size)

So I could see that things had been pretty “messy” during the initial confusion of the incident.  An hour of outage is worth a lot of money to a site like target.com, and cryptic errors couldn’t have been good for inspiring confidence, but an hour is just an hour and perhaps many people hadn’t even tried to visit the site yet. 

The next question I wanted to answer was, “What percentage of Target.com’s traffic was able to make it to the home page throughout the rest of the day?”

Here’s a view of my spreadsheet based on running the Object Trending report for the period starting one hour after the incident (9:00am EDT) and ending at midnight that evening:

(click image to view full size)

So taking the big view of that entire day up to midnight EDT (which is 9pm Pacific time), I determined that 93% got the fail doggie and no more than 7% got through to the real home page.

Not pretty no matter how you slice it

I re-ran the report with various time windows, and the results varied only slightly.  Most surprisingly, even pushing the end time all the way to 9am the following day, the stats still showed that 85% of all visits got the fail doggie page elements.  Focusing on Midnight 9/14/11 to 9:00am 9/14/11 the number was still high at 75%.  Was this all cleared up by the start of business on 9/14/11?  Looking at just the hour between 8:00am and 9:00am that second day 9/14/11, the number was still an amazing 50%.

By this time my boss was wondering why I was spending so much time with all those spreadsheets and charts, so I stopped my investigation there and moved on to share updated results with all the folks that had been asking for data.  I annotated a screenshot of a scatterplot graph and wrote a narrative to go with it, sharing the details with several media outlets and even conducting a radio interview with Wall Street Journal Radio.  Here are a couple of links to places the results showed up, along with a copy of that annotated scatter-plot graph.

Investor’s Business Daily: Target Website Crash Offers Lessons

Retail Online Integration: What You Can Learn From Target's Site Crash

(click image to view full size)

Epilog:

The one question I left unanswered was just how many truly useful pages there had been in the 7% that were not “fail doggie” pages.  I’m curious how many would have actually allowed a customer to purchase.  Given the size of Target.com’s typical traffic this time of year (reported as 29.5 million for the month of the prior October by Investor’s Business Daily), I was pretty content to stop with the work I had done over those two days, observing with a long sigh that the folks at Target, who are no amateurs at online retail, had missed out on a LOT of potential transactions by turning away something north of 93% of all visitor attempts that first day.

Filtering Out Web Performance Monitoring Traffic From Google Analytics

If you're a webmaster, site owner, or e-commerce business manager, picture this scenario. You open your Google Analytics dashboard and you see a big spike in traffic. At first, you're really excited - wow, look at all those site visitors! You run off to tell your boss that she can give you that bonus she promised if you got that online marketing campaign to work. And then... you learn that someone in the Site Ops team added web performance monitors, and so it's all so-called robots - synthetic traffic generated for the purpose of keeping tabs on your site's response time and availability. 



You think, no problem, I could filter that out from Google Analytics, and then you learn you can't. It's always going to be there, like, forever. You go crazy, emailing the GA team, posting on forums, and then slowly resign yourself to forever dealing with that spike. It will disappear from your default view, maybe after a month, but that's a long month to wait. And heaven forbid if your execs ask you for a site traffic report for the past year, try explaining why you can't filter out that spike - what, you weren't thinking ahead? What kind of guy did we hire to run our online business, anyway?

Don't let this happen to your career. Plan ahead and learn how to use Google Analytics to filter out all web performance monitors from your site analytics reports. Here's the recipe:

STEP 1: Find Out the User Agent String for your Web Performance Monitors

Keynote's monitors, like all other web performance monitors, insert a special marker in the browser, called an user agent string. An Advanced Segment in Google Analytics allows you to filter out all traffic that contains this marker. So all you have to know are the user agent strings that Keynote adds to the browser. Keynote has several performance monitors - using real IE and Firefox browsers, or mobile browsers. Each performance monitor comes with its own special marker, so you have to construct a regular expression to filter all of these out.

Here are the browser markers that you have to use to filter out web performance monitors:

Keynote - Use "KHTE" (for Application Perspective monitors), "KTXN" (for the real browser Transaction Perspective monitors). Other companies whose user agent strings I know of are: "AlertSite" (AlertSite), "GomezAgent" (Gomez), "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" (Pingdom), "YottaMonitor" (Yottaa). 

If you are using a web performance monitor not listed above, Google "<insert your monitoring vendor> user agent string" and you will definitely find the user agent string you need to know.

Next, we will create an Advanced Segment in Google Analytics to filter out this synthetic traffic.

STEP 2: Create An Advanced Segment to Filter Out the Web Performance Monitors

Make sure you are using the new Google Analytics version, by clicking on the "New Version" link at the top of your GA page. You should now see the word "beta" right below "Google Analytics". Ensuring that you are in the MySite tab, click on Advanced Segments.

In the bottom right area of the Advanced Segments dialog box, click on this button:    and name it "Real People". We will now create a segment that filters out the synthetic traffic from web performance monitors. Here's the Advanced Segments dialog box:

Now, here's the tricky part - writing the regular expression that the Advanced Segment requires. Here is the regexp that filters out both the Keynote and Gomez monitors. Be careful to use the string exactly as shown, with the periods and asterisks: .*(KHTE|KTXN|GomezAgent).*

It's critical that you use a regular expression correctly, and getting it wrong is why I suspect many people believe that Google Analytics can't filter out this traffic. Once an advanced segment is created, then all traffic AFTER today will be filtered in the reports, but this will not apply to traffic that was generated prior to your creating this advanced segment. That's what I believe, from trial and error, though Google Analytics help says that you can filter out historical traffic. In any case, it's important you setup these monitors anyway, because you have no control over someone else setting up web performance monitors - even if your company didn't create performance monitors, your competitors could be monitoring your site's performance and creating all this traffic to your site - it is the world wide web, after all.

STEP 3: Select the Real People Advanced Segment When Viewing Data

Drop down the Advanced Segments dialog box, and select "All Visits" on the left hand side, and the advanced segment that you created, "Real People" on the right hand side.

Click on Apply, and view your data:

Now, it would be nice if I could choose "Real People" as the default segment to apply to all my reports, but I can't do that in Google Analytics yet. Nevertheless, you now have a handy way to view all the traffic and exclude web performance monitors, including Keynote.

After the Storm: Of Aftershocks and Hurricanes

What a week for the people of the Eastern Seaboard. Earthquake, then hurricane within a matter of days. Both events dominated the local and national news cycles, and struck in the very same geography. It appears that resulting damage was relatively low despite the historic magnitude of the quake, and the sheer scale of hurricane Irene. Interestingly, the impact of both events on the Internet and usage of the Web was dramatically different.

As we reported last week in our Mobility blog, the 5.8 magnitude Virginia earthquake demonstrated what studies have suggested in regards to the way people use the Web for accessing breaking news and information: they reach for their smartphones. Within hours of the quake, the Washington Post experienced outages and a huge spike in response time due to overwhelming demand across the major wireless carriers in the region, as well as its fixed-web property.
 
By contrast, the Washington Post, and other major news and weather websites did not experience the same degradation in performance when Hurricane Irene hit. This graph shows the availability of a prominent news network's mobile site throughout the week.


We thought maybe the major weather sites would also show signs of stress as the hurricane approached. Here you can see the response time of one weather site throughout the event that shows a steady ramp over the week to its peak on Saturday, but no dramatic spike. (Post-event power outages in the region are reported to have since diminished Internet access.)

So why didn’t people grab their smartphones to continuously check on the advance of the hurricane? Many probably did. But the quake was a complete surprise, catching people outside of their homes in their schools, offices and shops. Warnings of Irene’s disaster potential were well reported. So although Irene has wrought an estimated $3-billion in damage—10 times more than the losses estimated from the Virginia earthquake—its impact on news and weather sites was minor in comparison.

Of Aftershocks and Hurricanes

What a week for the people of the Eastern Seaboard—earthquake, then hurricane within a matter of days. Both events dominated the local and national news cycles, and struck in the very same geography. It appears that resulting damage was relatively low despite the historic magnitude of the quake, and the sheer scale of hurricane Irene. Interestingly, the impact of both events on the Internet and usage of the Web was dramatically different.

As we reported last week in our Mobility blog, the 5.8 magnitude Virginia earthquake demonstrated what studies have suggested in regards to the way people use the Web for accessing breaking news and information: they reach for their smartphones. Within hours of the quake, the Washington Post experienced outages and a huge spike in response time due to overwhelming demand across the major wireless carriers in the region, as well as its fixed-web property.

By contrast, the Washington Post, and other major news and weather websites did not experience the same degradation in performance when Hurricane Irene hit.

We thought maybe the major weather sites would also show signs of stress as the hurricane approached. Here you can see the response time of one weather site throughout the event that shows a steady ramp over the week to its peak on Saturday, but no dramatic spike. (Post-event power outages in the region have since diminished Internet access.)

New MyKeynote Version: Hot Fun In The Summertime!

Keynote's crack Engineering and Operations teams released version 10.4 of the MyKeynote portal on Wednesday night.  We've made some great improvements in the user-friendliness of our graphs, bringing all the controls you need to tweak your visualization directly to the forefront.

 
We've also added the option to include only error datapoints in scatter plot graphs, clustered 3D bar graphs, page-level trending in long-term graphs, and much, much more!

Deep Link to Keynote Website Performance Data

By Ian Withrow

MyKeynote is designed so you can easily move from high level trends and events down to detailed data points. Users start with an alert or a scatter plot and then iterate down into successive levels of detail, typically culminating in a waterfall chart and details for a specific measurement that we took for them. However, sometimes a customer already knows what specific data point they wish to drill down onto.  For example, if an APM or RUM solution that you have captures a Keynote transaction you would naturally want to compare the user perspective of that transaction side by side with the system perspective provided by your other tools.  In this way you could see both endpoints.  Deep linking to detailed Keynote data points could also be handy in highly automated operations enviroments.  Imagine if your trouble tickets included direct links to poorly performing data points associated with an alarm?

As you may have guessed by now, in this post I’m going to show how you can link directly to a data point or waterfall in Keynote. I expect this will prove handy for operations tools, devops, and performance analyst types who want to compare the Keynote data with APM data or possibly want to iteratively retrieve waterfall visualizations of data that meets specific criteria. In fact Keynote partners with a number of APM vendors like OpTier, ExtraHop and OPNET so you may soon see the ability to link from one of these APM solutions to Keynote visualizations.

Before we proceed I must mention that you will need our Data Feed or DataPulse service.  DataPulse provides a near real stream of Keynote data. Data Feed provides the same data but every 15 minutes.

Keep in mind that I'm appropiating functionality here that wasn't designed for this purpose. Yes it's a creative hack, but don't let that stop you from enjoying the benefits.

 

How To Deep Link Into Keynote

MyKeynote offers two types of visualizations that you can link to directly if you know when the data point was collected, which agent took it, and which transaction or script it is associated with. Happily all of this information can be retrieved from the aforementioned data services. In the below example I use excerpts from Data Feed XML files. DataPulse has all the same information; the XML file is just formatted slightly differently. Either way Keynote has user guides for both of these services if you want more details on them. From this point I’m going to assume some familiarity with these XML files.

 

Step 1 – Gather the Variables Needed for the MyKeynote Request

In XML Data Feed, each record starts with the following tag. The variables we care about are colored green.

<TXN_MEASUREMENT agent="45537" slot="845480" datetime="2011-JUL-05 22:37:51" target="1034270" agent_inst="45538" profile="0">

Additionally if we have a multiple page transaction we’ll need to know what page we care about. Each page record in Data Feed has the following tag which specifies page information.

<TXN_PAGE page_seq="1">

The hardest part is that we need to convert ‘datetime’ to a base UTD number. In English, this is the number of seconds that had passed since 01-01-1997 12AM (GMT) when the data point was captured. This may sound hard to figure out but writing a program to do this conversion is fairly straightforward. Here is how to quickly do the conversion by hand.

1. Go to: http://www.epochconverter.com/ a handy site that lets you quickly convert data and time formats between human readable version and Unix Time
2. Copy ‘datetime’ from your XML file, in our case the value is 2011-JUL-05 22:37:51
3. Enter this value into the “Human date to timestamp” converter and run it as shown below
4. Subtract 852076800 from the result; this converts the number from Unix Epoch time to base UTD.
5. In this case we get the value 457853871, save this result for later since we will need it in the next step.

 

Step 2 – Build the MyKeynote Request

First, a caveat these requests are only fulfilled if the user is already logged into MyKeynote. If not they’ll be asked to login and will need to try again. As mentioned previously we have two options.

1. Display a page waterfall
2. Display a transaction summary which compares multiple pages in a single transaction.

If what these two options represent isn’t immediately clear to you that’s ok. Below I describe how to form a request for each along with a screenshot that shows what you’ll get for your efforts.

 

Waterfalls

A waterfall can be requested using the below URL with the variables you need to fill in denoted in {brackets}.

http://my.keynote.com/newmykeynote/transpagesingledrilldown.aspx?transid={target}&agentid={agent}&butd={UTD}&ksid=5&profid={profile}&pageid={page_seq}&mskin=0

As you may recall in the previous step I described how to find each of these inputs.  Remember if you have only one page in the transaction then the value there is of course 1.  Below is a completed example and the resulting output.

http://my.keynote.com/newmykeynote/transpagesingledrilldown.aspx?transid=1034270&agentid=45537&butd=457853871&ksid=5&profid=0&pageid=1&mskin=0

Transaction Drill Down

For many transactions each data point will be comprised of multiple pages. If you want a detailed view comparing each page, instead of a waterfall for one page, the following request will do the trick.  Again variables are specified in {brackets}.

http://my.keynote.com/newmykeynote/scatterplotdrilldown.do?transid={target}&agentid={agent}&butd={UTD}&profid={profile}&pseq=-1

Note that in this case we don’t care about page number, if you did specify a pseq value it would actually take you to the waterfall for that page.  A completed example below:

http://my.keynote.com/newmykeynote/scatterplotdrilldown.do?transid=1034270&agentid=45537&butd=457853871&profid=0&pseq=-1

Building a Bridge from a Log

Now some of you may be saying wait a minute, what if I only have the log or the header from the Keynote requests to start with?  How am I going to get from that to the Data Feed or DataPulse record?  Fortunately we add a header to each Keynote agent request formatted as below.  In the header is the target and UTD value which should be sufficient for you to find the right XML record.  Example:

X-KTRACE

457853871|45538|1034270|0|1\r\n

Or rewritten using our variable names:

UTD|45538|Target|0|1\r\n

Wouldn’t it be nice if the agent number was in there too?  Sorry I warned you this functionality was developed for another purpose and that we are being opportunistic here.

 

Conclusion

I recognize this won’t be for everyone. For those operational groups that are trying to shave off every bit of time they can from their MTR numbers then being able to jump directly to detailed Keynote data points from say a Nagios or an APM system will save you valuable time. Do let your account team know if you like this an want an expanded solution.

Contrarian View: Amazon Outage Proves the Promise of Cloud Computing

By Ian Withrow

Those who know me well can tell you that I’m hardly a frothy fan boy, indeed I’m a died in the wool skeptic. So it may come as a surprise to you to hear that I view the fallout of the recent Amazon Web Services (AWS) outage as a very positive sign for Cloud Computing. Sure some sites got taken down, including one of my personal favorites Quora. However, another favorite site of mine managed to survive the incident with comparatively minor hiccups: Netflix. This is the bright spot I want to highlight. I just happened to have a performance measurement for Netflix in my Keynote account. On the east coast starting at 12am April 21^st, Netflix’s performance for successful transactions stayed a consistent couple of seconds and was available 96% of the time. Granted this isn’t flawless execution, note that the 27 failed data points are all timeouts resulting in just a red screen. However, compared to what happened to many sites, this is outstanding. (Y-axis details obscured)

It’s not dumb luck that got Netflix off this easy. It’s the product of hard work and engineering time invested in building their Amazon Web Services deployment the right way. As Netflix has been touting in various cloud conferences this year, they’ve been forced to fully embrace AWS due to their tremendous growth. Basically they only run credit card transactions in their private network. To ensure they always have enough capacity (and incidentally are highly available) they have turned provisioning decisions over to their operational systems.  Whenever an Amazon instance is poorly performing they terminate it and get a new one.  Likewise if there is an availability zone acting up (like what happened) then they automatically switch over to another.

This is how real high availability has always been done in networking: ensure that you can automatically failover to logically, physically, and geographically separate resources.  Any real engineer will tell you that problems and failures will happen.  Your availability track record is not based on how frequently this occurs but how gracefully you recover from them.

Herein is the promise of Cloud Computing: namely the favoreable relationship between cost and failover capabilities. In a private network world you would have to build and pay for a lot of stuff yourself: multiple data centers, double the hardware, internet access connections on opposite sides of the building, etc.  Very quickly the cost of high availability gets prohibitive, locking out all but the deepest of pockets.  Netflix explicitly said at Cloud Connect they came to the conclusion that they, even with all their growth, just weren’t big enough to justify building their own network of redundant data centers.

Enter Cloud Computing.  Now having access to redundant data centers is just a matter of purchasing the right performance monitoring tools and the engineering time in programming your applications and operational systems to take full advantage of on demand resources.  In the end you only pay for what you use of the infrastructure, not what you might need as is the case when doing it yourself. That’s what the real shame and promise highlighted by this outage is, young companies like Quora and Foursquare could easily have done just what Netflix has done.  The barrier to entry here isn’t a huge budget but the knowledge and priorities to do the work. The next step of course after fully leveraging Amazon is to be able to failover to different cloud providers, I’d bet you $100 Netflix is working on exactly this right now.

In a way this drives home a point we’ve known all along.  Cloud Computing is not outsourcing, this implies a transfer of risk and responsibility. You, not Amazon or Microsoft or Google etc., are responsible for the performance of your applications whether they are in the cloud or not.  Cloud Computing is a powerful tool to increase performance and availability many fold while reducing costs, if it’s used correctly. If you don’t use the tool properly then an outage isn’t Amazon’s fault, it’s yours.  I'll leave you with this thought, Amazon seems to agree: according to Gartner Analyst Lidya Leong this isn’t an outage that generates service credits. (Quote at very end of article)

How To: Setup First Mile Monitoring with Cloud Application Perspective

By Ian Withrow

One of the most straightforward uses for our new Cloud Application Perspective (CApP) product is monitoring the performance of a website or API up to the edge of your private network. To be sure other use cases, which I plan to talk about soon, focus more purely on internal monitoring. However, this is a good one to start with if you are already using or familiar with Keynote services for monitoring user experience from outside your firewall. In fact if you already have Keynote then this is a great complement for your existing monitoring via our Transaction or Application Perspective products. At a high level you will learn the following: how to request a new CApP agent, where to place it, how to create and deploy a measurement, and of course how you can make use of the resulting data.

Ok but first, why should you care? Well there is a direct and an indirect benefit here. By monitoring first mile performance you can quickly identify if performance problems are inside your data center or outside of it. This need not be limited to consumer facing applications, you could easily monitor the first mile performance of B2B applications or even internal/partner applications. In fact, these scenarios might be more compelling as they are often associated with SLAs or clear expectations of performance. Moreover, I talk about monitoring a private data center here but it could just as easily be a private or public cloud. This brings us to the second benefit: proof. In short, gathering first mile performance data with CApP gives you evidence of where a problem is located. Who doesn’t want to avoid finger pointing?

Hopefully by now you are asking how you can get one of these. Well for starters you have to be a Keynote customer, but assuming you are you can easily request one online in the Keynote Service Center. First, select the “Agent” tab. If you don’t have this tab in your portal then you haven’t been setup for CApP, we’ll address that in a second. Second, click on “Request CApP Agent”. Finally, fill in and submit the resulting form. The only input that involves some research is supplying the public IP address range from which the agent will contact our servers. A full Class A is ok though. If you don’t know the public IP address of a given device then a number of free websites can answer that question, like IP Chicken.

So what if you have Keynote service but haven’t signed up for CApP? Well in the “Add Measurement” screen you may have noticed that you now have a CApP section. You can select this option and choose “Getting Started.” This takes you to a form that will nudge your sales team to get in touch with you to get your contract amended. I know not very SaaS’y, but we wanted to get version 1.0 out quickly. In the future, look for this process to be further automated.

 

After going through all of this you’ll get an email from us with a download link and a license key for your brand spanking new CApP agent. So where to put it? Typically you’d deploy CApP for this use case behind your firewall or possibly in the DMZ, this would provide visibility up to the edge of your network. You can then compare this data with measurements from Keynote’s global test and measurement network. Since CApP is designed to operate on end user machines for remote testing, it doesn’t need much in the way of horse power and can easily be run on shared or virtualized systems. Fair warning, it is Windows only right now. The setup itself is a breeze; you basically speed click through a Windows installer. You know, where you quickly click “yes” and “next” while you unwittingly consign your first born over to Keynote? After that there is a simple dialogue window where you enter the license key we provided you. From here on out CApP runs as a Windows service behind the scenes and only requires outbound Port 443 access, so it can communicate with our SaaS portal.

So speaking of instructions, how do you create a measurement for CApP? CApP, if you didn’t already guess, generates synthetic measurements based on predefined instructions we call scripts. You use KITE to create an Application Perspective (ApP) script. In fact if you already have ApP scripts you can skip this step and use them with CApP. For in depth KITE tutorials or to download this free tool, then check out its web page here. What I’ll show you here is how to create a basic script using the point and click functionality of KITE.

Launch KITE and choose the “Record” drop down menu, not the big red button tempting though it is.  Then select “Record Simulated Browser Script”.

KITE will ask you for a URL to start out with, I chose Keynote’s website for this demo.

Next you’ll be presented with the KITE web browser, which loads the URL you provided. Now simply navigate through this browser session as normal and click “stop” once you’ve completed the transaction you wish to record.

Once you are done, KITE will test out the script you’ve defined right away. Afterwards you can add advanced steps, logic, rules, or edit any errors that are present. If you’ve purchased CApP you should know that you also have a support contract with Keynote, this allows you to get script help from us among other things.

 

For now let’s keep things simple and just save and exit. With this we are ready to provision a measurement in the Keynote Service Center. CApP is provisioned like other Keynote measurements, except that it has its own product box, shown below. The only other important difference is that measurements can be as frequent as 1 minute or infrequent as 60 minutes.

With a measurement, like this one, setup you can make use of the data in all the current ways that are possible in MyKeynote, whether it’s the dashboard, graphs, or alarms. Below I copied a graph that highlights the value of CApP for this particular use case: a graph that compares inside (You) and outside (Me) measurements.

Just to wrap up, you’ve seen at a high level how to setup CApP from scratch in order to get improved visibility into how your website or API performs up to the edge of your private network or cloud. This is a great addition for any customer who is already monitoring their properties externally.