Web Performance Watch

Did You Fall When Facebook Stumbled?

Thursday's outage of Facebook was notable for the ripple it created across the Web. Many online businesses were impacted indirectly as a result of failing content integrated into their websites. The nearly ubiquitous "Like" button and Facebook's other social plugins became a drag on the performance and availability of some websites. We saw notable failures across media, travel and retail sites. Yet, other some appeared to have avoided significant impact.

The websites of CNN, USA Today and Expedia--members of the Keynote Business 40 Index--demonstrate how different approaches to integrating the Facebook social plugin can be the difference between disaster and only a minor setback when 3rd party content fails.

Here are graphs of their performance and availability during the outage (along with Facebook's for reference):

Notice USA Today's performance (the blue line). Their home page continued to happily build despite the lack of content availability from Facebook.

Incidents like this are good reminders of the importance of using page construction best practices that accomodate third party content failures. If you feature ads, widgets or plugins on your site, do they represent a potential single point of failure if the service becomes unavailable?

Even if you take care with integrating third party content into your pages, it's important to continuously monitor each source indepently if your site changes frequently. Keynote helps companies uniquely monitor 3rd party content such as ads, social widgets and plugins so you can quickly take action to mitigate performance issues and focus business improvement with actionable data.

So how did your site perform Thursday evening?

Making IT Matter

Yesterday, I read an article reporting about a company's potential plans to upgrade one of its key business applications. The article featured perspectives from the application's end users, management and of course, the IT organization that developed and now supports it. The dynamic at issue was the app's lack of compatibility across browsers.

When asked about the likelihood of a change, the company's IT director responded with the following:

"[My organization has] been shying away from upgrading a major version of [biz app] because every time we’ve done it, it’s been tremendously difficult." (emphasis added)

Super. And once again, we have another reason why 
"IT doesn't matter."

While the end users could not understand why the application wouldn't run reliably on Safari and Chrome in this case, IT was not convinced it was necessary to change the app to support them. Of course, cross-browser compatibility is a thorny issue. We've explored the reasons why site owners need to consider multi-browser strategies, and considerations for getting there. In this instance, there may no real viable solution for improving this application's end-user experience with regard to browser support.

But saying that it won't be done because "it’s been tremendously difficult" exemplifies the challenges that IT continues to have with finding relevance in a cloudy, mobile, and increasingly consumerized IT landscape. Today, the end user matters more than ever. IT organizations that recognize this, inculcate a user-first approach into how they manage their portfolio as well as tool choices. They adopt standards that support user oriented approaches. And the good news for IT is that these standards and tools exist.

It's possible that the choice of words of this IT manager in this instance belies other realities. But on the surface, the sentiment provides exactly the motivation that will drive this IT organization's customers and business stakeholders to a competitive SaaS application.

The end-user makes IT matter

Making IT to matter means placing the user at the forefront of the priority matrix, from choices in platform to its ongoing monitoring. Your users are changing... rapidly. When an architecture cannot support the expectations today's user, it will not survive.

 

Cyber-Awesome: Killer Performance Helps Drive Record Revenue

For years, Keynote has been monitoring the ‘net’s best websites and reporting on notable outages during the peak Black Friday and Cyber Monday online shopping period. Typically, we find that many online retail sites—both pure Internet and those with brick and mortar stores—experience some issues. But the trend has been towards better performance and higher availability. This year marks yet more progress toward the goal of a trouble-free and virtually instant shopping experience. And the numbers suggest the payoff for retailers has been huge: a record setting $1.25 billion in online sales for Cyber Monday, 2011.

The Keynote Top Retailers Index tracks 47 online shopping destinations. This year, the average website response time was 3.05 seconds. According to a study by IBM, online sales peaked at about mid-day, and spiked again later in the evening. Average response times across the Top Retailer Index reflected this pattern as peaking demand slowed sites. But the absolute impact even at its worst was only about 1 second of delay. That’s great performance across the industry as a whole.

Additionally, the availability of these websites remained extremely high throughout the period. Out of the 47 sites tracked, only 10 (21%) experienced any downtime at all and their average availability was around 99.0%.

So how did online retailers get to this pinnacle? We see that customers are maturing the technology, processes and culture of their Web Operations around a passion for performance and deep preparation. One example of this is Keynote customer Karmaloop.com who rang up their biggest day in company history and over 100% more sales than the same period last year. Though careful planning and web load testing, they were able prepare their site for the demands of such a dramatic increase in traffic and order transactions.

The improving trend in website performance throughout the increasingly demanding Black Friday and Cyber Monday period is encouraging. But it also means that if your site experiences failure or less than stellar performance, your competitor is in a much better position to attract and retain your abandoning customers. Congratulations to the online retail industry for hitting a remarkable milestone over the past six days. Keep pushing performance higher through December!

Filtering Out Web Performance Monitoring Traffic From Google Analytics

If you're a webmaster, site owner, or e-commerce business manager, picture this scenario. You open your Google Analytics dashboard and you see a big spike in traffic. At first, you're really excited - wow, look at all those site visitors! You run off to tell your boss that she can give you that bonus she promised if you got that online marketing campaign to work. And then... you learn that someone in the Site Ops team added web performance monitors, and so it's all so-called robots - synthetic traffic generated for the purpose of keeping tabs on your site's response time and availability. 



You think, no problem, I could filter that out from Google Analytics, and then you learn you can't. It's always going to be there, like, forever. You go crazy, emailing the GA team, posting on forums, and then slowly resign yourself to forever dealing with that spike. It will disappear from your default view, maybe after a month, but that's a long month to wait. And heaven forbid if your execs ask you for a site traffic report for the past year, try explaining why you can't filter out that spike - what, you weren't thinking ahead? What kind of guy did we hire to run our online business, anyway?

Don't let this happen to your career. Plan ahead and learn how to use Google Analytics to filter out all web performance monitors from your site analytics reports. Here's the recipe:

STEP 1: Find Out the User Agent String for your Web Performance Monitors

Keynote's monitors, like all other web performance monitors, insert a special marker in the browser, called an user agent string. An Advanced Segment in Google Analytics allows you to filter out all traffic that contains this marker. So all you have to know are the user agent strings that Keynote adds to the browser. Keynote has several performance monitors - using real IE and Firefox browsers, or mobile browsers. Each performance monitor comes with its own special marker, so you have to construct a regular expression to filter all of these out.

Here are the browser markers that you have to use to filter out web performance monitors:

Keynote - Use "KHTE" (for Application Perspective monitors), "KTXN" (for the real browser Transaction Perspective monitors). Other companies whose user agent strings I know of are: "AlertSite" (AlertSite), "GomezAgent" (Gomez), "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" (Pingdom), "YottaMonitor" (Yottaa). 

If you are using a web performance monitor not listed above, Google "<insert your monitoring vendor> user agent string" and you will definitely find the user agent string you need to know.

Next, we will create an Advanced Segment in Google Analytics to filter out this synthetic traffic.

STEP 2: Create An Advanced Segment to Filter Out the Web Performance Monitors

Make sure you are using the new Google Analytics version, by clicking on the "New Version" link at the top of your GA page. You should now see the word "beta" right below "Google Analytics". Ensuring that you are in the MySite tab, click on Advanced Segments.

In the bottom right area of the Advanced Segments dialog box, click on this button:    and name it "Real People". We will now create a segment that filters out the synthetic traffic from web performance monitors. Here's the Advanced Segments dialog box:

Now, here's the tricky part - writing the regular expression that the Advanced Segment requires. Here is the regexp that filters out both the Keynote and Gomez monitors. Be careful to use the string exactly as shown, with the periods and asterisks: .*(KHTE|KTXN|GomezAgent).*

It's critical that you use a regular expression correctly, and getting it wrong is why I suspect many people believe that Google Analytics can't filter out this traffic. Once an advanced segment is created, then all traffic AFTER today will be filtered in the reports, but this will not apply to traffic that was generated prior to your creating this advanced segment. That's what I believe, from trial and error, though Google Analytics help says that you can filter out historical traffic. In any case, it's important you setup these monitors anyway, because you have no control over someone else setting up web performance monitors - even if your company didn't create performance monitors, your competitors could be monitoring your site's performance and creating all this traffic to your site - it is the world wide web, after all.

STEP 3: Select the Real People Advanced Segment When Viewing Data

Drop down the Advanced Segments dialog box, and select "All Visits" on the left hand side, and the advanced segment that you created, "Real People" on the right hand side.

Click on Apply, and view your data:

Now, it would be nice if I could choose "Real People" as the default segment to apply to all my reports, but I can't do that in Google Analytics yet. Nevertheless, you now have a handy way to view all the traffic and exclude web performance monitors, including Keynote.

New MyKeynote Version: Hot Fun In The Summertime!

Keynote's crack Engineering and Operations teams released version 10.4 of the MyKeynote portal on Wednesday night.  We've made some great improvements in the user-friendliness of our graphs, bringing all the controls you need to tweak your visualization directly to the forefront.

 
We've also added the option to include only error datapoints in scatter plot graphs, clustered 3D bar graphs, page-level trending in long-term graphs, and much, much more!

Deep Link to Keynote Website Performance Data

By Ian Withrow

MyKeynote is designed so you can easily move from high level trends and events down to detailed data points. Users start with an alert or a scatter plot and then iterate down into successive levels of detail, typically culminating in a waterfall chart and details for a specific measurement that we took for them. However, sometimes a customer already knows what specific data point they wish to drill down onto.  For example, if an APM or RUM solution that you have captures a Keynote transaction you would naturally want to compare the user perspective of that transaction side by side with the system perspective provided by your other tools.  In this way you could see both endpoints.  Deep linking to detailed Keynote data points could also be handy in highly automated operations enviroments.  Imagine if your trouble tickets included direct links to poorly performing data points associated with an alarm?

As you may have guessed by now, in this post I’m going to show how you can link directly to a data point or waterfall in Keynote. I expect this will prove handy for operations tools, devops, and performance analyst types who want to compare the Keynote data with APM data or possibly want to iteratively retrieve waterfall visualizations of data that meets specific criteria. In fact Keynote partners with a number of APM vendors like OpTier, ExtraHop and OPNET so you may soon see the ability to link from one of these APM solutions to Keynote visualizations.

Before we proceed I must mention that you will need our Data Feed or DataPulse service.  DataPulse provides a near real stream of Keynote data. Data Feed provides the same data but every 15 minutes.

Keep in mind that I'm appropiating functionality here that wasn't designed for this purpose. Yes it's a creative hack, but don't let that stop you from enjoying the benefits.

 

How To Deep Link Into Keynote

MyKeynote offers two types of visualizations that you can link to directly if you know when the data point was collected, which agent took it, and which transaction or script it is associated with. Happily all of this information can be retrieved from the aforementioned data services. In the below example I use excerpts from Data Feed XML files. DataPulse has all the same information; the XML file is just formatted slightly differently. Either way Keynote has user guides for both of these services if you want more details on them. From this point I’m going to assume some familiarity with these XML files.

 

Step 1 – Gather the Variables Needed for the MyKeynote Request

In XML Data Feed, each record starts with the following tag. The variables we care about are colored green.

<TXN_MEASUREMENT agent="45537" slot="845480" datetime="2011-JUL-05 22:37:51" target="1034270" agent_inst="45538" profile="0">

Additionally if we have a multiple page transaction we’ll need to know what page we care about. Each page record in Data Feed has the following tag which specifies page information.

<TXN_PAGE page_seq="1">

The hardest part is that we need to convert ‘datetime’ to a base UTD number. In English, this is the number of seconds that had passed since 01-01-1997 12AM (GMT) when the data point was captured. This may sound hard to figure out but writing a program to do this conversion is fairly straightforward. Here is how to quickly do the conversion by hand.

1. Go to: http://www.epochconverter.com/ a handy site that lets you quickly convert data and time formats between human readable version and Unix Time
2. Copy ‘datetime’ from your XML file, in our case the value is 2011-JUL-05 22:37:51
3. Enter this value into the “Human date to timestamp” converter and run it as shown below
4. Subtract 852076800 from the result; this converts the number from Unix Epoch time to base UTD.
5. In this case we get the value 457853871, save this result for later since we will need it in the next step.

 

Step 2 – Build the MyKeynote Request

First, a caveat these requests are only fulfilled if the user is already logged into MyKeynote. If not they’ll be asked to login and will need to try again. As mentioned previously we have two options.

1. Display a page waterfall
2. Display a transaction summary which compares multiple pages in a single transaction.

If what these two options represent isn’t immediately clear to you that’s ok. Below I describe how to form a request for each along with a screenshot that shows what you’ll get for your efforts.

 

Waterfalls

A waterfall can be requested using the below URL with the variables you need to fill in denoted in {brackets}.

http://my.keynote.com/newmykeynote/transpagesingledrilldown.aspx?transid={target}&agentid={agent}&butd={UTD}&ksid=5&profid={profile}&pageid={page_seq}&mskin=0

As you may recall in the previous step I described how to find each of these inputs.  Remember if you have only one page in the transaction then the value there is of course 1.  Below is a completed example and the resulting output.

http://my.keynote.com/newmykeynote/transpagesingledrilldown.aspx?transid=1034270&agentid=45537&butd=457853871&ksid=5&profid=0&pageid=1&mskin=0

Transaction Drill Down

For many transactions each data point will be comprised of multiple pages. If you want a detailed view comparing each page, instead of a waterfall for one page, the following request will do the trick.  Again variables are specified in {brackets}.

http://my.keynote.com/newmykeynote/scatterplotdrilldown.do?transid={target}&agentid={agent}&butd={UTD}&profid={profile}&pseq=-1

Note that in this case we don’t care about page number, if you did specify a pseq value it would actually take you to the waterfall for that page.  A completed example below:

http://my.keynote.com/newmykeynote/scatterplotdrilldown.do?transid=1034270&agentid=45537&butd=457853871&profid=0&pseq=-1

Building a Bridge from a Log

Now some of you may be saying wait a minute, what if I only have the log or the header from the Keynote requests to start with?  How am I going to get from that to the Data Feed or DataPulse record?  Fortunately we add a header to each Keynote agent request formatted as below.  In the header is the target and UTD value which should be sufficient for you to find the right XML record.  Example:

X-KTRACE

457853871|45538|1034270|0|1\r\n

Or rewritten using our variable names:

UTD|45538|Target|0|1\r\n

Wouldn’t it be nice if the agent number was in there too?  Sorry I warned you this functionality was developed for another purpose and that we are being opportunistic here.

 

Conclusion

I recognize this won’t be for everyone. For those operational groups that are trying to shave off every bit of time they can from their MTR numbers then being able to jump directly to detailed Keynote data points from say a Nagios or an APM system will save you valuable time. Do let your account team know if you like this an want an expanded solution.

Contrarian View: Amazon Outage Proves the Promise of Cloud Computing

By Ian Withrow

Those who know me well can tell you that I’m hardly a frothy fan boy, indeed I’m a died in the wool skeptic. So it may come as a surprise to you to hear that I view the fallout of the recent Amazon Web Services (AWS) outage as a very positive sign for Cloud Computing. Sure some sites got taken down, including one of my personal favorites Quora. However, another favorite site of mine managed to survive the incident with comparatively minor hiccups: Netflix. This is the bright spot I want to highlight. I just happened to have a performance measurement for Netflix in my Keynote account. On the east coast starting at 12am April 21^st, Netflix’s performance for successful transactions stayed a consistent couple of seconds and was available 96% of the time. Granted this isn’t flawless execution, note that the 27 failed data points are all timeouts resulting in just a red screen. However, compared to what happened to many sites, this is outstanding. (Y-axis details obscured)

It’s not dumb luck that got Netflix off this easy. It’s the product of hard work and engineering time invested in building their Amazon Web Services deployment the right way. As Netflix has been touting in various cloud conferences this year, they’ve been forced to fully embrace AWS due to their tremendous growth. Basically they only run credit card transactions in their private network. To ensure they always have enough capacity (and incidentally are highly available) they have turned provisioning decisions over to their operational systems.  Whenever an Amazon instance is poorly performing they terminate it and get a new one.  Likewise if there is an availability zone acting up (like what happened) then they automatically switch over to another.

This is how real high availability has always been done in networking: ensure that you can automatically failover to logically, physically, and geographically separate resources.  Any real engineer will tell you that problems and failures will happen.  Your availability track record is not based on how frequently this occurs but how gracefully you recover from them.

Herein is the promise of Cloud Computing: namely the favoreable relationship between cost and failover capabilities. In a private network world you would have to build and pay for a lot of stuff yourself: multiple data centers, double the hardware, internet access connections on opposite sides of the building, etc.  Very quickly the cost of high availability gets prohibitive, locking out all but the deepest of pockets.  Netflix explicitly said at Cloud Connect they came to the conclusion that they, even with all their growth, just weren’t big enough to justify building their own network of redundant data centers.

Enter Cloud Computing.  Now having access to redundant data centers is just a matter of purchasing the right performance monitoring tools and the engineering time in programming your applications and operational systems to take full advantage of on demand resources.  In the end you only pay for what you use of the infrastructure, not what you might need as is the case when doing it yourself. That’s what the real shame and promise highlighted by this outage is, young companies like Quora and Foursquare could easily have done just what Netflix has done.  The barrier to entry here isn’t a huge budget but the knowledge and priorities to do the work. The next step of course after fully leveraging Amazon is to be able to failover to different cloud providers, I’d bet you $100 Netflix is working on exactly this right now.

In a way this drives home a point we’ve known all along.  Cloud Computing is not outsourcing, this implies a transfer of risk and responsibility. You, not Amazon or Microsoft or Google etc., are responsible for the performance of your applications whether they are in the cloud or not.  Cloud Computing is a powerful tool to increase performance and availability many fold while reducing costs, if it’s used correctly. If you don’t use the tool properly then an outage isn’t Amazon’s fault, it’s yours.  I'll leave you with this thought, Amazon seems to agree: according to Gartner Analyst Lidya Leong this isn’t an outage that generates service credits. (Quote at very end of article)

How To: Setup First Mile Monitoring with Cloud Application Perspective

By Ian Withrow

One of the most straightforward uses for our new Cloud Application Perspective (CApP) product is monitoring the performance of a website or API up to the edge of your private network. To be sure other use cases, which I plan to talk about soon, focus more purely on internal monitoring. However, this is a good one to start with if you are already using or familiar with Keynote services for monitoring user experience from outside your firewall. In fact if you already have Keynote then this is a great complement for your existing monitoring via our Transaction or Application Perspective products. At a high level you will learn the following: how to request a new CApP agent, where to place it, how to create and deploy a measurement, and of course how you can make use of the resulting data.

Ok but first, why should you care? Well there is a direct and an indirect benefit here. By monitoring first mile performance you can quickly identify if performance problems are inside your data center or outside of it. This need not be limited to consumer facing applications, you could easily monitor the first mile performance of B2B applications or even internal/partner applications. In fact, these scenarios might be more compelling as they are often associated with SLAs or clear expectations of performance. Moreover, I talk about monitoring a private data center here but it could just as easily be a private or public cloud. This brings us to the second benefit: proof. In short, gathering first mile performance data with CApP gives you evidence of where a problem is located. Who doesn’t want to avoid finger pointing?

Hopefully by now you are asking how you can get one of these. Well for starters you have to be a Keynote customer, but assuming you are you can easily request one online in the Keynote Service Center. First, select the “Agent” tab. If you don’t have this tab in your portal then you haven’t been setup for CApP, we’ll address that in a second. Second, click on “Request CApP Agent”. Finally, fill in and submit the resulting form. The only input that involves some research is supplying the public IP address range from which the agent will contact our servers. A full Class A is ok though. If you don’t know the public IP address of a given device then a number of free websites can answer that question, like IP Chicken.

So what if you have Keynote service but haven’t signed up for CApP? Well in the “Add Measurement” screen you may have noticed that you now have a CApP section. You can select this option and choose “Getting Started.” This takes you to a form that will nudge your sales team to get in touch with you to get your contract amended. I know not very SaaS’y, but we wanted to get version 1.0 out quickly. In the future, look for this process to be further automated.

 

After going through all of this you’ll get an email from us with a download link and a license key for your brand spanking new CApP agent. So where to put it? Typically you’d deploy CApP for this use case behind your firewall or possibly in the DMZ, this would provide visibility up to the edge of your network. You can then compare this data with measurements from Keynote’s global test and measurement network. Since CApP is designed to operate on end user machines for remote testing, it doesn’t need much in the way of horse power and can easily be run on shared or virtualized systems. Fair warning, it is Windows only right now. The setup itself is a breeze; you basically speed click through a Windows installer. You know, where you quickly click “yes” and “next” while you unwittingly consign your first born over to Keynote? After that there is a simple dialogue window where you enter the license key we provided you. From here on out CApP runs as a Windows service behind the scenes and only requires outbound Port 443 access, so it can communicate with our SaaS portal.

So speaking of instructions, how do you create a measurement for CApP? CApP, if you didn’t already guess, generates synthetic measurements based on predefined instructions we call scripts. You use KITE to create an Application Perspective (ApP) script. In fact if you already have ApP scripts you can skip this step and use them with CApP. For in depth KITE tutorials or to download this free tool, then check out its web page here. What I’ll show you here is how to create a basic script using the point and click functionality of KITE.

Launch KITE and choose the “Record” drop down menu, not the big red button tempting though it is.  Then select “Record Simulated Browser Script”.

KITE will ask you for a URL to start out with, I chose Keynote’s website for this demo.

Next you’ll be presented with the KITE web browser, which loads the URL you provided. Now simply navigate through this browser session as normal and click “stop” once you’ve completed the transaction you wish to record.

Once you are done, KITE will test out the script you’ve defined right away. Afterwards you can add advanced steps, logic, rules, or edit any errors that are present. If you’ve purchased CApP you should know that you also have a support contract with Keynote, this allows you to get script help from us among other things.

 

For now let’s keep things simple and just save and exit. With this we are ready to provision a measurement in the Keynote Service Center. CApP is provisioned like other Keynote measurements, except that it has its own product box, shown below. The only other important difference is that measurements can be as frequent as 1 minute or infrequent as 60 minutes.

With a measurement, like this one, setup you can make use of the data in all the current ways that are possible in MyKeynote, whether it’s the dashboard, graphs, or alarms. Below I copied a graph that highlights the value of CApP for this particular use case: a graph that compares inside (You) and outside (Me) measurements.

Just to wrap up, you’ve seen at a high level how to setup CApP from scratch in order to get improved visibility into how your website or API performs up to the edge of your private network or cloud. This is a great addition for any customer who is already monitoring their properties externally.

The Website Sausage Factory and Impact on Performance – a TechCrunch Case Study

By Ian Withrow

As we’ve discussed in various blog posts, websites are like a sausage.  Ok maybe not so directly, but like a sausage they are made of ingredients than come from many sources even though they are presented in one tidy package to the user.  Today I’m going to break apart the sausage that is the TechCrunch blog, show how this can be easily done for any website using KITE, and as a special bonus show how a sausage maker can monitor all the pieces of their links using a Keynote technology called Virtual Pages. 

TechCrunch Composition

First a fair warning: just like with sausage making, finding out what is inside your favorite website is not always pretty.  If you feel you are of a squeamish disposition then you have been warned.  Second, note that all details in the post are from the time of writing and the balance of ingredients is likely to change overtime.

TechCrunch is a behemoth of a site, weighing in at just over 4 MB of data, 329 page elements, and a whopping 65 domains.  About half of this comes from them directly or really via Wordpress who is evidently the platform they use to power their blog.  The rest comes from over 20 3^rd parties.  You read that correctly, 65 domains in total and half the content originates from someone else.  After direct content the next biggest category is from social sites; Facebook, Twitter, and tools related to these properties total about 1 MB.  Google is about a 500 KB and the ‘Misc.’ category of various 3^rd party tools that TechCrunch uses to improve user experience is about 200 KB.  Ads and ad related content are about 180 KB.  Below is a snapshot from KITE breaking it out for you.  (Note you need to complete the process in the next section to actually get this view)

Now it’s time for a few fun observations.  The amount of content from Facebook and Twitter is huge!  Each alone is bigger than most websites are in total.  Digging into this is unfortunately off topic for this post but it is definitely something on my radar screen for the future.  Another interesting area is the level of user tracking that goes on.  I could identify at least 6 different 3^rd parties that were tracking TechCrunch visitors, not including Google and Facebook.  TechCrunch knows what you are, if not who.

Finally, while these stats make it seem like TechCrunch is hardly advertising, understand that TechCrunch is a very, very, very long page (vertically) and all the ad content is at the top where the user is most likely to see it.  They aren’t dummies giving away their yummy sausage for free.

Scripting Sites for 3^rd Party Monitoring & Analysis

To make sense of this mess I used Keynote’s KITE product.  There are a lot of other great, free products out there that one can use to view all the content and domains of a page.  However, KITE has the ability to permanently parcel out these domains into what we call Virtual Pages for ongoing monitoring and analysis.  Note this section won’t be a detailed how-to; I’m going to focus on highlighting what is possible with the tool.  After which you should be prepared to experiment or watch this training video depending on your learning style.

After downloading TechCrunch in KITE I organized the content by domain as shown below.

This lets me easily see the composition and breakout of a page in a manual fashion.  If I just want to see the domains I can simply collapse the domain groupings.  There are tons of options that I can add to this view like content size and various time breakouts based on my interest.  Here is a complete list:

With just this you can see that I can casually learn a lot about the page.  However, if I’m serious about how TechCrunch and its 3^rd parties perform then I need ongoing data points.  If I’m going to gather a lot of data then I don’t want to do this parsing and analysis manually, it just won’t scale.  The solution is to organize this content into permanent logical pieces.  For example, in a simple scenario I’d carve out a Virtual Page for my advertising so I could monitor and analyze the performance of that content separately from my content with Keynote.  As you can imagine the more complicated your site becomes, the more important this exercise is.  True you can always pick through a waterfall manually to see who did it in the event of the problem but if you want to have ongoing data about 3^rd party performance or be proactive with alerts then you’ll need something like Virtual Pages.  The nice thing is once you’ve designated content into a Virtual Page you can monitor and analyze it like a regular page.

Let’s discuss how I broke-up and organized TechCrunch.  Please note I’m not holding this up as the standard for the best or only way to use Virtual Pages.  One thing we need to keep in mind is cost.  Each page (virtual or otherwise) adds to the cost of the measurement and so in the real world we probably can’t go hog wild with these.  Given an unlimited budget I’d define a Virtual Page for each 3^rd party, possibly even one for each domain if I was especially crazy for detailed data.  My guess is you live in the real world and even if your site isn’t as complicated as this one you’ll need to create some buckets.  Most likely you’d start with prior experience, defining Virtual Pages where you knew or suspected there was a problem.  Here I simply broke the site into the following logical categories:

• TechCrunch direct content plus AOL
• Google (but not Google owned advertising)
• Facebook
• Twitter (and related tools like Postup)
• Wordpress (even though this is the core of the site, I want to evaluate my vendor here)
• Misc. Tools and Widgets for the users
• Analytics and User Tracking
• Ads and Ad related content

Why no CDN category? We certainly encounter CDN’s here but each is tied to a specific 3^rd party.  Facebook has its own CDN, the ad platforms have CDNs and so forth.  So instead I left the CDN’s with their respective masters.

Here is a brief teaser for how this is done in KITE

Step 1) Pick the URL you want to virtualize and run the page once (we did this already)

Step 2) Right click on the page in question and select ‘Insert Virtual Action’

Step 3) We now have a new Virtual Page at the bottom of your script.  Right click ‘Match Page Elements’ and select ‘Add URL Match’.  Here I’ve used the naming convention “vp:TechCrunch” to distinguish Virtual Pages from real pages.  You can name them anything you want in practice though.  There are other options that you can use to construct Virtual Pages, such as content type, that have interesting possibilities but to address 3^rd party content, URL seems ideal to me.  As you see below I’ve created a list of URL matches that should capture all the differently named TechCrunch domains.

Step 4) In the Script Properties Editor you can create the settings for each URL Match.  Note in my script I used a variety of regular expressions so that I could get away with far fewer rules than the 65 domains and still cover all of the page content.

Note that to do this I never had to write any code or do any advanced scripting.  It was all point, click, and form completion.  Hopefully by now you can see how easy it is to create Virtual Pages in KITE and have an idea of its possibilities.

How can I Benefit from Virtual Pages?

The obvious and immediate answer is you can now isolate and monitor the performance of certain 3^rd parties or subsections of you website.  If Facebook slows down you’ll know immediately and explicitly that this is case regardless of the overall impact on your performance.  Moreover, you can easily track and directly report on the performance of these guys overtime without needing to manually crunch the data and objects yourself.  Another interesting possibility is you could monitor your own additions to your site to see how they fair.  Finally, another angle might be to isolate and monitor all the Javascript that your site utilizes.  There are a countless number of ways that Virtual Pages might be used, and my list probably just scratches the surface.  Have fun with it!

 

Elevate Performance Awareness in your Company by Publishing Keynote Graphs

By Ian Withrow

I’ll warn you right off this isn’t my typical post about a concept or a market trend.  I’m going to straight up show you how to do something cool in MyKeynote that’s available right now: export a saved graph.  Why you ask would anyone want to do this?  It’s the perfect way to raise visibility within your organization about the valuable information that your Keynote measurements generate.

In many of the organizations I talk to, Keynote data is an invaluable tool for IT and operational groups but it’s a shame in my opinion that this information doesn’t get out to groups like product management and sales as readily as it should. So today I’m going to show you how to create a graph in MyKeynote that will provide an ongoing view into the performance of your website or application.  I’ll also suggest some types of graphs different business functions might be interested in.  Finally, I’ll spit ball some ideas on how you might distribute this information.

First Create and Save a Graph

First, before we can become heroes (or villains who dare speak the truth) we need to figure out how to actually share a graph.  The first part of this is a routine MyKeynote process so stick with me or skip down to the next section if you already know how to save a graph.

 To start, we head to the graph section and perform the following steps:

1. Select the measurement we want to graph.  This could be more than one measurement if we wanted to make a comparison.  I’ve gone with the Keynote Business 40, The Keynote Business 40 Internet Performance Index (KB40) measures the average download time for the home pages of 40 important US-based business Web sites.  To find out more go here.
2. Pick the graph type, time history is a great option to show performance overtime.
3. Time period is up to you but its important use to a relative period if you want your graph to have permanent relevance.
4. Finally I’ve elected just to show performance to keep things simple in this post.
5. Ok let’s generate the graph, this graph is a big one with a lot of data and measurements in it so it takes some time to generate.  Keep in mind how long your audience will wait when you setup your graph.

Once generated, save the graph using the menu in the top right hand corner.

The next step is important not to mess up so read carefully.  Name your graph whatever you want and click save.  You’ll get a dialog window like the below.  If you want a ‘relative’ graph you need to choose “Cancel”.

Retrieving the URL for your Graph

Ok now we are done with MyKeynote and for the experienced users out there you may note that we haven’t done anything unusual or new yet.  Now for the trick!  Keynote actually publishes your saved graphs in two heretofore undocumented RSS feeds.  One feed publishes the entire MyKeynote graph page, complete with legend and the other publishes just the graphs themselves.  Those feeds are:

1. For the full graph page: http://my.keynote.com/newmykeynote/mykeynoterss.do
2. For just the graph:  http://my.keynote.com/newmykeynote/mykeynoteembed.do

You’ll find that visiting one of these feeds requires you to login in to your MyKeynote account.  Do so with a browser like IE or a feed reader that can handle authenticated RSS feeds and you’ll see something like this.

The URL behind Keynote Business 40 and any other graph is permanent (as long as the graph is saved) and can be requested by clients without logging in.  Note that you can also get this link from the MyKeynote home page “Saved Graphs” widget.  There you can find an RSS feed button that will provide the same view above.  Below is an example of what the link provides:

Putting your Data to Work

This is a handy feature to be sure but how best to make use of it? Well at the simplest level you could simply distribute the link internally to interested parties.  Better might be to embed this page within another page, such as your companies Intranet.  Perhaps the cleverest idea I’ve encountered so far is to actually serve different graphs based on the organizational function of the user logging into an internal knowledge base site.

Presumably different groups want to know different things.  A product manager or marketing manager needs to know how their key pages or transactions are performing and for a large company there may be multiple product areas with different interests.  Developers may want to track their pages to see the impact of any code changes.  In contrast sales may wish to see the company’s page benchmarked against key competitors.  Any outage in a competitor’s site can provide an immediate piece of FUD to use.  Similarly a problem in their site is a heads up that they may be getting customer complaints.  As you can see timescale, graph type, and level of detail can easily vary based on the user’s interest.

As mentioned earlier it’s important to consider load time.  Generating graphs of measurements from many agents over a long period of time can take a few minutes.  You may want to limit the scope of your graphs to something that can be quickly generated to maximize the amount of attention it gets from viewers.  A drill down graph can easily be referenced to for users who are willing to wait for more details.

This feature is new and we see a lot of areas to make it even better.  Let us know how you make use of this feature and what improvements you think are most important to improving its impact.