Web Performance Watch

Another Win for Keynote Customers

When Keynote introduced Cloud Application Perspective last year, we believed that it addressed a critical gap in the way companies monitored the performance of their Web applications. As “cloud” technologies have emerged, websites have become more a coupling of components from multiple assembly lines, rather than a product manufactured in a dedicated IT application factory. Monitoring the performance of a website in its end-user state—the sum of the many parts—makes sense. But when issues arise, quickly finding their location in the “supply chain” can be a challenge.

What if you could drop something into that “supply chain”, at the key points of potential failure, giving you insight into your Web app’s performance? What if this was so easy to install, a non-IT person could do it in minutes? And what if the cost was just pennies per measurement?

Last week, the SIIA CODiE awards recognized Keynote Cloud Application Perspective for best Cloud Application/Service, along with Google and Scribe Software. They did so because Cloud Application Perspective offers customers a radically new way to monitor their websites and applications. Unlike traditional APM options, Cloud Application Perspective is uber-portable, monkey-simple to implement and really affordable.

Although we’re honored to have won this award, our customers are the real winners. Taking advantage of the cloud can be complex. Monitoring it should be easy. If you’d like to see how easy it can be with Cloud Application Perspective, get a free trial today!

Speed and Tenacity: the Apple iPad Outage

We’ve heard a lot recently about the importance of speed and performance when it comes to online retail. The New York Times highlighted research from Microsoft claiming that 250 milliseconds—a mere eye blink—could make the difference between a repeat visitor and a lost customer. And a popular infographic touts that Amazon would stand to lose $1.6 billion in sales per year from a 1 second web page delay. Our friends at Walmart.com have also shared some awesome research linking web performance to conversion.

These statistics are welcome news for the web performance community. But sometimes they don’t apply. With Apple, a lot of rules don’t apply.

 

This past weekend, Apple sold a record 3 million new iPad 3 tablets. That’s pretty phenomenal. Yet, it came on the back of a pretty bad outage only 10 days before.

On March 7, Apple announced the new iPad 3. For effectively the entire day, the Apple Store was unavailable. That meant no one could check out the new iPad, nor purchase iPhones, MacBooks or anything else.  

To Apple’s credit, the Apple store normally runs very quickly—averaging well less than 2 seconds for total User Experience Time and less a second for Time to First Paint. (The Apple Store is a member of the Keynote Retail Performance Index, measured with Keynote Transaction Perspective.)

We’ve written previously about the concept of tenacity. A website visitor’s tolerance for errors, or delays, is a major factor when balancing the cost and benefit of building capacity and engineering performance into Web applications. While Apple’s fanatic customer base is an extreme, it illustrates the point that there’s a continuum of performance expectations for users.

Your product/service is unique. And your customers are also unique. Keynote web load testing consultants dig into web analytics to model user behavior. They consider familiarity, tenacity, interaction speed and connection speed when developing virtual user profiles. It may be unrealistic for you to understand how different levels of performance impact your various customer types across all these variables. But if you can begin to understand them, you’ll be in a better position for setting ongoing performance goals and SLAs—especially around tolerances for outliers from your averages.

Cyber-Awesome: Killer Performance Helps Drive Record Revenue

For years, Keynote has been monitoring the ‘net’s best websites and reporting on notable outages during the peak Black Friday and Cyber Monday online shopping period. Typically, we find that many online retail sites—both pure Internet and those with brick and mortar stores—experience some issues. But the trend has been towards better performance and higher availability. This year marks yet more progress toward the goal of a trouble-free and virtually instant shopping experience. And the numbers suggest the payoff for retailers has been huge: a record setting $1.25 billion in online sales for Cyber Monday, 2011.

The Keynote Top Retailers Index tracks 47 online shopping destinations. This year, the average website response time was 3.05 seconds. According to a study by IBM, online sales peaked at about mid-day, and spiked again later in the evening. Average response times across the Top Retailer Index reflected this pattern as peaking demand slowed sites. But the absolute impact even at its worst was only about 1 second of delay. That’s great performance across the industry as a whole.

Additionally, the availability of these websites remained extremely high throughout the period. Out of the 47 sites tracked, only 10 (21%) experienced any downtime at all and their average availability was around 99.0%.

So how did online retailers get to this pinnacle? We see that customers are maturing the technology, processes and culture of their Web Operations around a passion for performance and deep preparation. One example of this is Keynote customer Karmaloop.com who rang up their biggest day in company history and over 100% more sales than the same period last year. Though careful planning and web load testing, they were able prepare their site for the demands of such a dramatic increase in traffic and order transactions.

The improving trend in website performance throughout the increasingly demanding Black Friday and Cyber Monday period is encouraging. But it also means that if your site experiences failure or less than stellar performance, your competitor is in a much better position to attract and retain your abandoning customers. Congratulations to the online retail industry for hitting a remarkable milestone over the past six days. Keep pushing performance higher through December!

Deconstructing the Target.com “Fail Doggie”: A Keynote Perspective

Have you met the Target.com “fail doggie?”

He’s cute, but if you are a Target customer, you don’t really want to encounter him in a browser.  I met the “fail doggie” while using the MyKeynote portal to research the nature and extent of the major outage that occurred on Tuesday, September 13^th 2011.  That was the day that Target began allowing online purchases of “Missoni by Target” items.

A lot at stake

According to one source, the Target site re-build from scratch, launched just weeks before this incident, drew on the talents of over 20 vendors, including many of the biggest names in the e-commerce technology space.

Imagine spending millions of dollars and two years of time to “create a more user-friendly, reliable experience” and then have this happen.  Not fun.  When a major event like this outage happens, it can be difficult to get complete details from any one participant.

Passions are high when a crisis like this occurs.  Where can you go to get an objective vantage point from which to make accurate assessments and analysis?  At Keynote Systems, we’ve long been looked to as a neutral third-party with accurate and actionable Internet and mobile performance data, and that day proved to be no exception.

In this blog post, I’ll explain how I used the tools that every Keynote customer has in MyKeynote along with measurements being run for two of our public web performance indices to determine what users were seeing that day and the following morning and to determine just how extensive the outage was.

Target.com appears in a number of our public index measurements, so I had several to pick from – note that we never publish insights based on a customers’ private data, but only based on publicly available data that we collected without payment by any company. Two were particularly useful for figuring out what was going on and capturing screenshots as the day went on. The first measurement visited their home page only, while the second arrived at the home page and then performed a multiple step transaction, just as a customer would when shopping, placing items in a cart and checking out. 

First a brief bit of background: both scripts were written in the Keynote Internet Testing Environment, known as KITE and both measurements were being run with our real browser product, Keynote Transaction Perspective. I point that out so you’ll know that none of what you will see here was retrieved by a “bot” or other emulation system; we were getting the experience of a user launching an actual Internet Explorer browser to go visit Target.com.

How things went down that day

As word of the outage quickly spread, we started taking calls from various news media companies asking if we had data.  We took a look at the home page monitoring scatter plot below and saw what looked like a brief spike in response times followed by a restoration of reasonable response times shortly thereafter. 

(click image to view full size)

At first glance, it was tempting to say that the outage had been brief and thus was relatively uneventful.  But a deeper look at the chart revealed telltale signs that something was not right.  Notice the distribution of the dots before and after the spike?  See how they are randomly distributed somewhere between two and six seconds before the spike (time on the network is on the left scale) but then they are all tightly packed down below the one-second mark afterword?

Each of those dots represents a full visit with a real browser, so I could click down on any of them to see a listing of what was on each of those pages and how long it took to get to the browser.  I did that eventually, but much like an operations team technician would be at such a moment, I was in triage mode at this point.  The first thing I wanted to do was to look for clues as to why those dots were organized as they were.

First, I hovered my mouse over a datapoint from before the spike in performance.   This would be my “normal” baseline to compare to.  Notice that the page contained 147 elements (separate downloaded objects) and a total size of about 2 MB.  The time on the network to download the page and all elements was 5.156 seconds. 

Next, I took a look at the two red triangles, which represent pages where we know there was an error of some sort or another:

In this case, the element count and page size were lower than the baseline but the time to download was skyrocketing to a number eight times higher. The lower object count and page size were due to timeouts being hit… we couldn’t get all of the objects into the browser before time was up, so the agent running the browser quit trying and reported the error.

What was really interesting was the object count from the very next green dot after the red triangles.

That data point showed only one page element and a very small download size of 640 bytes. 

One element?  I knew that if we had downloaded only one element, that it must be the base page of html, but a page with only 640 bytes couldn’t possibly have much to say.  That was my first clue that visitors were probably getting raw error messages.

I quickly scanned over the remaining “spiky” datapoints after the incident began and found more of the same: just one page element and a size of 639 or 640 bytes.   Here’s the last point caught during that initial spike:

So that was all fairly consistent with a site failure; something went very wrong around 8:00am EDT and the server took a long time to send out a very small page that was probably just an error message. 

What about all of those green dots hugging the bottom line after the spike had subsided?   I hovered over a few and got the same thing each time: 5 page elements and a page size of 31550 bytes. 

This wasn’t some random subset of the real page and no error was being recorded, so clearly these speedy responses were something altogether different. 

Now it was finally time to start drilling in for some details.  I clicked one of those data points and made my way to the page detail waterfall graph:

 

(click image to view full size)

Welcome to our waiting room

I hovered over each of the bars and noticed that each object came from a folder called “spawaitingroom.” The image files consisted of a red stripe, a Target logo and a photo of the Target dog posing next to a tool box.  I had met the “fail doggie” for the first time:

 

(click image to view full size)

So let’s recap what I had learned so far:

1. Initial data points have no page objects… just the base page, and that base page was TINY (640 bytes) in comparison to the pages that preceded it, but the amount of time the server took to return that page was HUGE.
2. The datapoints following the big spike downloaded much faster and had larger base pages but only had five page elements instead of the 140+ found in the normal pages.
3. Drilling in on the five-element page datapoints, I found that all five of the objects came from a folder called “spawaitingroom” – the “fail doggie” was part of a “waiting room” feature that was being served in lieu of the real home page. 

Getting the whole picture: what were people actually seeing?

I was making good progress, but I still had a lot of questions to answer.  I could guess that a Target dog next to a toolbox was probably some variation of an “under construction” page, but I didn’t have the text of the page yet.  I really wanted to know what those pages looked like but all I had were some pieces to the puzzle.

The thing that was complicating my sleuthing was that neither the original 640-byte server error pages nor the fail doggie pages were being sent as an “error” (http status of 4xx or 5xx) – they were being sent as successful pages (http status 200).  That prevented me from getting as much diagnostic information as I otherwise might have.  I could poke around at each scatterplot and piece together what I was seeing, but without an error, I wasn’t going to have the html from the page or any screenshots, both of which MyKeynote stores on a fatal error. Fortunately, all was not lost.  There's a benefit in more than 400,000,000 objects per day stored away inside MyKeynote.

Web Content Trending to the rescue

Here’s where the other measurement that was scripted to go five steps deep into the target.com site came in very handy.  The second monitoring script was set up to do the following:

1. Go to the target.com home page.
2. Perform a search for “lil wayne”
3. Filter the search results to the “music” category.
4. Click on the first album in the resulting list to view the details.
5. Click the “add to cart.” button on the details page and then confirm that “1 item added to cart.” appears on the screen.

Without the real site being served up, there was no way the script could complete the search for an album.  When the Keynote agent piloting the Internet Explorer browser went to find the search box and type “lil wayne” into it, there was no search box. The resulting error provided a steady stream of screenshots of the home page throughout the day.

Let me explain a little more about why I got the screenshots.  With Keynote’s Web Content Trending option turned on, every screen is proactively captured and if an error is detected in subsequent steps, all captured screenshots are saved to the MyKeynote portal to support troubleshooting by our customers.  If there is no error, the proactive screenshots are discarded before ever being sent to the database.  Every time the second step failed, (which was on EVERY visit at this point), the screenshot of the home page taken on arrival was stored.

To get those screenshots, I simply had to drill into the scatterplot chart and I saw this:

(click image to view full size)

I clicked on that “E” which is the error recorded for the second step and saw the page details screen below. I have added callouts so you can see where the links to the screenshot and html are:

(click image to view full size)

Things were about to get a lot clearer in a hurry. 

The wrong kind of "direct communication"

I clicked on the thumbnail of the screen snapshot and sighed as I saw what visitors had seen in the moments when the site became unusable around 7:58-8:00am:

We captured the above screenshot at 8:01am.  It shows what the 640 and 639 byte html pages with no images looked like in the browser.  This is a raw server error that was passed through all the way to users’ browsers (something developers and operations teams work very hard to avoid).

In the minutes that followed, the target.com team took rapid action to replace the cryptic server error with something more friendly.

By 8:14 am, we had captured the first of those friendlier images; the “fail doggie” page made its debut. About 17 minutes later we captured a new version of the page.  We saw additional changes again at 11:30 and 12:38.  It should be noted that these are times when we visited with the transaction-based measurement and that particular measurement was only set to visit ten times per hour.  The point is that these times are when we observed the pages and made captures, not necessarily the exact times that they changed.

Here, then, is the full gallery of “fail doggie” pages we recorded:

8:14 am EDT – “Oh no”

This page requested the user to “please try again” and provided a single link to “Target help”

(click image to view full size)

8:31am EDT – “Hello”

This page let folks know the team was “hard at work making the site better” and dropped the “please try again” in favor of “Sorry for the inconvenience – we’ll be back up and running shortly.  It also featured links to three services that were still online: redcard, weekly ad, and find a store.

(click image to view full size)

11:30 – “Woof” #1

Around 11:30 the message changed to “We are suddenly extremely popular.”  Visitors were also asked to “Please stay here and we’ll try to get you in as soon as we can!”  Finally, this version also explained what the three links that began appearing in the previous version were, saying “We are up and running here” just above the links.

(click image to view full size)

12:38pm EDT – “Woof” #2

This version added a plea to not keep hitting refresh, something that can make bringing a site back up very difficult when a large group is all doing it at the same time: “Please know that there is no need to refresh your browser.  Your request will automatically retry in 30 seconds.”  This was the most well organized page, broken into four separate paragraphs, and adding back in an apology with the sentence, “Thank you and our apologies for the inconvenience.”

(click image to view full size)

Sizing up the impact: just how bad was it, and for how long?

I was starting to put it all together.  Now I knew what the raw server error looked like and I had a play-by-play set of screen captures showing how the “waiting room” page evolved over time that morning. 

What I still wanted to know was just how “unavailable” the site had been.  Were any users getting through to the real home page at any point?  The fail doggie page said it would auto-refresh in 30 seconds and implied that perhaps one might eventually get through.  Was the real home page ever turning up, and if so how often? 

I waited until the next morning to size up the duration and intensity of the outage.  My goal was to confidently establish an “end point” for the incident and then do the numbers.

How can you measure “availability” when the site is displaying “OK” pages, but not the right ones?

The home page only measurement had the highest frequency (about 40 per hour) so I wanted to use that to calculate availability.  There are many different reports and graphs in the MyKeynote portal, and most of them will tell you at a glance what your “availability” is – that is, what percentage of the measurements succeeded versus failed in some way.  The catch here was that server error and “fail doggie” pages were sent to the browser as “OK” (http status 200) pages, not errors. 

Scripts can be easily enhanced with validations that look either for required text that should be there or error text that should not be there.  Either validation option would have marked the server message or fail doggie pages as errors and impacted the built-in availability calculation, but the index measurement I was using didn’t have that validation in place.  In practice, validation is usually used at the end of a multi-page script to be sure the right final page had been reached.  Fortunately the information we needed to answer the questions was readily available anyway (more on that below). We just had to step back and consider the available tools to size it up.

The tool I chose to use was MyKeynote’s Object Trending report.  This report is available for any measurement that has been configured with our Web Content Trending (WCT) option.  WCT stores performance information for every object in the page, not just a roll-up for the page as a whole. Once again, storing all those details day and night was about to become very handy.

The Object Trending report has several options, all of which provide ways to view the performance of page elements over time.  To display it, I chose the Target measurement, selected Object Trending and set the date range to the 24 hour period after the incident had begun:

(click image to view full size)

Here’s what that report looks like by default, which is a separate line for each domain that objects originated from:

(click image to view full size)

The above default view is great for determining if one or more domains are particularly slow or unavailable, but I needed even more detail than that, so I dropped down the menu at the bottom of the graph and changed it to “Object data by object without parameters:”

This option would give me each separate element in the page, ignoring any query-string data that might be appended to the object name.  Think of the home page as a stage with a cast of characters.  The object trending report was about to show me who had appeared in what number of performances. 

I clicked “Generate Graph Now” and scrolled down to the table below the graph. Now I had a listing of the frequency of every object that had appeared on the home page from 8:00am EDT onward:

(click image to view full size)

What I was particularly interested in was the object name on the left and the “Included datapoints” on the right.   Every visit always had retrieved at least the base page (www.target.com). By comparing the count of base pages observed to the count of all the other objects, I could make some meaningful conclusions about how often the “fail doggie” had been turning up. 

I needed to do a little math, so I pulled the results into an Excel spreadsheet with a simple copy and paste, sorted by the datapoints column and added a new column to compare the count of each element to the count of the base page.  I did this a number of times, varying the time period a bit to narrow in on useful takeaways.

For starters, I re-ran the report to look just at the objects observed in the first hour between 8:00am EDT and 9:00am EDT.  The cast of characters is quite small; we either got just the base page of html, or the base page plus the elements of the fail doggie page.  Remember, these aren’t observations of all user traffic, they are the results observed by the visits of our real browser agents. While we were just a drop in the bucket of actual visitors, we did make it through to the site 31 times in that first hour.  Here’s what we saw: The “fail doggie” appeared just 55% of the time and we got nothing but the base page in the remaining 45% of visits.  The hundreds of thousands (or millions) of users attempting to visit during that same period likely saw a similar mix.

(click image to view full size)

So I could see that things had been pretty “messy” during the initial confusion of the incident.  An hour of outage is worth a lot of money to a site like target.com, and cryptic errors couldn’t have been good for inspiring confidence, but an hour is just an hour and perhaps many people hadn’t even tried to visit the site yet. 

The next question I wanted to answer was, “What percentage of Target.com’s traffic was able to make it to the home page throughout the rest of the day?”

Here’s a view of my spreadsheet based on running the Object Trending report for the period starting one hour after the incident (9:00am EDT) and ending at midnight that evening:

(click image to view full size)

So taking the big view of that entire day up to midnight EDT (which is 9pm Pacific time), I determined that 93% got the fail doggie and no more than 7% got through to the real home page.

Not pretty no matter how you slice it

I re-ran the report with various time windows, and the results varied only slightly.  Most surprisingly, even pushing the end time all the way to 9am the following day, the stats still showed that 85% of all visits got the fail doggie page elements.  Focusing on Midnight 9/14/11 to 9:00am 9/14/11 the number was still high at 75%.  Was this all cleared up by the start of business on 9/14/11?  Looking at just the hour between 8:00am and 9:00am that second day 9/14/11, the number was still an amazing 50%.

By this time my boss was wondering why I was spending so much time with all those spreadsheets and charts, so I stopped my investigation there and moved on to share updated results with all the folks that had been asking for data.  I annotated a screenshot of a scatterplot graph and wrote a narrative to go with it, sharing the details with several media outlets and even conducting a radio interview with Wall Street Journal Radio.  Here are a couple of links to places the results showed up, along with a copy of that annotated scatter-plot graph.

Investor’s Business Daily: Target Website Crash Offers Lessons

Retail Online Integration: What You Can Learn From Target's Site Crash

(click image to view full size)

Epilog:

The one question I left unanswered was just how many truly useful pages there had been in the 7% that were not “fail doggie” pages.  I’m curious how many would have actually allowed a customer to purchase.  Given the size of Target.com’s typical traffic this time of year (reported as 29.5 million for the month of the prior October by Investor’s Business Daily), I was pretty content to stop with the work I had done over those two days, observing with a long sigh that the folks at Target, who are no amateurs at online retail, had missed out on a LOT of potential transactions by turning away something north of 93% of all visitor attempts that first day.

Script and Monitor RESTful APIs in Public and Private Clouds

By Ian Withrow

In the 4.1 release of KITE we believe that we’ve significantly improved the ease and power with which users can create scripts to monitor APIs. Web APIs, aka Web Services, are an important way for companies to integrate 3^rd party data and service providers both in the browser and in the data center. Web API’s that need to be monitored tend to fall into one of three categories.

• First, all the widgets and mash-ups from companies with Facebook, Google, and Twitter are built upon web APIs.  However, even traditional companies like Best Buy, Hoovers, and the New York Times offer APIs.
• Second, most companies talk to one or more 3^rd parties directly from their data centers.  Examples from this category might include payment vendors like Zuora and PayPal and range to specialized service providers like Quova (IP Geolocation) and Netbiscuits (Mobile site delivery).
• Third, many companies are starting to build their applications using a Web Oriented Architecture, which calls for modularizing different components of the application and exposing them via a Web API. 

Regardless of the scenario, the reason for monitoring these APIs is the same. API performance can have a significant impact on user experience especially when it drives critical transactions like checkout or the delivery of customized content. Unlike other monitoring solutions Keynote actually interacts with the underlying API, enabling you to test not only performance but to ensure responses conform to specific parameters, e.g. number of widgets in inventory is a number between 1 and 100. In this way you can reproduce the logic found in the actual application that consumes the API.

In this two part blog series I’ll walk you through exactly how to script and monitor APIs with Keynote both inside and outside of your cloud. This post will cover RESTful APIs. The second post will cover SOAP APIs. This article is long and in depth so below I’ve listed the major sections and the associated topics:

• Step 1: Create a basic web services script
• Step 2: Techniques for iterative development
• Step 3: Advanced script editing
• Selecting a Keynote monitoring product for your script

 

Step 1: Create a basic web services script

For the purposes of this demonstration we need an API, I’ve chosen the Active.com API because it’s a real, working API that has rich features like developer keys and multiple format responses.

To get started you’ll need KITE 4.1 or newer. If you don’t have KITE a copy can be obtained for free here. Launch KITE, and in the top left corner there should be a button that looks like a big red record button. Click the bottom half of this button which opens a drop down menu and then select ‘Create Web Service Script’ as shown below.

You will be asked to select a service type, go ahead and select REST.

Upon doing this you’ll be presented with a wizard, as shown below, that will gather input from you in order to construct an advanced script. First, you enter the base URL for the RESTful API function you wish to access. Next you specify the response format type you will receive, such as JSON, XML, or unstructured text. Then you can proceed to fill in all the URL query parameters that will be passed with the request. The Wizard supports all standard HTTP request such as POST and DELETE. You can also enter any special headers at the bottom; standard headers will be created for you. In this API, the license key is added as a parameter but other APIs may call for a header.

Don’t worry if you don’t have everything perfect just yet. You can manually edit the script later or simply recreate the Web Services action and the Wizard will remember the last inputs you gave it. When you are ready, click ok and you’ll see a new action added for you in the Script Viewer. 

 

The best practice suggests that at this time you should rename the action something logical by clicking on ‘Call method’ and edit the Script Property shown below. My API call retrieves events in the San Diego area.

 

Now we are ready to view our completed script, you can do this by clicking on ‘REST Script’ on the left, note that this too can be renamed. When you do you’ll get a new window, called Advanced Scripting. Please note if you have a lot of windows open in your KITE instance you may need to close or resize some to get the expansive view that I have below. We’ll explore the different elements of this script later but for now the key portion is boxed in red, this is the RESTful query that you’ve just created. You can now double check and edit it manually using JavaScript.

 

Once done, you click the ‘run’ button as normal to test out the script and see what results you get.  If everything works you’ll get a screen like the below shaded in green.

 

For comparison here is what failure looks like, in this case I deliberately mistyped my license key resulting in a 403 error.

Step 2: Iterative Development

In addition to the Wizard we’ve added three great tools for iterative development, which we’ll examine in turn within this section:

1. Advanced script log
2. HTTP request/response header payload viewer
3. Using saved variables in subsequent actions

Advanced Script Log

After you run a script, the Advanced Script Log will be automatically updated and added to your screen as shown below. The log displays the request made as well as any error response details received. If the response was a successful, then the results are store into variables. For example, 0.42716 is saved in the variable [searchTime], more on how to use this later.

 

HTTP Request/Response Header Payload Viewer

Sometimes I find it helps to see the exact request and response payloads. Now there is an easy way to do this. In the Transaction Performance Details window, which is typically right below the Advanced Script Log, right click the step or object that you are interested in and select View HTTP Payload from the context menu.

Back on top you’ll get the below screen which shows you the exact headers and payloads, up to a character limit.

 

Using Variables in Subsequent Steps

The real power of this tool is to create a sequence of API calls that utilize results received in prior steps, just like a real program would. Let’s say that after our query of local events in San Diego we now want to get the event details for one of the results returned. A quick review of the API’s developer documentation shows that this is possible via the ‘assets’ API method which uses the variable assetId in its base URL. Let’s add that step using a saved variable from our first step. First we should copy and save the variable we want to use, the assetId of the first event in the list. We can find our quarry in the Advanced Script Log as shown below; we want everything inside and including the brackets.

 

Next we need to add a new step. This is done, as normal in KITE, by right clicking the script name in the Script Viewer window and selecting a new REST action as shown below.

Now we are back at the Web Services Wizard. We again start by entering the base URL, but this time we need to append an assetId at the end of the base URL. As you can see below, we can paste the bracketed saved variable directly into the wizard. Next we need to set the Results Format, let’s try XML this time. Finally we need to input our developer key and any other relevant parameters.

Now when we check the results of our new script we see that the code generated is retrieving our saved variable. This could also be done manually, as sharp readers will no doubt guess, but this way saves a lot of time and effort.

We can now run this two-step script and check the HTTP Payload of the second object to confirm we got the results we expected, in this case by double checking the assetId.  As we’ll explore in the next section, it is possible to extend the script to add your own custom validation logic on elements like this and throw errors depending on the results.

The script is now done so we can save and upload it to Keynote for deployment to a monitoring agent.

Step 3: Advanced Script Editing

As I alluded to earlier, it’s entirely possible to manually rewrite the script created by the wizard and even modify the template itself. First though let me explain the organization of the built-in script template. Each script, both REST and SOAP, is organized into three logical sections as shown below. The red section is the parser which takes the results and saves them in variables. By default the script will usually only save one version of a variable, i.e. if you have multiple assetID’s it will save only one, the last. This is fine in most cases, however there may be times when you want to enhance this behavior and that can be done here. The blue section is where the actual query is built, this code block will obviously be pretty different between REST and SOAP but the organization is the same. Finally, the purple section is where error handling is done. The built in error handling can catch HTTP as well as API errors; however you may wish to extend this functionality further. As mentioned previously, all of this is done in JavaScript.

Let’s say you find yourself frequently updating the built in script functionality and want to make a global change. This can be done by modifying the files named wsdl.template and rest.template. These files are found in your application data folder under …/Keynote/Record/Config/. So for example in Windows 7 this would be username/AppData/Roaming/Keynote/Recorder/Config/. I encourage you to back up the originals before experimenting with this though.

Selecting a Keynote Monitoring Agent for Your Script

Now that you have created a brand new Web API monitoring script you’ll need to select some agents to run it. You have two choices, our Application Perspective (ApP) and Cloud Application Perspective (CApP) products. ApP is a collection of globally distributed monitoring agents that are ideal for monitoring your own APIs that are accessed by a broad range of consumers and those 3^rd party APIs integrated at the client or browser. However, for APIs and partners that your own applications require direct access too you can deploy your own CApP agent. CApP is your very own Keynote agent software which you can install in your own private cloud or even within an important business partner’s cloud. The short rule of thumb is you want to monitor the API from where it will actually be used.  If it’s consumed by end users around the world go with ApP. If it’s consumed by applications within a private cloud or network then deploy your own CApP agent there.

 

Script and Monitor SOAP APIs inside Your Private Cloud

By Ian Withrow

SOAP APIs, or Web Services, frequently form the backbone of private enterprise to enterprise communication.  Companies use these APIs to integrate value chains, share information, and integrate applications that depend on partners. While RESTful APIs may be taking the technology world by storm, SOAP still has an established place in private enterprise communications. Even as organizations migrate from dedicated physical resources to private and public clouds, SOAP APIs are likely to continue to play a role for now. As a result, web applications often depend on the information delivered by a SOAP API. As such if the API is slow then so too shall the application be. Thus it makes sense for businesses serious about performance to monitor their partner APIs, whether they are SOAP or otherwise.

In this post I will show you how to easily create an advanced SOAP script using the new enhanced functionality available in KITE 4.1, topics covered here include:

• Step 1: Create a basic web services script
• Step 2: Techniques for iterative development
• Step 3: Advanced script editing
• Putting your script to work with a Keynote monitoring agent

Note that I discuss how to script RESTful APIs in another post.

Step 1: Create a basic web services script

Public SOAP APIs are rare these days, most are private and require authorization.  For the purposes of this demonstration I will utilize the free, public Holiday Web Service API which supplies holiday dates for a small selection of countries.

To get started you’ll need KITE 4.1 or newer. If you don’t have KITE a copy can be obtained for free here. Launch KITE, and in the top left corner there should be a button that looks like a big red record button. Click the bottom half of this button which opens a drop down menu and then select ‘Create Web Service Script’ as shown below.

You will be asked to select a service type, go ahead and select SOAP.

After clicking ok a Wizard will pop-up on your screen. This wizard will automatically generate an advanced script for you that will make a properly formatted SOAP request, parse the results into a log, and capture errors. First, load the WSDL file for your API. Next you select the function you want to call; in this case I want a list of supported countries. A sharp eye might to detect what seems to be a duplicate list of functions. In fact there are multiple versions of SOAP and you will see duplicates when the WSDL specifies the same method for multiple versions. For Keynote monitoring purposes they work the same.

When you are ready, click ok and you’ll see a new action added for you in the Script Viewer. 

The best practice suggests that at this time you should rename the action something logical by clicking on [Action] and editing the Script Property shown below. 

Now we are ready to view our completed script, you can do this by clicking on ‘SOAP Script’ on the left, note that this too can be renamed. When you do you’ll get a new window, called Advanced Scripting. Please note if you have a lot of windows open in your KITE instance you may need to close or resize some to get the expansive view that I have below. We’ll explore the different elements of this script later but for now the key portion is shown below from row 36 to 53, this is the SOAP query that you’ve just created.  You can now double check and edit it manually using JavaScript. Note that the RequestXML variable contains just the SOAP envelope body if you are comparing it to a reference example.  If you needed to add login and authentication information this is the section of the code to do so, either be adding an additional KNWeb.AddHeader command or by including it in the RequestXML string.

Once done, you click the ‘run’ button as normal to test out the script and see what results you get.  If everything works you’ll get a screen like the below shaded in green.

For comparison here is what failure looks like, in this case I deliberately messed up the envelope by deleting a tag.

Step 2: Iterative Development

In addition to the Wizard we’ve added three great tools for iterative development, which we’ll examine in turn within this section:

1. Advanced script log
2. HTTP request/response header payload viewer
3. Using saved variables in subsequent actions

Advanced Script Log

After you run a script, the Advanced Script Log will be automatically updated and added to your screen as shown below. The log displays the request made as well as any error response details received. If the response was a successful, then the results are store into variables. For example, country code GBSCT is saved in the variable:

[GetCountriesAvailableResponse.GetCountriesAvailableResult.diffgr:diffgram.NewDataSet.Countries.Code]

I’ll show you more on how to use this in a second.

HTTP Request/Response Header Payload Viewer

Sometimes I find it helps to see the exact request and response payloads. Now there is an easy way to do this. In the Transaction Performance Details window, which is typically right below the Advanced Script Log, right click the step or object that you are interested in and select View HTTP Payload from the context menu.

Back on top you’ll get the below screen which shows you the exact headers and payloads, up to a character limit.

Using Variables in Subsequent Steps

Retrieving the list of supported countries is nice, but next let’s try to get some real information like a list of holiday’s for one of those countries. In this way our script can behave much like a real program on whose behalf we are monitoring performance. We know, from the parsing of the WSDL, that there is a GetHolidaysAvailable function. Let’s add a step using that method together with a saved variable from our first step. This would require the country code shown in the previous section from the Advanced Script Log: GBSCT.

Next we need to add a new step. This is done, as normal in KITE, by right clicking the script name in the Script Viewer window and selecting a new SOAP action as shown below.

We are at the Web Services Wizard again. This time we click on the ‘GetHolidaysAvailable’ function and a second popup is displayed asking us for input parameters. In this screen we will use the saved parameter as the input variable as shown below. Be sure to include the brackets then proceed as before by clicking ok.

Now when we check the results of our new script we see that the code generated is retrieving our saved variable.  This could also be done manually, as sharp readers will no doubt guess, but this way saves a lot of time and effort.

We can now run this two-step script and check the HTTP Payload of the second object to confirm we got the results we expected, in this case by double checking the countryCode.  As we’ll explore in in the next section, it is possible to extend the script to add your own custom validation logic on elements like this and throw errors depending on the results.

The script is now done so we can save and upload it to Keynote for deployment to a monitoring agent.

Step 3: Advanced Script Editing

As I alluded to earlier, it’s entirely possible to manually rewrite the script created by the wizard and even modify the template itself. First though let me explain the organization of the built-in script template. Each script, both REST and SOAP, is organized into three logical sections as shown below. The red section is the parser which takes the results and saves them in variables. By default the script will usually only save one version of a variable, i.e. if you have multiple countryCode’s it will save only one, the last. This is fine in most cases, however there may be times when you want to enhance this behavior and that can be done here. The blue section is where the actual query is built, this code block will obviously be pretty different between REST and SOAP but the organization is the same. Finally, the purple section is where error handling is done. The built in error handling can catch HTTP as well as API errors; however you may wish to extend this functionality further. As mentioned previously, all of this is done in JavaScript.

Let’s say you find yourself frequently updating the built in script functionality and want to make a global change. This can be done by modifying the files named wsdl.template and rest.template. These files are found in your application data folder under …/Keynote/Record/Config/. So for example in Windows 7 this would be username/AppData/Roaming/Keynote/Recorder/Config/. I encourage you to back up the originals before experimenting with this though.

Putting Your Script to Work with a Keynote Monitoring Agent

Now that you have created a brand new SOAP script you’ll need to setup some agents to run it. Since most SOAP APIs involve private application to application communication, Keynote’s Cloud Application Perspective (CApP) product is the ideal tool for the job. CApP is your very own Keynote agent which you can deploy in your own private or public cloud and even within an important business partner’s network. In this way you can monitor the SOAP API from where it is actually queried in order to get an accurate picture of performance.

 

Monitoring Internal Applications with Cloud Application Perspective

By Ian Withrow

Keynote is typically used to monitor customer facing production systems today. This is a side effect of the twin facts that our monitoring agents are on the public internet and of course many companies understand they lose a lot of money from poorly performing or unavailable websites. However, with Cloud Application Perspective (CApP), Keynote can now be used to monitor internal and private applications just easily as one might monitor a public one. Since CApP is a customer installable monitoring agent, it can be easily deployed in branch offices, customer locations, and of course data centers to monitor internal private communications. Much of what the First Mile and API How To posts cover with respect to scripting and deploying measurements are equally applicable to internal applications. In fact, in this entry I won’t walk you through these steps assuming you’ve read one of the previous posts. What I will highlight here are some of the nuances and other considerations that crop up when you are monitoring internally. So this is more of a grab bag of issues than a step by step guide.

An important detail when deciding to monitor internal applications is why. Well the obvious answer is that internal users are impacted by performance issues, which in turn influences their productivity and satisfaction. These are factors that a sophisticated IT operation should care about in the age of cloud computing and web applications.  However, another reason is vendor selection and contract renewal. Monitoring services with Cloud Application Perspective provides a wealthy of good data that can be used to better choose among vendors, inform contract renewals, and in general keep them honest.

A key step before monitoring these types of applications is to find a system to run your Cloud Application Perspective agent(s) on. It’s easy to find a server in a data center, but not all locations are this flexible. CApP supports old operating systems like XP and needs minimal system resources (CPU, RAM, and Disk) to operate. Probably your biggest consideration is network I/O which is fortunately easy to estimate and manage since you specify both the frequency and volume of content your agent will download. In fact since CApP runs silently as a Windows Service some of our customers have elected to load it onto an end user machine or point of sale system where no other computer was available.

While we are on the topic of end user devices, many customers may want to monitor an application that is destined for a device that isn’t Windows, for example an iPad. Cloud Application Perspective can easily emulate any browser or device type on a per script basis. To do so simply open any script in KITE and select “Settings” under the script name at the top of the “Script Viewer” pane. Then choose the drop down menu next to “Browser” and pick the agent type you want.

Keynote has a number of preconfigured device profiles but you can also make a custom one. If you are uncertain what the custom user agent should be then either this website or a simple Google search should answer the question. If you happen to have the device then you could also visit a site like this to find out. Note that IE7 is the default in KITE at the time of writing.

Another sticky issue you are likely to run into is two-factor authentication, which for web applications typically utilizes cookies or digital certificates. Cookies can be easily added by right clicking on the script name as shown below.

Once it is created you can click on the blank cookie to edit the details.

Certificates are little trickier. Fortunately, these can be imported from a local browser once they have been created on the machine you are writing the script on. As before, select the script name and in the “Script Properties” pane below there is an add button next to “Certificates.” This creates a pop-up window that displays all the certificates available on the machine, these can then be imported to the script.

Another application that uses two-factor authentication is Salesforce. After the first login Salesforce will prompt you to authorize a new a machine. In these types of cases you’ll need to do this not only for the device on which you are running KITE but also the device(s) on which you’ll run the CApP agent. As for Salesforce, this is a particularly challenging but important site to script so look for a future post all about them. Always bear in mind that Cloud Application Perspective comes with a support contract from Keynote so if you get stuck then help is just an email away at support@keynote.com.

I’ll close out this post with a general suggestion: monitoring from multiple locations increases your ability to quickly triangulate on the location of a problem. The graph below best highlights this advantage. You can see that a performance problem in Beijing is not felt by other offices so we know that this isn’t likely to be a data center or application problem. 

This type of at a glance data can be invaluable when an angry VIP is blaming your application for a performance problem. Correlation like this is why it’s important to accurately specify agent location when requesting a new CApP agent in the Keynote Service Center.

 

How To: Monitor Web APIs with Cloud Application Perspective

By Ian Withrow

Keynote is known as the end user experience monitoring expert, specifically for humans browsing a website with a web browser. However, there is another set of users that we can monitor: applications. More specifically we can monitor the experience of any application consuming a Web API or Web Service. Of course many of our customers understand that we can monitor any HTTP traffic and have been using us to monitor RESTful API and SOAP calls for a long time, but it has always been a subset of our business.  In the immediate future we see that changing with Cloud Application Perspective. (CApP) I believe two cloud trends will drive this change, which I’ll touch on next. However, the ultimate goal of this post is to show you an example of a RESTful API I scripted for our CApP product.  (Although it would work for our public agents too)

The first trend that will foster an increase in critical API traffic is the adoption of hybrid clouds, specifically applications distributed across them. By hybrid clouds I mean organizations with elements of their applications both on a public cloud and a private cloud computing platforms. This is a popular deployment model for business reasons ranging from the regulatory to the religious to the practical. For example, a company might plan to deploy the core of their application in a public cloud to take maximum advantage of scale, geographic expansion, elasticity, and bursting for unexpected demand etc. However, a key piece of the application will remain in a private cloud or possibly on dedicated hardware. It could be that migrating some legacy code isn’t feasible or perhaps there is data there that the company wants/needs to keep under its direct control. Whatever the reason, the public cloud portion must now send queries back to the enterprise data center. A robust and manageable way of handling this is via a Web API. In this case CApP is then deployed at one or both sides of this link to monitor performance.

The second trend will be driven by companies who aggressively embrace public cloud computing solutions and design their applications for them. A core tenant of this approach is to design your application to scale and contract automatically, as well as terminate and replace instances that become unhealthy or even need to be updated. How will all of this be done? The cloud computing provider’s APIs. Interestingly the performance and availability of the management APIs now become very important and thus merit monitoring. Imagine what happens to your application if you can’t complete requests to this API in a timely fashion or at all? To monitor this interaction, CApP is deployed wherever you need to query these APIs, likely in the cloud itself and near your operations center or devops group.

There are of course many ways to monitor APIs. Many are rather superficial, like a ping or response time test. This of course tells you that the physical system is up and responding in a basic sense, but it doesn’t mean you can actually do something useful. This is the strength of Keynote’s API monitoring, something which I’ll demonstrate here with an example. I’ve picked the Active.com RESTful API because it’s a good modern RESTful API with concepts like developer keys and nested queries with the added benefit that it’s free to use.

To start scripting a RESTful API, enter KITE and create a new ApP script as shown below.

Next you’ll be asked for a URL to start with.  You have two options here. You could input the entire URL for your first API request. KITE will understand that. However, if you aren’t sure yet of the parameters you want to use you can also just enter the base URL for the API query you want and define the variables in KITE later. In our case the base is:

http://api.amp.active.com/search

KITE will then bring up a browser in an attempt to visit this webpage. Just immediately stop the recording window and ignore the likely 403 error you get. At this point you’ll see something like this:

As a procedural note in future screen shots I may show only a subset of this window, in particular just the script viewer, but I started with a full screen shot so you can orient yourself within KITE. Now we want to add query parameters. To do so, right click “Navigate to” and select “Add Queries.”

Doing so will add a Queries section to this step in KITE which is basically a place to specify variables to include in the URL request. To add a name/value pair or just a name right click “Queries” and choose the option you desire.

If you choose to paste the exact request string for the first query when you create a script then these fields will be pre-populated. Now we can generate our full request, note that if you have an API key of some sort it can be specified here. For this query we are looking for activities in Active.com’s database near San Diego. To do this we add a Name/Value pair and edit as shown below in the “Script Properties Editor” box.

A sharp observer might have noticed that Type is static, implying we could pass a non-static piece of data here. In fact you can retrieve/save data from a previous step to use in subsequent actions. Let’s say we want to see all the events going on in San Diego and retrieve the event details for the first one. Active.com offers another query for retrieving event details based on assetid:

http://api.amp.active.com/resource/assetservice/asset/<put assetid here>

So how do we tell the script to pull this value from the first step for use in the second? Well for starters if you are a visual person like me it may help to actually run the first stage to see what we are dealing with.

I’ve highlighted the field we are interested in, also note that the first entry is the World Run Day at Chula Vista. Now we know what we need to in order to create the query. First we add a new action by right clicking the Script name and selecting “Add Navigate.”

Next click on the new step we just added which will have blank URL. Type in the base query URL in the script properties editor. When done click on “Edit” so we can add some logic to find the correct assetid and append it to this request.

You’ll get an “Edit Parameter” box, select “Append” and then “Ok.”

Now we’ll deviate from using static values. Click the drop down menu and select “Text found from page”. This will allow us to search the previous page for some information. Note that we have a number other options ranging from a saved value to Javascript parameters.

From our previous query we know we want text between two XML tags. In fact we can even copy and paste the tags out of that test query in order to avoid typos. There are other optional parameters here, for example if we wanted the second event we could increase the index value.            

One final step not picture here is that I needed to add a name/value pair to this query just like I did in the previous step for my API key. Once done I can run the query and see that I do in fact get the World Run Day record.

Ok that’s enough for our simple starter example. In a production script you may wish to consider adding other details, like error handling, depending on the API. For example what happens if we run a query that has no entries? Is that a failure to you? KITE provides you the flexibility to add this logic to the measurement and report/alarm on it in MyKeynote.

To review, whether you have a hybrid cloud or you are going whole hog into a full public cloud, entering this world means you’ll likely depend on some Web APIs. CApP can easily be scripted to monitor these which will reduce down time and increase visibility into your systems.

 

Performance and the Cloud – Does it Matter?

The Keynote team recently returned from the Cloud Connect conference in Santa Clara California having had an opportunity to meet lots of people looking for clarity on the concept of performance. While many of the attendees were there to learn about moving to a cloud computing model, those that have already adopted the cloud raised several interesting questions about what performance means and why (or whether) it matters.

At one panel, the moderator asked each of the panelists to describe “performance in the cloud.” After getting everyone’s answers, he suggested that they all sounded the same as how performance was described ten years ago. So what has changed? How can this be when, in a public cloud like Amazon Web Services EC2, you have minimal visibility and control over resources with varying characteristics that are unpredictably warped by the behavior of shared tenants?

The consensus of the panelists was clearly formed around managing to end user experience. Reddit, the social content linking service, commented that performance for them is all about driving page load speed. “If we can get 10% more performance, we immediately see 10% more traffic.”

So how do you go about achieving this? First, you’ve got to architect your application for the cloud. As one CTO suggested, anyone who believes they can take an application developed for use in a dedicated on-site environment and put it in the cloud without issue is delusional. And once your applications are properly architected for your cloud environment, you can exploit its elasticity and “mask” certain application performance deficiencies. Essentially, find performance improvements by scaling more capacity.

Hmmm. Is instant capacity the new performance? And what about SLAs? We’ll talk about cloud performance more in future posts, but let us know your thoughts!

IE No Longer #1 - Are You Keeping Pace?

Today the pace of change in how your customers are accessing your Web site is breathtaking.  The massive rate of change in the past 12 months in browser market share is staggering. Most notably, Firefox 3.5 is now used in Europe more than any other single browser version, and usage of Internet Explorer worldwide is dropping faster than Middle East autocracies.

Why is this important? Browsers matter when it comes to web performance. Different browsers display different behaviors, especially with regard to the AJAX and Flash applications used to create rich Internet experiences. These applications are dependent on client side scripting and the initialization of plug-ins which requires understanding how the browser behaves as it interacts with a Web site, not just the performance of the Web site itself. And although application behavior can be tested across browsers within your development and QA environment, performance issues often manifest once these applications are moved into production. Additionally, network latency and third-party content interactions at the “last mile” can contribute to an unacceptable user experience.

Keynote customer LinkedIn recognizes this and monitors its performance across both IE and Firefox using Keynote Transaction Perspective. Keeping a vigilant watch on the performance of their services from the end user’s perspective is a priority so they can quickly identify and resolve issues.

Is it also yours? I’d love to know if and how you are monitoring your rich Internet applications across multiple browsers.