Mobility

A Holistic Approach To Mobile Testing - Chapter 1

Most of us probably have the same "mental image" of what it means to do mobile testing. We envision a person sitting at their workstation holding a mobile phone in their hands and pecking out the sequence of steps needed to exercise some function of a mobile application or service. On their desk we see a whole bunch of different handsets laid out in a row. When you finish all the test cases on one handset, you set it down, pick up the next handset and repeat.

A more advanced variation of this approach is where you "rent time" to access a particular handset in a hosted service model. It is still testing using a real handset, you are just accessing it remotely. Testing with real handsets allows you to verify that what is shown on the LCD of the device "looks cosmetically correct" for each step of the test case. The downside of this approach is that you have limited access to any type of diagnostic information other than what is shown on the LCD of the real handset. If your question is "Why is the page slow to load?", testing with a real handset does not provide detailed root cause analysis information needed to solve this problem.

For testing mobile internet content, a second approach is based on a "virtual device" or emulator. In this approach, the handset under test is mostly based in software. Why would you want to test using this approach? One of the biggest reasons is that a virtual device approach lets you see what is happening "behind" the LCD. Since the virtual device is in control of its own software stack, you can collect important testing information on "each component" of a content page. For example, how long does it take for each image on the page to load? Does the page contain URL redirects that slow down the loading process? Are all the images returned in a size that is compatible with the target device? What is the full URL of a broken link on the page? These questions can all be easily answered using a virtual device testing approach.

Each approach offers answers to certain types of testing questions. Using both techniques offers a holistic approach to mobile internet testing.

For example if the testing goal is to validate that the LCD "looks cosmetically correct", then the right approach is to test with the real device. Any type of virtual device could never mimic the exact behavior of the real handset.

On the other hand, what if you are testing with a real device and you receive the dreaded "Page Cannot Be Displayed" error? If you are testing with a real device, you can not quickly determine the root cause, but with a virtual device you could access the markup code of the returned page, as well as the headers of the content request. For these types of errors the virtual device is superior in finding the root cause of the problem.

There is also the issue of economics. To test with a real device you need physical access to that device. If you have multiple testers, and-or multiple groups doing testing in different geographies, then providing access to physical devices becomes problematic. You could "rent time" on remote devices, but the costs of this activity used for every single test case could quickly add up to huge amounts.

For certain types of test cases where "cosmetic accuracy" is not the goal of the test, a much cheaper testing model based on a virtual device can be used. Many types of content compatibility tests such as verifying image sizes, or identifying broken links can be accurately and quickly performed using the virtual device model. In most cases virtual device testing tools are downloaded and installed on your local PC. There is no hourly rental fee.

Lastly, the virtual device approach solves the "device scarcity" problem. What if you need to test on a device that has not been released? What if you need to test on a device that is brand new in the market place and because of demand, few real devices are available for purchase? These types of problems are also solved using the virtual device approach. The virtual device approach allows you to quickly define and configure new device "profiles". The speed of deployment is much faster than waiting for a scarce physical real device. Multiple testers can also share virtual device definitions so that scalability is achieved within the testing group.

Keynote believes that both approaches should be used in concert to create a testing methodology that is thorough, practical, and economical. In the same way that a mechanic or carpenter uses different tools for different purposes, so should the mobile testing group use a collection of tools in order to produce the highest quality result.

Monitoring your ads . . . your print ads

Advertisers have long known that 1/2 of their expenditure on advertising doesn't work but they've not known which half.  And publishers desperately seek a way to monitor the effectiveness of prints ads so that they can keep their customers spending through a recession.

What if there was a way to track advertising using a mobile phone?

Here's how ad monitorng would work:

1) The publisher of the magazine or newspaper runs a promotion encouraging its readers to take pictures of ads they see in the publication and send them in.  In return the reader gets entered into a raffle for big prizes.

2) Readers snap picture of ads (presumably ones they like) and send them in via MMS.

3) The publishers sends back a response to the reader's phone via SMS confirming entry into the raffle and maybe a coupon for future purchase.

4) The publisher shares pictures received with advertisers to prove that readers are viewing the ads. Oh and one more thing. The advertiser now would know the telephone number of potential consumers for its products. 

Is this really possible?  Yes it's already happening enabled by a company called SnapNow.

Another example of how messaging via MMS and SMS has quickly become the killer mobile app changing our commercial world.

Keynote MITE

   

US Presidential Election 2009 - Many Firsts

There have been many firsts in the 2009 US presidential election.  The one that we have been tracking closely is the use of SMS for campaign news and voter outreach. 

This has been made possible by the tremendous popularity of texting in the United States where adoption came late but when once it did, it forever changed electronic communications. 

Texting has made email obsolete for an entire generation.  And the Obama campaign both recognized and capitalized on the change as voters signed up in droves to stay in touch with the campaign.

But what we and the presidential campaign have found on several occasions is that texting does not enjoy the same certainty of deliverability as email.   (Read more about results from Keynote mobile monitoring for texting online.)

A colleague recently wrote a column exposing the most recent issues he found on October 28th with SMS deliverability. You can read it online at Mobile Marketer.

This is perhaps understandable given the fact that the use of texting for mass communications is still in its infancy. The Obama campaign has raised the visibility of this medium.  Now SMS and short code service providers can benefit from increased interest but first they need to address issues with deliverability of text messages.

And the issue of SMS reliability isn't isolated to just SMS service providers (see list of companies online), reliability impacts the entire mobile ecosystem because of its popularity, versatility and revenue generating potential (read more about SMS revenue and mobile carriers).   

Allegory of the screen: Plato, the Matrix, and mobile testing. (Part II)

Part II

   Plato's cave & the Wachowskis' matrix: how do they relate to the world of mobile development and testing? Well, ...fasten your seat belt Dorothy, 'cause Kansas is going bye-bye.

 

  Content developers are not afforded the luxury of ignorance. They are "enlightened" to the magic world behind the curtain and are tasked to make it better for the rest of us.  Their unique challenge is to create a better experience in our matrix-like view of the Internet. Multiply this challenge by the number of mobile handsets that can impact the user experience and a developer's challenge is truly daunting.  When code does not render consistently on the thousands of types of screens out there, the developer must identify the issue, fix it, and validate that the fix addressed the issue.  Now compound all of this with geographical and network differences.

   Until recently the options for testing and validation were few. They were solely dependent on real devices.  This solution can be costly, cumbersome, and slow-- dirty words when speed to market is key. Recently, new emulated testing tools have hit the market that help address these issues and provide quick and accurate solutions for content developers.

   The characteristics of hundreds of devices can instantly be downloaded to your desktop. From there you can choose a device and carrier, pull up a website and have direct access to the raw content components of a page from the network before they are rendered by the handset's browswer. More importantly, you can pull up the source code and find error, redirects, broken-links, and over-sized images. 

Neo: "Link, where am I?"
Link: "You're not gonna believe this, but you're in the mountains."
[Neo looks around. He is surrounded by grassy hills and snow-topped peaks]
Neo: [Sarcastically] "Really?"

   Now here's where it gets really cool.  Record the script you've run and you can re-run over and over again on the other devices.  What you're able to accomplish in a short amount of time is no less incredible than what Neo was able to accomplish in the world of Thomas Anderson (in Plato's world, Neo's slave name). One can proverbially fly through tasks and make sweeping changes that seamlessly alter and enhance our visual experience.

Neo:"Choice, the problem is choice." 

   But the choice for an emulated solution is not that simple.  Shaved headed Neo looks pretty close to his cool looking persona in the Matrix world... but not quite as good looking. And more importantly to the rest of us, it's not as we see things.  For many in the content development cycle there is no alternative to a seeing content on a real device.  For them, validation on a real device will continue to be needed. 

Morpheus: "Free your mind."

  The fatal flaw is to think that real device testing is the only means of improving the develop process.  It isn't an either-or proposition. Emulated devices can be used be used in conjunction with real devices in the development cycle and still result in faster time to market, and a better end user experience. Much like Neo, toggling back and forth within the Matrix, you can use emulation to rapidly test content and optimize it across many different user screens. Then, if needed, run a validation test on real devices. You should still find yourself doing more, in less time, and at a lower cost.

   Emulated device technology is powerful and can greatly impact time to market and the user experience for any content provider.  We at Keynote are proud to be at the forefront of this with our newest desktop testing product MITE.  Like Plato, we end this conversation without concluding that emulated is better than "real" but we will stand by his position that persuing knowledge is the ultimate goal.

Try it for free today and see what you think: www.mite.keynote.com

Neo: "Why do my eyes hurt?"
Morpheus: "You've never used them before."

"Help" means help. "Stop" means stop. "Logomachy" means ???

My family likes to watch "Dancing With The Stars". One of the fun things about the show is that you can vote for your favorite dance couple by SMS text. The way that it works is that you send the text "vote" to the short code that represents your favorite couple. So for instance send "vote" to 3405 like the following:

A few seconds after sending your vote, you get back a response saying your vote has been received.

Most TV viewers would probably leave it at that. But since this blog is about "mobile", and our topic is really about SMS, I decided to explore the limits of this service and test what other keywords are supported by the "Dancing" short codes.

The Mobile Marketing Association (MMA) publishes a best practices document that is considered to be the standard description of how SMS short code programs should work. The document recommends that all short code programs recognize standard keywords that are meant to help the consumer use the service in a way that promotes full disclosure. The document also recommends that all short code programs recognize certain keywords like "HELP" and "STOP" as a way to give users proactive control over how they interact with the service.

Just for the fun of it, I sent both "HELP" and "STOP" to the short codes of a few of the other dancers on the show, here are the results:

"Help" to 3407, and "Stop" to 3412 didn't return the results I was expecting.

Instead of returning a text message describing what options are available, it looks like "Help" is being processed as a vote. Same for "Stop".

So in my example "Help" and "Stop" seem to do the same thing as "Vote". In this particular instance, the "Dancing" short code program doesn't seem to be following the guidelines described by the MMA.

Just for the fun of it I decided to send one last text value to short code. I would pick a word that the service would definitely not understand and I hoped I would get back some type of helpful message.

Here is the screen shot, I'll leave it up to you to see what I did.

Keynote has automation technologies that can help test how a short code responds to different keywords sent to it. Maybe you work for a carrier, sms aggregator, or application house that has responsibility for 100's, or 1000's of short codes. If you need to audit all of them for MMA compliance, that might not be a fun task if you have to do it manually. Or are you exposing your company to some type of "risk" if you are not doing it all?

If you haven't seen our other recent blog posts, check out what we call "MITE" which is an automated testing tool that supports the ability to send and receive SMS messages.

In case you are wondering click here to find out what my "word" means.

Allegory of the screen: Plato, the Matrix, and mobile testing. (Part I)

Part I   

   Some 2,400 years ago Plato wrote The Republic, his most famous work and one that to this day has inspired philosophical thought and challenged our concepts of reality and truth.  In Book 7 of The Republic Plato's main character, Socrates presents the Allegory of the Cave.  Imagine a deep, dark cave hidden from the light of the outside world.  The cave is filled with slaves, they are bound from birth in a manner in which the cannot move.  Their heads are positioned by their unknown captors to face a wall upon which shadows form a puppet show of the outside world which they have never seen.  They are not permitted to see anything but this wall in front of them. These shadowy silhouettes they see on the wall are cast from a fire behind them and voices are added which may or may not accurately align with the images they intend to portray.  The quality of the shows are determined by the slaves based on their perceptions of what they should see with no concept of what of what is real and how the image got there.  One day a slave breaks loose and escapes to the outside world.  Bright light, truth, and perhaps some information overload destroy the comfort that the slave had once known.  And once truth is known going back his former "reality" becomes impossible.  

    Jump ahead to 1999 where Andy and Larry Wachowski gave Plato's story a makeover in their movie The Matrix.  In it, the main character Neo is the former slave who becomes enlightened to the realities of his world.  Once exposed he makes the conscious decision that he cannot go back to the false world in which the rest of us live.  In a Platonistic way he seeks truth and takes the red pill.  He's able to navigate behind the source code of the perceived world and manipulate it to act in ways we cannot conceive.  However, we are left with the question of whether our lives are better and richer than that lived within the matrix.  The character Cypher (bad guy) makes this perfectly clear when he argues that had he known truth he never would have chosen it and the red pill.

Cypher: "You know, I know this steak doesn't exist. I know that when I put it in my mouth, the Matrix is telling my brain that it is juicy and delicious. After nine years, you know what I realize? [Takes a bite of steak] Ignorance is bliss."

   The Matrix successfully took Plato's dark, subterranean world and adapted to the modern world of the Internet.  We all understand that everything we see on the Internet isn't real and know the sarcastic joke "it must be true, I saw it on the Internet."  The Matrix's use of the Internet allowed common people a medium to understand Plato's main philosophical question: if reality didn't exist as we know it, would we be happier knowing, or not? 

   People generally understand (hopefully) that the Internet, whether on your desktop of handset, isn't more than representation of millions of lines of code quickly assembled to give us a visual representation of what the content developer had intended.  Yet, most us are happier with what we see on the screen. More over, we're bothered when errors in the code or limitations of desktops / handsets create graphical experiences that don't conform to what we consider "real."

Next a much greater leap, one of thought. 

Smile, You’re on Candid Camera

CTIA IT & Entertainment was held in San Francisco this week. It is the 2^nd tradeshow hosted annually by the mobile industry’s main US trade group, which means it is carefully observed for new announcements and predictions.

If you didn’t attend and instead caught one of the event roundup articles you’ve read that attendees groused about everything from the sessions to the multi-floor layout and that the most some could say was that it showed maturity. In other words compared to previous years attendees felt there weren’t many new announcements or predictions. (And it seems that Apple might have stolen media attention in the lead up not only by its absence but also by hosting another event simultaneously in the city where, BTW, there was no new mobile news.)

Does this mean the mobile industry has in fact reached maturity? Hardly.

The mobile industry ecosystem is not standing still. Mobile carriers, content providers and handset makers are all intently working on improving the mobile experience to move you and me from talkers and texters to surfers and buyers. And the top of mind question is: what mobile app will ignite consumer imagination? At CTIA there were plenty of demonstrations on turn-key mobile portals and widgets that promise a Web-like experience. But are consumers just waiting to mobilize their Web world before they dive in?

One answer might come in the form of a San Francisco company a couple of years old founded by 2 Europeans. www.kyte.com has taken what is unique about the mobile experience and married it with what is contagious about the Web. Namely portability and social networks respectively.

If you have a Nokia camera phone running the S60 OS, Kyte allows you to stream video live, right from your phone, wherever you are to your community – your entire community. Musicians can connect with their fan base, politicians with their constituency.

Kyte enables revenue for everyone: mobile subscribers will need to get an unlimited data plan on 3G networks from their mobile operator; savvy artists, media and exhibitionist consumers will flock to higher-end Nokia phones such as the N95; and content providers and online communities will benefit from enabling Kyte-streams to make their site more sticky for their base.

If you thought texting and sharing photos online was limiting your ability to express yourself, with Kyte you can broadcast yourself 24 x 7 – or at least until the batteries run out. Thoughts? For my household at least, it’s best summed up by “wow, so much for privacy”.

[Footnote: Kyte bears no relation to a similar sounding product from Keynote known by the acronym KITE. And of course all other trademarks are the property of their respective owners.]

"Tackle" your mobile testing problems

Steve Largent (Pro Football Hall of Fame, President & CEO of CTIA) takes some time to walk the exhibitor floor at CTIA in San Francisco. 

Here Nisheeth Mohan, Keynote Mobile Product Manager, gives Mr. Largent a demo of our new mobile testing tool called MITE.

What can you do with MITE? Here is just one scenario:

Let's say you are manual testing with a real handset, you load your content in the mobile browser, but it is really slow to load. If you load other sites, the pages load very quickly, only your page is slow. How do you figure out what is going on?

MITE uses the concept of a "virtual handset" that allows you understand the "construction" of a mobile page. This means you can examine image download times, network redirects, and other actions that can cause your page to load slowly. The ability to examine individual components of a page gives you a better chance of figuring why your page is slow. This is just one of many things you can do with MITE. If you happen to be at CTIA this week, stop by our booth, we are in the middle of the 900 row on the lower level. Take a look at MITE and learn how you can get a free iPhone.

It's not you, it's me