Dara Kerr with C|NET wrote an interesting article based on an Incapsula study that showed how bots are now running the Internet with 61% of Web traffic.
* Source: Bot vs. human Web traffic distribution. (Credit: Incapsula)
I manage Keynote's web and online marketing team (yes, a new website is coming soon!), and I don't dispute the magnitude of this number. It's the bane of my existence.
As a data-driven marketing geek, bots are ruining my ability to report on the reality of what's working and what's not, especially when it comes to conversion percentages.
If You See an Agent, You Do What We Do...
Agent traffic, as we at Keynote perfer to call it in a very Matrix-esch way, is at the heart of how our we monitoring products work. Since, in effect, I am causing my own problems, if anyone is going to have some insight into solving the problem, it's me.
The question I asked on my first week on the job at Keynote was, "How do we filter out all of the Agent traffic from our Google Analytics web reporting?"
The answer was, "it's not as easy as you might think."
A Short Term Solution
Back in 2012 we posted an article to this blog about how to filter out Agent traffic by making changes to the tracking code you deploy on your website. This method worked by identifying andf filtering out certain unique types of browser agent strings that Keynote and other performance monitoring tools use. All was well and good until Google made an update to GA that broke this hack.
I needed a way to remove Agent visits from GA altogether without relying on complex Segments.
The Real Solution
After much back and forth with my operations team, the only way we know (thus far) is to filter out the traffic based on IP ranges at the View level in GA. It's well worth the time and effort. As proof, here is my web traffic after filtering out the Keynote agents in late October:
If you are a Keynote customer and use our services to test and monitor your website, you can contact support to get a list of the IP Address ranges to filter out.
There's Got to Be A Better Way
IP filtering is a blunt tool, and while this solution works for Keynote Agent traffic, what about the myriad of other bots and agents? I think Google Analytics needs a better way to remove this traffic - it shouldn't be too hard. I'd love to work with them to make this happen. What do you think? Can you think of a better way? Feel free to comment, I'd love to discuss.