Date: February 2023
In uMarketingSuite 1.21 we have decided to not store bot/spider/crawler traffic anymore. We also have a nice list of bugfixes and are still working behind the scenes on the Reporting feature.
Curious about the latest uMarketingSuite version? Go to Our Umbraco or install the latest package via NuGet.
No more bots...
We have decided to not store bot/spider/crawler traffic anymore. Feedback from our customers and partners but also our own investigations on our sites showed that approx. 20% - 60% of all the traffic was coming from (unidentified)bots/spiders and crawlers. This causes polluting of data, unnecessary growth of the database and possible slow running reports. The tabs Analytics - Bots and Profiles - Bots are still there but no new data will be added.
How is bot detection currently implemented?
We have implemented a two-step approach. At first we check the user agent against a well known list of user agents using DeviceDetector. So if we see that the user agent contains for example 'Googlebot' we will mark this request as a non-person request and we will track/personalize nothing. The request is not stored anymore as of uMarketingSuite 1.21.
If the user agent is not on the list you will 'enter' the next detection mechanism (which is in place since uMarketingSuite 1.13). What we need from the client is a client side request to the server, this is the so called ping mechanism. A javascript method is executing a request to /umbraco/umarketingsuite/pagedata/ping (make sure this url is accessible for all front-end server(s)!). The ping url will receive the client side 'ping' which we combine with the initial server request and then say 'ok, you are a person'. If this client side ping is not received we will mark the request as non-person and the request is not stored as of uMarketingSuite 1.21. This mechanism works because a lot (not all!) non-legit bots/spiders/crawlers won't execute javascript For example the 'Googlebot' will execute javascript but is identified by the first detection mechanism using the user agent. Any questions?, contact us.
Is this 'ping' mechanism 100% 'bot proof' ?
No, we can't guarantee that there is no bot/spider/crawler traffic in your analytics data because non-legit bot/spiders/crawlers will introduce new possibilities to crawl your site and bypass any detection mechanism.
Is this the end-state of the uMarketingSuite bot detection?
No, this is not the end-state. Within the Analytics team we try to keep up with the innovations regarding identifying requests (person vs bot) and make sure that those innovations will be incorporated into the uMarketingSuite platform.
Sneak peek | Reporting
We are currently working on the new Reporting feature for Personalization so you can actually see the effectiveness of your personalizations. This feature is still in development and the first public beta is expected soon.
The reporting feature will include all the details that you need to know from your active segments.
Bugfixes / improvements in uMarketingSuite 1.21
- Bugfix | Save & preview button in Personalization does not work
- Bugfix | Backoffice refresh triggers duplicate GetById API calls
- Bugfix | uMarketingSuite tries to access nodes the user doesn't have access to
- Improvement | uMarketingSuiteRawPageview table is removed
If you find any bugs? Please report them to us on Github: https://github.com/uMarketingSolutions/uMarketingSuite
Do you want to know more?
Book your uMarketingSuite demo here! or contact us
Obtaining a license
Check out our pricing page on uMarketingSuite.com for a complete overview of the available bundles and pricing. Interested in becoming a uMarketingSuite partner with a lot of partner benefits? Go to our partner page.
What is next?
- Reporting for Personalization
- Improved reporting for Analytics