Through our ongoing conversion rate optimization work with Great Lakes Skipper, we theorized that certain visitors might not be finding the hard to find boat parts they were looking for because of substitute part numbers or other information that wasn’t always being found by the existing search engine powering their website. As a Klevu partner we had seen other cases where the advanced AI and deep product data indexing produced a lift. But like all things we recommend and evaluate, we wanted assurance that the cost of the service would represent solid ROI for our partner. This analysis can be tough if you don’t have a way to truly test a service against what’s already in place, because the metrics you would be tracking – things like per session value and search assisted revenue – are always fluctuating on their own anyway.
So we proposed that we use our Advanced A/B Tests Extension to send half of the website traffic to Klevu’s AI based search and smart recommendations, and half of the traffic to the existing website search that was powered by elastic search. To our knowledge this is the first time Klevu has ever been able to run a true A/B test of their entire suite of optimizations, comparing to the native options in Magento / Adobe Commerce.
Interestingly, the first few months the A/B test was completely flat. This had us scratching our heads. Was there really no difference between the Klevu optimized tools and the site search that was already in place? We went to work with the Klevu team to make sure all data attributes were being indexed properly and the data Klevu needed to optimize over time was being sent to their algorithm. We uncovered some important use cases that were not being captured by the Klevu indexer and after adjusting those we kicked off a fresh new A/B test and saw a great improvement in the 50% of visits that were getting the optimized Klevu site search and automated recommendations. After a month of testing and thousands of transactions per variation, we saw a 16.8% increase in revenue per visit for the variation running the Klevu site search and AI. The ROI case for Great Lakes Skipper was a home run, by our calculations in the neighborhood of 20X.
We appreciate the collaboration and the willingness of both Great Lakes Skipper and Klevu to go the extra mile of running an A/B test. Without it, we might not have uncovered the opportunities that lead to the big wins. But we’re also grateful for the open collaboration on the part of Klevu and willingness to dive deep when we weren’t seeing the results we all thought we should initially.
You can read more about the project in this case study from Klevu:
Great Lakes Skipper – Searching by SKU and Gaining Revenue
We had a follow up webinar with Jake Doro of Great Lakes Skipper and Lauren Jones of Klevu where we discussed some of the insights and takeaways: