View source for Discovery/Testing

====== Fewer Than 3 Results ======
{{Discovery-Test
 |name      = Fewer Than 3 Results
 |status    = Done
 |project   = Cirrus
 |analysis  = [https://github.com/wikimedia-research/LessThan3 GitHub]
 |report    = [[File:Second Language Switching Test.pdf|PDF on Commons|none|thumb]]
 |result    = Success
}}
Corresponding Phab tickets: [[phab:T118285|Measure clickthroughs from multiple wikis]], [[phab:T118286|Write test]], [[phab:T118290|Turn on]], [[phab:T118294|Verify]], [[phab:T118292|Turn off]], [[phab:T118295|Analyze]]

This was an A/B test to determine the impact of switching search languages in the case that a query produces zero results, and in the case that a query produces fewer than 3 results, depending on the test group. This
followed an earlier test that just applied this to queries producing zero results. Our hope was that applying language detection methods to failed queries would let us, for a subset, identify that they were using the wrong language, switch them over to the correct one, and get them results - and that new data collection methods would let us produce a more certain result than the last test.

To check our hypothesis we ran a second A/B test, from 21 November to 27 November 2015. This contained two test groups; one would have language detection applied with zero (<1) results, one with less than 3 (<3) results. In both cases, query metadata was added to indicate whether a search would have had language detection applied. We hoped that with these additional datapoints (and a wider population) we could see the impact. This test generated 33,282,741 queries.

We found very promising evidence that, for those queries the language detection could actually be applied to, this makes a remarkable difference to the zero results rate. We recommend that a user-side A/B test be run to look at the clickthrough rate when language detection is applied, to ensure that it produces useful results, not just "some" results.