Research:Reducing Zero Result Searches Through Elasticsearch Slop Parameter Changes

Contact

Mikhail Popov

Wikimedia Foundation

Duration:  2015-08 – 2015-09

 Wikimedia hosted

mw:Wikimedia Discovery

  Open access
via github.com

 Open source
via github.com

---

Research:Projects

This page documents a completed research project.

This report covers the second experiment run by the Discovery team. The slop parameter indicates how far apart terms are allowed to be while still considering the document a match. Far apart means: how many times do you need to move a term in order to make the query and document match? ^[1] We hypothesized that increasing the slop parameter would yield less zero results. This was based on our intuition that less restrictive (more flexible) phrase matching should find more documents that match the terms. In summary, we did not find a consistent improvement in outcome for users subject to the experimental conditions, and recommend disabling the experiment and returning users to the default search experience - coupled with some investigation suggestions.

We sampled 3% of the total queries over the course of 12 days.

• 1% of the total queries did not receive a change; they were the control group "A".
• 1% of the total queries were assigned a slop parameter of 1 to their queries; they were the experimental group "B".
• 1% of the total queries were assigned a slop parameter of 2 to their queries; they were the experimental group "C".

Even with 1% sampling rate, the final sample sizes were huge. Were we to perform our analyses on the full collected dataset, we would have seen significance for the tiniest effects. Instead, we chose to perform an a priori-style power analysis post-hoc. Which is to say, we calculated our sample sizes as one would before beginning the experiment, and then sampled down (using random sampling stratified on group to obtain equally-sized groups) to obtain a smaller dataset. A more detailed description of the power analysis follows.

From the preliminary analysis of the data from the morning of the test's launch, 84.2% of the control group were getting some results. This is the "prevalence within control group" that we used for this power analysis.

We also wanted to be able to detect an odds ratio of at least 1.5. That is, if the slop parameter does have an impact on the search results then the test group being 1.5 times more likely to get results than the control group is the smallest impact we'd be satisfied with.

We decided on 99% power to detect this effect and an α-level of 0.05.

wmf::sample_size_odds(odds_ratio = 1.5, p_control = 0.8423662,
                      power = 0.99, conf_level = 0.95,
                      sample_ratio = 1) # => 3950 => 4000

After removing known automata and nonsensical queries (e.g. queries to the donation website), we randomly sampled each day's big dataset down to a smaller dataset of just 6000 observations -- 2000 for each of the 3 groups, so that each of the two tests comparing controls to a test group had 4000 observations evenly split.

The large sample sizes were sufficient for us to perform Chi-squared tests of independence between the group (slop parameter) and outcome (none vs some results).

The number of full-text queries varies according to an established pattern of user activity throughout the week, particularly of the mobile users who are most active on Sundays.

These numbers reflect full-text, no-known-automata queries after sampling down. The only really problematic (read: disproportionately sampled queries) projects are Commons and Wiktionary. We reflect on this in the Discussion.

In the figure on the left, we suggestions of association. Some groups had better or worse outcomes than we would have expected them under the assumption of independence. Group C, for example, almost consistently had MORE zero results (denoted by blue), while groups A and B had less (denoted by red).

We performed Chi-squared tests of independence on each of the days and found evidence of statistically significant association between group membership and outcome:

Next, we looked into the strength and the direction of the relationships.

Overall, with the exception of a couple of days, the group with slop parameter of 1 was more likely to get nonzero results than the controls. The group with the slop parameter of 2, on the other hand, was significantly LESS likely to get nonzero results! In fact, on one of the days group C was actually 1/4 times less likely to get nonzero results, or to put it another way: 4 times more likely than to get zero results.

The numbers from the tests are provided in greater detail in the Appendix.

We have shown here that changing slop parameter from 0 to 1 yields significantly less zero results across some of the days this experiment was active. There were also a few days when the effect was significantly negative. Only once was group B more than 1.5 times more likely than group A to get nonzero results.

Group C, however, had the worst experience. Group C was less likely than A to get nonzero results and even less likelier than group B. This fundamentally contradicts our understanding of search -- how could we give our users LESS results by being LESS restrictive with phrase matching?

But we set a goal for ourselves: positive impact is at least 1.5 times, and the overall results do not suggest that changing the slop parameter from 0 to 1 gives us that impact. Therefore, we do not recommend switching from slop parameter of 0 to a slop parameter of 1. The results, while positive and nearly-but-not-quite up to our standards on some days, are not overwhelmingly convincing that switching is the way to go.

We improved a lot from our first A/B test in terms of how we design and conduct our experiments. The addition of power analysis prior to performing any statistical analyses was a wise decision, a trend we hope to establish going forward.

We recommend investigating the cause for the disproportionate sampling within Commons and Wiktionary. We originally noticed this when we did the Initial Analysis (PDF) and thought that perhaps the changes haven't had time to propagate to those servers. That we see the same sampling biases in the final dataset suggests a need to conduct an investigation.

Additionally, we recommend further research into the discrepancy between what we logically expect the results to be versus what the results actually are. We should not be seeing overwhelemingly worse outcomes with less restrictive matching parameters.

Final dataset after grooming. These numbers reflect the pre-downsampling dataset.

Note: We excluded "be_x_old wiki", "donation site", and "testwikidata" from the analysis.