Wikidata:SPARQL query service/WDQS backend update/September 2024 scaling update

Hello!

As you might be aware we are addressing the Wikidata Query Service stability and scaling issues. The service has issues in 4 areas: the amount of data, the amount of edits, the amount of queries and the maintenance status of the Blazegraph software. We have been working on several projects to address these issues. This announcement is about one of them, the WDQS Graph Split. It entails splitting out a part of Wikidata’s data (the Items for scholarly articles) into their own Blazegraph instance. This has an impact on the Wikidata Query Service only. Editing will continue to happen in one wiki, Wikidata. You can find out more about the WDQS Graph Split on-wiki.

As part of the WDQS Graph Split project, we have new SPARQL endpoints available for serving the “scholarly” and “main” subgraphs of Wikidata:

• https://query-scholarly.wikidata.org
• https://query-main.wikidata.org

The “query-scholarly” endpoint serves all entities related to scholarly articles, the “query-main” endpoint serves everything else. For more details, have a look at the WDQS Graph Split Rules.

The “query.wikidata.org” endpoint is still here and will continue to serve the full Wikidata graph for a transition period, with the period ending in March 2025. After the transition, it will only serve the main graph.

We are now entering a transition period until the end of March 2025. The three SPARQL endpoints will remain in place until the end of the transition. At the end of the transition, query.wikidata.org will serve the main Wikidata subgraph (without scholarly articles). The query-main and query-scholarly endpoints will continue to be available after the transition.

If your use case does not require Scholarly articles, you can stop reading here, nothing will change for you. If your use case requires Scholarly articles, please read our Federation Guide, and start rewriting your queries to use the new endpoints. If you need help, use the Request a Query page, contact the Search Platform team, or join our Office Hours.

We understand that the WDQS Graph Split will require work from our communities to adapt their queries and use cases. The Search Platform and Wikidata teams are available to support this transition to the best of our abilities. If you think your tool will be affected by the split, please add it to the list of affected tools we set up.

This split graph approach is new, we expect a few issues. If you find any problem, please let us know on wiki or with a Phabricator task tagged with “wikidata-query-service”. We already know of one intermittent connection issue related to SPARQL federation. In this case, re-running the failed query should work.

Thank you for your understanding!