View source for Dumps

You do not have permission to edit this page, for the following reasons:

The action you have requested is limited to users in the group: Users.
You must confirm your email address before editing pages. Please set and validate your email address through your user preferences.
Your IP address is in a range that has been blocked on all Wikimedia Foundation wikis. The block was made by ‪AntiCompositeNumber‬. The reason given is Open proxy/Webhost: Visit the FAQ if you are affected .
- Start of block: 13:55, 18 August 2022
- Expiry of block: 13:55, 18 April 2025
Your current IP address is 44.201.113.115. The blocked range is 44.201.0.0/16. Please include all above details in any queries you make. If you believe you were blocked by mistake, you can find additional information and instructions in the No open proxies global policy. Otherwise, to discuss the block please post a request for review on Meta-Wiki. You could also send an email to the stewards VRT queue at stewards@wikimedia.org including all above details.

You can view and copy the source of this page.

{{Navigation Wikimedia infrastructure|expand=mw}}
{{Hatnote|See [[Help:Shared storage#Dumps]] for information on using Dumps data from [[Portal:Toolforge|Toolforge]].}}
Documentation about '''Dumps''' is stored in a few different wikis:
* These Wikitech docs are for '''maintainers''' of the various dumps.
** Information about the <code>clouddumps</code> servers serving mirrors to various clients can be found on [[Portal:Data Services/Admin/Dumps]].
* Information for '''users''' of the dumps can be found at Meta-wiki's [[m:Data dumps]] page. 
* Information for '''developers''' can be found at MediaWiki-wiki's [[mw:SQL/XML Dumps]] page.

== Daily checks ==
Dumps maintainers should watch or check a few things every day:
* emails to the [https://groups.google.com/a/wikimedia.org/g/ops-dumps ops-dumps internal mailing list.].
* [[mail:xmldatadumps-l|xmldatadumps-l public mailing list]]
* [[phab:tag/dumps-generation/|Phabricator Dumps-Generation workboard]]
* https://dumps.wikimedia.org/ (mentions the current run, unless idle)
* icinga for dumps hosts: [https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=snapshot1 snapshots], [https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=dumpsdata dumpsdata]

== Dumps types ==
We produce several types of dumps. For information about deployment of updates, architecture of the dumps, and troubleshooting each dump type, check the appropriate entry below.

* [[Dumps/XML-SQL Dumps|xml/sql dumps]] which contain '''revision metadata and content''' for public Wikimedia projects, along with contents of select '''sql tables'''
* [[Dumps/Adds-changes_dumps|adds/changes dumps]] which contain a '''daily xml dump of new pages''' or pages with '''new revisions''' since the previous run, for public Wikimedia projects
* [[Dumps/WikidataDumps|Wikidata entity dumps]] which contain dumps of ''' 'entities' (Qxxx)''' in various formats, and a dump of '''lexemes''', run once a week.
* [[Dumps/CategoriesRDF|category dumps]] which contain weekly full and daily incremental '''category lists''' for public Wikimedia projects, in '''rdf format'''
* [[Dumps/OtherMisc|other miscellaneous dumps]] including '''content translation''' dumps, '''cirrus search''' dumps, and '''global block''' information.

Other datasets are also provided for download, such as page view counts; these datasets are managed by other folks and are not documented here.

== Service ==

=== Hardware ===
* [[Dumps/Snapshot hosts | Dumps snapshot hosts]] that run scripts to generate the dumps 
* [[Dumps/Dumpsdata hosts | Dumps datastores]] where the snapshot hosts write intermediate and final dump output files, which are later published to our web servers
* [[Dumps/Dump servers | Dumps servers]] that provide the dumps to the public, to our mirrors, and via nfs to Wikimedia Cloud Services and stats host users

=== Adding new dumps ===

If you are interested in adding a new dumpset, please check the [[Dumps/New dumps and datasets|guidelines]] (still in draft form).

If you are working with wikibase dumps of some sort, you might want to look at a code walkthrough; see [[Dumps/Wikibase dumps overview]].

=== Not an SLO but... ===

Dumps have never had an SLO. But current dumps maintainers have a set of unofficial standards for responsiveness and reliability.

We try to reply to incoming tasks filed, emails from folks interested in hosting mirrors, and requests for information, within 2 business days. This may be extended if the dumps maintainers are ill or out of the office for other reasons.

When the SQL/XML dumps for one or more wikis are broken, we do our best to respond to the breakage within 24 hours; this usually includes the filing of a task in Phabricator and some investigation of the problem. If changes to MediaWiki code are required, we will coordinate that work even when we don't write the patch, also arranging for a timely backport and deployment of the patch.

We do our best to ensure that all jobs for the SQL/XML dumps for every wiki are complete before the start of the next run. So for the run starting on the 1st of the month, all jobs on all wikis must be complete before the 20th of the month, and for the run starting on the 20th, all jobs for all wikis must be complete before the end of the month. This sometimes requires work on days off, or beyond the regular workday, in which case future workdays might be shortened to compensate.

=== Testing changes to the dumps or new scripts ===

See [[Dumps/Testing]] for more about this.

=== Dealing with problems ===

See [[Dumps/Troubleshooting]] for more about this.

=== Mirrors ===

If you are adding a mirror, see [[Dumps/Mirror status |Dumps Mirror setup]].

=== Source code ===

* [https://gerrit.wikimedia.org/r/q/project:operations%252Fdumps operations/dumps.git]

[[Category: Dumps]]

Templates used on this page:

Template:Hatnote (view source)
Template:Navigation Wikimedia infrastructure (view source)
Template:Navigation Wikimedia infrastructure/style.css (view source)
Template:Navigation sidebar (view source) (semi-protected)
Template:Navigation sidebar/styles.css (view source) (semi-protected)
Template:See (view source)
Template:See/styles.css (view source)

Return to Dumps.

Retrieved from "https://wikitech.wikimedia.org/wiki/Dumps"