UESPWiki:Data Dumps

Available Data Dumps

View a complete list of all available data dumps. The backup include the current version of all pages in the main namespace without their revision history. Does not include images.

Development Notes

• maintenance\dumpBackup.php

• The "server" parameter is currently ignored (in all MediaWiki versions as of June 2013). You can modify maintenance/backup.inc and change the two occurrences of DB_SLAVE to the specific database server index as specified in LocalSettings.php if a specific server is desired to generate the dump.
• Use the "--namespaces=0,2,6,8,10,14,102,104,106,108,110,112,114,116,118,120,122,124,126,128,130,132,134,136,138,140,142,144,146" option to output just the relevant content data from the wiki.
• A dump using "--current" from the primary slave DB takes around 2 minutes and is 145MB uncompressed (20MB compressed).
• A dump using "--current" from content3 slave DB takes around 18 minutes.
• A full dump will likely take many hours to complete (or days if run from the content3 slave DB) and be dozens of GB in size.
• Using the primary slave during a dump does impact site performance. The database slows down due to load which can result in connection "piling up" on the Apache instances and further issues for longer dump times. Random "pauses" during the dump also have the same affect. It is not recommended to use the primary database except for small, partial dumps. A "--current" dump _usually_ has minimal impact on the site.
• This script appears to have no built-in throttling to control the load it places on the database.
• Using the "--include-files" parameter results in a PHP error/warning message. See this bug report.

• Automatic dump creator

• content3:/etc/cron.weekly/uesp-dumpxml-current -- Creates the "current" version of the XML dump once a week.
• files1:/etc/cron.weekly/uesp-updatedumps -- Syncs all dump files from content3 once a day and regenerates the dump HTML file.
• files1:/uesp-setup/uesp-generate-dumphtml -- Called from uesp-updatedumps once a day. Creates the HTML page listing all available data dumps.
• files:/uesp/shared/dumps/ -- Location of all data dumps being served publicly.
• files:/uesp/shared/dumps/uesp-dumpxml-header.html
• files:/uesp/shared/dumps/uesp-dumpxml-footer.html -- Header and footer HTML excerpts used by uesp-generate-dumphtml to create the final HTML dump page.