Editing Encyclopedia:Database download (section)

==Help to parse dumps for use in scripts==

* [[Wikipedia:Computer help desk/ParseMediaWikiDump]] describes the [[Perl]] Parse::MediaWikiDump library, which can parse XML dumps.
* [https://web.archive.org/web/20070907074625/http://www.cs.technion.ac.il/~gabr/resources/code/wikiprep Wikipedia preprocessor (wikiprep.pl)] is a [[Perl]] script that preprocesses raw XML dumps and builds link tables, category hierarchies, collects anchor text for each article etc.
* [https://github.com/svick/Wikipedia-SQl-dump-parser Wikipedia SQL dump parser] is a .NET library to read MySQL dumps without the need to use MySQL database
* [https://github.com/MartinRichards23/WikiDumpParser WikiDumpParser] – a .NET Core library to parse the database dumps.
* [https://github.com/newca12/dictionary-builder Dictionary Builder] is a Rust program that can parse XML dumps and extract entries in files
* [https://github.com/napsternxg/WikiUtils Scripts for parsing Wikipedia dumps] – Python based scripts for parsing sql.gz files from wikipedia dumps.
* [https://crates.io/crates/parse-mediawiki-sql/ parse-mediawiki-sql] – a Rust library for quickly parsing the SQL dump files with minimal memory allocation
* [https://gitlab.com/tozd/go/mediawiki gitlab.com/tozd/go/mediawiki] – a Go package providing utilities for processing Wikipedia and Wikidata dumps.

===Doing Hadoop MapReduce on the Wikipedia current database dump===
You can do Hadoop MapReduce queries on the current database dump, but you will need an extension to the InputRecordFormat to
have each &lt;page&gt; &lt;/page&gt; be a single mapper input. A working set of java methods (jobControl, mapper, reducer, and XmlInputRecordFormat) is available at [//tpmoyer-gallery.appspot.com/hadoopWikipedia Hadoop on the Wikipedia]