Manual/Importing XML dumps

<languages/>

<translate> This page describes methods to import XML dumps.</translate> <translate> XML Dumps contain the content of a wiki (wiki pages with all their revisions), without the site-related data.</translate> <translate> A XML dump does not create a full backup of the wiki database, the dump does not contain user accounts, images, edit logs, etc.</translate>

<translate> The <tvar name=1>Special:Export</tvar> page of any MediaWiki site, including any Wikimedia site and Wikipedia, creates an XML file (content dump).</translate> <translate> See [[<tvar name=2>m:Special:MyLanguage/Data dumps</tvar>|meta:Data dumps]] and <tvar name=3></tvar>.</translate> <translate> XML files are explained more on [[<tvar name=4>Special:MyLanguage/Help:Export#Export format</tvar>|Help:Export]].</translate>

<translate>

How to import?

There are several methods for importing these XML dumps.

Using Special:Import

<tvar name=1>Special:Import</tvar> can be used by wiki users with <tvar name=2>import</tvar> permission (by default this is users in the <tvar name=3>sysop</tvar> group) to import a small number of pages (about 100 should be safe).

Trying to import large dumps this way may result in timeouts or connection failures. </translate>

• <translate> See <tvar name=1></tvar> for a basic description of how the importing process works.</translate><ref><translate> See <tvar name=1></tvar> for a C# code sample that manipulates an XML import file.</translate></ref>

<translate> You are asked to give an interwiki prefix.</translate> <translate> For instance, if you exported from the English Wikipedia, you have to type 'en'.</translate> <translate> XML importing requires the <tvar name="1">import</tvar> and <tvar name="2">importupload</tvar> permissions.</translate> <translate> See <tvar name=1></tvar>.</translate>

<translate>

Large XML uploads

Large XML uploads might be rejected because they exceed the PHP upload limit set in the <tvar name="1">php.ini</tvar> file. </translate>

<syntaxhighlight lang="ini">

; <translate nowrap> Maximum allowed size for uploaded files.</translate>
upload_max_filesize = 20M

</syntaxhighlight> <translate> Try changing these four settings in <tvar name="1">php.ini</tvar>: </translate> <syntaxhighlight lang="ini">

; <translate nowrap> Maximum size of POST data that PHP will accept.</translate>
post_max_size = 20M

max_execution_time = 1000  ; <translate nowrap> Maximum execution time of each script, in seconds</translate>
max_input_time = 2000	    ; <translate nowrap> Maximum amount of time each script may spend parsing request data</translate>

; <translate nowrap> Default timeout for socket based streams (seconds)</translate>
default_socket_timeout = 2000

</syntaxhighlight>

<translate>

Using <tvar name=1>importDump.php</tvar>, if you have shell access

</translate>

<translate> Recommended method for general use, but slow for very big data sets.</translate>

<translate> See: <tvar name=1></tvar>, including tips on how to use it for large wikis.</translate>

<translate>

Using <tvar name=1>importTextFiles.php</tvar> maintenance Script

</translate>

<translate> If you have a lot of content converted from another source (several word processor files, content from another wiki, etc), you may have several files that you would like to import into your wiki.</translate> <translate> In MediaWiki 1.27 and later, you can use the <tvar name=2></tvar> maintenance script.</translate>

<translate> You can also use the <tvar name=2></tvar> maintenance script for this purpose. </translate>

rebuildall.php

<translate> For large XML dumps, you can run <tvar name=1>rebuildall.php</tvar>, but, it will take a long time, because it has to parse all pages.</translate> <translate> This is not recommended for large data sets.</translate>

<translate>

Using pywikibot, <tvar name=1>pagefromfile.py</tvar> and Nokogiri

</translate> <translate> <tvar name=1></tvar> is a collection of tools written in python that automate work on Wikipedia or other MediaWiki sites.</translate> <translate> Once installed on your computer, you can use the specific tool <tvar name=3></tvar> which lets you upload a wiki file on Wikipedia or MediaWiki sites.</translate> <translate> The xml file created by <tvar name=1>dumpBackup.php</tvar> can be transformed into a wiki file suitable to be processed by <tvar name=2>pagefromfile.py</tvar> using a simple script similar to the following (here the program will transform all xml files which are on the current directory which is needed if your MediaWiki site is a family):</translate>

<translate>

On Python

</translate> <syntaxhighlight lang="python"> from pathlib import Path from lxml import etree import re from pywikibot import Link, Site

""" The exported XML preserves local namespace names. To import into a project in another language,

you need to convert them to canonical names. Otherwise the pages will be imported as originals
without recognizing their namespace.

XML contains namespace numbers, but mapping them can be problematic. Because they can differ between projects.

For example, in ru.wikisource NS Page is number 104, while in uk.wikisource it is 252.

To solve these, the Link() class from pywikibot is used.

The Link() required the language code and family of project. XML does not contain these as separate

values. Instead, they are in another format, like `enwikisource` or 'commonswiki', which needs to be parsed.    

"""

xml_folder = 'PATH_TO_XML_DUMP' for xml_file in Path(xml_folder).glob('*.xml'):

   root = etree.parse(xml_file).getroot()
   ns = {: root.nsmap[None]}  # xml namespace

   # Parsing language code and family of the project
   dbname = root.find('.//siteinfo/dbname', ns).text
   m = re.search(r'^(\w+?)(wiki|wikisource|wikiquote|wikinews|wikibooks)$', dbname)
   if m:
       site = Site(code=m.group(1), fam=m.group(2))
   elif 'commons' in dbname:
       site = Site('commons')
   elif 'mediawikiwiki' in dbname:
       site = Site('mediawiki')
   else:
       print("Site didn't recognized.")
       break

   wiki = []
   for page in root.findall('.//page', ns):
       title_source = page.find('title', ns).text
       title = Link(title_source, site).ns_title().replace(' ', '_')
       revision_id = page.find('.//revision/id', ns).text
       print(f'{title}, revision id: {revision_id}')
       text = page.find('.//revision/text', ns).text
       wiki.append("\n%s\n%s\n" % (title, text))
   Path(f'out_{xml_file.stem}.wiki').write_text('\n'.join(wiki))

</syntaxhighlight>

<translate>

On Ruby

</translate>

<syntaxhighlight lang="ruby">

1. -*- coding: utf-8 -*-
2. dumpxml2wiki.rb

require 'rubygems' require 'nokogiri'

1. <translate nowrap> This program <tvar name=1>dumpxml2wiki</tvar> reads MediaWiki xml files dumped by <tvar name=2>dumpBackup.php</tvar> on the current directory and transforms them into wiki files which can then be modified and uploaded again by <tvar name=3>pywikipediabot</tvar> using <tvar name=4>pagefromfile.py</tvar> on a MediaWiki family site.</translate>
2. <translate nowrap> The text of each page is searched with xpath and its title is added on the first line as an html comment: this is required by <tvar name=1>pagefromfile.py</tvar>.</translate>

Dir.glob("*.xml").each do |filename|

 input = Nokogiri::XML(File.new(filename), nil, 'UTF-8')

 puts filename.to_s  # prints the name of each .xml file

 File.open("out_" + filename + ".wiki", 'w') {|f| 
   input.xpath("//xmlns:text").each {|n|
     pptitle = n.parent.parent.at_css "title" # searching for the title
     title = pptitle.content
     f.puts "\n" << n.content  << "\n"
   }
 }

end </syntaxhighlight>

<translate> For example, here is an excerpt of a wiki file output by the command 'ruby dumpxml2wiki.rb' (two pages can then be uploaded by pagefromfile.py, a Template and a second page which is a redirect): </translate>

<syntaxhighlight lang="html4strict">

1. REDIRECTbadania demograficzne

</syntaxhighlight>

<translate> The program accesses each xml file, extracts the texts within <tvar name=1><text> </text></tvar> markups of each page, searches the corresponding title as a parent and enclosed it with the paired <!--'''Title of the page'''--> commands used by 'pagefromfile' to create or update a page.</translate> <translate> The name of the page is in an html comment and separated by three quotes on the same first start line.</translate> <translate> Please notice that the name of the page can be written in Unicode.</translate> <translate> Sometimes it is important that the page starts directly with the command, like for a <tvar name=2>#REDIRECT</tvar>; thus the comment giving the name of the page must be after the command but still on the first line.</translate>

<translate> Please remark that the xml dump files produced by <tvar name=1>dumpBackup.php</tvar> are prefixed by a namespace: </translate>

<syntaxhighlight lang="html4strict"><mediawiki xmlns="http://www.mediawiki.org/xml/export-0.4/"></syntaxhighlight>

<translate> In order to access the text node using Nokogiri, you need to prefix your path with 'xmlns': </translate>

<syntaxhighlight lang="html4strict">input.xpath("//xmlns:text")</syntaxhighlight>

<translate> Nokogiri is an HTML, XML, SAX, & Reader parser with the ability to search documents via XPath or CSS3 selectors from the last generation of XML parsers using Ruby.

Example of the use of 'pagefromfile' to upload the output wiki text file: </translate>

<syntaxhighlight lang="bash"> python pagefromfile.py -file:out_filename.wiki -summary:"Reason for changes" -lang:pl -putthrottle:01 </syntaxhighlight>

<translate>

How to import logs?

Exporting and importing logs with the standard MediaWiki scripts often proves very hard; an alternative for import is the script <tvar name=1>pages_logging.py</tvar> in the [<tvar name=2>https://github.com/glimmerphoenix/WikiDAT</tvar> WikiDAT tool], as [[<tvar name=3>mailarchive:xmldatadumps-l/2013-July/000876.html</tvar>|suggested by Felipe Ortega]].

Troubleshooting

Merging histories, revision conflict, edit summaries, and other complications

</translate> <translate> When importing a page with history information, if the user name already exists in the target project, change the user name in the XML file to avoid confusion.</translate> <translate> If the user name is new to the target project, the contributions will still be available, but no account will be created automatically.</translate>

<translate> When a page is referenced through a link or a URL, generic namespace names are converted automatically.</translate> <translate> If the prefix is not a recognized namespace name, the page will default to the main namespace.</translate> <translate> However, prefixes like <tvar name=1>mw:</tvar> might be ignored on projects that use them for interwiki linking.</translate> <translate> In such cases, it might be beneficial to change the prefix to "Project:" in the XML file before importing.</translate>

<translate> If a page with the same name already exists, importing revisions will combine their histories.</translate> <translate> Adding a revision between two existing ones can make the next user's changes appear different.</translate> <translate> To see the actual change, compare the two original revisions, not the inserted one.</translate> <translate> So, only insert revisions to properly rebuild the page history.</translate>

<translate> A revision won't be imported if there's already a revision with the same date and time.</translate> <translate> This usually happens if it's been imported before, either to the current, a previous wiki, or both, possibly from a third site.</translate>

<translate> An edit summary might reference or link to another page, which can be confusing if the page being linked to has not been imported, even though the referencing page has been.</translate>

<translate> The edit summary doesn't automatically indicate whether the page has been imported, but in the case of an upload import, you can add this information to the edit summaries in the XML file before the import.</translate> <translate> This can help prevent confusion. When modifying the XML file with find/replace, keep in mind that adding text to the edit summaries requires differentiating between edits with and without an edit summary.</translate> <translate> If there are multiple comment tags in the XML file, only the last set will be applied.</translate>

<translate>

Interwikis

</translate> </tvar> linking.</translate> |1= }} <translate> For example, ones with a prefix of 'Meta:' would conflict with the interwiki prefix <tvar name=2>meta:</tvar> which by default links to <tvar name=3>https://meta.wikimedia.org</tvar>.</translate>

<translate> You can do any of the following. </translate>

• <translate> Remove the prefix from the <tvar name=2></tvar> table.</translate> <translate> This will preserve page titles, but prevent interwiki linking through that prefix.</translate>

  <translate> Example: you will preserve page titles 'Meta:Blah blah' but will not be able to use the prefix 'meta:' to link to meta.wikimedia.org (although it will be possible through a different prefix).</translate>

  <translate> How to do it: before importing the dump, run the query <tvar name=1>DELETE FROM interwiki WHERE iw_prefix='prefix'</tvar> (note: do not include the colon in the prefix).</translate> <translate> Alternatively, if you have [[<tvar name=2>Special:MyLanguage/Manual:Guide to setting up interwiki linking</tvar>|enabled editing the interwiki table]], you can simply go to <tvar name=3>Special:Interwiki</tvar> and click the 'Delete' link on the right side of the row belonging to that prefix.</translate>

• <translate> Replace the unwanted prefix in the XML file with "Project:" before importing. This will preserve the functionality of the prefix as an interlink, but will replace the prefix in the page titles with the name of the wiki where they're imported into, and might be quite a pain to do on large dumps.</translate>

  <translate> Example: replace all 'Meta:' with 'Project:' in the XML file. MediaWiki will then replace 'Project:' with the name of your wiki during importing.</translate>

<translate>

See also

</translate>

• meta:Data dumps/Other tools
• – <translate> May come in handy if you are doing massive imports</translate>
• – <translate> Settings that may need to be changed if you are doing massive imports</translate>
• – <translate> for importing images.</translate>

<translate>

References

</translate> <references/>

---

Sample content adapted from mediawikiwiki:Manual:Importing_XML_dumps on mediawiki.org, licensed under CC BY-SA 4.0.