Editing Manual/Importing XML dumps (section)

==How to import?== <!--T:2-->

<!--T:7-->
There are several methods for importing these XML dumps.

===Using Special:Import=== <!--T:3-->

<!--T:22-->
<tvar name=1>[[Special:Import]]</tvar> can be used by wiki users with <tvar name=2><code>import</code></tvar> permission (by default this is users in the <tvar name=3><code>sysop</code></tvar> group) to import a small number of pages (about 100 should be safe).

<!--T:23-->
''Trying to import large dumps this way may result in timeouts or connection failures''.
</translate>

* <translate><!--T:57--> See <tvar name=1></tvar> for a basic description of how the importing process works.</translate><ref><translate><!--T:25--> See <tvar name=1></tvar> for a C# code sample that manipulates an XML import file.</translate></ref>

<translate><!--T:27--> You are asked to give an interwiki prefix.</translate>
<translate><!--T:163--> For instance, if you exported from the English Wikipedia, you have to type 'en'.</translate>
<translate><!--T:58--> XML importing requires the <tvar name="1"><code>import</code></tvar> and <tvar name="2"><code>importupload</code></tvar> permissions.</translate>
<translate><!--T:10--> See <tvar name=1></tvar>.</translate>

<translate>
=== Large XML uploads === <!--T:59-->

<!--T:60-->
Large XML uploads might be rejected because they exceed the PHP upload limit set in the <tvar name="1"><code>php.ini</code></tvar> file.
</translate>

<syntaxhighlight lang="ini">
 ; <translate nowrap><!--T:61--> Maximum allowed size for uploaded files.</translate>
 upload_max_filesize = 20M
</syntaxhighlight>
<translate>
<!--T:62-->
Try changing these four settings in <tvar name="1"><code>php.ini</code></tvar>:
</translate>
<syntaxhighlight lang="ini">
 ; <translate nowrap><!--T:63--> Maximum size of POST data that PHP will accept.</translate>
 post_max_size = 20M

 max_execution_time = 1000  ; <translate nowrap><!--T:64--> Maximum execution time of each script, in seconds</translate>
 max_input_time = 2000	    ; <translate nowrap><!--T:65--> Maximum amount of time each script may spend parsing request data</translate>

 ; <translate nowrap><!--T:66--> Default timeout for socket based streams (seconds)</translate>
 default_socket_timeout = 2000
</syntaxhighlight>

<translate>
===Using <tvar name=1>importDump.php</tvar>, if you have shell access=== <!--T:6-->
</translate>
:<translate><!--T:31--> ''Recommended method for general use, but slow for very big data sets.''</translate>
: <translate><!--T:32--> ''See: <tvar name=1></tvar>, including tips on how to use it for large wikis.''</translate>

<translate>
===Using <tvar name=1>importTextFiles.php</tvar> maintenance Script=== <!--T:19-->
</translate>


<translate><!--T:33--> If you have a lot of content converted from another source (several word processor files, content from another wiki, etc), you may have several files that you would like to import into your wiki.</translate>
<translate><!--T:67--> In MediaWiki 1.27 and later, you can use the <tvar name=2></tvar> maintenance script.</translate>

<translate>
<!--T:34-->
You can also use the <tvar name=2></tvar> maintenance script for this purpose.
</translate>

===rebuildall.php===
<translate><!--T:36--> For large XML dumps, you can run <tvar name=1><code>[[Special:MyLanguage/Manual:rebuildall.php|rebuildall.php]]</code></tvar>, but, it will take a  ''long'' time, because it has to parse all pages.</translate>
<translate><!--T:85--> This is not recommended for large data sets.</translate>

<translate>
=== Using pywikibot, <tvar name=1>pagefromfile.py</tvar> and Nokogiri === <!--T:37-->
</translate>
<translate><!--T:38--> <tvar name=1></tvar> is a collection of tools written in python that automate work on Wikipedia or other MediaWiki sites.</translate>
<translate><!--T:68--> Once installed on your computer, you can use the specific tool <tvar name=3></tvar> which lets you upload a wiki file on Wikipedia or MediaWiki sites.</translate>
<translate><!--T:69--> The xml file created by <tvar name=1>dumpBackup.php</tvar> can be transformed into a wiki file suitable to be processed by <tvar name=2>pagefromfile.py</tvar> using a simple script similar to the following (here the program will transform all xml files which are on the current directory which is needed if your MediaWiki site is a family):</translate>

<translate>
==== On Python ==== <!--T:161-->
</translate>
<syntaxhighlight lang="python">
from pathlib import Path
from lxml import etree
import re
from pywikibot import Link, Site

""" The exported XML preserves local namespace names. To import into a project in another language,
 you need to convert them to canonical names. Otherwise the pages will be imported as originals
 without recognizing their namespace.
XML contains namespace numbers, but mapping them can be problematic. Because they can differ between projects.
 For example, in ru.wikisource NS Page is number 104, while in uk.wikisource it is 252.
To solve these, the Link() class from pywikibot is used.

The Link() required the language code and family of project. XML does not contain these as separate
 values. Instead, they are in another format, like `enwikisource` or 'commonswiki', which needs to be parsed.    
"""

xml_folder = 'PATH_TO_XML_DUMP'
for xml_file in Path(xml_folder).glob('*.xml'):
    root = etree.parse(xml_file).getroot()
    ns = {'': root.nsmap[None]}  # xml namespace

    # Parsing language code and family of the project
    dbname = root.find('.//siteinfo/dbname', ns).text
    m = re.search(r'^(\w+?)(wiki|wikisource|wikiquote|wikinews|wikibooks)$', dbname)
    if m:
        site = Site(code=m.group(1), fam=m.group(2))
    elif 'commons' in dbname:
        site = Site('commons')
    elif 'mediawikiwiki' in dbname:
        site = Site('mediawiki')
    else:
        print("Site didn't recognized.")
        break

    wiki = []
    for page in root.findall('.//page', ns):
        title_source = page.find('title', ns).text
        title = Link(title_source, site).ns_title().replace(' ', '_')
        revision_id = page.find('.//revision/id', ns).text
        print(f'{title}, revision id: {revision_id}')
        text = page.find('.//revision/text', ns).text
        wiki.append("\n'''%s'''\n%s\n" % (title, text))
    Path(f'out_{xml_file.stem}.wiki').write_text('\n'.join(wiki))
</syntaxhighlight>

<translate>
==== On Ruby ==== <!--T:162-->
</translate>

<syntaxhighlight lang="ruby">
# -*- coding: utf-8 -*-
# dumpxml2wiki.rb

require 'rubygems'
require 'nokogiri'

# <translate nowrap><!--T:140--> This program <tvar name=1>dumpxml2wiki</tvar> reads MediaWiki xml files dumped by <tvar name=2>dumpBackup.php</tvar> on the current directory and transforms them into wiki files which can then be modified and uploaded again by <tvar name=3>pywikipediabot</tvar> using <tvar name=4>pagefromfile.py</tvar> on a MediaWiki family site.</translate>
# <translate nowrap><!--T:143--> The text of each page is searched with xpath and its title is added on the first line as an html comment: this is required by <tvar name=1>pagefromfile.py</tvar>.</translate>
# 
Dir.glob("*.xml").each do |filename|
  input = Nokogiri::XML(File.new(filename), nil, 'UTF-8')

  puts filename.to_s  # prints the name of each .xml file

  File.open("out_" + filename + ".wiki", 'w') {|f| 
    input.xpath("//xmlns:text").each {|n|
      pptitle = n.parent.parent.at_css "title" # searching for the title
      title = pptitle.content
      f.puts "\n<!--'''" << title.to_s << "'''-->" << n.content  << "\n"
    }
  }
end
</syntaxhighlight>

<translate>
<!--T:39-->
For example, here is an excerpt of a wiki file output by the command 'ruby dumpxml2wiki.rb' (two pages can then be uploaded by pagefromfile.py, a Template and a second page which is a redirect):
</translate>

<syntaxhighlight lang="html4strict">
<!--'''Template:Lang_translation_-pl'''--><includeonly>Tłumaczenie</includeonly>


#REDIRECT[[badania demograficzne]]<!--'''ilościowa demografia'''-->
<noinclude>
[[Category:Termin wielojęzycznego słownika demograficznego (pierwsze wydanie)|ilościowa demografia]]
[[Category:Termin wielojęzycznego słownika demograficznego (pierwsze wydanie) (redirect)]]
[[Category:10]]</noinclude>

</syntaxhighlight>

<translate><!--T:40--> The program accesses each xml file, extracts the texts within <tvar name=1><nowiki><text> </text></nowiki></tvar> markups of each page, searches the corresponding title as a parent and enclosed it with the paired <nowiki><!--'''Title of the page'''--> </nowiki> commands used by 'pagefromfile' to create or update a page.</translate>
<translate><!--T:70--> The name of the page is in an html comment and separated by three quotes on the same first start line.</translate>
<translate><!--T:71--> Please notice that the name of the page can be written in Unicode.</translate>
<translate><!--T:72--> Sometimes it is important that the page starts directly with the command, like for a <tvar name=2><nowiki>#REDIRECT</nowiki></tvar>; thus the comment giving the name of the page must be ''after'' the command but still on the first line.</translate>

<translate>
<!--T:41-->
Please remark that the xml dump files produced by <tvar name=1>dumpBackup.php</tvar> are prefixed by a namespace:
</translate>

<syntaxhighlight lang="html4strict"><mediawiki xmlns="http://www.mediawiki.org/xml/export-0.4/"></syntaxhighlight>

<translate>
<!--T:42-->
In order to access the text node using Nokogiri, you need to prefix your path with 'xmlns':
</translate>

<syntaxhighlight lang="html4strict">input.xpath("//xmlns:text")</syntaxhighlight> 

<translate>
<!--T:12-->
Nokogiri is an HTML, XML, SAX, & Reader parser with the ability to search documents via XPath or CSS3 selectors from the last generation of XML parsers using Ruby.

<!--T:43-->
Example of the use of 'pagefromfile' to upload the output wiki text file:
</translate>

<syntaxhighlight lang="bash">
python pagefromfile.py -file:out_filename.wiki -summary:"Reason for changes" -lang:pl -putthrottle:01
</syntaxhighlight>

<translate>

=== How to import logs? === <!--T:13-->

<!--T:44-->
Exporting and importing logs with the standard MediaWiki scripts often proves very hard; an alternative for import is the script <tvar name=1><code>pages_logging.py</code></tvar> in the [<tvar name=2>https://github.com/glimmerphoenix/WikiDAT</tvar> WikiDAT tool], as [[<tvar name=3>mailarchive:xmldatadumps-l/2013-July/000876.html</tvar>|suggested by Felipe Ortega]].