Editing Manual/Importing XML dumps (section)

=== Using pywikibot, <tvar name=1>pagefromfile.py</tvar> and Nokogiri === <!--T:37-->
</translate>
<translate><!--T:38--> <tvar name=1></tvar> is a collection of tools written in python that automate work on Wikipedia or other MediaWiki sites.</translate>
<translate><!--T:68--> Once installed on your computer, you can use the specific tool <tvar name=3></tvar> which lets you upload a wiki file on Wikipedia or MediaWiki sites.</translate>
<translate><!--T:69--> The xml file created by <tvar name=1>dumpBackup.php</tvar> can be transformed into a wiki file suitable to be processed by <tvar name=2>pagefromfile.py</tvar> using a simple script similar to the following (here the program will transform all xml files which are on the current directory which is needed if your MediaWiki site is a family):</translate>

<translate>
==== On Python ==== <!--T:161-->
</translate>
<syntaxhighlight lang="python">
from pathlib import Path
from lxml import etree
import re
from pywikibot import Link, Site

""" The exported XML preserves local namespace names. To import into a project in another language,
 you need to convert them to canonical names. Otherwise the pages will be imported as originals
 without recognizing their namespace.
XML contains namespace numbers, but mapping them can be problematic. Because they can differ between projects.
 For example, in ru.wikisource NS Page is number 104, while in uk.wikisource it is 252.
To solve these, the Link() class from pywikibot is used.

The Link() required the language code and family of project. XML does not contain these as separate
 values. Instead, they are in another format, like `enwikisource` or 'commonswiki', which needs to be parsed.    
"""

xml_folder = 'PATH_TO_XML_DUMP'
for xml_file in Path(xml_folder).glob('*.xml'):
    root = etree.parse(xml_file).getroot()
    ns = {'': root.nsmap[None]}  # xml namespace

    # Parsing language code and family of the project
    dbname = root.find('.//siteinfo/dbname', ns).text
    m = re.search(r'^(\w+?)(wiki|wikisource|wikiquote|wikinews|wikibooks)$', dbname)
    if m:
        site = Site(code=m.group(1), fam=m.group(2))
    elif 'commons' in dbname:
        site = Site('commons')
    elif 'mediawikiwiki' in dbname:
        site = Site('mediawiki')
    else:
        print("Site didn't recognized.")
        break

    wiki = []
    for page in root.findall('.//page', ns):
        title_source = page.find('title', ns).text
        title = Link(title_source, site).ns_title().replace(' ', '_')
        revision_id = page.find('.//revision/id', ns).text
        print(f'{title}, revision id: {revision_id}')
        text = page.find('.//revision/text', ns).text
        wiki.append("\n'''%s'''\n%s\n" % (title, text))
    Path(f'out_{xml_file.stem}.wiki').write_text('\n'.join(wiki))
</syntaxhighlight>

<translate>
==== On Ruby ==== <!--T:162-->
</translate>

<syntaxhighlight lang="ruby">
# -*- coding: utf-8 -*-
# dumpxml2wiki.rb

require 'rubygems'
require 'nokogiri'

# <translate nowrap><!--T:140--> This program <tvar name=1>dumpxml2wiki</tvar> reads MediaWiki xml files dumped by <tvar name=2>dumpBackup.php</tvar> on the current directory and transforms them into wiki files which can then be modified and uploaded again by <tvar name=3>pywikipediabot</tvar> using <tvar name=4>pagefromfile.py</tvar> on a MediaWiki family site.</translate>
# <translate nowrap><!--T:143--> The text of each page is searched with xpath and its title is added on the first line as an html comment: this is required by <tvar name=1>pagefromfile.py</tvar>.</translate>
# 
Dir.glob("*.xml").each do |filename|
  input = Nokogiri::XML(File.new(filename), nil, 'UTF-8')

  puts filename.to_s  # prints the name of each .xml file

  File.open("out_" + filename + ".wiki", 'w') {|f| 
    input.xpath("//xmlns:text").each {|n|
      pptitle = n.parent.parent.at_css "title" # searching for the title
      title = pptitle.content
      f.puts "\n<!--'''" << title.to_s << "'''-->" << n.content  << "\n"
    }
  }
end
</syntaxhighlight>

<translate>
<!--T:39-->
For example, here is an excerpt of a wiki file output by the command 'ruby dumpxml2wiki.rb' (two pages can then be uploaded by pagefromfile.py, a Template and a second page which is a redirect):
</translate>

<syntaxhighlight lang="html4strict">
<!--'''Template:Lang_translation_-pl'''--><includeonly>Tłumaczenie</includeonly>


#REDIRECT[[badania demograficzne]]<!--'''ilościowa demografia'''-->
<noinclude>
[[Category:Termin wielojęzycznego słownika demograficznego (pierwsze wydanie)|ilościowa demografia]]
[[Category:Termin wielojęzycznego słownika demograficznego (pierwsze wydanie) (redirect)]]
[[Category:10]]</noinclude>

</syntaxhighlight>

<translate><!--T:40--> The program accesses each xml file, extracts the texts within <tvar name=1><nowiki><text> </text></nowiki></tvar> markups of each page, searches the corresponding title as a parent and enclosed it with the paired <nowiki><!--'''Title of the page'''--> </nowiki> commands used by 'pagefromfile' to create or update a page.</translate>
<translate><!--T:70--> The name of the page is in an html comment and separated by three quotes on the same first start line.</translate>
<translate><!--T:71--> Please notice that the name of the page can be written in Unicode.</translate>
<translate><!--T:72--> Sometimes it is important that the page starts directly with the command, like for a <tvar name=2><nowiki>#REDIRECT</nowiki></tvar>; thus the comment giving the name of the page must be ''after'' the command but still on the first line.</translate>

<translate>
<!--T:41-->
Please remark that the xml dump files produced by <tvar name=1>dumpBackup.php</tvar> are prefixed by a namespace:
</translate>

<syntaxhighlight lang="html4strict"><mediawiki xmlns="http://www.mediawiki.org/xml/export-0.4/"></syntaxhighlight>

<translate>
<!--T:42-->
In order to access the text node using Nokogiri, you need to prefix your path with 'xmlns':
</translate>

<syntaxhighlight lang="html4strict">input.xpath("//xmlns:text")</syntaxhighlight> 

<translate>
<!--T:12-->
Nokogiri is an HTML, XML, SAX, & Reader parser with the ability to search documents via XPath or CSS3 selectors from the last generation of XML parsers using Ruby.

<!--T:43-->
Example of the use of 'pagefromfile' to upload the output wiki text file:
</translate>

<syntaxhighlight lang="bash">
python pagefromfile.py -file:out_filename.wiki -summary:"Reason for changes" -lang:pl -putthrottle:01
</syntaxhighlight>

<translate>