YaCy '#[clientname]#': MediaWiki Dump Import

#%env/templates/metas.template%# #(refresh)#:: #(/refresh)#
#%env/templates/header.template%# #%env/templates/submenuIndexImport.template%#

## MediaWiki Dump Import

#(import)#

#(prevReport)#::Check the last import report.#(/prevReport)#

#(status)#

No import thread is running, you can start a new thread here
::
Error : dump URL is malformed.
::
Error : file not found "#[sourceFile]#"
::
Error : can not read file "#[sourceFile]#"
::
Error : you selected a directory ("#[sourceFile]#")
::
Error : dump file ("#[sourceFile]#") was not modified since last import (#[lastImportDate]#).
#(/status)#
MediaWiki Dump File Selection

You can import MediaWiki dumps here. An example is the file https://dumps.wikimedia.org/dewiki/latest/dewiki-latest-pages-articles.xml.bz2.

Dumps can be stored in the local file system or on a remote server in XML format and may be compressed in gz or bz2.

Dump file path or URL
Import only when modified since last import
When checked, the dump file is imported only if its last modified date is unknown or is after the last import execution date on this same file (see recorded API calls with the "dump" type).

When the import is started, the following happens:

- The dump is extracted on the fly and wiki entries are translated into Dublin Core data format. The output looks like this:

          <?xml version="1.0" encoding="UTF-8"?>
      <surrogates xmlns:dc="http://purl.org/dc/elements/1.1/">
        <record>
          <dc:Title><![CDATA[Alan Smithee]]></dc:Title>
          <dc:Identifier>http://de.wikipedia.org/wiki/Alan%20Smithee</dc:Identifier>
          <dc:Description><![CDATA[Der als Filmregisseur oft genannte Alan Smithee ist ein Anagramm]]></dc:Description>
          <dc:Language>de</dc:Language>
          <dc:Date>2009-05-07T06:03:48Z</dc:Date>
        </record>
        <record>
          ...
        </record>
      </surrogates>

- Each 10000 wiki records are combined in one output file which is written to /DATA/SURROGATES/in into a temporary file.
- When each of the generated output file is finished, it is renamed to a .xml file
- Each time a xml surrogate file appears in /DATA/SURROGATES/in, the YaCy indexer fetches the file and indexes the record entries.
- When a surrogate file is finished with indexing, it is moved to /DATA/SURROGATES/out
- You can recycle processed surrogate files by moving them from /DATA/SURROGATES/out to /DATA/SURROGATES/in

::

#(status)#::

Error encountered : #[message]#
#(/status)#
Import Process

Thread:
  #(thread)#terminated. You can check the next step in the Crawl Results page or start a new import.::started::running#(/thread)#
Dump:
  #[dump]#
Processed:
  #[count]# Wiki Entries
Speed:
  #[speed]# articles per second
Running Time:
  #[runningHours]# hours, #[runningMinutes]# minutes
Remaining Time:
  #[remainingHours]# hours, #[remainingMinutes]# minutes

#(/import)# #%env/templates/footer.template%#