User:Podbrushkin/Wikibase: Local Instance

This is a comprehensive guide on how to install your own instance of MediaWiki with Wikibase extension with the help of Docker. It is intended for unexperienced Windows users who had never seen Docker or bash before, but can be followed by Linux users as well. Original installation instruction can be found here: Wikibase/Docker. Instruction is relevant as for November, 2022.

=Installing MediaWiki with extensions= Wikibase is an extension to MediaWiki, which you will get by default, if you will choose to install MediaWiki via Docker images. Installed MediaWiki will be of version 1.37.6, which is not the latest release, but it will have multiple extensions preinstalled, so you wouldn't need to look for compatible versions yourself.
 * First of all, make sure you have Docker Desktop installed, if not - install it from offical website.
 * In CMD, run  to check if it's installed.
 * You should have at least 10 GB of free disk space.
 * Create a directory which you will use for Docker-related files, give it a name of your choice, for example . Note, name of this directory will be used as a part of a name for Docker containers, which will be created later. Avoid using non-aplhanumeric characters.
 * Copy docker-compose.yml, docker-compose.extra.yml and template.env files to this directory from Github: wikibase-release-pipeline/example.
 * In template.env put your own values for this variables:
 * MW_ADMIN_PASS, DB_PASS: create your own password. Make sure it's not very common and is longer than 10 symbols, otherwise container won't start.
 * MW_SECRET_KEY: use a random combination of letters and digits;
 * WIKIBASE_HOST, WDQS_FRONTEND_HOST, QUICKSTATEMENTS_HOST: use your local ip address, which can be found here:
 * Or here:
 * Probably it's something like 192.168.0.1 or 192.168.1.3.
 * Rename file template.env to .env
 * Open this directory in CMD and run this command:
 * For this command to work, Docker Desktop should be running. Execution can take up to 15 minutes.
 * Visit  in your browser, you should see your MediaWiki Main page, installation is complete.
 * If it's not available, go to  and Start your wikibaselocal container.
 * You can login with credentials you've set in your .env file.
 * Give a closer look to your .env file: it contains ports on which MediaWiki extensions are hosted. You can add following links to yor Main Page for quick access:
 * localhost:8834 - SparQL endpoint;
 * localhost:8840 - QuickStatements;
 * Check your  folder, it can take up to 8 Gb. This disk space is reserved by Docker and wouldn't be freed up on containers, images or volumes deletion.
 * Probably, search function doesn't work. Check section on this page to fix it.
 * Check your  folder, it can take up to 8 Gb. This disk space is reserved by Docker and wouldn't be freed up on containers, images or volumes deletion.
 * Probably, search function doesn't work. Check section on this page to fix it.

=Import data= Consult this sources:
 * Wikibase/Importing
 * wikibase.consulting
 * mw:Manual:Parameters_to_Special:Export
 * MediaWiki_Administrator%27s_Handbook/Importing

Native MediaWiki export/import
Don't follow this section, it is not ready. If you know, how to finish it, please help.

There is no way to export only properties or some range of items.

By default, native import of Wikibase entities is not enabled. You can enable it by adding this line

to the end of this file:.

You can do it by running this command:
 * Argument to printf is just contents of new line enclosed with newline-characters and escaped $-sign.
 * Argument to printf is just contents of new line enclosed with newline-characters and escaped $-sign.

Export some sample item from Wikidata, Neo for example:
 * If you don't have curl available, you can open link from the command above in browser and download .xml by yourself.
 * If you need multiple items, on the d:Special:Export page type in textfield qids you need (one per line), check all three checkboxes and click Export.
 * If you need multiple items, on the d:Special:Export page type in textfield qids you need (one per line), check all three checkboxes and click Export.

Follow this link, you will get an empty dump file from your own MediaWiki.

Replace  section of the dataToImport.xml file with similar section from dump of your own MediaWiki.

In dataToImport.xml file replace all  with.
 * If you're importing properties, they will have  already, replace them with.
 * You can use Powershell:
 * First is for properties, second is for items.
 * First is for properties, second is for items.
 * First is for properties, second is for items.

Your dumpfile is ready, copy it into Docker container:

Import it with importDump.php:

Run maintenance scripts: php /var/www/html/maintenance/rebuildall.php php /var/www/html/maintenance/runJobs.php php /var/www/html/maintenance/initSiteStats.php --update Visit your imported item: wiki/Item:Q247120

Visiting pages of some imported items (e.g. Q1-Q4) can bring this error: Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 499712 bytes) in /var/www/html/includes/libs/rdbms/database/Database.php on line 2605 Dispose of them here: Special:Nuke And run

Replicator
As stated in tool's Github repository: 2022 note: this tool is somewhat out of date This tool is outdated, doesn't work and shouldn't be used. It is an attempt to import via tool Replicator which has been obtained from this page.

Following this instruction, download and install Replicator:

cd /var/local git clone https://github.com/JeroenDeDauw/Replicator.git cd /var/local/Replicator composer install Got  error? Execute  and try   again.

Next, you will need to provide this utility with your actual database credentials, which you set up in your .env file. It reads them from  file, and you will create it on the base of. You will need to provide DB_NAME, DB_USER, DB_PASS variables from your .env file and the name of your mysql container. There are two ways to do it.

First method - using sed utility, which allows you to change file from the shell. You may need to change bold values for your actual ones.

cd /var/local/Replicator/config cp db-example.json db.json sed -i "s/vagrant_replicator/my_wiki/i" db.json sed -i "s/root/sqluser/i" db.json sed -i "s/PASSWORD_HERE/yourDB_PASSfromEnvFile/i" db.json sed -i "s/localhost/wikibaselocal-mysql-1/i" db.json

Second method - copying file to your filesystem, changing it with your favorite text editor, and loading it back to container again:
 * Open your WikibaseLocal directory in CMD and run following command. It will copy file from Docker container to your actual filesystem.
 * Open this file and change this values:.
 * Save changes and execute following command. It will move edited file back to container.
 * Save changes and execute following command. It will move edited file back to container.

Now db.json contains the details of your database:

Install MySQL driver:

Proceed Replicator installation: cd /var/local/Replicator composer install php replicator install Import something:
 * - import one item;
 * - import range of items;

Run maintenance scripts: php /var/www/html/maintenance/rebuildall.php php /var/www/html/maintenance/runJobs.php php /var/www/html/maintenance/initSiteStats.php --update Changes will not be visible. Import was unsuccessfull.

Also it's possible you will encounter character set issues on the  step. In this case, in your MySQL container's shell, run, it will find you settings file of mysql. Use. Maybe this SO will help: Or this:

=Additional information=

Search function
If you will try to use a searchbar in your MediaWiki, you will see a message:. This is because search extensions are disabled by default. This section will show you how to enable them. It is based upon this instruction: Wikibase/Suite. It means you have everything ready to enable search in your MediaWiki. wfLoadExtension( 'Elastica' ); wfLoadExtension( 'CirrusSearch' ); wfLoadExtension( 'WikibaseCirrusSearch' );
 * Check Special:Version to see what extensions installed. You will see Elasticsearch in Installed software section.
 * Execute  to see available extensions. You will see folders such as CirrusSearch, Elastica and WikibaseCirrusSearch.
 * Check your LocalSettings.php file with
 * Probably it doesn't have lines, mentioned in linked instruction;
 * To add this lines, in your WikibaseLocal directory, execute
 * Open this file and add following lines to the end of it. Replace  with MW_ELASTIC_HOST variable from your .env file, for example

$wgCirrusSearchServers = [ '' ]; $wgSearchType = 'CirrusSearch'; $wgCirrusSearchExtraIndexSettings['index.mapping.total_fields.limit'] = 5000; $wgWBCSUseCirrus = true; php /var/www/html/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --startOver php /var/www/html/maintenance/runJobs.php --wiki my_wiki --wait & php /var/www/html/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --queue --maxJobs 10000 --pauseForJobs 1000 --skipLinks --indexOnSkip
 * Save it and execute following command. It will move edited file back to container.
 * You can check if changes were saved:
 * Execute following commands one by one:
 * Execute following commands one by one:

Now your search works. Note, this changes can be lost on Docker restart.

Clean restart
Do you want to start it all over again, but don't want to wait for 15 minutes? Following this instruction found here, you can do this:
 * In your WikibaseLocal directory, execute, it will stop all containers;
 * Run, it will remove all containers and networks from Docker ;
 * Run, it will remove all volumes. At this point, in Docker Desktop you will see it has only images, but no containers or volumes.
 * will re-create everything up.

MySQL
There is a MySQL server runnning in one of your containers in Docker.

You can find out the name of the container by executing  in directory with your .yml files. Throughout this page, directory is  and container name is wikibaselocal-mysql-1. You can gain access to it's tables this way:
 * 1) Open your MySQL container's shell in Docker:
 * Or just execute  in CMD. Replace wikibaselocal-mysql-1 if needed.
 * 1) Run command
 * Your database name, address and credentials can be found in  file you've created earlier.
 * 1) Input your SQL commands
 * Your database name, address and credentials can be found in  file you've created earlier.
 * 1) Input your SQL commands

Zip installation
and  commands should be available in your wikibaselocal-wikibase-1 shell. Run  to see if it's installed. If it's not installed, execute this commands: apt-get update apt install zip unzip Now  should return infromation about installed zip utility.

Source:

Composer installation
Before installing Composer, make sure you have  already installed.

In your wikibaselocal-wikibase-1 container shell,  should return version of installed Composer utility. If it's not, install it by executing following commands: php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" php -r "if (hash_file('sha384', 'composer-setup.php') === '55ce33d7678c5a611085589f1f3ddf8b3c52d662cd01d4ba75c0ee0459970c2200a51f492d557530c71c15d8dba01eae') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" php composer-setup.php php -r "unlink('composer-setup.php');" mv composer.phar /usr/local/bin/composer

Now  should work. You will also need a composer-merge-plugin, to install it execute: cd /var/www composer require wikimedia/composer-merge-plugin Sources:
 * Composer:
 * getcomposer.org
 * composer-merge-plugin:
 * Composer
 * Github/composer-merge-plugin

Bash commands
You will interact with the filesystem of your containers through shell. Each container has it's own shell. Run  to get into your wikibase-shell. Replace wikibaselocal-wikibase-1 with your actual wikibase container's name, if it's different. In shell, run  to return to CMD. 
 * - show current directory
 * - move to directory
 * - list contents of directory
 * - find and print all files with specified name
 * - find and delete all files (but not directories) with specified name
 * - delete single file
 * - delete single directory and all of it's contents
 * - print contents of a file
 * - clear screen