User:Kine/Upgrades/1.31.0

From Gineipaedia, the Legend of Galactic Heroes wiki

Jump to: navigation, search

I'm in the process of migrating our shitty old server to a fancy new one. As part of this process i'll be, amongst other things, upgrading MediaWiki from 1.16.0 to 1.31.0. This whole thing is fairly complicated and time-consuming, so i figured i should document it in case i need to do something similar in the future — and what better place is there to do that than the wiki itself?

Contents

Current status

January 2019

I regret to report that i did not get around to working on this over the break. :(

MediaWiki 1.32 was delayed, and i think that maybe combining Bootstrap with OOUI (the stuff used on the admin/editing pages) is not viable, so i probably have to rethink that. But i do want to get this done. If nothing else, i'm still paying to keep the new server running. I'm not sure when i'll get back to it though; hard to predict my whims.  ♥ kine @ 12:40, 3 January 2019 (UTC)

October 2018

I ended up getting burnt out on this in August or September. The new theme is probably 2/3 of the way done as far as the 'content' features go (i.e., stuff needed just to read the articles), but there's quite a bit left to do when it comes to admin/editing pages. Search works, appearance lists work, image uploads work (though thumb-nailing may need further adjustments), the main navigation menu is bare-bones but usable. The date template needs re-implemented, and the main page is still broken, and a few features like image galleries remain unstyled.

I'm hoping i'll be able to finish the job in December, when i have a bunch of time off. By then, MediaWiki 1.32.0 should be out, so that's good timing i guess. Will update again later.  ♥ kine @ 02:54, 19 October 2018 (UTC)

Background

Gineipaedia was first set up in late 2010 / early 2011. The progress i made on the site early on was documented at Gineipaedia:News (maybe this page should be moved, idk). The version of MediaWiki we used was the latest at the time, 1.16.0. The most notable feature of this version was the introduction of the Vector skin (Wikipedia's current default). Gineipaedia's skin is a hacked-up version of Vector and dates from that time.

In addition to the huge changes i made to Vector, i also modified the MediaWiki core, mostly to override page output i didn't like (links, titles, recent changes, site map, sign-ups, &c.). Obviously upgrading means we'll lose those changes unless otherwise accounted for. Fortunately, a lot of them are no longer necessary, or, given the benefit of experience, i know how to implement them in a less stupid way.

Even then though we have a bunch of extensions that need upgraded or replaced.

On top of all that, a lot has changed on the Web since 2011, and Gineipaedia's general design now looks and behaves like shit. It doesn't work well on mobile, it doesn't account for high-DPI (Retina) displays, some of the JavaScript stuff is broken, and it just looks dated and ugly. Furthermore, since i changed everything so drastically, it's very hard to keep up with upgrades (even if nothing else was an issue), because i have to micro-manage every single little aspect of the interface to make it look right. Even with the current skin i've failed at that — see the broken tabs on Special:Preferences for example.

So that's probably the biggest thing i need to address: We need a new skin.

Architecture overview

Gineipaedia is hosted on a server called odin. odin has been upgraded exactly once since 2011 — in December 2015, when i migrated it from Debian 6.0 (squeeze) to Ubuntu 14.04 (trusty). At that time i also moved us from MySQL to MariaDB, and from Apache to nginx. I also (finally) set up HTTPS. (These were only changes to the lower-level architecture — i did nothing with MediaWiki itself.)

As part of the 1.31 migration i decided to spin up an entirely new server, called fezzan. fezzan is running Ubuntu 18.04 (bionic), still with MariaDB and nginx, but everything is generally much newer and nicer.

Content migration

I've succeeded in migrating the basic wiki content over, and so far it looks as good as one can expect. For future reference, here are the steps i took: 

###################
# Old server (odin)
###################
# Dump database
~ % dump="gineipaedia_com.$( date +%Y%m%d ).sql"
~ % mysqldump \
      -h localhost \
      -u gineipaediacom \
      -p \
      --default-character-set=binary \
      gineipaedia_com \
    > $dump

# Transfer database dump to new server
~ % scp $dump fezzan:

# Transfer image file structure to new server
~ % rsync -ave 'ssh -p 666' /srv/www/gineipaedia/public/w/images fezzan:

#####################
# New server (fezzan)
#####################
# Massage database dump to conform to new database structure
~ % dump="gineipaedia_com.$( date +%Y%m%d ).sql"
~ % LC_ALL=C sed -Ei.bak0 's/\<gineipaedia_com-wiki_:/gineipaedia:/g;' $dump
~ % LC_ALL=C sed -Ei.bak1 's/\<gineipaedia_?com\>/gineipaedia/g;' $dump
~ % LC_ALL=C sed -Ei.bak2 's/`wiki_(\w+)`/`\1`/g;' $dump

# Re-create existing database (if applicable)
~ % mysqladmin -u gineipaedia -p drop   gineipaedia
~ % mysqladmin -u gineipaedia -p create gineipaedia

# Import database dump
~ % mysql -u gineipaedia -p gineipaedia < $dump

# Run database-upgrade script
~ % sudo -u www-data php /srv/www/gineipaedia/public/w/maintenance/update.php

# Move over image file structure
~ % sudo rsync -av --delete \
      --exclude=README --exclude=tmp/ --exclude=lockdir/ \
      images/ \
      /srv/www/gineipaedia/public/w/images/
~ % sudo chown -R www-data:www-data /srv/www/gineipaedia/public/w/images

# Clear any caches
~ % sudo -u www-data touch /srv/www/gineipaedia/public/w/LocalSettings.php
~ % sudo systemctl restart php7.2-fpm nginx

See also:

Broken parts

Here are some things that immediately stand out as broken or otherwise needing changed:

Appearances

I've re-implemented appearance lists as a tag extension. The <appearances> tag now generates the basic list, with the accompanying JavaScript handling just the toggle links. Pages can be updated to use the new tag as follows: 

# Replace old tags by new (requires ReplaceText extension obv)
% sudo -u www-data php /srv/www/gineipaedia/*/w/e*/ReplaceText/m*/replaceAll.php \
    --debug \
    --user 'Rudolf von Goldenbot' \
    --nsall \
    --regex '(?s)<div class="appear">(.+?)<\/div>' '<appearances>$1</appearances>'

# Refresh category links (since the new tag adds categories by default)
sudo -u www-data php /srv/www/gineipaedia/*/w/m*/refreshLinks.php --e 1000
sudo -u www-data php /srv/www/gineipaedia/*/w/m*/refreshLinks.php --e 2000 1000
sudo -u www-data php /srv/www/gineipaedia/*/w/m*/refreshLinks.php --e 3000 2000
# ... and so on (should script the 'chunking')

Redirects vis-à-vis search suggestions

I can't remember why i never set up search suggestions originally — probably i just didn't know what i was doing. In any case, i tried to work around the lack of a useful search function by adding hundreds of superfluous redirects for various misspellings and alternative capitalisations — see Category:Episode_redirects for example. Once search suggestions are implemented properly, these should probably go away; if nothing else, they pollute the suggestions with redundant nonsense.

That said, i do think we need to keep redirects for legit name variations (right?), so it would be nice if i could figure out a way to mark certain pages as non-suggestible. Not sure if that's possible by default. Hmm.

I got full-text search set up today (at least it seems to work). Fortunately, i don't think we need to do anything special for redirects — search suggestions are prioritised based on some kind of quality metric, which takes into account whether the page is just a redirect or whatever. So, for example, if you're looking for João Rebelo, you can type in joa and João Rebelo will be the only result, until you type in joan, and then you'll get Joan Rebelo, and so on. Even the dumb capitalisation redirects i mentioned are hidden (though i still think we should get rid of them). That's pretty cool.
Elasticsearch uses a shit-tonne of RAM though. Maybe too much for this server :/  ♥ kine @ 01:15, 12 August 2018 (UTC)
Retrieved from "https://gineipaedia.com/wiki/User:Kine/Upgrades/1.31.0"
Personal tools
Namespaces
Variants
Actions
Miscellany
Common
Tool box