Opasnet update 2017

This page contains notes produced while updating Opasnet server systems during the summer of 2017.

Remaining tasks

• CreateBox to InputBox conversion for page creation
  • Redirect to MediaWiki:Create exists does not work
    • Or rather it is a feature of CreatBox and will be dropped
    • Default behaviour is to open Editor for existing page
  • Also used on these pages
• RTools map functionality and OpasnetUtilsExt
  • Some datasets (first example on Opasnet map) involve a PostgreSQL/PostGIS database
    • Datatypes make migration between versions difficult
      • custom dump (pg_dump -Fc) + hard upgrade (perl utils/postgis_restore.pl in root directory) seems to work
        • readOGR fails for migrated data, different versions of gdal and rgdal produce different errors traceable to data types
    • database source name / connection string does not work
  • Sorvi package has been updated
    • LoadData replaced by load_sorvi_data
    • Need to install a non-master branch for the data used in the example: install_github("ropengov/sorvi", ref = "data-from-gh")
  • KML is not drawn
    • Page produces "ReferenceError: jQuery is not defined", but map loads and gives message about adding KML, which suggests the relevant part works?
    • Could probably use the mediawiki Maps extension instead and draw KML using that
      • Very hard to implement since the output (map definition by R-code on the RTools-server) would have to be parsed on the Wiki-server running the RTools-extension. Currently there is an escaping mechanism which allows raw html (including javascript) to be injected into RTools output-blocks. This might have been made harder by changes to MediaWiki ResourceLoader in newer versions (to make malicious cross-site-scripting more difficult) resulting in the jQuery error.
    • Could also be a versioning issue, as code on original page runs
    • KML file needs to be publicly hosted for Google Maps API, which was not the case for the test wiki.
• Iron out minor bugs in RTools-wiki extension, t2b, OpasnetBase-viewer
• Update live servers
  • Update OS
  • Update software
  • Update extensions
• New server running latest OS and software
  • english Opasnet duplicate
    • Basic features work
    • TODO: install updated in-house extensions
  • TODO: duplicate rest of the wikis
  • Issues:
    • Page load counters were removed in 1.26
      • DPL using hit counters not possible anymore
    • Contributors extension is broken
    • Apache Virtual server configuration breaks all functionality (likely fixed once using proper domain names)
    • Edit toolbar is missing
      • Version 1.27.3 without usebetatoolbar=1 works
      • TODO: fix custom toolbar buttons
  • TODO: lock old server, sync changes, update address, shut down old server
• Other notable issues
  • Backup scripts
    • Users and keys
  • Server users and permissions
  • SVN
    • On Wiki server
    • Contains source code of all software written by us

RTools server

CentOS 6.4: full support ended, but still get maintenance updates.

Key software, upgrade paths and related issues:

• apache 2.2.15 -> 2.2.32 or 2.4.25
  • newer branch not particularly important
• PHP 5.3.3 -> 5.6.x or 7.1.x
  • incompatibilities in mysql functions
    • "mysql" -> "mysqli" (5.5)
  • should upgrade to latest version
• MySQL 5.1 -> 5.7
• mongoDB 2.4 -> 3.2
  • should be relatively easy as we are not using sharding etc.
• rtools_server PHP application
  • starts on test server after minor changes
  • further testing needed

OpasnetBase2

Runs on Apache as a PHP web application. Updating PHP and MongoDB causes a few incompatibilities and other issues with old code:

• MongoDB driver updates
• "Mongo" PHP library deprecated, new "MongoDB" PHP library
  • Mongo-class no longer available
    • Use MongoDB\Client instead
  • Other classes changed to some extent
    • cursor does not handle query options -> everything needs to be supplied to collection.find()
    • Collections do not implement __get(), hence they cannot be chained like before. db->ident->subset->dat should be combined to db->{ident.subset.dat}.
• Use pecl to install driver
• Use composer to download library to be included in web app
• Actually, a lot of code (~1000 lines)
  • "mysql" to "mysqli" patching
    • some static functions may need new parameters e.g. OpasnetBaseActiveRecord::parse_qmarks needed a database link where one was not needed before (due to mysqli_real_escape_string requiring a database link parameter now)
  • The MongoDB drivers (both old and new) overload __get() to give shortcuts to queries and other functionality, which makes debugging difficult.
    • It might be useful to change them to use proper method calls instead.
  • "insert" to "insertOne" and "ensureIndex" to "createIndex" patching
  • "MongoDate" type changed to "MongoDB\BSON\UTCDateTime"
    • Different methods: "date('Y-M-d H:i:s',$obj[$index->ident]->sec)" to "$obj[$index->ident]->toDateTime()->format('Y-M-d H:i:s')"
  • Tracked issue with invalid times in $data to MySQL column data type issue (datetime instead of timestamp)
    • MySQL table download_sessions contains the keys, chunk counters and download times of every download ever
      • HUGE (3.5M rows), pretty much useless after download is complete. This should probably be cleaned regularly.
• Current issues:
  • Server throws no exceptions, but returns garbage when attempting to prepare uploads using unmodified OpasnetUtils.
    • Using RCurl postForm() for header works.
      • Upload, replace, append, download, include, exclude all work.
    • Return header seems fine
  • test.php upload works
  • Read APIs return errors ERROR: Load by columns: record not found!
    • MongoDB\Driver\Cursor and other driver classes methods are not exposed to PHP?
      • Cursor methods not implemented in the new PHP library -> have to do foreach loop for results and manually chunk them into files.
        • Reasonable to implement Done, performance might be worse however.
    • Collection->find(...) executes the given query which means options such as batchSize have to be included in find() as options.
      • Irrelevant with foreach loop. The driver handles everything.
• WORKS (well enough)

Opasnet-wiki server

CentOS 6.5: full support ended, but still get maintenance updates.

Key software, upgrade paths and related issues:

• apache 2.2.15 -> 2.2.32 or 2.4.25
• PHP 5.4.28 -> 5.6.x or 7.1.x
  • 5.5.9+ required by currently supported mediawiki releases
  • latest php also supported
  • incompatibilities likely to occur in 5.4->5.5, hence might as well upgrade to latest
• MySQL 5.5.37 -> 5.5.56 or 5.7.x
• MediaWiki 1.20.0 -> 1.27 (LTS) or 1.28.2 (latest)
  • need to check supported extensions
  • MyExtension.php no longer supported, need extension.json for registering extensions
  • Composer might be useful for managing PHP dependencies

RTools MediaWiki extension

• RTools special page class (now named SpecialRtools)
  • Creates instance of rtools_client
  • separate config file, which is included by SpecialRTools instead of rtools_client for some reason
  • fixed: "mysql" to "mysqli"
  • fixed: "wfMsg" to "wfMessage->text()"
• <rcode> tag
  • Detected by parser hooks
  • RToolsParser class
    • fixed: "wfMsg" to "wfMessage->text()"
    • fixed: class instead of global functions
      • create class instance in parser given by hook in order to retain variable values
    • fixed: added "wgOut->addModules( 'ext.RTools.parser' );" which enables the on-page UI
    • fixed: changed efRCodeMBInArray to a static function and changed reference in templates/inputs/checkbox.php
• rtools_client
  • handles connections and job submissions etc. to server
  • xml-rpc library
    • communicates with rtools_server API
• javascripts
  • script.aculo.us + prototype
    • UI
    • ajax
    • old versions, probably still acceptable
  • loaded by mediawiki resource loader
    • now set up in extension.json
• wiki database
  • rtools_inputs
    • added schema update hook to create the necessary table if missing
• Current issues:
  • Show code button does not work
  • Run page throws: Didn't receive 200 OK from remote server. (HTTP/1.0 500 Internal Server Error), which is a xml-rpc related error
  • Runs and shows outputs (oprint and cat), but code is not visible, embed also does not work
  • Input forms are slightly broken: conditional categories do not work, checkboxes crash the whole page
  • Rtools page does not update, shows error until refreshed once the run is over. Error on page: "jQuery is not defined", but it is loaded by mediawiki by default. Caused by some linking problem between the special page and mediawiki.
    • SpecialRTools-class problem with include of "template/index.php" in method "output" not scoping as expected. Including javascript dependencies separately resolves the first issue, but results in "mw not defined" indicating no access to mediawiki.
    • Special page includes javascript for both parser and special page, which does not happen on the current server. It is loaded as part of parser initialization on every page.
    • Works 1% of the time ...

Extensions

Our extensions

ExternalSourceBrowser: Einari

• Started in 2010, looks incomplete
• Special:ExternalSourceBrowser - missing config
• Still loaded for some reason

OpasnetBase: Einari, Juha

• Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
• wfMsg() to wfMessage()->text()
• Issue: row counter does not always work
  • javascript autoloader does not appear to work on all included parts of the page
• Seems to work otherwise

OpasnetBaseImport: Einari, Juha

• Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
  • Parser stuff put in a new class
  • Added all classes to Autoload
  • Load configuration file at start of all class definition files
    • Added "global $wgArticlePath, $wgScript;" at config start to give access to mw globals in the new scope
  • Corrected extension documentation URL
• wfMsg() to wfMessage()->text()
• Current issues:
  • configuration variable $obImportUploadsPath becomes null or somehow
    • Changed config to use constants instead of global variables
• Works

Table2Base: Juha, Einari

• Uses OpasnetBaseImporters OpasnetBaseUpload class
• Converted to extension registeration
• Created Table2BaseParser-class
• Writes to both OpasnetBase and Wiki database
  • Need to figure out wiki database string escapes, may be unnecessary with proper use of wrapper functions. Can't use mysqli_real_escape_string due to mediawiki wrapping.
    • Wikimedia\Rdbms\Database::insert handles escaping
  • Need to add database update hook
  • OpasnetBase queries: 'mysql_real_escape_string' to 'mysqli_real_escape_string' patching
• Current issues:
  • No index for t2b uploads in wiki base may make queries slow (affects performance on every page with t2b)?
  • Displays data normally, displays errors in data, but does not upload.
    • Uploads most of the time, never shows the message: "Data updated successfully!" though.

OpasnetRater: Einari

• Rater
• Has seen very limited use
• Low priority

PageID

• Not ours actually. Created by Jim R. Wilson in 2008.
• Implements {{PAGEID}}
  • Which is a core variable in MediaWiki 1.20+

Other Extensions

Currently installed extensions, still maintained unless specified otherwise. Almost all have been updated since last time. Nothing tested yet.

• AuthorProtect
  • Git maintained
• CategoryTree [1]
• CheckUser [2]
• ConfirmEdit is shipped with MediaWiki
  • ReCaptcha should be replaced with ReCaptcha(NoCaptcha)
• ContactPage [3]
  • Need to reconfigure local settings when upgrading
• Contribution Scores [4]
• Contributors [5]
• DataValues [6]
  • DataValues Geo [7]
  • DataValues Time [8]
  • DataValues Validators [9]
  • DataValues Common [10]
  • DataValues Interfaces [11]
  • Still updated
  • Loaded via Composer
  • Used by Maps extension
• DynamicPageList [12]
  • Very old version, might not work on latest MediaWiki and PHP
• Google Analytics Integration [13]
• ImageMap is shipped with MediaWiki
• Maps [14]
  • Very active development
  • Possible incompatibilities with RTools when upgraded
  • Opasnet_map
    • Need to check compatibility with OpasnetUtilsExt and the RTools extension
• Math [15]
• MsLinks [16]
  • Used to display files on pages using "{{#l:Testfile.pdf}}"
    • Has this been used at all?
  • Very old version, likely broken
• MsUpload [17]
  • Upload files from editor
    • Has this seen any use?
• NewPageCSS [18]
  • Very old version
  • Security risk?
    • Install by copying source from wikipage might be considered hazardous, however there are no issues with code.
• PageNotice [19]
• SimpleCalendar
• User Merge and Delete [20]
  • Has MediaWiki 1.29+ listed on extension page, but probably works on 1.28
• Validator [21]
• Widgets [22]
  • Works

Obsolete or otherwise dead

Not maintained anymore, very old, security risk or integrated to MediaWiki.

• Ajax Show Editors [23]
  • Archived
• CreateBox [24]
  • Archived
  • Option: InputBox [25]
    • This is included with new installations of MediaWiki 1.21 and above
  • Used on Create_article
    • Any others?
• EmailAddressImage [26]
  • access to extensions folder is restricted in new default installation, which breaks this extension
    • might work in Opasnet
• Google Co-op Extension [27] ?
  • Search functionality that was never used and is now obsolete?
• NiceCategoryList [28]
  • Security risk
  • Implements ncl-tag e.g. <ncl>Category:Traffic</ncl>
    • DynamicPageList dpl-tag does the same thing with <dpl>category = Traffic</dpl>, is in active development, does not have known vulnerabilities and seems to be in wider use in Opasnet as well.
• Todo Tasks [29]
  • Not maintained since 2009, expect to break.

Not listed in Special:Version

Have to check local settings to see if these are installed. Not in use.

• AccessControl [30]
  • Maintained
• ArticleComments [31]
  • Archived
• CategoryStepper [32]
  • Unmaintained
• CategorySuggest [33]
  • Maintained
• Discussion [34]
  • Not maintained, download link dead.
• DiscussionThreading [35]
  • Maintained
• Header Footer [36]
  • Maintained
• Lockdown [37]
  • Maintained
• mfiles, mfiles2
• NamespacePermissions [38]
  • Suggested: Lockdown, which is also present on the server
• PageDisqus [39]
  • Still in beta, git has some recent commits ...
• PdfHandler [40]
  • Shipped with current version of MediaWiki
• Renameuser
  • Shipped with current version of MediaWiki
• SelectCategory [41]
  • Maintained
• SimpleAntiSpam [42]
  • Merged into MediaWiki
• SpamBlacklist [43]
  • Maintained
• StubManager [44]
  • Archived
• Survey [45]
  • Unmaintained
  • Alternative if still in use: QuickSurveys [46]?
• TitleBlacklist [47]
  • Maintained
• WhoIsWatching [48]
  • Maintained
• WhosOnlineAjax [49]
  • Archived
  • Alternative: WhosOnline [50]

See Also

Opasnet:Technical_documentation