Opasnet update 2017

From Opasnet
Jump to navigation Jump to search

This page contains notes produced while updating Opasnet server systems during the summer of 2017.

Remaining tasks

  • CreateBox to InputBox conversion for page creation
    • Redirect to MediaWiki:Create exists does not work
      • Or rather it is a feature of CreatBox and will be dropped
      • Default behaviour is to open Editor for existing page
    • Also used on these pages
  • RTools map functionality and OpasnetUtilsExt
    • Some datasets (first example on Opasnet map) involve a PostgreSQL/PostGIS database
      • Datatypes make migration between versions difficult
        • custom dump (pg_dump -Fc) + hard upgrade (perl utils/postgis_restore.pl in root directory) seems to work
          • readOGR fails for migrated data, different versions of gdal and rgdal produce different errors traceable to data types
      • database source name / connection string does not work
    • Sorvi package has been updated
      • LoadData replaced by load_sorvi_data
      • Need to install a non-master branch for the data used in the example: install_github("ropengov/sorvi", ref = "data-from-gh")
    • KML is not drawn
      • Page produces "ReferenceError: jQuery is not defined", but map loads and gives message about adding KML, which suggests the relevant part works?
      • Could probably use the mediawiki Maps extension instead and draw KML using that
        • Very hard to implement since the output (map definition by R-code on the RTools-server) would have to be parsed on the Wiki-server running the RTools-extension. Currently there is an escaping mechanism which allows raw html (including javascript) to be injected into RTools output-blocks. This might have been made harder by changes to MediaWiki ResourceLoader in newer versions (to make malicious cross-site-scripting more difficult) resulting in the jQuery error.
      • Could also be a versioning issue, as code on original page runs
      • KML file needs to be publicly hosted for Google Maps API, which was not the case for the test wiki.
  • Iron out minor bugs in RTools-wiki extension, t2b, OpasnetBase-viewer
  • Update live servers
    • Update OS
    • Update software
    • Update extensions
  • New server running latest OS and software
    • english Opasnet duplicate
      • Basic features work
      • TODO: install updated in-house extensions
    • TODO: duplicate rest of the wikis
    • Issues:
      • Page load counters were removed in 1.26
        • DPL using hit counters not possible anymore
      • Contributors extension is broken
      • Apache Virtual server configuration breaks all functionality (likely fixed once using proper domain names)
      • Edit toolbar is missing
        • Version 1.27.3 without usebetatoolbar=1 works
        • TODO: fix custom toolbar buttons
    • TODO: lock old server, sync changes, update address, shut down old server
  • Other notable issues
    • Backup scripts
      • Users and keys
    • Server users and permissions
    • SVN
      • On Wiki server
      • Contains source code of all software written by us

RTools server

CentOS 6.4: full support ended, but still get maintenance updates.

Key software, upgrade paths and related issues:

  • apache 2.2.15 -> 2.2.32 or 2.4.25
    • newer branch not particularly important
  • PHP 5.3.3 -> 5.6.x or 7.1.x
    • incompatibilities in mysql functions
      • "mysql" -> "mysqli" (5.5)
    • should upgrade to latest version
  • MySQL 5.1 -> 5.7
  • mongoDB 2.4 -> 3.2
    • should be relatively easy as we are not using sharding etc.
  • rtools_server PHP application
    • starts on test server after minor changes
    • further testing needed

OpasnetBase2

Runs on Apache as a PHP web application. Updating PHP and MongoDB causes a few incompatibilities and other issues with old code:

  • MongoDB driver updates
  • "Mongo" PHP library deprecated, new "MongoDB" PHP library
    • Mongo-class no longer available
      • Use MongoDB\Client instead
    • Other classes changed to some extent
      • cursor does not handle query options -> everything needs to be supplied to collection.find()
      • Collections do not implement __get(), hence they cannot be chained like before. db->ident->subset->dat should be combined to db->{ident.subset.dat}.
  • Use pecl to install driver
  • Use composer to download library to be included in web app
  • Actually, a lot of code (~1000 lines)
    • "mysql" to "mysqli" patching
      • some static functions may need new parameters e.g. OpasnetBaseActiveRecord::parse_qmarks needed a database link where one was not needed before (due to mysqli_real_escape_string requiring a database link parameter now)
    • The MongoDB drivers (both old and new) overload __get() to give shortcuts to queries and other functionality, which makes debugging difficult.
      • It might be useful to change them to use proper method calls instead.
    • "insert" to "insertOne" and "ensureIndex" to "createIndex" patching
    • "MongoDate" type changed to "MongoDB\BSON\UTCDateTime"
      • Different methods: "date('Y-M-d H:i:s',$obj[$index->ident]->sec)" to "$obj[$index->ident]->toDateTime()->format('Y-M-d H:i:s')"
    • Tracked issue with invalid times in $data to MySQL column data type issue (datetime instead of timestamp)
      • MySQL table download_sessions contains the keys, chunk counters and download times of every download ever
        • HUGE (3.5M rows), pretty much useless after download is complete. This should probably be cleaned regularly.
  • Current issues:
    • Server throws no exceptions, but returns garbage when attempting to prepare uploads using unmodified OpasnetUtils.
      • Using RCurl postForm() for header works.
        • Upload, replace, append, download, include, exclude all work.
      • Return header seems fine
    • test.php upload works
    • Read APIs return errors ERROR: Load by columns: record not found!
      • MongoDB\Driver\Cursor and other driver classes methods are not exposed to PHP?
        • Cursor methods not implemented in the new PHP library -> have to do foreach loop for results and manually chunk them into files.
          • Reasonable to implement Done, performance might be worse however.
      • Collection->find(...) executes the given query which means options such as batchSize have to be included in find() as options.
        • Irrelevant with foreach loop. The driver handles everything.
  • WORKS (well enough)

Opasnet-wiki server

CentOS 6.5: full support ended, but still get maintenance updates.

Key software, upgrade paths and related issues:

  • apache 2.2.15 -> 2.2.32 or 2.4.25
  • PHP 5.4.28 -> 5.6.x or 7.1.x
    • 5.5.9+ required by currently supported mediawiki releases
    • latest php also supported
    • incompatibilities likely to occur in 5.4->5.5, hence might as well upgrade to latest
  • MySQL 5.5.37 -> 5.5.56 or 5.7.x
  • MediaWiki 1.20.0 -> 1.27 (LTS) or 1.28.2 (latest)
    • need to check supported extensions
    • MyExtension.php no longer supported, need extension.json for registering extensions
    • Composer might be useful for managing PHP dependencies

RTools MediaWiki extension

  • RTools special page class (now named SpecialRtools)
    • Creates instance of rtools_client
    • separate config file, which is included by SpecialRTools instead of rtools_client for some reason
    • fixed: "mysql" to "mysqli"
    • fixed: "wfMsg" to "wfMessage->text()"
  • <rcode> tag
    • Detected by parser hooks
    • RToolsParser class
      • fixed: "wfMsg" to "wfMessage->text()"
      • fixed: class instead of global functions
        • create class instance in parser given by hook in order to retain variable values
      • fixed: added "wgOut->addModules( 'ext.RTools.parser' );" which enables the on-page UI
      • fixed: changed efRCodeMBInArray to a static function and changed reference in templates/inputs/checkbox.php
  • rtools_client
    • handles connections and job submissions etc. to server
    • xml-rpc library
      • communicates with rtools_server API
  • javascripts
    • script.aculo.us + prototype
      • UI
      • ajax
      • old versions, probably still acceptable
    • loaded by mediawiki resource loader
      • now set up in extension.json
  • wiki database
    • rtools_inputs
      • added schema update hook to create the necessary table if missing
  • Current issues:
    • Show code button does not work
    • Run page throws: Didn't receive 200 OK from remote server. (HTTP/1.0 500 Internal Server Error), which is a xml-rpc related error
    • Runs and shows outputs (oprint and cat), but code is not visible, embed also does not work
    • Input forms are slightly broken: conditional categories do not work, checkboxes crash the whole page
    • Rtools page does not update, shows error until refreshed once the run is over. Error on page: "jQuery is not defined", but it is loaded by mediawiki by default. Caused by some linking problem between the special page and mediawiki.
      • SpecialRTools-class problem with include of "template/index.php" in method "output" not scoping as expected. Including javascript dependencies separately resolves the first issue, but results in "mw not defined" indicating no access to mediawiki.
      • Special page includes javascript for both parser and special page, which does not happen on the current server. It is loaded as part of parser initialization on every page.
      • Works 1% of the time ...

Extensions

Our extensions

ExternalSourceBrowser: Einari

OpasnetBase: Einari, Juha

  • Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
  • wfMsg() to wfMessage()->text()
  • Issue: row counter does not always work
    • javascript autoloader does not appear to work on all included parts of the page
  • Seems to work otherwise

OpasnetBaseImport: Einari, Juha

  • Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
    • Parser stuff put in a new class
    • Added all classes to Autoload
    • Load configuration file at start of all class definition files
      • Added "global $wgArticlePath, $wgScript;" at config start to give access to mw globals in the new scope
    • Corrected extension documentation URL
  • wfMsg() to wfMessage()->text()
  • Current issues:
    • configuration variable $obImportUploadsPath becomes null or somehow
      • Changed config to use constants instead of global variables
  • Works

Table2Base: Juha, Einari

  • Uses OpasnetBaseImporters OpasnetBaseUpload class
  • Converted to extension registeration
  • Created Table2BaseParser-class
  • Writes to both OpasnetBase and Wiki database
    • Need to figure out wiki database string escapes, may be unnecessary with proper use of wrapper functions. Can't use mysqli_real_escape_string due to mediawiki wrapping.
      • Wikimedia\Rdbms\Database::insert handles escaping
    • Need to add database update hook
    • OpasnetBase queries: 'mysql_real_escape_string' to 'mysqli_real_escape_string' patching
  • Current issues:
    • No index for t2b uploads in wiki base may make queries slow (affects performance on every page with t2b)?
    • Displays data normally, displays errors in data, but does not upload.
      • Uploads most of the time, never shows the message: "Data updated successfully!" though.

OpasnetRater: Einari

  • Rater
  • Has seen very limited use
  • Low priority

PageID

  • Not ours actually. Created by Jim R. Wilson in 2008.
  • Implements {{PAGEID}}

Other Extensions

Currently installed extensions, still maintained unless specified otherwise. Almost all have been updated since last time. Nothing tested yet.

  • AuthorProtect
    • Git maintained
  • CategoryTree [1]
  • CheckUser [2]
  • ConfirmEdit is shipped with MediaWiki
    • ReCaptcha should be replaced with ReCaptcha(NoCaptcha)
  • ContactPage [3]
    • Need to reconfigure local settings when upgrading
  • Contribution Scores [4]
  • Contributors [5]
  • DataValues [6]
    • DataValues Geo [7]
    • DataValues Time [8]
    • DataValues Validators [9]
    • DataValues Common [10]
    • DataValues Interfaces [11]
    • Still updated
    • Loaded via Composer
    • Used by Maps extension
  • DynamicPageList [12]
    • Very old version, might not work on latest MediaWiki and PHP
  • Google Analytics Integration [13]
  • ImageMap is shipped with MediaWiki
  • Maps [14]
    • Very active development
    • Possible incompatibilities with RTools when upgraded
    • Opasnet_map
      • Need to check compatibility with OpasnetUtilsExt and the RTools extension
  • Math [15]
  • MsLinks [16]
    • Used to display files on pages using "{{#l:Testfile.pdf}}"
      • Has this been used at all?
    • Very old version, likely broken
  • MsUpload [17]
    • Upload files from editor
      • Has this seen any use?
  • NewPageCSS [18]
    • Very old version
    • Security risk?
      • Install by copying source from wikipage might be considered hazardous, however there are no issues with code.
  • PageNotice [19]
  • SimpleCalendar
  • User Merge and Delete [20]
    • Has MediaWiki 1.29+ listed on extension page, but probably works on 1.28
  • Validator [21]
  • Widgets [22]
    • Works

Obsolete or otherwise dead

Not maintained anymore, very old, security risk or integrated to MediaWiki.

  • Ajax Show Editors [23]
    • Archived
  • CreateBox [24]
    • Archived
    • Option: InputBox [25]
      • This is included with new installations of MediaWiki 1.21 and above
    • Used on Create_article
      • Any others?
  • EmailAddressImage [26]
    • access to extensions folder is restricted in new default installation, which breaks this extension
      • might work in Opasnet
  • Google Co-op Extension [27] ?
    • Search functionality that was never used and is now obsolete?
  • NiceCategoryList [28]
    • Security risk
    • Implements ncl-tag e.g. <ncl>Category:Traffic</ncl>
      • DynamicPageList dpl-tag does the same thing with <dpl>category = Traffic</dpl>, is in active development, does not have known vulnerabilities and seems to be in wider use in Opasnet as well.
  • Todo Tasks [29]
    • Not maintained since 2009, expect to break.

Not listed in Special:Version

Have to check local settings to see if these are installed. Not in use.

  • AccessControl [30]
    • Maintained
  • ArticleComments [31]
    • Archived
  • CategoryStepper [32]
    • Unmaintained
  • CategorySuggest [33]
    • Maintained
  • Discussion [34]
    • Not maintained, download link dead.
  • DiscussionThreading [35]
    • Maintained
  • Header Footer [36]
    • Maintained
  • Lockdown [37]
    • Maintained
  • mfiles, mfiles2
  • NamespacePermissions [38]
    • Suggested: Lockdown, which is also present on the server
  • PageDisqus [39]
    • Still in beta, git has some recent commits ...
  • PdfHandler [40]
    • Shipped with current version of MediaWiki
  • Renameuser
    • Shipped with current version of MediaWiki
  • SelectCategory [41]
    • Maintained
  • SimpleAntiSpam [42]
    • Merged into MediaWiki
  • SpamBlacklist [43]
    • Maintained
  • StubManager [44]
    • Archived
  • Survey [45]
    • Unmaintained
    • Alternative if still in use: QuickSurveys [46]?
  • TitleBlacklist [47]
    • Maintained
  • WhoIsWatching [48]
    • Maintained
  • WhosOnlineAjax [49]
    • Archived
    • Alternative: WhosOnline [50]

See Also

Opasnet:Technical_documentation

Retrieved from "https://en.opasnet.org/index.php?title=Opasnet_update_2017&oldid=41317"