Opasnet update 2017

From Opasnet
Jump to navigation Jump to search

This page contains notes produced while updating Opasnet server systems during the summer of 2017.

RTools server

CentOS 6.4: full support ended, but still get maintenance updates.

Key software, upgrade paths and related issues:

  • apache 2.2.15 -> 2.2.32 or 2.4.25
    • newer branch not particularly important
  • PHP 5.3.3 -> 5.6.x or 7.1.x
    • incompatibilities in mysql functions
      • "mysql" -> "mysqli" (5.5)
    • should upgrade to latest version
  • MySQL 5.1 -> 5.7
  • mongoDB 2.4 -> 3.2
    • should be relatively easy as we are not using sharding etc.
  • rtools_server PHP application
    • starts on test server after minor changes
    • further testing needed

OpasnetBase2

Runs on Apache as a PHP web application. Updating PHP and MongoDB causes a few incompatibilities and other issues with old code:

  • MongoDB driver updates
  • "Mongo" PHP library deprecated, new "MongoDB" PHP library
    • Mongo-class no longer available
      • Use MongoDB\Client instead
    • Other classes changed to some extent
      • cursor does not handle query options -> everything needs to be supplied to collection.find()
      • Collections do not implement __get(), hence they cannot be chained like before. db->ident->subset->dat should be combined to db->{ident.subset.dat}.
  • Use pecl to install driver
  • Use composer to download library to be included in web app
  • Actually, a lot of code (~1000 lines)
    • "mysql" to "mysqli" patching
      • some static functions may need new parameters e.g. OpasnetBaseActiveRecord::parse_qmarks needed a database link where one was not needed before (due to mysqli_real_escape_string requiring a database link parameter now)
    • The MongoDB drivers (both old and new) overload __get() to give shortcuts to queries and other functionality, which makes debugging difficult.
      • It might be useful to change them to use proper method calls instead.
    • "insert" to "insertOne" and "ensureIndex" to "createIndex" patching
    • "MongoDate" type changed to "MongoDB\BSON\UTCDateTime"
      • Different methods: "date('Y-M-d H:i:s',$obj[$index->ident]->sec)" to "$obj[$index->ident]->toDateTime()->format('Y-M-d H:i:s')"
    • Tracked issue with invalid times in $data to MySQL column data type issue (datetime instead of timestamp)
      • MySQL table download_sessions contains the keys, chunk counters and download times of every download ever
        • HUGE (3.5M rows), pretty much useless after download is complete. This should probably be cleaned regularly.
  • Current issues:
    • Server throws no exceptions, but returns garbage when attempting to prepare uploads using unmodified OpasnetUtils.
      • Using RCurl postForm() for header works.
        • Upload, replace, append, download, include, exclude all work.
      • Return header seems fine
    • test.php upload works
    • Read APIs return errors ERROR: Load by columns: record not found!
      • MongoDB\Driver\Cursor and other driver classes methods are not exposed to PHP?
        • Cursor methods not implemented in the new PHP library -> have to do foreach loop for results and manually chunk them into files.
          • Reasonable to implement Done, performance might be worse however.
      • Collection->find(...) executes the given query which means options such as batchSize have to be included in find() as options.
        • Irrelevant with foreach loop. The driver handles everything.
  • WORKS (well enough)

Opasnet-wiki server

CentOS 6.5: full support ended, but still get maintenance updates.

Key software, upgrade paths and related issues:

  • apache 2.2.15 -> 2.2.32 or 2.4.25
  • PHP 5.4.28 -> 5.6.x or 7.1.x
    • 5.5.9+ required by currently supported mediawiki releases
    • latest php also supported
    • incompatibilities likely to occur in 5.4->5.5, hence might as well upgrade to latest
  • MySQL 5.5.37 -> 5.5.56 or 5.7.x
  • MediaWiki 1.20.0 -> 1.27 (LTS) or 1.28.2 (latest)
    • need to check supported extensions
    • MyExtension.php no longer supported, need extension.json for registering extensions
    • Composer might be useful for managing PHP dependencies

RTools MediaWiki extension

  • RTools special page class (now named SpecialRtools)
    • Creates instance of rtools_client
    • separate config file, which is included by SpecialRTools instead of rtools_client for some reason
    • fixed: "mysql" to "mysqli"
    • fixed: "wfMsg" to "wfMessage->text()"
  • <rcode> tag
    • Detected by parser hooks
    • RToolsParser class
      • fixed: "wfMsg" to "wfMessage->text()"
      • fixed: class instead of global functions
        • create class instance in parser given by hook in order to retain variable values
      • fixed: added "wgOut->addModules( 'ext.RTools.parser' );" which enables the on-page UI
      • fixed: changed efRCodeMBInArray to a static function and changed reference in templates/inputs/checkbox.php
  • rtools_client
    • handles connections and job submissions etc. to server
    • xml-rpc library
      • communicates with rtools_server API
  • javascripts
    • script.aculo.us + prototype
      • UI
      • ajax
      • old versions, probably still acceptable
    • loaded by mediawiki resource loader
      • now set up in extension.json
  • wiki database
    • rtools_inputs
      • added schema update hook to create the necessary table if missing
  • Current issues:
    • Show code button does not work
    • Run page throws: Didn't receive 200 OK from remote server. (HTTP/1.0 500 Internal Server Error), which is a xml-rpc related error
    • Runs and shows outputs (oprint and cat), but code is not visible, embed also does not work
    • Input forms are slightly broken: conditional categories do not work, checkboxes crash the whole page
    • Rtools page does not update, shows error until refreshed once the run is over. Error on page: "jQuery is not defined", but it is loaded by mediawiki by default. Caused by some linking problem between the special page and mediawiki.
      • SpecialRTools-class problem with include of "template/index.php" in method "output" not scoping as expected. Including javascript dependencies separately resolves the first issue, but results in "mw not defined" indicating no access to mediawiki.
      • Special page includes javascript for both parser and special page, which does not happen on the current server. It is loaded as part of parser initialization on every page.
      • Works 1% of the time ...

Extensions

Our extensions

ExternalSourceBrowser: Einari

OpasnetBase: Einari, Juha

  • Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
  • wfMsg() to wfMessage()->text()
  • Issue: row counter does not always work
    • javascript autoloader does not appear to work on all included parts of the page
  • Seems to work otherwise

OpasnetBaseImport: Einari, Juha

  • Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
    • Parser stuff put in a new class
    • Added all classes to Autoload
    • Load configuration file at start of all class definition files
      • Added "global $wgArticlePath, $wgScript;" at config start to give access to mw globals in the new scope
    • Corrected extension documentation URL
  • wfMsg() to wfMessage()->text()
  • Current issues:
    • configuration variable $obImportUploadsPath becomes null or somehow
      • Changed config to use constants instead of global variables
  • Works

Table2Base: Juha, Einari

  • Uses OpasnetBaseImporters OpasnetBaseUpload class
  • Converted to extension registeration
  • Created Table2BaseParser-class
  • Writes to both OpasnetBase and Wiki database
    • Need to figure out wiki database string escapes, may be unnecessary with proper use of wrapper functions. Can't use mysqli_real_escape_string due to mediawiki wrapping.
      • Wikimedia\Rdbms\Database::insert handles escaping
    • Need to add database update hook
    • OpasnetBase queries: 'mysql_real_escape_string' to 'mysqli_real_escape_string' patching
  • Current issues:
    • No index for t2b uploads in wiki base may make queries slow (affects performance on every page with t2b)?
    • Displays data normally, displays errors in data, but does not upload.
      • Uploads most of the time, never shows the message: "Data updated successfully!" though.

OpasnetRater: Einari

Other Extensions

Currently installed extensions, still maintained unless specified otherwise. Almost all have been updated since last time. Nothing tested yet.

  • AuthorProtect
    • Git maintained
  • CategoryTree [1]
  • ConfirmEdit is shipped with MediaWiki
  • ContactPage [2]
    • Need to reconfigure local settings when upgrading
  • Contribution Scores [3]
  • Contributors [4]
  • DataValues [5]
    • DataValues Geo [6]
    • DataValues Time [7]
    • DataValues Validators [8]
    • DataValues Common [9]
    • DataValues Interfaces [10]
    • Still updated
  • Google Analytics Integration [11]
  • ImageMap is shipped with MediaWiki
  • Maps [12]
    • Very active development
    • Possible incompatibilities with RTools when upgraded
  • Math [13]
  • MsUpload [14]
    • Upload files from editor
      • Has this seen any use?
  • PageNotice [15]
  • SimpleCalendar [16]
  • User Merge and Delete [17]
    • Has MediaWiki 1.29+ listed on extension page, but probably works on 1.28
  • Validator [18]
  • Widgets [19]
    • Works

Obsolete or otherwise dead

Not maintained anymore, very old, security risk or integrated to MediaWiki.

  • Ajax Show Editors [20]
    • Archived
  • CheckUser [21]
    • Integrated functionality?
  • CreateBox [22]
    • Archived
    • Option: InputBox [23]
      • This is included with new installations of MediaWiki 1.21 and above
    • Used on Create_article
      • Any others?
  • DynamicPageList [24]
    • Very old version, might not work on latest MediaWiki and PHP
  • EmailAddressImage [25]
  • Google Co-op Extension [26] ?
    • Search functionality that was never used and is now obsolete?
  • MsLinks [27]
    • Used to display files on pages using "{{#l:Testfile.pdf}}"
      • Has this been used at all?
    • Very old version, likely broken
  • NewPageCSS [28]
    • Very old version
    • Security risk?
  • NiceCategoryList [29]
    • Security risk
  • Todo Tasks [30]
    • Not maintained since 2009, expect to break.

Not listed in Special:Version

Have to check local settings to see if these are installed.

  • AccessControl [31]
    • Maintained
  • ArticleComments [32]
    • Archived
  • CategoryStepper [33]
    • Unmaintained
  • CategorySuggest [34]
    • Maintained
  • Discussion [35]
    • Not maintained, download link dead.
  • DiscussionThreading [36]
    • Maintained
  • Header Footer [37]
    • Maintained
  • Lockdown [38]
    • Maintained
  • mfiles, mfiles2
  • NamespacePermissions [39]
    • Suggested: Lockdown, which is also present on the server
  • PageDisqus [40]
    • Still in beta, git has some recent commits ...
  • PageID
    • One of ours?
  • PdfHandler [41]
    • Shipped with current version of MediaWiki
  • Renameuser
    • Shipped with current version of MediaWiki
  • SelectCategory [42]
    • Maintained
  • SimpleAntiSpam [43]
    • Merged into MediaWiki
  • SpamBlacklist [44]
    • Maintained
  • StubManager [45]
    • Archived
  • Survey [46]
    • Unmaintained
    • Alternative if still in use: QuickSurveys [47]?
  • TitleBlacklist [48]
    • Maintained
  • WhoIsWatching [49]
    • Maintained
  • WhosOnlineAjax [50]
    • Archived
    • Alternative: WhosOnline [51]

See Also

Opasnet:Technical_documentation

Retrieved from "https://en.opasnet.org/index.php?title=Opasnet_update_2017&oldid=41197"