Opasnet update 2017: Difference between revisions
Line 13: | Line 13: | ||
****custom dump (pg_dump -Fc) + hard upgrade (perl utils/postgis_restore.pl in root directory) seems to work | ****custom dump (pg_dump -Fc) + hard upgrade (perl utils/postgis_restore.pl in root directory) seems to work | ||
*****readOGR fails for migrated data, different versions of gdal and rgdal produce different errors traceable to data types | *****readOGR fails for migrated data, different versions of gdal and rgdal produce different errors traceable to data types | ||
***database source name / connection string does not work | ***<s>database source name / connection string does not work</s> | ||
**Sorvi package has been updated | **Sorvi package has been updated | ||
***LoadData replaced by load_sorvi_data | ***LoadData replaced by load_sorvi_data |
Revision as of 12:27, 28 July 2017
This page contains notes produced while updating Opasnet server systems during the summer of 2017.
Remaining tasks
- CreateBox to InputBox conversion for page creation
- Redirect to MediaWiki:Create exists does not work
- Or rather it is a feature of CreatBox and will be dropped
- Default behaviour is to open Editor for existing page
- Also used on these pages
- Redirect to MediaWiki:Create exists does not work
- RTools map functionality and OpasnetUtilsExt
- Some datasets (first example on Opasnet map) involve a PostgreSQL/PostGIS database
- Datatypes make migration between versions difficult
- custom dump (pg_dump -Fc) + hard upgrade (perl utils/postgis_restore.pl in root directory) seems to work
- readOGR fails for migrated data, different versions of gdal and rgdal produce different errors traceable to data types
- custom dump (pg_dump -Fc) + hard upgrade (perl utils/postgis_restore.pl in root directory) seems to work
database source name / connection string does not work
- Datatypes make migration between versions difficult
- Sorvi package has been updated
- LoadData replaced by load_sorvi_data
- Need to install a non-master branch for the data used in the example: install_github("ropengov/sorvi", ref = "data-from-gh")
KML is not drawnPage produces "ReferenceError: jQuery is not defined", but map loads and gives message about adding KML, which suggests the relevant part works?Could probably use the mediawiki Maps extension instead and draw KML using thatVery hard to implement since the output (map definition by R-code on the RTools-server) would have to be parsed on the Wiki-server running the RTools-extension. Currently there is an escaping mechanism which allows raw html (including javascript) to be injected into RTools output-blocks. This might have been made harder by changes to MediaWiki ResourceLoader in newer versions (to make malicious cross-site-scripting more difficult) resulting in the jQuery error.
Could also be a versioning issue, as code on original page runs- KML file needs to be publicly hosted for Google Maps API, which was not the case for the test wiki.
- Some datasets (first example on Opasnet map) involve a PostgreSQL/PostGIS database
- Iron out minor bugs in RTools-wiki extension, t2b, OpasnetBase-viewer
- Update live servers
- Update OS
- Update software
- Update extensions
- Other notable issues
- Backup scripts
- Users and keys
- Server users and permissions
- SVN
- On Wiki server
- Contains source code of all software written by us
- Backup scripts
RTools server
CentOS 6.4: full support ended, but still get maintenance updates.
Key software, upgrade paths and related issues:
- apache 2.2.15 -> 2.2.32 or 2.4.25
- newer branch not particularly important
- PHP 5.3.3 -> 5.6.x or 7.1.x
- incompatibilities in mysql functions
- "mysql" -> "mysqli" (5.5)
- should upgrade to latest version
- incompatibilities in mysql functions
- MySQL 5.1 -> 5.7
- mongoDB 2.4 -> 3.2
- should be relatively easy as we are not using sharding etc.
- rtools_server PHP application
- starts on test server after minor changes
- further testing needed
OpasnetBase2
Runs on Apache as a PHP web application. Updating PHP and MongoDB causes a few incompatibilities and other issues with old code:
- MongoDB driver updates
- "Mongo" PHP library deprecated, new "MongoDB" PHP library
- Mongo-class no longer available
- Use MongoDB\Client instead
- Other classes changed to some extent
- cursor does not handle query options -> everything needs to be supplied to collection.find()
- Collections do not implement __get(), hence they cannot be chained like before. db->ident->subset->dat should be combined to db->{ident.subset.dat}.
- Mongo-class no longer available
- Use pecl to install driver
- Use composer to download library to be included in web app
- Actually, a lot of code (~1000 lines)
- "mysql" to "mysqli" patching
- some static functions may need new parameters e.g. OpasnetBaseActiveRecord::parse_qmarks needed a database link where one was not needed before (due to mysqli_real_escape_string requiring a database link parameter now)
- The MongoDB drivers (both old and new) overload __get() to give shortcuts to queries and other functionality, which makes debugging difficult.
- It might be useful to change them to use proper method calls instead.
- "insert" to "insertOne" and "ensureIndex" to "createIndex" patching
- "MongoDate" type changed to "MongoDB\BSON\UTCDateTime"
- Different methods: "date('Y-M-d H:i:s',$obj[$index->ident]->sec)" to "$obj[$index->ident]->toDateTime()->format('Y-M-d H:i:s')"
- Tracked issue with invalid times in $data to MySQL column data type issue (datetime instead of timestamp)
- MySQL table download_sessions contains the keys, chunk counters and download times of every download ever
- HUGE (3.5M rows), pretty much useless after download is complete. This should probably be cleaned regularly.
- MySQL table download_sessions contains the keys, chunk counters and download times of every download ever
- "mysql" to "mysqli" patching
- Current issues:
- Server throws no exceptions, but returns garbage when attempting to prepare uploads using unmodified OpasnetUtils.
- Using RCurl postForm() for header works.
- Upload, replace, append, download, include, exclude all work.
Return header seems fine
- Using RCurl postForm() for header works.
- test.php upload works
Read APIs return errorsERROR: Load by columns: record not found!MongoDB\Driver\Cursor and other driver classes methods are not exposed to PHP?- Cursor methods not implemented in the new PHP library -> have to do foreach loop for results and manually chunk them into files.
Reasonable to implementDone, performance might be worse however.
- Cursor methods not implemented in the new PHP library -> have to do foreach loop for results and manually chunk them into files.
Collection->find(...) executes the given query which means options such as batchSize have to be included in find() as options.- Irrelevant with foreach loop. The driver handles everything.
- Server throws no exceptions, but returns garbage when attempting to prepare uploads using unmodified OpasnetUtils.
- WORKS (well enough)
Opasnet-wiki server
CentOS 6.5: full support ended, but still get maintenance updates.
Key software, upgrade paths and related issues:
- apache 2.2.15 -> 2.2.32 or 2.4.25
- PHP 5.4.28 -> 5.6.x or 7.1.x
- 5.5.9+ required by currently supported mediawiki releases
- latest php also supported
- incompatibilities likely to occur in 5.4->5.5, hence might as well upgrade to latest
- MySQL 5.5.37 -> 5.5.56 or 5.7.x
- MediaWiki 1.20.0 -> 1.27 (LTS) or 1.28.2 (latest)
- need to check supported extensions
- MyExtension.php no longer supported, need extension.json for registering extensions
- Composer might be useful for managing PHP dependencies
RTools MediaWiki extension
- RTools special page class (now named SpecialRtools)
- Creates instance of rtools_client
- separate config file, which is included by SpecialRTools instead of rtools_client for some reason
- fixed: "mysql" to "mysqli"
- fixed: "wfMsg" to "wfMessage->text()"
- <rcode> tag
- Detected by parser hooks
- RToolsParser class
- fixed: "wfMsg" to "wfMessage->text()"
- fixed: class instead of global functions
- create class instance in parser given by hook in order to retain variable values
- fixed: added "wgOut->addModules( 'ext.RTools.parser' );" which enables the on-page UI
- fixed: changed efRCodeMBInArray to a static function and changed reference in templates/inputs/checkbox.php
- rtools_client
- handles connections and job submissions etc. to server
- xml-rpc library
- communicates with rtools_server API
- javascripts
- script.aculo.us + prototype
- UI
- ajax
- old versions, probably still acceptable
- loaded by mediawiki resource loader
- now set up in extension.json
- script.aculo.us + prototype
- wiki database
- rtools_inputs
- added schema update hook to create the necessary table if missing
- rtools_inputs
- Current issues:
Show code button does not workRun page throws: Didn't receive 200 OK from remote server. (HTTP/1.0 500 Internal Server Error), which is a xml-rpc related errorRuns and shows outputs (oprint and cat), but code is not visible, embed also does not workInput forms are slightly broken: conditional categories do not work, checkboxes crash the whole page- Rtools page does not update, shows error until refreshed once the run is over. Error on page: "jQuery is not defined", but it is loaded by mediawiki by default.
Caused by some linking problem between the special page and mediawiki.SpecialRTools-class problem with include of "template/index.php" in method "output" not scoping as expected. Including javascript dependencies separately resolves the first issue, but results in "mw not defined" indicating no access to mediawiki.Special page includes javascript for both parser and special page, which does not happen on the current server. It is loaded as part of parser initialization on every page.- Works 1% of the time ...
Extensions
Our extensions
ExternalSourceBrowser: Einari
- Started in 2010, looks incomplete
- Special:ExternalSourceBrowser - missing config
- Still loaded for some reason
OpasnetBase: Einari, Juha
- Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
- wfMsg() to wfMessage()->text()
- Issue: row counter does not always work
- javascript autoloader does not appear to work on all included parts of the page
- Seems to work otherwise
OpasnetBaseImport: Einari, Juha
- Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
- Parser stuff put in a new class
- Added all classes to Autoload
- Load configuration file at start of all class definition files
- Added "global $wgArticlePath, $wgScript;" at config start to give access to mw globals in the new scope
- Corrected extension documentation URL
- wfMsg() to wfMessage()->text()
- Current issues:
configuration variable $obImportUploadsPath becomes null or somehow- Changed config to use constants instead of global variables
- Works
Table2Base: Juha, Einari
- Uses OpasnetBaseImporters OpasnetBaseUpload class
- Converted to extension registeration
- Created Table2BaseParser-class
- Writes to both OpasnetBase and Wiki database
- Need to figure out wiki database string escapes, may be unnecessary with proper use of wrapper functions. Can't use mysqli_real_escape_string due to mediawiki wrapping.
- Wikimedia\Rdbms\Database::insert handles escaping
- Need to add database update hook
- OpasnetBase queries: 'mysql_real_escape_string' to 'mysqli_real_escape_string' patching
- Need to figure out wiki database string escapes, may be unnecessary with proper use of wrapper functions. Can't use mysqli_real_escape_string due to mediawiki wrapping.
- Current issues:
- No index for t2b uploads in wiki base may make queries slow (affects performance on every page with t2b)?
- Displays data normally, displays errors in data,
but does not upload.- Uploads most of the time, never shows the message: "Data updated successfully!" though.
OpasnetRater: Einari
- Rater
- Has seen very limited use
- Low priority
PageID
- Not ours actually. Created by Jim R. Wilson in 2008.
- Implements {{PAGEID}}
- Which is a core variable in MediaWiki 1.20+
Other Extensions
Currently installed extensions, still maintained unless specified otherwise. Almost all have been updated since last time. Nothing tested yet.
- AuthorProtect
- Git maintained
- CategoryTree [1]
- CheckUser [2]
- ConfirmEdit is shipped with MediaWiki
- ReCaptcha should be replaced with ReCaptcha(NoCaptcha)
- ContactPage [3]
- Need to reconfigure local settings when upgrading
- Contribution Scores [4]
- Contributors [5]
- DataValues [6]
- DynamicPageList [12]
Very old version, might not work on latest MediaWiki and PHP
- Google Analytics Integration [13]
- ImageMap is shipped with MediaWiki
- Maps [14]
- Very active development
- Possible incompatibilities with RTools when upgraded
- Opasnet_map
- Need to check compatibility with OpasnetUtilsExt and the RTools extension
- Math [15]
- MsLinks [16]
- Used to display files on pages using "{{#l:Testfile.pdf}}"
Has this been used at all?
Very old version, likely broken
- Used to display files on pages using "{{#l:Testfile.pdf}}"
- MsUpload [17]
- Upload files from editor
- Has this seen any use?
- Upload files from editor
- NewPageCSS [18]
Very old versionSecurity risk?- Install by copying source from wikipage might be considered hazardous, however there are no issues with code.
- PageNotice [19]
- SimpleCalendar
- User Merge and Delete [20]
- Has MediaWiki 1.29+ listed on extension page, but probably works on 1.28
- Validator [21]
- Widgets [22]
- Works
Obsolete or otherwise dead
Not maintained anymore, very old, security risk or integrated to MediaWiki.
- Ajax Show Editors [23]
- Archived
- CreateBox [24]
- Archived
- Option: InputBox [25]
- This is included with new installations of MediaWiki 1.21 and above
- Used on Create_article
- Any others?
- EmailAddressImage [26]
- access to extensions folder is restricted in new default installation, which breaks this extension
- might work in Opasnet
- access to extensions folder is restricted in new default installation, which breaks this extension
- Google Co-op Extension [27] ?
- Search functionality that was never used and is now obsolete?
- NiceCategoryList [28]
- Security risk
- Implements ncl-tag e.g. <ncl>Category:Traffic</ncl>
- DynamicPageList dpl-tag does the same thing with <dpl>category = Traffic</dpl>, is in active development, does not have known vulnerabilities and seems to be in wider use in Opasnet as well.
- Todo Tasks [29]
- Not maintained since 2009, expect to break.
Not listed in Special:Version
Have to check local settings to see if these are installed. Not in use.
- AccessControl [30]
- Maintained
- ArticleComments [31]
- Archived
- CategoryStepper [32]
- Unmaintained
- CategorySuggest [33]
- Maintained
- Discussion [34]
- Not maintained, download link dead.
- DiscussionThreading [35]
- Maintained
- Header Footer [36]
- Maintained
- Lockdown [37]
- Maintained
- mfiles, mfiles2
- NamespacePermissions [38]
- Suggested: Lockdown, which is also present on the server
- PageDisqus [39]
- Still in beta, git has some recent commits ...
- PdfHandler [40]
- Shipped with current version of MediaWiki
- Renameuser
- Shipped with current version of MediaWiki
- SelectCategory [41]
- Maintained
- SimpleAntiSpam [42]
- Merged into MediaWiki
- SpamBlacklist [43]
- Maintained
- StubManager [44]
- Archived
- Survey [45]
- Unmaintained
- Alternative if still in use: QuickSurveys [46]?
- TitleBlacklist [47]
- Maintained
- WhoIsWatching [48]
- Maintained
- WhosOnlineAjax [49]
- Archived
- Alternative: WhosOnline [50]