Wikibase/Wikimedia verification GUI thread/tracking

[kyrieb-ekat]

Warning

I'm going to use this discussion thread as a way to track my progress, choices, and direction in getting this branch of verification option testing going, with the other "branch" being undertaken by Mai Lyn using Django/Postgres.

My initial experimentation was using Wikibase Cloud, which is a smaller and much simpler version of Wikibase and Wikimedia tools proper. I'm going to now switch over to setting up Wikimedia and then Wikibase proper, and probably implement the commonly used Semantic Mediawiki approach, which has basic and additional setups for getting a semantic and organized wiki and database up and running. If you're curious about Semantic Mediawiki, you can see more about them and other projects which use them here. Special shoutout to CODECS which seems especially rad (and is manuscripts!!!).

So, this means getting a webserver up and going. I'm going to select things we're already using in the lab, so nginx etc. Fortunately they have a very comprehensive docker file, so I'm otherwise just going to build from the box, as discussed.

Once this is up and going, I'm going to export all data currently in the Wikibase Cloud to the separate what I'm going to call "local" instance of Wikibase (which will be the umbrella term for the Wikimedia > Wikibase > Semantic MediaWiki conglomeration this setup will be).

From there I'm going to look into:

• the various user hierarchy's and permission which can be employed, as well as t
• he special pages that will need to be created. In this case, it's possible to produce a page which will be something like Verification Portal, which will on open automatically fetch a list of items labelled "Needs Review" via the SPARQL endpoint/backend native to the Wikibase setup (hooray!).

Wikibase (etc) also comes with some built in options for using Postgres etc, which if needed I'll also opt into given that's part of our current plan/what the lab already uses.

From there, attached and extended from the Verification Portal (which will only be able to be accessed with a special permission/user type), when a submission is reviewed and then submitted there will be some kind of "review the entry" > "Submit" > "are you sure?" etc. and then maybe some kind of field for verifier notes.

First swing at this will be the agreed upon test to only address adding a new label to an existing instrument.

Note

Using this package and directions, building from Docker. https://github.com/wmde/wikibase-release-pipeline

[kyrieb-ekat]

List of likely Extensions for Wikibase Implementation

Note

Core MediaWiki + Wikibase already gives you:
Custom user groups (beyond the standard Admin/User)
Granular permissions per group
Page protection levels

User and Page Management:

Extension:UserMerge - manage user accounts
Extension:UserPageEditProtection - control who can edit what
Extension:TitleBlacklist - prevent creation of certain pages

Preventing Bulk Editing

Extension:AbuseFilter - create rules to block bulk operations
Rate limiting built into MediaWiki core
Custom user rights - you can create a "verifier" group with limited permissions

Special Pages for Verification Workflow

Extension:Page Forms - create structured forms for data entry
Extension:Admin Links - custom admin/workflow interfaces
Custom special pages - you can write your own for specific verification tasks

Keeping in mind moving forward:
https://www.mediawiki.org/wiki/Extension:Page_Schemas#Usage

[kyrieb-ekat]

A set of scripts have been set up to work with the API to automatically import everything held in the UMILex cloud wikibase into this larger local wikimedia wikibase. A special page is being implemented for loading in items from a sparql query and for a verifier to interact with it.

two things occurred today; 1) addressing the special page approach and 2) discovering something is wonky with elastic search, and while properties complete in an automatic import (and imported statement will correctly link "Instance of" to the wikibase property P2 = instance of), the entities for whatever reason aren't: Q519 is showing up as a string in the field, and not linking to the existing wikibase property Q519 "musical instrument". Further weirdness with elasticsearch is confirmed with the searhbar, which isn't autofilling and also isn't returning results. Ugh. Whee.

In other news, the special page looks great. Screenshot below. Not hooked up yet.

Continuing to address the issue with statement imports, which are technically excellent, but stymied by the elasticsearch/cirrusSearch issues. Both exist in docker and are healthy, and so from here am going to try editing localsettings.php etc. and see what happens from there.

[kyrieb-ekat]

the issue is with elastic search and cirrussearch which, in the foundation/clone from the more recent wikibase instance, did not have them successfully/wholly installed; editing and indexing the items anew.

once fixed, will retrace steps on re-importing another test batch of items (or retest that same initial batch) to check for automatic mappings and test search functionality. this is apparently a known issue inthe wikibase-docker community, which I've learned after the fact (whups), but is helpful for looking forward at other extensions and functionalities the base will need.

[kyrieb-ekat]

A good chunk of this might be because I'm relatively new to elasticsearch and docker wrangling, as a heads up to the aficionados who may or may not be perusing this. 😁

WikiBase Elasticsearch Configuration Fix

🐞 Problem Summary

After attempting to configure Elasticsearch search functionality in a WikiBase installation, two critical issues occurred:

1. Website broken: PHP warnings caused complete loss of CSS/formatting, making the site unusable
2. Search non-functional: Elasticsearch connection failures prevented search, autocomplete, and entity matching

Important Context: WikiBase Default Configuration ⚠️

The WikiBase installation from the official tarball/clone does not include Elasticsearch configuration by default. The search functionality was completely missing from the initial setup, which is why manual configuration was necessary. The original LocalSettings.php contained no CirrusSearch or Elasticsearch references - all search-related configuration had to be added manually to enable this functionality.

🔍 Root Causes Identified

1. PHP Variable Name Mismatch

• Issue: Defined $elasticsearchHost (lowercase 's') but referenced $elasticSearchHost (uppercase 'S')
• Impact: Undefined variable warnings output before HTML headers, breaking page rendering
• Location: Line 148 in LocalSettings.php

2. Elasticsearch URL Configuration Issues

• Issue: MediaWiki's CirrusSearch extension couldn't properly parse the Elasticsearch connection string
• Impact: "Malformed URL" errors preventing index creation and search functionality

🏁 Solution Steps

Step 1: Fix PHP Variable Name Mismatch

Problem: Variable name inconsistency causing undefined variable warnings

Context: The unused variable $elasticsearchHost = 'elasticsearch:9200'; was a remnant from initial configuration attempts. Since we ultimately used the explicit IP-based configuration, this variable became irrelevant and was safely removed.

Original problematic code:

$elasticsearchHost = 'elasticsearch';  // lowercase 's' - unused variable
$wgCirrusSearchClusters = [
    'default' => [ $elasticSearchHost ]  // uppercase 'S' - UNDEFINED!
];

Note

The default $elasticsearchHost = 'elasticsearch'; is all that comes with this version of wikibase "out of the box", and ended up being rendered invalid. Just a warning, for future editors, etc.

Fix: Use explicit IP configuration and remove unused variables

Step 2: Use Explicit IP Address Configuration

Problem: Hostname-based connection strings causing "Malformed URL" errors

Find Elasticsearch container IP:

docker inspect wbs-deploy-elasticsearch-1 | grep IPAddress

Final working configuration in LocalSettings.php:

wfLoadExtension( 'Elastica' );
wfLoadExtension( 'CirrusSearch' );

$wgSearchType = 'CirrusSearch';
$wgCirrusSearchClusters = [
    'default' => [
        [
            'host' => '172.18.0.4',  // Use actual container IP
            'port' => 9200,
            'transport' => 'Http'
        ]
    ]
];
$wgCirrusSearchDefaultCluster = 'default';
$wgCirrusSearchIndexBaseName = 'wikibase';

Step 3: Proper Maintenance Script Sequence

Important: Run scripts in correct order for proper index setup

# 1. First: Update search index configuration
php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --startOver

# 2. Then: Force index population
php extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip

Key Troubleshooting Techniques Used

1. Systematic Error Analysis

• Identified PHP warnings as root cause of page rendering issues
• Distinguished between site functionality vs. search functionality problems

2. Configuration Format Testing

• Tried multiple Elasticsearch connection formats:
  • Simple string: 'elasticsearch:9200'
  • Array format: ['elasticsearch:9200']
  • Object format with explicit host/port (final solution)

3. Container Network Debugging

• Verified Elasticsearch service health: curl elasticsearch:9200/_cluster/health
• Found container IP address for direct connection
• Tested connectivity from within WikiBase container

4. Incremental Validation

• Fixed site rendering first (PHP warnings)
• Then addressed search functionality separately
• Validated each step before proceeding

What Each Configuration Element Does

wfLoadExtension( 'Elastica' );           // Loads Elasticsearch PHP client
wfLoadExtension( 'CirrusSearch' );       // Loads MediaWiki search extension

$wgSearchType = 'CirrusSearch';          // Sets CirrusSearch as search backend

$wgCirrusSearchClusters = [              // Defines Elasticsearch cluster connections
    'default' => [                       // Default cluster name
        [
            'host' => '172.18.0.4',      // Elasticsearch container IP
            'port' => 9200,              // Standard Elasticsearch port
            'transport' => 'Http'        // Connection protocol
        ]
    ]
];

$wgCirrusSearchDefaultCluster = 'default';      // Which cluster to use by default
$wgCirrusSearchIndexBaseName = 'wikibase';      // Prefix for index names

Results Achieved

✅ Website Functionality Restored

• Page rendering and CSS loading works correctly
• No more PHP warnings breaking the interface

✅ Full Search Functionality

• Autocomplete suggestions as you type
• Search results for existing content
• Entity matching for imported items
• Proper indexing of new content

✅ Elasticsearch Integration

• Successful connection to Elasticsearch cluster
• Proper index creation and population
• Real-time search capabilities

Prevention Tips for my future self/others jumping into wikibase from scratch (not a prebuilt version)

1. Always use consistent variable naming in PHP configuration
2. Test configuration changes incrementally rather than multiple changes at once
3. Use explicit IP addresses for Docker container connections when hostname resolution fails
4. Run maintenance scripts in proper sequence for index setup
5. Verify Elasticsearch health before configuring MediaWiki integration

Additional Notes

• The explicit array format with host/port keys was required for proper URL parsing by the Elastica library and took extensive debugging
• Using container IP addresses instead of hostnames resolved DNS resolution issues in this Docker environment
• The --startOver flag ensures clean index recreation when troubleshooting

[kyrieb-ekat]

Continuing to work with Docker on getting settings corrected and connected; entity matching inside of a statement block (wherein you add a statement > add a property like P2/"Instance of" > add an entity to designate the place in the property, "Musical Instrument" Q519). For general SPARQL queries this isn't a problem: you can amend your search to return a string instead of the entity number itself, so it'll look for all occurrences of the string Q519 inside of an attribute block instead of all linked occurrences of the entity. I do want this fixed, but there's the short term solution/workaround for now. Otherwise the static/special pages for verifiers to land, retrieve, and work are up, and I'm refining their unique privileges today.

I might end up tearing down this docker instance and relaunching from a different image run by the Wikimedia Deutschland folks. Reading a lot about some of their procedures and previous issues encountered with this; a significant issue is that their most recent stable release isn't so stable, and includes a deprecated and no longer supported version of jquery.ui which is what is actually causing the search issues, which compounds the indexing issues due to the structure of the LocalSettings.php file in what gets executed first. I'm continuing to work on that.

[kyrieb-ekat]

Summary: ReviewDashboard Extension - SPARQL Integration Complete! 🎉

Successfully integrated a fully functional ReviewDashboard extension with live SPARQL queries to display items needing review in Wikibase.

Stumbling points:

1. Docker Container Management

• Resolved wikibase container startup/restart issues
• Fixed missing WDQS (Wikibase Query Service) containers
• Stabilized multi-service Docker environment (ugh)

2. SPARQL Query Development

• Built working SPARQL query: ?item wdt:P2 "Musical Instrument"
• Debugged WDQS indexing pipeline (nuclear reset of SPARQL data) (got screwed up from elastic search indexing previously)
• Verified query returns correct results (Q106, Q214, Q534)

3. CORS Security Resolution

• CORS policy blocking SPARQL requests
• Attempted multiple solutions (direct ports, headers, different endpoints)
• Final solution: PHP proxy bypassing CORS entirely (sparql-proxy.php) (temp fix)

4. API Integration

• Established working MediaWiki API calls for item creation
• Successfully created test items via API with P2 statements
• Pattern established for bulk operations (ensuing bulk attribute and statement loading if needed)

Final Architecture

• Frontend: ReviewDashboard extension with JavaScript interface
• Backend: PHP proxy making internal Docker network calls
• Data Flow: Browser → Wikibase → PHP Proxy → WDQS → SPARQL results (users won't see this remotely)
• No CORS issues: Server-to-server communication via Docker internal network

Current Functionality

✅ Load items needing review (SPARQL query working)
✅ Display in clean interface (3 items: Superbone, Test Item, musical bow)
✅ View/Edit/Mark Reviewed buttons (ready for workflow)
✅Added label functionality for crowdsourcing instrument names: verifiers can add, view, and edit.

Key Technical Learnings

• Docker service orchestration for Wikibase ecosystem
• SPARQL query debugging and WDQS data pipeline
• CORS security in browser-to-service communication
• MediaWiki extension development and API integration

Bottom line: Core verification pipeline infrastructure is now functional and ready for instrument name collection workflow! 🎵

And I've learned how to create a horrifying combination of javascript and wikitext. It's weird.

[kyrieb-ekat]

Final update! 🫡 🎉 🐳

The Final Dashboard

Items viewed on the Review Dashboard are items which have not been reviewed. Labels can be added in languages within the dropdown (only grabbed a few for testing). If a label in that language already exists, the user will be informed and asked if they consent to their label being added as an alias. Once an item has had edits made to it, it automatically gains another internal flag—needs review—which would then be viewable by a reviewer on another page to verify the information. The reviewer then clicked Approve Disapprove or Edit, which progressed the item along the pipeline for final approval by the senior reviewer, who would apply a final Verified tag, which would clear the previous entity IDs affiliated with the item. The item would then be ready for submission to Wikidata.

Post-Docker Reset: Reconstructing the ReviewDashboard Pipeline

After a fresh Docker environment reset, the Wikibase ReviewDashboard extension was not functioning, primarily due to network communication issues between Docker containers and mixed content errors when accessing the dashboard via HTTPS. This document outlines the systematic steps taken to diagnose and resolve these issues.

Problem Overview:

Network Resolution : The wikibase.local and query.wikibase.local hostnames were not correctly routing to their respective Docker services.
Internal SPARQL Proxy Communication : The PHP proxy (running inside the Wikibase container) was unable to connect to the WDQS SPARQL endpoint; CORS errors and issues.
Mixed Content Errors : The browser, accessing the main Wikibase via HTTPS, blocked insecure HTTP requests made by the ReviewDashboard's JavaScript to the SPARQL proxy and Wikibase API.

Solution Steps Taken:

Verified Docker Container Status (docker ps):
Confirmed all core Wikibase, WDQS, database, and Traefik containers were Up and healthy. This established a baseline of running services.

Configured Host Machine's hosts File:
Ensured 127.0.0.1 wikibase.local and 127.0.0.1 query.wikibase.local entries were present in /etc/hosts (or equivalent) to direct these hostnames to the local machine where Traefik was listening.

Configured Traefik Routing in docker-compose.yml:

• Set .env variables: Confirmed WIKIBASE_PUBLIC_HOST=wikibase.local and WDQS_PUBLIC_HOST=query.wikibase.local in the .env file.
• Removed direct ports mapping from wikibase service: Traefik now handles external exposure (e.g., 8080:80 was removed).
• Added Traefik labels to wikibase service: Configured Traefik to route https://${WIKIBASE_PUBLIC_HOST} to the wikibase container's internal port 80.
• Added Traefik labels to wdqs-frontend service: Configured Traefik to route https://${WDQS_PUBLIC_HOST} to the wdqs-frontend container's internal port 80.
  - subnote: CORS, why 😭
• Corrected wdqs-frontend labels indentation: Fixed a YAML syntax error that was preventing wdqs-frontend's Traefik labels from being correctly parsed.

Adjusted Internal SPARQL Endpoint URL in sparql-proxy.php:

• Modified extensions/ReviewDashboard/api/sparql-proxy.php to use the internal Docker service name and port for the WDQS endpoint: http://wdqs:9999/bigdata/namespace/wdq/sparql. This ensures direct, efficient container-to-container communication.

Updated Frontend URLs for HTTPS in ext.reviewdashboard.js:

• Changed all instances of http://wikibase.local to https://wikibase.local within extensions/ReviewDashboard/modules/ext.reviewdashboard.js for all AJAX/Fetch requests (SPARQL proxy, Wikibase API token, wbgetentities, wbsetaliases, wbsetlabel).
  • This resolved the critical "Mixed Content" security errors reported by the browser.
• Added missing event listener: Bound confirmAddLabel to its button click.

Rebuilt and Restarted Docker Services:

• Executed docker-compose down followed by docker-compose up -d --build (or docker-compose up -d --force-recreate ) to apply all docker-compose.yml and extension code changes.
• Performed hard browser cache clears (Ctrl+Shift+R / Cmd+Shift+R) during testing to ensure updated JavaScript was loaded.

Outcome:

The Wikibase ReviewDashboard is now fully operational. The wikibase.local and query.wikibase.local hostnames correctly resolve via Traefik to their HTTPS endpoints, the PHP SPARQL proxy successfully fetches data from the internal WDQS service, and all frontend JavaScript calls are made securely over HTTPS, eliminating mixed content issues.

Notes in reflection

Wikibase is highly versatile and very flexible in what you can set up, however it requires some investment of time in finding all appropriate documentation and the updating of small dependencies. It's highly versatile but requires extensive customization, which can be highly advantageous for personalizing how we want to store and query data, and utilizes SQL for its basic database organization, and allows for SPARQL querying on the front end of the built-in SPARQL endpoint.

The extension methodology is, once you figure out where the documentation is and how everything connect, exceptionally powerful. I was able to put up several special pages which run on these SPARQL queries, with items automatically updating as a false user-reviewer-editor progressed editing it. So, once it's up and running, very easy to use for both the user and the maintainer. Getting there, however, is an adventure, and requires some notable immersion in Docker and its quirks to then get into and work with Wikibase's.