Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions Docker/SettingsTemplate.php
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,13 @@
$wgDebugComments = false;
}

// MediaWiki routes no log channel anywhere by default, so every diagnostic NeoWiki emits would reach
// nobody: a store refusing a page, a store name it will not accept, a projection failing on save.
// Container stderr is where a Docker install's logs are read. No level floor, in either mode: a rebuild
// records a skipped page at info, and that is a store left stale. Volume is bounded by the log driver's
// rotation in docker-compose.yml rather than by dropping entries here.
$wgDebugLogGroups['NeoWiki'] = 'php://stderr';

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

Expand Down
7 changes: 7 additions & 0 deletions Docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@ services:
mediawiki:
image: ghcr.io/professionalwiki/neowiki:latest
restart: unless-stopped
# LocalSettings routes the NeoWiki log channel here, and a stack whose EDM projection has no Mapping
# page reports a failure on every save. Rotate, so a long-running stack cannot fill the host with it.
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
ports:
- "${MW_SERVER_PORT:-8484}:80"
volumes:
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,9 @@ wfLoadExtension( 'SomeExtension' );
$wgDebugLogGroups['NeoWiki'] = '/tmp/neowiki-debug.log';
```

The stack already routes the `NeoWiki` channel ([Logging](docs/operations/installation.md#logging)), so setting
`$wgDebugLogGroups['NeoWiki']` here replaces that destination rather than adding to it.

### Try-it-out and server deployment

For the prebuilt try-it-out stack or server deployment with Caddy, see
Expand Down
8 changes: 3 additions & 5 deletions docs/extending/extending.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,11 +157,9 @@ class MyGraphDatabasePlugin implements GraphDatabasePlugin {
Register with `NeoWikiRegistrar::addGraphDatabasePlugin( $name, $plugin )`. Example:
[`src/RedHerbGraphDatabasePlugin.php`](https://github.com/ProfessionalWiki/NeoWiki/blob/master/tests/RedHerb/src/RedHerbGraphDatabasePlugin.php).

The name is what [`--store`](../operations/maintenance.md#rebuilding-one-store) addresses, and what a rebuild files
its run records under. Pick a stable one and namespace it to your extension. A name is refused with a warning on the
`NeoWiki` channel when another backend already holds it, when it is `neo4j` in any casing — reserved for the bundled
Neo4j backend — or when it is longer than 255 bytes, which is all a run record can hold. A refused backend receives
no page changes and cannot be rebuilt.
The name is what [`--store`](../operations/maintenance.md#rebuilding-one-store) addresses. Pick a stable one and
namespace it to your extension. A refused name drops the plugin, which then projects nothing and cannot be rebuilt;
the [`NeoWiki` log channel](../operations/installation.md#logging) says why and what to change.

`savePage` hands you the page with all of its Subjects and the Page Properties contributed by every
`PagePropertyProvider`, and runs for every revision, so subject edits, undeletions and page moves all reach you as a
Expand Down
26 changes: 22 additions & 4 deletions docs/operations/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,24 @@ These are the settings you are most likely to change. For the full list with des
| `$wgNeoWikiSparqlStores` | SPARQL 1.1 graph stores to keep in sync and query, e.g. QLever | `[]` | No |
| `$wgNeoWikiAutoRebuildOnMappingChange` | Rebuilds every store holding a Mapping's projection when that Mapping changes | `false` | No |

## Logging

NeoWiki logs on the `NeoWiki` channel: a graph store failing on save, the pages a rebuild could not project, a store
entry or name it will not accept, and RDF a projection had to drop. Below `warning` it also records rebuild decisions
and denied page reads. MediaWiki routes no channel anywhere by default, so none of it reaches you until you route it:

```php
$wgDebugLogGroups['NeoWiki'] = '/var/log/mediawiki/neowiki.log';
```

Add a `level` to drop everything below it:

```php
$wgDebugLogGroups['NeoWiki'] = [ 'destination' => '/var/log/mediawiki/neowiki.log', 'level' => 'warning' ];
```

The [Docker install](#method-a-docker) routes the channel to container stderr. `make logs` tails it.

## User rights

`neowiki-admin` allows viewing and rebuilding the wiki's graph stores, through
Expand Down Expand Up @@ -246,13 +264,13 @@ $wgNeoWikiSparqlStores = [
];
```

A store entry whose `updateUrl` is missing or empty is skipped with a warning rather than failing the wiki.

Each store's `name` identifies it when [rebuilding one store](maintenance.md#rebuilding-one-store), so no two entries
may share one, and none may be `neo4j` in any casing — reserved for the bundled Neo4j backend. Since the name defaults
to the projection, two entries holding the same projection — mirroring it to a second endpoint, say — collide until
one of them sets an explicit `name`. An entry whose name cannot identify it is skipped with a warning, so its store
receives no page changes.
one of them sets an explicit `name`.

An entry NeoWiki cannot use is skipped rather than failing the wiki; the [`NeoWiki` log channel](#logging) says which
entry and why.

### Oxigraph

Expand Down
17 changes: 7 additions & 10 deletions docs/operations/maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,12 +55,12 @@ A run that finished but left individual pages behind has nothing to continue: re

Pass `--batch-size` to change how many pages are projected between recordings; it defaults to 200.

A page the store rejects is counted and the rebuild carries on; the `NeoWiki` channel says which pages failed and why.
A store that stops answering partway is another matter: when a whole batch fails the rebuild reopens the store, and
ends the run for `--resume` to retry that batch only if it cannot. A store that still opens means its pages were at
fault, so they are counted and the walk goes on past them. The script exits non-zero whenever a store was left out of
sync, which covers both a failed run and one that finished having left pages behind. Only the first has anything to
resume; rebuild the store for the second.
A page the store rejects is counted and the rebuild carries on; the [`NeoWiki` log channel](installation.md#logging)
says which pages failed and why. A store that stops answering partway is another matter: when a whole batch fails the
rebuild reopens the store, and ends the run for `--resume` to retry that batch only if it cannot. A store that still
opens means its pages were at fault, so they are counted and the walk goes on past them. The script exits non-zero
whenever a store was left out of sync, which covers both a failed run and one that finished having left pages behind.
Only the first has anything to resume; rebuild the store for the second.

A rebuild killed outright — `kill -9`, or the machine going down — leaves its run recorded as still going, and every
later rebuild of that store refuses to start while it is. So does a background rebuild whose first batch never
Expand Down Expand Up @@ -105,14 +105,11 @@ rebuild somebody started by hand is left to finish rather than restarted; the st
## What happens during a Neo4j outage

- **Editing pages works.** Edits, deletions and undeletions all commit. NeoWiki logs the projection failure on the
`NeoWiki` channel.
[`NeoWiki` channel](installation.md#logging).
- **Editing and displaying Subjects fails**, along with queries and anything else that reads the graph.

Once Neo4j is back, [rebuild the graph](#rebuilding-the-graph): it repairs both a failed save and a failed delete.

Route the `NeoWiki` log channel somewhere you read. On a default MediaWiki install it goes nowhere, and it carries
both the outage and the pages a rebuild could not reconcile.

## Backups

Back up the MediaWiki database as usual; it holds the canonical data. Neo4j and any SPARQL store need no backup — they
Expand Down
3 changes: 2 additions & 1 deletion docs/rdf/ontology-mapping.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,8 @@ to an IRI containing an IRIREF-illegal character (`< > " { } | ^ \` backtick, sp

The same checks re-run at **projection time**: a class, predicate, datatype, or prefix that does not re-expand
safely is dropped, an unusable node takes everything below it with it, an invalid language tag falls back to a
plain literal, and each is logged. The projection degrades rather than aborting the export.
plain literal, and each is logged on the [`NeoWiki` channel](../operations/installation.md#logging). The projection
degrades rather than aborting the export.

## What gets emitted

Expand Down
2 changes: 1 addition & 1 deletion docs/rdf/rdf-export.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ an `xsd:anyURI` literal, so nothing is lost. The other value types map to `xsd`
registered mapper — including an unregistered type — is omitted from the projection.

A Subject whose Schema cannot be loaded (for example, its Schema page was deleted) is omitted from the projection; a
warning is logged for each.
warning is logged for each on the [`NeoWiki` channel](../operations/installation.md#logging).

## Endpoint

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ private function logProjectionFailure( string $operation, PageId $pageId, Except
'NeoWiki failed to ' . $operation . ' page ' . $pageId->id . ' in graph backend '
. $this->plugin::class . '. The triggering operation was not aborted, but this backend is '
. 'now out of sync for that page. Run the RebuildGraphDatabases maintenance script to '
. 'reconcile it. Underlying error: ' . $e->getMessage(),
. 'reconcile it. Underlying error: ' . BackendFailureMessage::withoutCredentials( $e->getMessage() ),
[ 'exception' => $e ]
);
}
Expand Down
7 changes: 6 additions & 1 deletion src/EntryPoints/REST/ExportPageRdfApi.php
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
use MediaWiki\Rest\Response;
use MediaWiki\Rest\SimpleHandler;
use ProfessionalWiki\NeoWiki\Application\Rdf\RdfPageProjector;
use ProfessionalWiki\NeoWiki\Domain\GraphDatabase\BackendFailureMessage;
use ProfessionalWiki\NeoWiki\Domain\Page\PageId;
use ProfessionalWiki\NeoWiki\NeoWikiExtension;
use Wikimedia\ParamValidator\ParamValidator;
Expand Down Expand Up @@ -74,7 +75,11 @@ public function run( int $pageId ): Response {
} catch ( Exception $e ) {
LoggerFactory::getInstance( 'NeoWiki' )->error(
'NeoWiki could not export page {pageId} as RDF: {message}',
[ 'pageId' => $pageId, 'message' => $e->getMessage(), 'exception' => $e ]
[
'pageId' => $pageId,
'message' => BackendFailureMessage::withoutCredentials( $e->getMessage() ),
'exception' => $e,
]
);

return $this->noDataResponse( $pageId );
Expand Down
7 changes: 6 additions & 1 deletion src/FailureIsolatingPagePropertiesSource.php
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
use Exception;
use MediaWiki\Revision\RevisionRecord;
use MediaWiki\User\UserIdentity;
use ProfessionalWiki\NeoWiki\Domain\GraphDatabase\BackendFailureMessage;
use ProfessionalWiki\NeoWiki\Domain\Page\PageProperties;
use Psr\Log\LoggerInterface;
use Wikimedia\Rdbms\DBError;
Expand Down Expand Up @@ -41,7 +42,11 @@ public function getPagePropertiesFor( RevisionRecord $revision, ?UserIdentity $u
$this->logger->error(
'NeoWiki did not project page {pageId} because its page properties could not be built: '
. '{message}. The graph is out of sync for that page until the cause is resolved.',
[ 'pageId' => $revision->getPageId(), 'message' => $e->getMessage(), 'exception' => $e ]
[
'pageId' => $revision->getPageId(),
'message' => BackendFailureMessage::withoutCredentials( $e->getMessage() ),
'exception' => $e,
]
);

return null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,24 @@ public function testFailingSaveIsSwallowedAndLoggedWithAnActionableMessage(): vo
$this->assertStringContainsString( 'triggering operation', $message, 'stays operation-neutral' );
}

/**
* The Neo4j client reports what it could not reach by quoting the bolt URI it tried, userinfo and
* all, so relaying its message verbatim writes the store's password to wherever the NeoWiki channel
* is routed — on every save, for as long as the backend is down.
*/
public function testTheBackendPasswordIsKeptOutOfTheLoggedFailure(): void {
$plugin = new ThrowingGraphDatabasePlugin(
'Cannot connect to any server on alias: default with Uris: '
. "('bolt://neo4j:s3cr3t@neo.example:7687')"
);

$this->newDecorator( $plugin )->savePage( TestPage::build( id: 42 ) );

$message = $this->logger->records[0]['message'];
$this->assertStringNotContainsString( 's3cr3t', $message );
$this->assertStringContainsString( 'bolt://neo.example:7687', $message, 'still names the unreachable server' );
}

public function testFailingDeleteIsSwallowedAndLoggedForTheDeleteOperation(): void {
$this->newDecorator( new ThrowingGraphDatabasePlugin() )->deletePage( new PageId( 42 ) );

Expand Down
Loading