Skip to content

chore: development with easy db restore#74

Open
k-f-a wants to merge 1 commit into4TUResearchData:mainfrom
k-f-a:development-db-restore
Open

chore: development with easy db restore#74
k-f-a wants to merge 1 commit into4TUResearchData:mainfrom
k-f-a:development-db-restore

Conversation

@k-f-a
Copy link
Copy Markdown
Contributor

@k-f-a k-f-a commented Apr 17, 2026

Summary

Add the ability to restore a Virtuoso database backup when starting the development environment, making it easy to test against specific production data.

Contributors can restore a backup in a single command:

just db_backup=<filename_#N.bp> dev

Specific state-graph is also possible

just state_graph=https://my-env.nl db_backup=<filename_#N.bp> dev

just dev without arguments continues to work as before

This eliminates the need for manual database restore steps when a developer needs to work with a specific dataset.

Example

> just db_backup=../test-2026-03-02_#1.bp dev
Backup file: /Users/kdearaujo/dev/4TUReserchData/test-2026-03-02_#1.bp
Restore name: test-2026-03-02_#
State graph:  https://data.4tu.nl
==> Ensuring virtuoso.ini exists (initializing if needed)...
[+] up 3/3
 ✔ Network docker_default      Created                                                                                                              0.0s
 ✔ Volume docker_virtuoso-data Created                                                                                                              0.0s
 ✔ Container docker-virtuoso-1 Started                                                                                                              0.2s
[+] stop 1/1
 ✔ Container docker-virtuoso-1 Stopped                                                                                                              1.2s
==> Removing old database files (keeping virtuoso.ini)...
Container docker-virtuoso-run-b79320745a7b Creating 
Container docker-virtuoso-run-b79320745a7b Created 
==> Restoring backup...
Container docker-virtuoso-run-f131d130e719 Creating 
Container docker-virtuoso-run-f131d130e719 Created 

                Fri Apr 17 2026
12:55:00 { Loading plugin 1: Type `plain', file `graphql' in `../hosting'
12:55:00   GraphQL/SPARQL Bridge version 0.9.3 (a1fd8195bf) from OpenLink Software
12:55:00   Support functions for GraphQL/SPARQL Bridge
12:55:00   SUCCESS plugin 1: loaded from ../hosting/graphql.so }
12:55:00 { Loading plugin 2: Type `plain', file `shapefileio' in `../hosting'
12:55:00   ShapefileIO version 0.1virt71 from OpenLink Software
12:55:00   Shapefile support based on Frank Warmerdam's Shapelib
12:55:00   SUCCESS plugin 2: loaded from ../hosting/shapefileio.so }
12:55:00 OpenLink Virtuoso Universal Server
12:55:00 Version 07.20.3240-pthreads for Linux as of Jun 10 2024 (a1fd8195bf)
12:55:00 uses OpenSSL 1.1.1f  31 Mar 2020
12:55:00 uses parts of PCRE, Html Tidy
12:55:00 Begin to restore with file prefix test-2026-03-02_#
12:55:00 --> Backup file # 1 [0x38F0-0xB5-0xFC]
12:55:21 End of restoring from backup, 559929 pages
12:55:21 Server exiting
==> Updating state-graph in dev config...
    Set state-graph to: https://data.4tu.nl
==> Starting development environment...
[+] Building 2.8s (19/19) FINISHED                                                                                                                      
...
[+] up 3/3
 ✔ Image docker-djehuty       Built                                                                                                                 3.1s
 ✔ Volume docker_djehuty-data Created                                                                                                               0.0s
 ✔ Container docker-djehuty-1 Created                                                                                                               0.1s
Attaching to djehuty-1, virtuoso-1
virtuoso-1  | Starting the Virtuoso Server
virtuoso-1  | 
virtuoso-1  |           Fri Apr 17 2026
virtuoso-1  | 12:55:26 { Loading plugin 1: Type `plain', file `graphql' in `../hosting'
virtuoso-1  | 12:55:26   GraphQL/SPARQL Bridge version 0.9.3 (a1fd8195bf) from OpenLink Software
virtuoso-1  | 12:55:26   Support functions for GraphQL/SPARQL Bridge
virtuoso-1  | 12:55:26   SUCCESS plugin 1: loaded from ../hosting/graphql.so }
virtuoso-1  | 12:55:26 { Loading plugin 2: Type `plain', file `shapefileio' in `../hosting'
virtuoso-1  | 12:55:26   ShapefileIO version 0.1virt71 from OpenLink Software
virtuoso-1  | 12:55:26   Shapefile support based on Frank Warmerdam's Shapelib
virtuoso-1  | 12:55:26   SUCCESS plugin 2: loaded from ../hosting/shapefileio.so }
virtuoso-1  | 12:55:26 OpenLink Virtuoso Universal Server
virtuoso-1  | 12:55:26 Version 07.20.3240-pthreads for Linux as of Jun 10 2024 (a1fd8195bf)
virtuoso-1  | 12:55:26 uses OpenSSL 1.1.1f  31 Mar 2020
virtuoso-1  | 12:55:26 uses parts of PCRE, Html Tidy
virtuoso-1  | 12:55:26 Database version 3126
virtuoso-1  | 12:55:26 SQL Optimizer enabled (max 1000 layouts)
Container docker-virtuoso-1 Waiting 
virtuoso-1  | 12:55:27 Compiler unit is timed at 0.000073 msec
virtuoso-1  | 12:55:28 Roll forward started
virtuoso-1  | 12:55:28 Roll forward complete
virtuoso-1  | 12:55:28 PL LOG: Installing Virtuoso Conductor version 1.00.8869 (DAV)
virtuoso-1  | 12:55:28 PL LOG: Installing with dependencies Virtuoso Conductor version 1.00.8869/2024-06-10 15:04 (DAV)
virtuoso-1  | 12:55:28 Checkpoint started
virtuoso-1  | 12:55:28 Checkpoint finished, log reused
virtuoso-1  | 12:55:29 Checkpoint started
virtuoso-1  | 12:55:29 Checkpoint finished, log reused
virtuoso-1  | 12:55:29 PL LOG: Installation with dependencies complete
virtuoso-1  | 12:55:29 Checkpoint started
virtuoso-1  | 12:55:29 Checkpoint finished, log reused
virtuoso-1  | 12:55:29 HTTP/WebDAV server online at 8890
virtuoso-1  | 12:55:29 Server online at 1111 (pid 1)
Container docker-virtuoso-1 Healthy 
djehuty-1   | [INFO] 2026-04-17 12:55:32,493 - djehuty.web.ui: Reading config file: /config/djehuty/djehuty-dev-config.xml
djehuty-1   | [INFO] 2026-04-17 12:55:32,493 - djehuty.web.ui: Clearing cache.
djehuty-1   | [INFO] 2026-04-17 12:55:32,493 - djehuty.web.cache: Removing 0 files.
djehuty-1   | [WARNING] 2026-04-17 12:55:32,493 - djehuty.web.ui: Assuming to run in a non-production environment.
djehuty-1   | [WARNING] 2026-04-17 12:55:32,493 - djehuty.web.ui: Set <production> to 1 in your configuration file for hardened security settings.
djehuty-1   | [INFO] 2026-04-17 12:55:32,494 - djehuty.web.database: Using external RDF store.
djehuty-1   | [WARNING] 2026-04-17 12:55:32,494 - djehuty.web.ui: No menu structure provided.
djehuty-1   | [INFO] 2026-04-17 12:55:32,494 - djehuty.web.ui: SPARQL endpoint:         http://virtuoso:8890/sparql
djehuty-1   | [INFO] 2026-04-17 12:55:32,494 - djehuty.web.ui: State graph:             https://data.4tu.nl
djehuty-1   | [INFO] 2026-04-17 12:55:32,494 - djehuty.web.ui: Storage path:            /data
djehuty-1   | [INFO] 2026-04-17 12:55:32,494 - djehuty.web.ui: Secondary storage path:  None
djehuty-1   | [INFO] 2026-04-17 12:55:32,494 - djehuty.web.ui: Cache storage path:      /data/cache
djehuty-1   | [INFO] 2026-04-17 12:55:32,494 - djehuty.web.ui: Static pages loaded:     0
djehuty-1   | [INFO] 2026-04-17 12:55:32,494 - djehuty.web.ui: Handle registration is disabled.
djehuty-1   | [INFO] 2026-04-17 12:55:32,494 - djehuty.web.ui: Using AUTOMATIC-LOGIN as identity provider.
djehuty-1   | [WARNING] 2026-04-17 12:55:32,494 - djehuty.web.ui: Sending notification e-mails has been disabled.
djehuty-1   | [WARNING] 2026-04-17 12:55:32,494 - djehuty.web.ui: Please configure a mail server to enable it.
djehuty-1   | [INFO] 2026-04-17 12:55:32,606 - djehuty.web.ui: Invalidating caches ...
djehuty-1   | [INFO] 2026-04-17 12:55:32,606 - djehuty.web.cache: Removing 0 files.
djehuty-1   | [INFO] 2026-04-17 12:55:32,606 - djehuty.web.ui: Initializing RDF store ...
djehuty-1   | [INFO] 2026-04-17 12:55:33,465 - djehuty.web.ui: Initialization completed.
djehuty-1   | [INFO] 2026-04-17 12:55:33,772 - djehuty.web.database: Created account for dev@localhost.
djehuty-1   | [INFO] 2026-04-17 12:55:33,777 - djehuty.web.ui: Refreshing groups configuration.
djehuty-1   | [INFO] 2026-04-17 12:55:34,603 - djehuty.web.ui: Account dev@localhost already exists.
djehuty-1   | [INFO] 2026-04-17 12:55:35,124 - djehuty.web.database: Using external RDF store.
djehuty-1   | [INFO] 2026-04-17 12:55:35,124 - djehuty.web.ui: Reloaded.

Changes

  • justfile: an optional db_backup variable in dev that auto-initializes Virtuoso config on fresh volumes before restoring
    just dev without arguments continues to work as before
  • etc/djehuty/djehuty-dev-config.xml: just fix the state-graph to be like data.4tu.nl

Approval Checklist

  • I agree to follow Djehuty's code of conduct.
  • [ x I have read and I have follow the code contribution workflow.
  • Code style and conventions were respected.
  • Documentation has been updated where needed (README, docs, or examples).
  • Review approved by at least one maintainer.
  • Merge readiness (PR is squashed into a single commit and follows the commit template).

Issue Reference (optional - PRs may not be associated with an issue)

Closes #ISSUE_NUMBER

Screenshots (optional)

Before/After visuals, UI changes, or relevant logs.

Notes (optional)

Additional context, caveats, or follow-up tasks.

@k-f-a
chore: development/easy db restore
1f8d4d4 
it implements a easy way during development restore virtuoso database to
test specific data

* justfile: extend it adding parameters to use specific db backup as dev
  data

Signed-off-by: Kairo de Araujo <k.f.dearaujo@tudelft.nl>
@k-f-a k-f-a requested a review from gabrielakuhn April 17, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gabrielakuhn gabrielakuhn Awaiting requested review from gabrielakuhn

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@k-f-a