docs: Context and Storage by ethanholz · Pull Request #712 · OpenFreeEnergy/gufe

ethanholz · 2026-01-21T19:27:13Z

This PR closes #695, providing documentation on ExternalStorage and Context.

Checklist

Added a news entry

Developers certificate of origin

I certify that this contribution is covered by the MIT License here and the Developer Certificate of Origin at https://developercertificate.org/.

Signed-off-by: Ethan Holz <ethan.holz@omsf.io>

codecov · 2026-01-21T19:29:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (feat/warehouse@c932c8b). Learn more about missing BASE report.

Additional details and impacted files

@@                Coverage Diff                @@
##             feat/warehouse     #712   +/-   ##
=================================================
  Coverage                  ?   98.33%           
=================================================
  Files                     ?       42           
  Lines                     ?     2526           
  Branches                  ?        0           
=================================================
  Hits                      ?     2484           
  Misses                    ?       42           
  Partials                  ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ethanholz · 2026-01-21T23:56:37Z

Closes #695 . @atravitz ready for review whenever you get to it.

docs/concepts/context.rst

docs/concepts/storage.rst

Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com>

…docs/context

This adds new information on how storage is handled, and how storage gets passed around by units.

ethanholz · 2026-01-28T20:25:11Z

@atravitz pinging on the feedback from #718, we already demonstrate the workflow for loading data from the a result and using it in a protocol so I think this is already covered.

ethanholz · 2026-01-28T20:42:37Z

One thing that is clearly missing here is an update to the Protocol How-To, is this work that someone on the OpenFE side would like to tackle? I am more than happy to assist but I think it might be a good exercise in knowledge transfer.

atravitz · 2026-01-28T23:28:29Z

One thing that is clearly missing here is an update to the Protocol How-To, is this work that someone on the OpenFE side would like to tackle? I am more than happy to assist but I think it might be a good exercise in knowledge transfer.

@IAlibay is the person to sync with on this

docs/concepts/storage.rst

Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com>

ethanholz · 2026-01-29T21:26:01Z

We are in the home stretch, do we want to have @IAlibay take a look as well?

atravitz · 2026-01-29T23:14:45Z

docs/concepts/context.rst

+This means each ``ProtocolUnit``'s ``shared`` and ``permanent`` object are not paths, and should not be treated as such.
+Both of these are registries that track if a file should be transferred from its location in ``scratch`` to its final location after completing a unit.
+
+If you want to use some from ``shared`` or ``permanent``, you can use ``ctx.shared.load`` or ``ctx.permanent.load``.


Suggested change

If you want to use some from ``shared`` or ``permanent``, you can use ``ctx.shared.load`` or ``ctx.permanent.load``.

To access data from ``shared`` or ``permanent``, you can use ``ctx.shared.load`` or ``ctx.permanent.load``.

you were missing a word I think

github-actions · 2026-01-29T23:16:27Z

No API break detected ✅

atravitz · 2026-01-29T23:17:38Z

We are in the home stretch, do we want to have @IAlibay take a look as well?

@IAlibay - you can review here, or pass on reviewing here and give your edits when evaluating migration things, up to you!

IAlibay

Some initial thoughts / questions ahead of tomorrow's meeting.

docs/concepts/context.rst

IAlibay · 2026-02-04T22:30:49Z

docs/concepts/context.rst

+                "result_path": result_path_final_location,
+            }
+
+The example above showcases how


Incomplete sentence?

Bumping this.

docs/concepts/context.rst

IAlibay · 2026-02-04T22:38:38Z

docs/concepts/context.rst

+
+.. code-block:: python
+
+    ctx.shared.register("myfile.dat")


What if your underlying code cannot handle Path like objects? For example, the multistate sampler stores the checkpoint file using a filename that is relative to the main storage path, so when you pass the file it's just checkpoint.chk.

How do I get the full path?

So you can use registration as well, however we assume that everything is stored in scratch. Remember that we explicitly make it such that the registered files are not Path-like objects to ensure that people don't use them as paths. One thing we could do, is remove the requirement that everything is stored in scratch and then you can register any file to be transferred at the end of the run.

Potentially add just dumping loaded objects as Pathlib objects.

docs/concepts/context.rst

IAlibay · 2026-02-04T23:38:06Z

docs/concepts/storage.rst

+``ExternalStorage`` Base Class
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :class:`.ExternalStorage` abstract base class defines the interface that all storage implementations must provide. This class provides:


I can see a get_filename method, but what if I want the full path / operate on the Pathlib object (i.e. if I want to get the file extension or the parent directory)?

ExternalStorage objects intentionally don't use Pathlib so that we can abstract over the storage medium. This allows execution engines the choice of how they want to store files (which might not always be a filesystem).

docs/concepts/storage.rst

IAlibay · 2026-02-16T20:39:48Z

docs/concepts/context.rst

+                "result_path": result_path_final_location,
+            }
+
+The example above showcases how


Bumping this.

docs: add information on storage

0800b03

Signed-off-by: Ethan Holz <ethan.holz@omsf.io>

ethanholz added 3 commits January 21, 2026 13:34

docs: initial draft on context

5c428eb

docs: improve context docs

edd1d8b

docs: migrate how to write a custom implementation

66a40fb

ethanholz changed the title ~~[WIP] docs: Context and Storage~~ docs: Context and Storage Jan 21, 2026

ethanholz marked this pull request as ready for review January 21, 2026 23:55

atravitz self-requested a review January 22, 2026 21:04

atravitz requested changes Jan 23, 2026

View reviewed changes

ethanholz and others added 6 commits January 23, 2026 10:12

doc: update with comments from review

adbad03

Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com>

fix docs build

94469fd

Merge branch 'feat/warehouse' of github.com:OpenFreeEnergy/gufe into …

c4a910d

…docs/context

Merge branch 'feat/warehouse' of github.com:OpenFreeEnergy/gufe into …

a2a4b80

…docs/context

docs: Add information on changes in feat/return-storage-handles

355a4ce

This adds new information on how storage is handled, and how storage gets passed around by units.

docs: add information on pre-namespaced objects from other branch

497eb8c

ethanholz mentioned this pull request Jan 27, 2026

feat: return storage handles from StorageManager #718

Merged

2 tasks

Merge remote-tracking branch 'origin/feat/warehouse' into docs/context

e42fadf

ethanholz added 3 commits January 28, 2026 13:30

docs: add suggestion to back link to Context docs

200973a

docs: add becomes in migration guide per suggestion

ba76579

docs: add link to protocol how to

3fbe18f

ethanholz requested a review from atravitz January 28, 2026 20:42

atravitz reviewed Jan 28, 2026

View reviewed changes

docs/concepts/storage.rst Outdated Show resolved Hide resolved

ethanholz added 2 commits January 28, 2026 17:09

docs: add more informaton on StorageManagers

8f1ce78

docs: clean up storage code example

83ee929

atravitz reviewed Jan 29, 2026

View reviewed changes

docs/concepts/storage.rst Outdated Show resolved Hide resolved

Apply suggestions from code review

be26dfc

Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com>

atravitz reviewed Jan 29, 2026

View reviewed changes

fix typo

bdc284a

IAlibay reviewed Feb 5, 2026

View reviewed changes

IAlibay approved these changes Feb 16, 2026

View reviewed changes

docs/concepts/context.rst

"result_path": result_path_final_location,

}

The example above showcases how

Copy link

Member

IAlibay Feb 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bumping this.

	If you want to use some from ``shared`` or ``permanent``, you can use ``ctx.shared.load`` or ``ctx.permanent.load``.
	To access data from ``shared`` or ``permanent``, you can use ``ctx.shared.load`` or ``ctx.permanent.load``.

Conversation

ethanholz commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Developers certificate of origin

Uh oh!

codecov bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ethanholz commented Jan 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ethanholz commented Jan 28, 2026

Uh oh!

ethanholz commented Jan 28, 2026

Uh oh!

atravitz commented Jan 28, 2026

Uh oh!

Uh oh!

Uh oh!

ethanholz commented Jan 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 29, 2026

Uh oh!

atravitz commented Jan 29, 2026

Uh oh!

IAlibay left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ethanholz commented Jan 21, 2026 •

edited

Loading

codecov bot commented Jan 21, 2026 •

edited

Loading