feat: interactive plot for pointcloud hypothesis space #15

ramyamounir · 2025-11-30T13:12:22Z

This introduces a new hypothesis space visualization with a focus on burst sampling. The line plot shows when a burst is triggered, as well as the hypothesis space size and the max global slope over time. I've also introduced toggle switches to view the history of agent and sensor patch locations. There are some other nice features such as alpha slider for the mesh, and coloring the hyp space point cloud by different metrics. The video below showcases some of those features.

FYI @hlee9212, I've added your Shim Unpickler solution to the interactive library as part of a PretrainedModelsLoader class that returns a point cloud of the pretrained model. Feel free to adjust your code to use that directly if you like.

line_plot.mp4

fix: pretrained models class descriptions and shim unpickler fix: remove print message in widgets.py feat: extract_slider_state round parameter feat: add extract button state helper function feat: add interactive pointcloud plot

nielsleadholm · 2025-12-01T14:59:46Z

Love the visualization! Just some high level comments before I have a chance to review the code. These are generally focused on how we can make it more user friendly or the improve the aesthetics:

Would it make more sense to call it "agent" (path) rather than sensor? I think that might be more intuitive for people seeing it for the first time.
For most of the buttons, I think it would be clearer to have drop-downs (if possible) that present the different options. If this is not possible, something like "Agent Path: On" vs "Agent Path: Off" - basically not assuming that someone knows what is going to happen when they click a button if they have never clicked it before.
Changing "Alpha" to "Mesh Transparency"

Stylistically, I wonder if you could try some subtle changes to improve it a bit? E.g.

Changing all the font to be consistent and clean (Arial?)
?rounded corners on the buttons
Use the TBP colors (starting with Blumenta and then pink) for the buttons and plotting elements
Increasing the resolution of the line-plot, it looks kind of blurry vs the other elements?
You could also try passing the plotting code to an LLM and asking for graphic design tips to make it look more professional (obviously the content is already great, but I know my web dev friends have found LLMs great for this kind of thing).

ramyamounir · 2025-12-02T19:03:41Z

Thanks @nielsleadholm! I addressed the majority of these concerns.

Would it make more sense to call it "agent" (path) rather than sensor?

Yes, that makes a lot of sense, especially because the patch can also be called "sensor patch" sometimes, which can be confused with the sensor. Updated in d5427f1c032a

it would be clearer to have drop-downs (if possible) that present the different options. If this is not possible, something like "Agent Path: On" vs "Agent Path: Off"

Unfortunately, I only have a few very basic widgets in vedo and even customization can be difficult, drop down menus are not supported. I updated to the pattern Name: [On/Off] as you suggested in c85b16c761fd

Changing "Alpha" to "Mesh Transparency"

Updated in 9a9132555099

Changing all the font to be consistent and clean (Arial?)

I'm trying to avoid system fonts and only use ones that come builtin in vedo to make sure it works out of the box for everyone. I personally think Calco (the one we're using now) looks good, but here is a list of the available ones. Would you prefer, e.g., Theemim?

rounded corners on the buttons

This is one of the customizations that's not supported in Vedo.

Use the TBP colors (starting with Blumenta and then pink) for the buttons and plotting elements

Nice idea, I added the TBP color palette as part of the interactive library in 784d6fb0bffd and used it in the plot in b9aa9135fa09. I tried to use it wherever possible, hopefully the colors are not too distracting. What do you think?

Increasing the resolution of the line-plot

I fixed this in 8495f5d04f10

You could also try passing the plotting code to an LLM

I tried this and didn't get any good suggestions that can be done in Vedo. I think the main problem is that Vedo is limited in the types of widgets and possible customizations, we are already using all the options that can be used and pushing vedo to it's limits. Check out their gallery, it's very basic.

nielsleadholm · 2025-12-03T11:17:19Z

Nice thanks, that's looking great!

Re. the font, is it possible to try Comae and Normografo for comparison? Otherwise all the changes / responses look good.

nielsleadholm · 2025-12-03T11:22:42Z

As an aside/not something we need to sort now, these Vedo examples in the gallery made me think that it could eventually be interesting to have something where the mesh is gradually revealed in the same way as it is to Monty, basically showing only the parts that it has uncovered via its path. It would give the user a more intuitive sense for how much/little Monty has explored.

ramyamounir · 2025-12-03T11:48:48Z

try Comae and Normografo for comparison?

Sure, first image is "Comae" and one after is "Normografo". I'm not very good with Fonts and don't usually spot the differences when they are close 😅, but let me know if there's one you prefer. Note that the font changes will only apply to the vedo widgets, the matplotlib figure uses the matplotlib default "DejaVu Sans". Vedo fonts are not available in matplotlib.

interesting to have something where the mesh is gradually revealed in the same way as it is to Monty

Yeah, I agree. That would be interesting to try out. We would need the size of the patch to uncover the correct surface area. Vedo also has a ray collision system where it will give you the intersection point on the mesh based on the camera's FOV, etc.

nielsleadholm · 2025-12-03T11:57:43Z

Nice thanks! Personally I think Comae looks best. It might just be worth decreasing the font size ever so slightly, the text looks a bit too big for the boxes.

Would it be possible to also update the fonts below (either to match Matplotlib default or Comae?) - both the text and numbers.

(sorry I know I'm nitpicking, but just want to do these figures justice!)

ramyamounir · 2025-12-03T13:36:34Z

Nice catch, I forget to set a font variable to the slider. This is all changed now. I made some other changes to the scalar bar and the button inside the hypothesis space subplot. I also found an "Arial" font in Vedo and I think it looks nice, what do you think about these changes?

I can't change the text font size inside the box and leave the button box size the same. It's a more involved change because it looks like vedo rasterizes the button with the text on it, so we can't change the text properties after the button has been created. They also don't allow changing the font size independent of the box size, they shrink together.

ramyamounir · 2025-12-08T16:01:44Z

Added some more modifications to the line plot suggested by Viviane.

ramyamounir · 2025-12-12T11:11:14Z

I have renamed "Max slope" to "Max Recent Evidence Growth" as suggested by Niels, I also renamed "Burst" to "Sampling Burst" to be more specific.

nielsleadholm

Very nice and clean, looks great! Just a few minor comments.

nielsleadholm · 2025-12-12T15:10:47Z

src/tbp/plot/plots/interactive_hypothesis_space_pointcloud.py

+                label="Current step",
+            )
+
+        ax_left.set_ylabel("Max slope")


This would also be worth changing to "Max Recent Evidence Growth".

Nice catch, thanks. Fixed in 54517032c4fb

nielsleadholm · 2025-12-12T15:13:30Z

src/tbp/interactive/data.py

+        """Load a torch checkpoint with optional fallback for tbp.monty shimming.
+
+        Try a standard torch.load first (weights_only=False because we need objects).
+        If tbp.monty isn't installed and the checkpoint references it, optionally


Is there a reason tbp.monty wouldn't be installed if using these plotting tools? Was this something you came across?

Yeah, tbp.plot uses a different uv environment that does not contain tbp.monty in it. One reason is the mismatch in python versions, tbp.plot uses python 3.13 (not 3.8). But also, ideally, we wouldn't want to depend on the tbp.monty codebase since everything we need is the log files and pretrained models.

In the future, I imagine it would be really cool to have visualizations that actually run a Monty episode step by step from Vedo. This would allow us to change parameters, swap/manipulate objects, move sensors, etc., between steps. At this point, we would have to install tbp.monty.

nielsleadholm · 2025-12-12T15:16:46Z

src/tbp/plot/plots/interactive_hypothesis_space_pointcloud.py

+class StepMapper:
+    """Bidirectional mapping between global step indices and (episode, local_step).
+
+    Global steps are defined as the concatenation of all local episode steps:


Out of interest, did you manage to identify the cause of the issue where, after a jump-to-goal-state, the agent seemingly moves to a new location, while the sensory location remains un-updated?

Not yet, it seems like a problem on the logging side of tbp.monty. I'll look into it this week, shortcut ticket created.

nielsleadholm · 2025-12-12T15:20:16Z

src/tbp/plot/plots/interactive_hypothesis_space_pointcloud.py

+        }
+
+        self.step_mapper = step_mapper
+        self.current_episode = -1


Is there a significance to setting it to -1? Could it just be None? Seeing this value makes me nervous that there is some unexpected logic dependent on it 😅

For some reason, I get a little nervous when comparing None to int because I think python will error out and say that it cannot compare these two different types. I'm wrong, python is fine with it. I still do prefer to limit the number of types a variable can be (int vs int | None), but I guess that's just nitpicking on my side. I changed it to None in de62bdd9bea6.

nielsleadholm · 2025-12-12T15:25:56Z

src/tbp/plot/plots/interactive_hypothesis_space_pointcloud.py

+        patch_pos = self.data_parser.extract(
+            self._locators["patch_location"],
+            episode=str(episode_number),
+            step=step_number,


Is there a reason that the agent-pose uses int(mapping[step_number]), while here we just use step_number?

Yes, the agent_location locator is loaded from the motor_system logs, those are not filtered by which ones are processed by the LMs. So, I get the mask of which steps are processed first and use this mapping to extract the correct agent locations for the steps processed by the LM. The sensor locations come from the buffer, so we don't need this mapping since it only logs the ones processed by the LM.

nielsleadholm · 2025-12-12T15:31:45Z

src/tbp/plot/plots/interactive_hypothesis_space_pointcloud.py

+        ax_left.set_ylim(-1.0, 2.0)
+        ax_right.set_ylim(0, 10000)
+
+        # Burst locations (red dashed lines)


Suggested change

# Burst locations (red dashed lines)

# Burst locations (violet dashed lines)

Fixed in 54517032c4fb

nielsleadholm · 2025-12-12T15:31:55Z

src/tbp/plot/plots/interactive_hypothesis_space_pointcloud.py

+    """WidgetOps implementation for the line plot.
+
+    This widget shows a line plot for the max global slope and hypothesis space size
+    over time. It also shows the sampling bursts locations as vertical dashed red lines.


Suggested change

over time. It also shows the sampling bursts locations as vertical dashed red lines.

over time. It also shows the sampling bursts locations as vertical dashed violet lines.

Fixed in 54517032c4fb

nielsleadholm · 2025-12-12T15:36:05Z

src/tbp/plot/plots/interactive_hypothesis_space_pointcloud.py

+    - Buttons to activate or deactivate the history of agent and/or patch locations.
+    - A plot of the pretrained model for the primary target with a button to show the
+        hypothesis space for this object.
+    - The hypothesis space pointcloud can be coloured by different metrics, such as


Suggested change

- The hypothesis space pointcloud can be coloured by different metrics, such as

- The color-map of the hypothesis space pointcloud can reflect different metric values, such as

Fixed in 54517032c4fb

ramyamounir · 2025-12-13T14:09:26Z

Great, thanks @nielsleadholm! I think I addressed your comments. Can you take another look?

feat: interactive plot for pointcloud hypothesis space

ba6c91b

fix: pretrained models class descriptions and shim unpickler fix: remove print message in widgets.py feat: extract_slider_state round parameter feat: add extract button state helper function feat: add interactive pointcloud plot

ramyamounir requested a review from nielsleadholm November 30, 2025 13:12

ramyamounir assigned nielsleadholm Nov 30, 2025

ramyamounir added the triaged label Nov 30, 2025

ramyamounir added 8 commits December 2, 2025 07:27

refactor!: alpha to mesh transparency

9a91325

refactor!: rename sensor to agent

d5427f1

refactor!: rename buttons to "Name: [on|off]"

c85b16c

chore: reposition some buttons

97faf91

feat: add tbp color palette

784d6fb

refactor: change colors to use tbp palette

b9aa913

fix: minor typing fix

7a96a50

feat: increase figure dpi to 200

8495f5d

refactor: extract FONT constant

ec627f7

ramyamounir added 2 commits December 3, 2025 08:38

refactor: change scalarbar font and font sizes extraction

39ba30d

feat: add rectangles and plotting mods

f71204a

refactor!: label renames for lineplot

2daeb65

ramyamounir mentioned this pull request Dec 12, 2025

refactor!: Improve interactive correlation plot #16

Draft

nielsleadholm reviewed Dec 12, 2025

View reviewed changes

ramyamounir added 2 commits December 13, 2025 08:50

docs: change docstrings and fix typos

5451703

refactor: change current_episode default from -1 to None

de62bdd

ramyamounir requested a review from nielsleadholm December 13, 2025 14:08

	# Burst locations (red dashed lines)
	# Burst locations (violet dashed lines)

	over time. It also shows the sampling bursts locations as vertical dashed red lines.
	over time. It also shows the sampling bursts locations as vertical dashed violet lines.

	- The hypothesis space pointcloud can be coloured by different metrics, such as
	- The color-map of the hypothesis space pointcloud can reflect different metric values, such as

feat: interactive plot for pointcloud hypothesis space #15

Are you sure you want to change the base?

feat: interactive plot for pointcloud hypothesis space #15

Uh oh!

Conversation

ramyamounir commented Nov 30, 2025

Uh oh!

nielsleadholm commented Dec 1, 2025

Uh oh!

ramyamounir commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nielsleadholm commented Dec 3, 2025

Uh oh!

nielsleadholm commented Dec 3, 2025

Uh oh!

ramyamounir commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nielsleadholm commented Dec 3, 2025

Uh oh!

ramyamounir commented Dec 3, 2025

Uh oh!

ramyamounir commented Dec 8, 2025

Uh oh!

ramyamounir commented Dec 12, 2025

Uh oh!

nielsleadholm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ramyamounir commented Dec 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ramyamounir commented Dec 2, 2025 •

edited

Loading

ramyamounir commented Dec 3, 2025 •

edited

Loading