Skip to content

Deferred compression of frontend objects #494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JerryI merged 3 commits into from
Apr 3, 2026
Merged

Deferred compression of frontend objects #494

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
252a879
Deferred compression to avoid contexts missmatch when using Expressio…
JerryI Apr 2, 2026
b5fcfe6
fix container layout offset & added Controls disable
JerryI Apr 2, 2026
8c0ae57
updated release notes and readme
JerryI Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,16 @@ Typeset math input, syntactic sugar, multiple languages, and a fast, granular ev

See how at [wljs.io](https://wljs.io/frontend/setup)

## Star History

<a href="https://www.star-history.com/?repos=WLJSTeam%2Fwljs-notebook&type=date&legend=top-left">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/image?repos=WLJSTeam/wljs-notebook&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/image?repos=WLJSTeam/wljs-notebook&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://api.star-history.com/image?repos=WLJSTeam/wljs-notebook&type=date&legend=top-left" />
</picture>
</a>

## Highlights

### Feels like Mathematica and Jupyter
Expand Down
145 changes: 82 additions & 63 deletions modules/wljs-demos-archive/Demos/Release notes/3.0.3.wln
View file
Open in desktop

Large diffs are not rendered by default.

47 changes: 35 additions & 12 deletions modules/wljs-editor/src/FrontendObject.wl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,26 +26,45 @@ Objects = <||>
Symbols = <||>


Compressed[string_String, {"ExpressionJSON", "ZLIB"}] := ImportByteArray[ByteArray[Developer`RawUncompress[BaseDecode[string]//Normal]], "ExpressionJSON"]
(* ::: Compression for large frontend objects :::*)

Compressed[string_String, {"ExpressionJSON", "ZLIB"}] := ImportByteArray[ByteArray[Developer`RawUncompress[BaseDecode[string]//Normal]], "ExpressionJSON"] // ReleaseHold

compression;

(* [NOTE] This is a deferred compression method, i.e. it is only applied when the object is requested via net / link *)
(* Otherwise ExpressionJSON uncontrollably lifts the context from symbols depending where forntend object was created, *)
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in comment: “forntend” should be “frontend”.

Suggested change
(* Otherwise ExpressionJSON uncontrollably lifts the context from symbols depending where forntend object was created, *)
(* Otherwise ExpressionJSON uncontrollably lifts the context from symbols depending where frontend object was created, *)

Copilot uses AI. Check for mistakes.
(* this leads to some symbols to be falsly assumed to be in Global`, which will throw errors on the frontend *)
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in comment: “falsly” should be “falsely”.

Suggested change
(* this leads to some symbols to be falsly assumed to be in Global`, which will throw errors on the frontend *)
(* this leads to some symbols to be falsely assumed to be in Global`, which will throw errors on the frontend *)

Copilot uses AI. Check for mistakes.
(* The only way to avoid this is to deffer compression and ExpressionJSON convertion. *)
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in comment: “deffer” should be “defer”.

Suggested change
(* The only way to avoid this is to deffer compression and ExpressionJSON convertion. *)
(* The only way to avoid this is to defer compression and ExpressionJSON convertion. *)

Copilot uses AI. Check for mistakes.
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in comment: “convertion” should be “conversion”.

Suggested change
(* The only way to avoid this is to deffer compression and ExpressionJSON convertion. *)
(* The only way to avoid this is to deffer compression and ExpressionJSON conversion. *)

Copilot uses AI. Check for mistakes.
(* [FIXME] for the future: switch to Compress and WXF formats instead of JSON !!! *)

(* apply only on large objects*)
compression[expr_, {"ExpressionJSON", "ZLIB", "Defer"}] := Hold[expr] /; (ByteCount[expr] < 0.1 * 1024 * 1024);

SetAttributes[releaseCompression, HoldAll]

releaseCompression[compression[expr_, {"ExpressionJSON", "ZLIB", "Defer"}]] := With[{arr = Normal[ExportByteArray[expr, "ExpressionJSON"] ]},
With[{data = BaseEncode[ByteArray[Developer`RawCompress[arr] ] ]},
Compressed[data, {"ExpressionJSON", "ZLIB"}] // Hold
]
]

releaseCompression[expr_] := expr



compress[expr_, f: {"ExpressionJSON", "ZLIB"}] := Hold[expr]
compress[expr_, f: {"ExpressionJSON", "ZLIB"}] := With[{arr = Normal[ExportByteArray[expr, "ExpressionJSON"] ]},
With[{data = BaseEncode[ByteArray[Developer`RawCompress[arr] ] ]},
Compressed[data, f] // Hold
]
] /; (ByteCount[expr] > 0.1 * 1024 * 1024);

CreateFrontEndObject[expr_, uid_String, OptionsPattern[] ] := With[{},
With[{
data = Switch[OptionValue["Store"]
, "Kernel"
, <|"Private" -> compress[expr, {"ExpressionJSON", "ZLIB"}]|>
, <|"Private" -> compression[expr, {"ExpressionJSON", "ZLIB", "Defer"}]|>

, "Frontend"
, <|"Public" -> compress[expr, {"ExpressionJSON", "ZLIB"}]|>
, <|"Public" -> compression[expr, {"ExpressionJSON", "ZLIB", "Defer"}]|>

,_
, <|"Private" -> compress[expr, {"ExpressionJSON", "ZLIB"}], "Public" :> Objects[uid, "Private"]|>
, <|"Private" -> compression[expr, {"ExpressionJSON", "ZLIB", "Defer"}], "Public" :> Objects[uid, "Private"]|>
]
},
If[!AssociationQ[Objects],
Expand All @@ -69,7 +88,9 @@ CreateFrontEndObject[expr_, opts: OptionsPattern[] ] := CreateFrontEndObject[exp
Options[CreateFrontEndObject] = {"Store" -> All}

FrontEndRef[uid_String] := If[KeyExistsQ[Objects, uid],
Objects[uid, "Private"] // ReleaseHold
With[{o = Objects[uid, "Private"]},
releaseCompression[o] (*Ehhhh Okay... we need to release it anyway *)
] // ReleaseHold
Comment on lines 90 to +93
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FrontEndRef now routes stored objects through releaseCompression, which (for deferred wrappers) performs an ExportByteArray[..., "ExpressionJSON"] + compress, then ReleaseHold immediately triggers Compressed[...] and re-imports/decompresses the JSON. This adds a costly JSON round-trip on every FrontEndRef call and can reintroduce the symbol-context issues described in the header comment (since ExpressionJSON conversion happens under the current $Context). Consider unwrapping the deferred wrapper for kernel-side access (e.g., return the original expr/Hold[expr]) and only run ExpressionJSON conversion when actually sending over the link.

Copilot uses AI. Check for mistakes.
,
$MissingHandler[uid, "Private"] // ReleaseHold
]
Expand All @@ -79,7 +100,9 @@ FrontEndExecutable /: MakeBoxes[FrontEndExecutable[uid_String], StandardForm] :=
GetObject[uid_String] := With[{},
(*Echo["Getting object >> "<>uid];*)
If[KeyExistsQ[Objects, uid],
Objects[uid, "Public"]
With[{ c = Objects[uid, "Public"] },
releaseCompression[c]
]
,
$Failed
]
Expand Down
10 changes: 7 additions & 3 deletions modules/wljs-editor/src/FrontendObjectSync.wl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,13 @@ syncMonitor = ImportComponent[FileNameJoin[{rootDir, "templates", "SyncMonitor.w
EventHandler[NotebookEditorChannel // EventClone, {
"FetchFrontEndObject" -> Function[data,
Echo["Sync >> requested from master kernel"];
With[{promise = data["Promise"], kernel = GenericKernel`HashMap[ data["Kernel"] ]},
With[{result = CoffeeLiqueur`Extensions`FrontendObject`Internal`Objects[data["UId"] ]},
GenericKernel`Async[kernel, EventFire[promise, Resolve, result["Public"] ] ];
With[{promise = data["Promise"], kernel = GenericKernel`HashMap[ data["Kernel"] ]},
(* [FIXME] Include these symbols normally using Needs[] *)
(* we release any possible deferred compression wrappers *)
With[{result = CoffeeLiqueur`Extensions`FrontendObject`Internal`Objects[data["UId"] ]["Public"]},
With[{c = CoffeeLiqueur`Extensions`FrontendObject`Internal`releaseCompression[result]},
GenericKernel`Async[kernel, EventFire[promise, Resolve, c ] ];
Comment on lines +28 to +33
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FetchFrontEndObject now calls CoffeeLiqueurExtensionsFrontendObjectInternalreleaseCompression, but this package file doesn’t declare a dependency on CoffeeLiqueurExtensionsFrontendObject`` (no BeginPackage[..., {"CoffeeLiqueurExtensionsFrontendObject", ...}] / Needs[...]). If Objects is present via restored notebook state but the FrontendObject package wasn’t loaded, releaseCompression will be undefined and you’ll resolve the promise with an unevaluated wrapper. Add an explicit dependency/Needs for the FrontendObject package before using releaseCompression (and then the FIXME can be removed).

Copilot uses AI. Check for mistakes.
];
];
];
]
Expand Down
32 changes: 20 additions & 12 deletions modules/wljs-graphics3d-threejs/dist/kernel.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5303,7 +5303,9 @@ if (!GUI && PathRendering) {
/**
* @type {HTMLElement}
*/
const container = env.element;
const container = document.createElement('div');
container.classList.add('relative');
env.element.appendChild(container);

/**
* @type {[Number, Number]}
Expand Down Expand Up @@ -5469,7 +5471,7 @@ env.local.renderer = renderer;

//fix for translate-50% layout
const layoutOffset = {x:0, y:0};
if (container.classList.contains('slide-frontend-object')) {
if (env.element.classList.contains('slide-frontend-object')) {
layoutOffset.x = -1.0;
}

Expand Down Expand Up @@ -5553,19 +5555,22 @@ if (PathRendering) {
}

let controlObject = {
init: (camera, dom) => {
controlObject.o = new OrbitControls( camera, domElement );
controlObject.o.addEventListener('change', wakeFunction);
controlObject.o.target.set( 0, 1, 0 );
controlObject.o.update();
},
init: (camera, dom) => {
controlObject.o = new OrbitControls( camera, domElement );
controlObject.o.addEventListener('change', wakeFunction);
controlObject.o.target.set( 0, 1, 0 );
controlObject.o.update();
},

dispose: () => {

}
};
dispose: () => {

}
};

if ('Controls' in options && !(await interpretate(options.Controls))) {
controlObject.disabled = true;
domElement.style.pointerEvents = 'none';
}

if (options.Controls) {

Expand Down Expand Up @@ -5755,6 +5760,8 @@ if (options.Controls) {
}
}



env.local.controlObject = controlObject;


Expand Down Expand Up @@ -6981,6 +6988,7 @@ core.Graphics3D.destroy = (args, env) => {
if (env.local.labelContainer) env.local.labelContainer.remove();
if (env.local.guiContainer) env.local.guiContainer.remove();
env.local.rendererContainer.remove();
env.local.element.remove();
};

core.Graphics3D.virtual = true;
Expand Down
2 changes: 1 addition & 1 deletion modules/wljs-graphics3d-threejs/dist/kernel.min.js
View file
Open in desktop

Large diffs are not rendered by default.

32 changes: 20 additions & 12 deletions modules/wljs-graphics3d-threejs/src/kernel.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5526,7 +5526,9 @@ if (!GUI && PathRendering) {
/**
* @type {HTMLElement}
*/
const container = env.element;
const container = document.createElement('div');
container.classList.add('relative');
env.element.appendChild(container);

/**
* @type {[Number, Number]}
Expand Down Expand Up @@ -5694,7 +5696,7 @@ env.local.renderer = renderer;

//fix for translate-50% layout
const layoutOffset = {x:0, y:0};
if (container.classList.contains('slide-frontend-object')) {
if (env.element.classList.contains('slide-frontend-object')) {
layoutOffset.x = -1.0;
}

Expand Down Expand Up @@ -5778,19 +5780,22 @@ if (PathRendering) {
}

let controlObject = {
init: (camera, dom) => {
controlObject.o = new OrbitControls( camera, domElement );
controlObject.o.addEventListener('change', wakeFunction);
controlObject.o.target.set( 0, 1, 0 );
controlObject.o.update();
},
init: (camera, dom) => {
controlObject.o = new OrbitControls( camera, domElement );
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

controlObject.init takes (camera, dom) but ignores the dom parameter and instead closes over the outer domElement variable. This makes the API misleading and can break if callers pass a different DOM element (e.g., for different control implementations). Use the dom parameter inside init (or remove it from the signature) to keep the control-object interface consistent.

Suggested change
controlObject.o = new OrbitControls( camera, domElement );
controlObject.o = new OrbitControls( camera, dom );

Copilot uses AI. Check for mistakes.
controlObject.o.addEventListener('change', wakeFunction);
controlObject.o.target.set( 0, 1, 0 );
controlObject.o.update();
},

dispose: () => {

}
};
dispose: () => {

}
}

if ('Controls' in options && !(await interpretate(options.Controls))) {
controlObject.disabled = true;
domElement.style.pointerEvents = 'none';
}
Comment on lines +5795 to +5798
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When Controls is explicitly set to a falsy value, you disable pointer events on domElement, but controlObject.init(...) is still called unconditionally later (and controls = controlObject.o assumes init ran). This still constructs OrbitControls and registers listeners even though controls are intended to be disabled. Consider gating the init call (and controls assignment) behind the same enabled/disabled condition, or explicitly set controls.enabled = false / configure OrbitControls’ enableRotate|enablePan|enableZoom depending on what "Controls" is meant to control (the current pointerEvents = 'none' disables all interactions, not just rotation/pan).

Copilot uses AI. Check for mistakes.

if (options.Controls) {

Expand Down Expand Up @@ -5980,6 +5985,8 @@ if (options.Controls) {
}
}



env.local.controlObject = controlObject;


Expand Down Expand Up @@ -7209,6 +7216,7 @@ core.Graphics3D.destroy = (args, env) => {
if (env.local.labelContainer) env.local.labelContainer.remove();
if (env.local.guiContainer) env.local.guiContainer.remove();
env.local.rendererContainer.remove();
env.local.element.remove();
}

core.Graphics3D.virtual = true
Expand Down
Loading