Skip to content

Fix 885: Make loading of dependencies always synchronous #894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Fix 885: Make loading of dependencies always synchronous #894

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f264c12
Update CLI commands to deprecate synchronous flag
Sep 16, 2025
fe5bbf4
Remove various LoadDependencies() calls as this should just be handle…
Sep 16, 2025
65382f4
Add unit test for paser and only include deprecated if relevant
Sep 17, 2025
e8c8540
Merge branch 'main' into fix-885
Oct 21, 2025
9c78274
Remove missed InstallContext refs
Oct 21, 2025
40b6112
Update changelog, remove $$$IsISCWorker references that are no longer…
Oct 22, 2025
a18535e
Update module version
Oct 22, 2025
e896271
remove debug breaks
Oct 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,25 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.10.5] - Unreleased

### Added

### Changed
- #885: Always synchronously load dependencies and let each module do multi-threading as needed
to load using multicompile instead of trying to do own multi-threading of item load which causes
lock contention by bypassing IRIS compiler.

### Deprecated
- #885: `-synchronous` flag since loading dependencies synchronously is now the default behavior.

### Removed

### Fixed

### Security

Copy link
Collaborator

@isc-jili isc-jili Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry but I think my recent merge to main has made your CHANGELOG stale so would require another pull from main


## [0.10.4] - 2025-10-21

### Added
Expand Down
92 changes: 50 additions & 42 deletions module.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,49 +1,57 @@
<?xml version="1.0" encoding="UTF-8"?>
<Export generator="Cache" version="25">
<Document name="ZPM.ZPM"><Module>
<Name>ZPM</Name>
<Version>0.10.4-SNAPSHOT</Version>
<ExternalName>IPM</ExternalName>
<Description>InterSystems Package Manager (IPM) provides development tools and infrastructure for defining, building, distributing, and installing modules and applications.</Description>
<Keywords>Package Manager</Keywords>
<Author>
<Organization>InterSystems Corporation</Organization>
</Author>
<Packaging>module</Packaging>
<SourcesRoot>src</SourcesRoot>
<Resource Name="%IPM.PKG"/>
<Resource Name="%IPM.Common.INC"/>
<Resource Name="%IPM.Formatting.INC"/>
<Resource Name="IPM.Installer.CLS" Preload="true" />
<UnitTest Name="/tests/unit_tests/" Package="Test.PM.Unit" Phase="test"/>
<UnitTest Name="/tests/integration_tests/" Package="Test.PM.Integration" Phase="verify"/>
<Invoke Class="IPM.Installer" Method="Map" Phase="Reload" When="Before" />
<Invoke Class="IPM.Installer" Method="MapIfLegacy" Phase="Compile" When="After" />
<Invoke Class="IPM.Installer" Method="InstallORASIfMissing" Phase="Activate" When="After" />
<Invoke Class="%IPM.Main" Method="UpdateLanguageExtensions" />
<Invoke Class="%IPM.Utils.Migration" Method="RunAll">
<Arg>${verbose}</Arg>
</Invoke>
<Document name="ZPM.ZPM">
<Module>
<Name>ZPM</Name>
<Version>0.10.5-SNAPSHOT</Version>
<ExternalName>IPM</ExternalName>
<Description>InterSystems Package Manager (IPM) provides development tools and
infrastructure for defining, building, distributing, and installing modules and
applications.</Description>
<Keywords>Package Manager</Keywords>
<Author>
<Organization>InterSystems Corporation</Organization>
</Author>
<Packaging>module</Packaging>
<SourcesRoot>src</SourcesRoot>
<Resource Name="%IPM.PKG" />
<Resource Name="%IPM.Common.INC" />
<Resource Name="%IPM.Formatting.INC" />
<Resource Name="IPM.Installer.CLS" Preload="true" />
<UnitTest Name="/tests/unit_tests/" Package="Test.PM.Unit" Phase="test" />
<UnitTest Name="/tests/integration_tests/" Package="Test.PM.Integration" Phase="verify" />
<Invoke Class="IPM.Installer" Method="Map" Phase="Reload" When="Before" />
<Invoke Class="IPM.Installer" Method="MapIfLegacy" Phase="Compile" When="After" />
<Invoke Class="IPM.Installer" Method="InstallORASIfMissing" Phase="Activate"
When="After" />
<Invoke Class="%IPM.Main" Method="UpdateLanguageExtensions" />
<Invoke Class="%IPM.Utils.Migration" Method="RunAll">
<Arg>${verbose}</Arg>
</Invoke>

<!-- Requires 2022.1+ due to Embedded Python -->
<SystemRequirements Version=">=2022.1" />
<!-- Requires 2022.1+ due to Embedded Python -->
<SystemRequirements Version=">=2022.1" />

<!-- List of python wheels for offline installation – oras and its transitive dependencies -->
<PythonWheel Name="attrs-25.1.0-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="certifi-2025.1.31-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="charset_normalizer-2.1.1-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="idna-3.10-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="jsonschema-4.23.0-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="jsonschema_specifications-2024.10.1-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="oras-0.1.30-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="referencing-0.36.2-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="requests-2.32.3-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="typing_extensions-4.12.2-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<PythonWheel Name="urllib3-2.3.0-py3-none-any.whl" ExtraPipFlags="--no-deps"/>
<!-- List of python wheels for offline installation – oras and its transitive
dependencies -->
<PythonWheel Name="attrs-25.1.0-py3-none-any.whl" ExtraPipFlags="--no-deps" />
<PythonWheel Name="certifi-2025.1.31-py3-none-any.whl" ExtraPipFlags="--no-deps" />
<PythonWheel Name="charset_normalizer-2.1.1-py3-none-any.whl" ExtraPipFlags="--no-deps" />
<PythonWheel Name="idna-3.10-py3-none-any.whl" ExtraPipFlags="--no-deps" />
<PythonWheel Name="jsonschema-4.23.0-py3-none-any.whl" ExtraPipFlags="--no-deps" />
<PythonWheel Name="jsonschema_specifications-2024.10.1-py3-none-any.whl"
ExtraPipFlags="--no-deps" />
<PythonWheel Name="oras-0.1.30-py3-none-any.whl" ExtraPipFlags="--no-deps" />
<PythonWheel Name="referencing-0.36.2-py3-none-any.whl" ExtraPipFlags="--no-deps" />
<PythonWheel Name="requests-2.32.3-py3-none-any.whl" ExtraPipFlags="--no-deps" />
<PythonWheel Name="typing_extensions-4.12.2-py3-none-any.whl" ExtraPipFlags="--no-deps" />
<PythonWheel Name="urllib3-2.3.0-py3-none-any.whl" ExtraPipFlags="--no-deps" />

<!-- Pure python implementation of rpds-py for offline installation or behind a firewall.
<!-- Pure python implementation of rpds-py for offline installation or behind a
firewall.
This intentionally and necessarily masks possible installation of the real rpds-py package
for the sake of working in a container environment with durable %SYS. -->
<FileCopy Name="modules/python/rpds.py" Target="${mgrdir}python/rpds.py"/>
</Module>
</Document></Export>
<FileCopy Name="modules/python/rpds.py" Target="${mgrdir}python/rpds.py" />
</Module>
</Document>
</Export>
21 changes: 21 additions & 0 deletions src/cls/IPM/CLI.cls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -401,7 +401,9 @@ ClassMethod %HelpForCommandExamples(
/// Parses a command, validating it based on the Commands XData block and structuring output as follows:
/// pCommandInfo = "<command name>"
/// pCommandInfo("modifiers","<modifier-name>") = "<modifier-value>"
/// pCommandInfo("modifiers","<modifier-name>","deprecated") = 0/1
/// pCommandInfo("parameters","<parameter-name>") = "<parameter-value>"
/// pCommandInfo("parameters","<parameter-name>","deprecated") = 0/1
/// pCommandInfo("data","<data-name>") = "<data-value>"
ClassMethod %ParseCommandInput(
pCommandString As %String,
Expand Down Expand Up @@ -651,6 +653,25 @@ ClassMethod %ParseCommandInput(
set tState = tPreEscapeState
}
}

// Add in deprecated info to parameters and modifiers
set tCommandName = $get(pCommandInfo)
if (tCommandName '= "") {
for tType="parameters","modifiers" {
set tKey = ""
while 1 {
set tKey = $order(pCommandInfo(tType,tKey))
if (tKey = "") {
quit
}
// Only include deprecated subscript if command is deprecated
set tIsDeprecated = $get(tCommandStructure(tCommandName,tType,tKey,"deprecated"),0)
if tIsDeprecated {
set pCommandInfo(tType,tKey,"deprecated") = 1
}
}
}
}
if ($get(pCommandInfo) '= "") {
set commandName = pCommandInfo
for i=1:1 {
Expand Down
21 changes: 16 additions & 5 deletions src/cls/IPM/CLI/Commands.cls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ XData Schema [ Internal, MimeType = application/xml ]
<!-- Parameter required? -->
<xs:attribute name="required" type="xs:boolean" use="optional" default="false"/>

<!-- Whether the modifier is deprecated -->
<xs:attribute name="deprecated" type="xs:boolean" use="optional" default="false"/>

<!-- Description of what this parameter does -->
<xs:attribute name="description" type="string" use="optional"/>

Expand Down Expand Up @@ -92,16 +95,19 @@ XData Schema [ Internal, MimeType = application/xml ]
<!-- Modifier name -->
<xs:attribute name="name" type="string" use="required"/>

<!-- Modifier name -->
<!-- Modifier required -->
<xs:attribute name="required" type="xs:boolean" use="optional" default="false"/>

<!-- Whether the modifier is deprecated -->
<xs:attribute name="deprecated" type="xs:boolean" use="optional" default="false"/>

<!-- Comma-separated list of aliases -->
<xs:attribute name="aliases" type="string" use="optional"/>

<!-- If true, the modifier name is followed in the command string by a value -->
<xs:attribute name="value" type="xs:boolean" use="optional" default="false"/>

<!-- If true, the modifier name is followed in the command string by a value -->
<!-- Description of the modifier -->
<xs:attribute name="description" type="string" use="optional"/>

<!-- If set, this provides a fixed list of valid values for the modifier -->
Expand Down Expand Up @@ -159,7 +165,7 @@ XData Schema [ Internal, MimeType = application/xml ]
</xs:schema>
}

/// Turns the Comomand XData block into a subscripted array as follows:
/// Turns the Command XData block into a subscripted array as follows:
/// pCommandStructure(1, "<alias>") = "<command name>"
/// pCommandStructure("<command-name>") ->
/// "description" = describes the command
Expand All @@ -173,11 +179,13 @@ XData Schema [ Internal, MimeType = application/xml ]
/// "aliases" = aliases for the modifier
/// "description" = describes the modifier
/// "required" = whether or not modifier is required for the command
/// "deprecated" = whether or not modifier is deprecated for the command
/// "trailing" = whether modifier is a trailing modifier
/// "parameters":
/// "<parameter>":
/// "description" = description of parameter
/// "required" = is parameter required
/// "deprecated" = whether or not parameter is deprecated for the command
ClassMethod %GetOneCommandStructure(Output pCommandStructure) [ CodeMode = objectgenerator ]
{
#define DefaultGroup "-"
Expand Down Expand Up @@ -315,9 +323,10 @@ ClassMethod %GetOneCommandStructure(Output pCommandStructure) [ CodeMode = objec
do %code.WriteLine(" Set pCommandStructure("_tName_","_$$$QUOTE(tMap(tCommChild))_") = "_tDescContent)
}
} else {
// Common: name, required, description
// Common to bother modifier and parameter
Copy link
Collaborator

@isc-dchui isc-dchui Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: typo?

set tChildName = $$$QUOTE(tMap(tCommChild,"a","name"))
set tRequired = $case($get(tMap(tCommChild,"a","required")),"true":1,:0)
set tDeprecated = $case($get(tMap(tCommChild,"a","deprecated")),"true":1,:0)

if (tMap(tCommChild) = "modifier") {
// Also: aliases, value, valueList
Expand All @@ -337,17 +346,19 @@ ClassMethod %GetOneCommandStructure(Output pCommandStructure) [ CodeMode = objec
do %code.WriteLine(" Set pCommandStructure("_tName_",""modifiers"","_tChildName_",""group"") = "_tGroupName)
do %code.WriteLine(" Set pCommandStructure("_tName_",""modifiers"","_tChildName_",""value"") = "_$case(tValue,"true":1,:0))
do %code.WriteLine(" Set pCommandStructure("_tName_",""modifiers"","_tChildName_",""required"") = "_tRequired)
do %code.WriteLine(" Set pCommandStructure("_tName_",""modifiers"","_tChildName_",""deprecated"") = "_tDeprecated)
do:(tDesc'=$$$empty) %code.WriteLine(" Set pCommandStructure("_tName_",""modifiers"","_tChildName_",""description"") = "_tDesc)
do:(tValueList'=$$$empty) %code.WriteLine(" Set pCommandStructure("_tName_",""modifiers"","_tChildName_",""valueList"") = "_tValueList)
do:(tDataAlias'=$$$empty) %code.WriteLine(" Set pCommandStructure("_tName_",""modifiers"","_tChildName_",""dataAlias"") = "_tDataAlias)
do:(tDataValue'=$$$empty) %code.WriteLine(" Set pCommandStructure("_tName_",""modifiers"","_tChildName_",""dataValue"") = "_tDataValue)
} elseif (tMap(tCommChild) = "parameter") {
} elseif (tMap(tCommChild) = "parameter") {
// Also: example
set tExample = $$$QUOTE($get(tMap(tCommChild,"a","example")))
set tTrailing = $case($get(tMap(tCommChild,"a","trailing")),"true":1,:0)
do %code.WriteLine(" Set pCommandStructure("_tName_",""parameters"",$i(pCommandStructure("_tName_",""parameters""))) = "_tChildName)
do %code.WriteLine(" Set pCommandStructure("_tName_",""parameters"","_tChildName_",""trailing"") = "_tTrailing)
do %code.WriteLine(" Set pCommandStructure("_tName_",""parameters"","_tChildName_",""required"") = "_tRequired)
do %code.WriteLine(" Set pCommandStructure("_tName_",""parameters"","_tChildName_",""deprecated"") = "_tDeprecated)
do:(tDesc'=$$$empty) %code.WriteLine(" Set pCommandStructure("_tName_",""parameters"","_tChildName_",""description"") = "_tDesc)
do:(tExample'=$$$empty) %code.WriteLine(" Set pCommandStructure("_tName_",""parameters"","_tChildName_",""example"") = "_tExample)
}
Expand Down
72 changes: 0 additions & 72 deletions src/cls/IPM/General/InstallContext.cls
View file
Open in desktop

This file was deleted.

23 changes: 2 additions & 21 deletions src/cls/IPM/Lifecycle/Base.cls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -900,10 +900,6 @@ Method %Compile(ByRef pParams) As %Status
$$$ThrowOnError(##class(%Studio.SourceControl.Interface).SourceControlCreate())
}

// Load dependencies correctly for the requested phases. (This will be a no-op in most cases.)
set tSC = ..Module.LoadDependencies(..PhaseList,.pParams)
$$$ThrowOnError(tSC)

// Compile items within the module that are compilable using OnPhase
set orderedResourceList = ..Module.GetOrderedResourceList()
set tKey = ""
Expand Down Expand Up @@ -1018,13 +1014,6 @@ Method %Compile(ByRef pParams) As %Status
}

set tNoCompileLock = '$get(pParams("NoLock"))
if $$$IsISCWorker && 'tNoCompileLock {
// Customized locking - takes out shared locks on dependencies to avoid conflicts between parallel module loads.
set tSC = ##class(%IPM.Utils.LockManager).LockClassesRecursive(.tLockManager,.tCompileArray,.tLockedClasses)
$$$ThrowOnError(tSC)
set tNoCompileLock = 1
}

// Actually do the compile
if $data(tCompileArray)>1 {
set tCompiledSomething = 1
Expand Down Expand Up @@ -1078,10 +1067,6 @@ Method %Test(ByRef pParams) As %Status
{
set tSC = $$$OK
try {
// Load dependencies correctly for the requested phases. (This will be a no-op in most cases.)
set tSC = ..Module.LoadDependencies(..PhaseList,.pParams)
$$$ThrowOnError(tSC)

set orderedResourceList = ..Module.GetOrderedResourceList()
set tKey = ""
for {
Expand Down Expand Up @@ -1129,7 +1114,7 @@ Method %Export(
}

set tExportDependencies = $get(pParams("ExportDependencies"), ..#EXPORTDEPENDENCIES)
set tSC = ..Module.GetResolvedReferences(.tResourceArray, tExportDependencies,..PhaseList,1,.pDependencyGraph)
set tSC = ..Module.GetResolvedReferences(.tResourceArray, tExportDependencies,..PhaseList,.pDependencyGraph)
if $$$ISERR(tSC) {
quit
}
Expand Down Expand Up @@ -1308,10 +1293,6 @@ Method %Verify(ByRef pParams) As %Status
}
}

// Load dependencies correctly for the requested phases. (This will be a no-op in most cases.)
set tSC = ..Module.LoadDependencies(..PhaseList,.pParams)
$$$ThrowOnError(tSC)

set orderedResourceList = ..Module.GetOrderedResourceList()
set tKey = ""
for {
Expand Down Expand Up @@ -1592,7 +1573,7 @@ Method %MakeDeployed(ByRef pParams) As %Status

// Default implementation: see which resources are expicitly flagged with Deploy = true.
// Build an array of those, then mark them as deployed.
$$$ThrowOnError(..Module.GetResolvedReferences(.tResourceArray,tLockedDependencies,..PhaseList,1,.pDependencyGraph))
$$$ThrowOnError(..Module.GetResolvedReferences(.tResourceArray,tLockedDependencies,..PhaseList,.pDependencyGraph))

set tModuleKey = ""
for {
Expand Down
Loading