Skip to content

Conversation

hiyuh
Copy link
Contributor

@hiyuh hiyuh commented Sep 10, 2025

  • SSIA.
    • included commits from Dispose Scalars implicitly created in Tensor operators #1434.
    • newly introduced TorchSharp.{Scalar,Tensor}LeakDetector can throw exceptions on implicit conversion.
      • if enabled, it would show stack trace for possible missing {TorchSharp.Scalar,torch.Tensor}.Dispose.
      • i'm not sure whether these leak detectors need retain even after all fixes in TorchSharp have been made though.
    • real fix for missing {TorchSharp.Scalar,torch.Tensor}.Dispose in TorchSharp is on going.
  • CC: @ds5678

masaru-kimura-hacarus and others added 16 commits September 10, 2025 15:18
Otherwise, dotnet build will fail.
In the following case, at least 266 exceptions are observed.
* allowImplicitConversionOperator = false
* dotnet test /p:SkipCuda=true /p:SkipNetFxBuild=true --blame test\TorchSharpTest\TorchSharpTest.csproj -c Release

* Update src/TorchSharp/Scalar.cs.
  + Introduce ScalarLeakDetector.
  + Update Scalar.
    - Use ScalarLeakDetector.ThrowIfImplicitConversionNotAllowed.
In the following case, at least 45 exceptions are observed.
* allowImplicitConversionOperator = false
* dotnet test /p:SkipCuda=true /p:SkipNetFxBuild=true --blame test\TorchSharpTest\TorchSharpTest.csproj -c Release

* Update src/TorchSharp/Tensor/Tensor.cs
  + Introduce TensorLeakDetector.
  + Update Tensor.
    - Use TensorLeakDetector.ThrowIfImplicitConversionNotAllowed.
* Update src/TorchSharp/Tensor/Tensor.Operators.cs.
  + Declare TorchSharp.Scalar more explicitly.
    - Use prefix for left or right.
    - Call ToScalar explicitly.
* Update src/TorchSharp/Tensor/Tensor.Math.cs.
  + Declare TorchSharp.Scalar explicitly.
  + Add FIXMEs.
* Update src/TorchSharp/Optimizers/Adadelta.cs.
  + Update Adadelta.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0 explicitly.
* Update src/TorchSharp/Optimizers/Adagrad.cs.
  + Update Adagrad.step.
    + Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    + Add FIXME for possible unsued initial_accumulator_value.
    + Cache weight_decay != 0.
* Update src/TorchSharp/Optimizers/Adam.cs.
  + Update Adam.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0.
    - Add FIXME for possible no denom disposing.
* Update src/TorchSharp/Optimizers/Adamax.cs.
  + Update Adamax.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0.
    - Add FIXME for CA1806.
* Update src/TorchSharp/Optimizers/ASGD.cs.
  + Update ASGD.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unsed weight_decay_scalar.
    - Cache weight_decay != 0.
* Update src/TorchSharp/Optimizers/NAdam.cs.
  + Update NAdam.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0.
    - Add FIXME for possible no denom disposing.
* Update src/TorchSharp/Optimizers/RAdam.cs.
  + Update RAdam.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0.
    - Add FIXME for possible torch.Tensor.sub_ use.
    - Add FIXME for possible no dispose for torch.Tensor.
      - bias_corrected_exp_avg
      - t6
      - adaptive_lr and its intermediates and derives
    - Add FIXME for possible no dispose on param.add_ if rho_t > 5.
* Update src/TorchSharp/Optimizers/RMSprop.cs.
  + Update RMSProp.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused momentum_scalar.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache momentum > 0.
    - Cache weight_decay != 0.
    - Add FIXME for possible no avg dispose.
* Update src/TorchSharp/Optimizers/SGD.cs.
  + Update SGD.step.
    - Declare TorchSharp.Scalar explicitly.
    - Cache momentum != 0.
    - Cache dampening != 1.
    - Cache weight_decay != 0.
    - Omit unused TorchSharp.Scalar construction.
@hiyuh hiyuh force-pushed the dispose-implicitly-converted-scalar-tensor branch from 4a7cbd2 to 64520bc Compare September 11, 2025 09:04
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 11, 2025

  • nuked torch.Tensor.add(torch.Tensor, Scalar) w/ implicit conversion to TorchSharp.Scalar.
    image
  • i'll continue same kind fix based on "Find All References" by Visual Studio.

* Update src/TorchAudio/Functional.cs.
  + Update griffinlim.
    - Declare TorchSharp.Scalar explicitly.
      - Introduce eps_scalar.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 11, 2025

  • nuked torch.Tensor.add(Scalar) w/ implicit conversion to TorchSharp.Scalar.
    image
    • intentionally, i won't touch any unit tests if possible, since leaking due to test code incorrectness is not critical, as long as CI/CD passed.

* Update src/TorchSharp/Optimizers/AdamW.cs.
  + Update AdamW.step.
    - Declare TorchSharp.Scalar explicitly.
    - Dispose denom explicitly.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 11, 2025

  • ditto for torch.Tensor.add_(torch.Tensor, Scalar).
    image

* Update src/TorchSharp/NN/Normalization/BatchNorm.cs.
  + Update BatchNorm.forward.
    - Declare TorchSharp.Scalar explicitly.
    - Add FIXME for cache over training.
* Update src/TorchSharp/Tensor/Factories/Tensor.Factories.cs.
  + Update torch.normal.
    - Declare TorchSharp.Scalar explicitly.
* Update src/TorchVision/Utils.cs.
  + Update torchvision.utils.save_image.
    - Declare TorchSharp.Scalar explicitly.
    - Add FIXME for possible torch.Tensor.round_ use.
    - Add FIXME for no torch.min_int_value.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 12, 2025

  • ditto for torch.Tensor.add_(Scalar).
    image
  • CI/CD errors on Azure DevOps are irrelevant to this MR.
    • Windows_x64_NetFX Release_Build fails on "Run Tests" due to network connection closed by remote host.
    • MacOS_arm64 Release_Build fails on "Build" due to cmake_minimum_required doesn't met.

* Update src/TorchSharp/Optimizers/Rprop.cs.
  + Update Rprop.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for unused lr.
    - Add FIXME for possible torch.Tensor.sign_ use.
    - Cache eta{minus,plus} and 1 as torch.Tensor.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 12, 2025

  • ditto for torch.Tensor.addcmul_(Tensor, Tensor, Scalar).
    image

* Update src/TorchVision/Ops/StochasticDepth.cs.
  + Update torchvision.ops.stochastic_depth.
    - Declare TorchSharp.Scalar explicitly.
* Update src/TorchVision/Utils.cs.
  + Update torchvision.utils.make_grid.
    - Declare TorchSharp.Scalar explicitly.
* Update src/TorchSharp/Distributions/Constraints.cs.
  + Update torch.distributions.constraints._PositiveSemiDefinite.check.
    - Declare TorchSharp.Scalar explicitly.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 16, 2025

  • ditto for torch.Tensor.ge(Scalar).
    image

* Update src/TorchSharp/Distributions/Constraints.cs.
  + Update torch.distributions.constraints._CorrCholesky.check.
    - Declare TorchSharp.Scalar explicitly.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 16, 2025

  • ditto for torch.Tensor.le(Scalar).
    image

* Update src/Examples/SequenceToSequence.cs.
  + Update TransformerModel.GenerateSquareSubsequentMask.
    - Declare TorchSharp.Scalar explicitly.
* Update src/TorchAudio/Modules/Tacotron2.cs.
  + Update TorchSharp.Modules.Tacotron2.forward.
    - Declare TorchSharp.Scalar explicitly.
* Update src/TorchAudio/Modules/Tacotron2.cs.
  + Update TorchSharp.Modules.Tacotron2.Attention.forward.
    - Declare TorchSharp.Scalar explicitly.
* Update src/TorchSharp/Distributions/NegativeBinomial.cs.
  + Update TorchSharp.Modules.NegativeBinomial.log_prob.
    - Declare TorchSharp.Scalar explicitly.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 16, 2025

  • ditto for torch.Tensor.masked_fill(torch.Tensor, Scalar).
    image

This is an exceptional change.
Since comparison operators are widely used, I'd have to give up fix one
by one for now.
Introducing operators take {int,long,float,double} would cover most cases
to prevent missing TorchSharp.Scalar.Dispose.
However, these would leave TorchSharp.Scalar construction cost as is.

* Update src/TorchSharp/Tensor/Tensor.cs.
  + Add more operator ==.
  + Add more operator !=.
  + Add more operator <.
  + Add more operator <=.
  + Add more operator >.
  + Add more operator >=.
Support other types which are covered by implicit conversion to
TorchSharp.Scalar; byte, sbyte, short, Half, bool, (float, float),
System.Numerics.Complex.

* Update src/TorchSharp/Tensor/Tensor.cs.
  + Add more operator ==.
  + Add more operator !=.
  + Add more operator <.
  + Add more operator <=.
  + Add more operator >.
  + Add more operator >=.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Call PrintValue w/ explicitly declared TorchSharp.Scalar.
* Update src/TorchSharp/Tensor/TensorExtensionMethods.cs.
  + Update TensorExtensionMethods.To*(this Tensor value).
    - Declare TorchSharp.Scalar explicitly.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.fill_ overloads.
* Update src/TorchSharp/Optimizers/Rprop.cs.
  + Update TorchSharp.Modules.Rprop.step.
    - Cosmetic.
    - Use torch.Tensor.masked_fill_.
  + Update TorchSharp.Modules.Rprop.State.Initialize.
    - Use a torch.Tensor.fill_ overload.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.index_put_ overloads.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.index_add{,_} overloads.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.index_fill{,_} overloads.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.threshold{,_} overloads.
* Update src/TorchSharp/NN/Activation/Threshold.cs.
  + Update torch.nn.functional.threshold.
    - Use torch.Tensor.threshold{,_} overloads.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.softplus overloads.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Add more torch.Tensor.celu{,_} overloads.
* Update src/TorchSharp/NN/Activation/CELU.cs.
  + Update torch.nn.functional.celu.
    - Use torch.Tensor.celu{,_} overloads.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Add more torch.Tensor.elu{,_} overloads.
* Update src/TorchSharp/NN/Activation/Hardtanh.cs.
  + Introduce torch.Tensor.hardtanh{,_} overloads.
* Update src/TorchSharp/NN/Activation/Hardtanh.cs.
  + Update torch.nn.functional.hardtanh.
    - Use torch.Tensor.hardtanh{,_} overloads.
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.leaky_relu{,_} overloads.
* Update src/TorchSharp/NN/Activation/LeakyReLU.cs.
  + Update torch.nn.functional.leaky_relu.
    - Use torch.Tensor.leaky_relu{,_} overloads.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 18, 2025

  • introducing overloads on callee side, rather than fix on caller side.
    • although it might be suboptimal (sometimes, caller side could be reorganized with explicitly cached scalars/tensors), changeset would be consist (lower review cost) and conservative (more robust against uncareful caller side).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants