Using generic implementation for 16-bit activations and 8 bit weights for linear in backends #15997

RahulC7 · 2025-11-26T22:12:15Z

Summary:

Context

We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions.

Current Behavior

Right now, we're composing two macros together, the ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16 macro:

https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25

and the function macro(quantized_linear chosen for example):

https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41

so together, it just becomes a switch statement, calling the quantized_linear function with the correct template parameter.

However, note that it assumes that both the input activations and weights are the same dtype, which is not the case.

This Diff

We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests.

Differential Revision: D87946776

pytorch-bot · 2025-11-26T22:12:19Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15997

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

Dr CI is temporarily not working due to API fairewall

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-codesync · 2025-11-26T22:12:31Z

@RahulC7 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D87946776.

Copilot

Pull request overview

This PR adds support for 16-bit activations with 8-bit weights in quantized linear operations across backends. Previously, the implementation required both activations and weights to have matching data types.

Key changes:

Added conditional handling for int16 activation + int8 weight combinations using generic implementations
Added unit tests for the new int16 activation support in the HiFi backend
Updated build configuration to include necessary dependencies for int16 support

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
backends/cadence/hifi/operators/tests/test_op_quantized_linear_out.cpp	New test file validating int16 activation quantized linear operations
backends/cadence/hifi/operators/targets.bzl	Updated build targets to add dependencies for int16 support in quantized linear operators
backends/cadence/hifi/operators/op_quantized_linear_out.cpp	Added conditional logic to dispatch to generic implementation for int16 activations with int8 weights

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-26T22:12:46Z

backends/cadence/hifi/operators/targets.bzl

        define_operator(op)
+
+    # quantized_linear_out and quantized_linear_per_tensor_out needs additional dependency for int16 support
+    define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])


Trailing comma after the last list element should be removed for consistency with Python style conventions.

Suggested change

define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])

define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers"])

Copilot · 2025-11-26T22:12:47Z

backends/cadence/hifi/operators/targets.bzl

+
+    # quantized_linear_out and quantized_linear_per_tensor_out needs additional dependency for int16 support
+    define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])
+    define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])


Trailing comma after the last list element should be removed for consistency with Python style conventions.

Suggested change

define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])

define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers"])

… for linear in backends (pytorch#15997) Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests. Reviewed By: hsharma35 Differential Revision: D87946776

github-actions · 2025-12-01T22:00:04Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

… for linear in backends (pytorch#15997) Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests. Reviewed By: hsharma35 Differential Revision: D87946776

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-01T22:04:48Z

backends/cadence/hifi/operators/op_quantized_linear_out.cpp

+  if (out.scalar_type() == ::executorch::aten::ScalarType::Short &&
+      in.scalar_type() == ::executorch::aten::ScalarType::Short &&
+      weight.scalar_type() == ::executorch::aten::ScalarType::Char) {
+    ::impl::generic::native::quantized_linear_per_tensor_out(
+        ctx,
+        in,
+        weight,
+        bias,
+        in_zero_point,
+        weight_zero_point,
+        out_multiplier,
+        out_shift,
+        out_zero_point,
+        offset,
+        out);
+  } else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {


Inconsistent namespace qualification. Lines 279-281 use fully qualified ::executorch::aten::ScalarType:: while line 294 uses just executorch::aten::ScalarType:: (without leading ::). For consistency and clarity, use the same namespace qualification pattern throughout the function. The fully qualified form (with leading ::) is preferred to avoid potential ambiguity.

Copilot · 2025-12-01T22:04:49Z

backends/cadence/hifi/operators/tests/test_op_quantized_linear_out.cpp

+  }
+};
+
+// Test quantized_linear_out with int16 activations (asym8s)


The comment "Test quantized_linear_out with int16 activations (asym8s)" is confusing. The term "asym8s" typically refers to asymmetric 8-bit signed quantization, but this test is for 16-bit activations with 8-bit weights. Consider updating the comment to clarify the actual quantization scheme, e.g., "Test quantized_linear_out with int16 activations and int8 weights".

Suggested change

// Test quantized_linear_out with int16 activations (asym8s)

// Test quantized_linear_out with int16 activations and int8 weights

Copilot · 2025-12-01T22:04:49Z

backends/cadence/hifi/operators/targets.bzl

+
+    # quantized_linear_out and quantized_linear_per_tensor_out needs additional dependency for int16 support
+    define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators/generic:op_quantized_linear"])
+    define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators/generic:op_quantized_linear"])


The define_operator("quantized_linear_per_tensor_out", ...) call will look for a source file named op_quantized_linear_per_tensor_out.cpp (based on line 27 which uses op_name + ".cpp"), but this file doesn't exist. The quantized_linear_per_tensor_out function is defined in op_quantized_linear_out.cpp along with quantized_linear_out. This will cause a build failure. Consider either:

Removing this line if quantized_linear_per_tensor_out is only exported as part of the quantized_linear_out target, or

Creating a separate op_quantized_linear_per_tensor_out.cpp file if it should be a separate build target.

Suggested change

define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators/generic:op_quantized_linear"])

Copilot · 2025-12-01T22:04:49Z

backends/cadence/hifi/operators/op_quantized_linear_out.cpp

+  if (out.scalar_type() == ::executorch::aten::ScalarType::Short &&
+      in.scalar_type() == ::executorch::aten::ScalarType::Short &&
+      weight.scalar_type() == ::executorch::aten::ScalarType::Char) {
+    ::impl::generic::native::quantized_linear_out(
+        ctx,
+        in,
+        weight,
+        bias,
+        in_zero_point,
+        weight_zero_point,
+        out_multiplier,
+        out_shift,
+        out_zero_point,
+        offset,
+        out);
+  } else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {


Inconsistent namespace qualification. Lines 222-224 use fully qualified ::executorch::aten::ScalarType:: while line 237 uses just executorch::aten::ScalarType:: (without leading ::). For consistency and clarity, use the same namespace qualification pattern throughout the function. The fully qualified form (with leading ::) is preferred to avoid potential ambiguity.

Copilot AI review requested due to automatic review settings November 26, 2025 22:12

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 26, 2025

meta-codesync bot added fb-exported meta-exported labels Nov 26, 2025

Copilot AI reviewed Nov 26, 2025

View reviewed changes

RahulC7 force-pushed the export-D87946776 branch from c2a6831 to 2b47352 Compare November 28, 2025 21:04

Copilot AI review requested due to automatic review settings December 1, 2025 21:59

RahulC7 force-pushed the export-D87946776 branch from 2b47352 to a68941f Compare December 1, 2025 21:59

Copilot started reviewing on behalf of RahulC7 December 1, 2025 21:59 View session

Copilot finished reviewing on behalf of RahulC7 December 1, 2025 22:03

Copilot AI reviewed Dec 1, 2025

View reviewed changes

hsharma35 approved these changes Dec 1, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Using generic implementation for 16-bit activations and 8 bit weights for linear in backends #15997

Using generic implementation for 16-bit activations and 8 bit weights for linear in backends #15997

RahulC7 commented Nov 26, 2025

Uh oh!

pytorch-bot bot commented Nov 26, 2025 •

edited

Loading

Uh oh!

meta-codesync bot commented Nov 26, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 26, 2025

Uh oh!

Copilot AI Nov 26, 2025

Uh oh!

github-actions bot commented Dec 1, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 1, 2025

Uh oh!

Copilot AI Dec 1, 2025

Uh oh!

Copilot AI Dec 1, 2025

Uh oh!

Copilot AI Dec 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])
	define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers"])

	define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])
	define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers"])

	// Test quantized_linear_out with int16 activations (asym8s)
	// Test quantized_linear_out with int16 activations and int8 weights

Using generic implementation for 16-bit activations and 8 bit weights for linear in backends #15997

Are you sure you want to change the base?

Using generic implementation for 16-bit activations and 8 bit weights for linear in backends #15997

Conversation

RahulC7 commented Nov 26, 2025

Context

Current Behavior

This Diff

Uh oh!

pytorch-bot bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15997

❗ 1 Active SEVs

Uh oh!

meta-codesync bot commented Nov 26, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 1, 2025

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pytorch-bot bot commented Nov 26, 2025 •

edited

Loading

This PR needs a `release notes:` label