Using generic implemntation for 16-bit activations and 8 bit weights for Conv2D in Backends #16007

RahulC7 · 2025-11-28T20:58:46Z

Summary:

Context

We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions.

Current Behavior

Right now, we're composing two macros together, the ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16 macro:

https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25

and the function macro(quantized_linear chosen for example):

https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41

so together, it just becomes a switch statement, calling the quantized_linear function with the correct template parameter.

However, note that it assumes that both the input activations and weights are the same dtype, which is not the case.

This Diff

We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests.

Differential Revision: D87993325

pytorch-bot · 2025-11-28T20:58:50Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16007

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d9f4835 with merge base e38734e ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-codesync · 2025-11-28T20:58:56Z

@RahulC7 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D87993325.

Copilot

Pull request overview

This PR adds support for 16-bit activations with 8-bit weights (W8A16) in quantized operations for the Cadence HiFi backend. Previously, 16-bit support required both activations and weights to be 16-bit.

Extends quantized_linear_out, quantized_linear_per_tensor_out, quantized_conv2d_nchw_out, and quantized_conv2d_nhwc_out to handle W8A16 heterogeneous types by delegating to generic implementations
Adds comprehensive unit tests for the new W8A16 support in linear and conv2d operations
Updates build configuration to include necessary dependencies for generic implementations

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
backends/cadence/hifi/operators/tests/test_op_quantized_linear_out.cpp	Adds unit test for quantized linear with int16 activations and int8 weights
backends/cadence/hifi/operators/tests/test_op_quantized_conv2d_out.cpp	Adds unit tests for quantized conv2d (NCHW/NHWC) with int16 activations and int8 weights
backends/cadence/hifi/operators/targets.bzl	Updates build targets to add generic implementation dependencies for W8A16 support
backends/cadence/hifi/operators/op_quantized_linear_out.cpp	Adds W8A16 type check and delegates to generic implementation for linear operations
backends/cadence/hifi/operators/op_quantized_conv2d_nhwc_out.cpp	Adds W8A16 type check and delegates to generic implementation for NHWC conv2d
backends/cadence/hifi/operators/op_quantized_conv2d_nchw_out.cpp	Adds W8A16 type check and delegates to generic implementation for NCHW conv2d
backends/cadence/aot/quantizer/quantizer.py	Adds new quantizer class for 16-bit conv activations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-28T20:59:20Z

backends/cadence/hifi/operators/op_quantized_linear_out.cpp

+  }
+  else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {


[nitpick] The else if should be on the same line as the closing brace according to standard C++ formatting conventions. Change to } else if for consistency with typical C++ style.

Suggested change

}

else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {

} else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {

Copilot · 2025-11-28T20:59:21Z

backends/cadence/hifi/operators/op_quantized_linear_out.cpp

+  }
+  else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {


[nitpick] The else if should be on the same line as the closing brace according to standard C++ formatting conventions. Change to } else if for consistency with typical C++ style.

Suggested change

}

else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {

} else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {

…for Conv2D in Backends (pytorch#16007) Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests. Differential Revision: D87993325

github-actions · 2025-12-01T21:36:25Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…for Conv2D in Backends (pytorch#16007) Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests. Differential Revision: D87993325

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-01T22:07:41Z

backends/cadence/hifi/operators/targets.bzl

+    define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators/generic:op_quantized_linear"])
+    define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators/generic:op_quantized_linear"])
+
+      # quantized_conv2d_nchw_out and quantized_conv2d_nhwc_out need additional dependency for int16 support


Incorrect indentation: This line has extra leading spaces (6 spaces) compared to the surrounding code (4 spaces). The comment should be aligned with the other comments in the file.

Suggested change

# quantized_conv2d_nchw_out and quantized_conv2d_nhwc_out need additional dependency for int16 support

# quantized_conv2d_nchw_out and quantized_conv2d_nhwc_out need additional dependency for int16 support

Copilot · 2025-12-01T22:07:41Z

backends/cadence/hifi/operators/op_quantized_linear_out.cpp

+  if (out.scalar_type() == ::executorch::aten::ScalarType::Short &&
+      in.scalar_type() == ::executorch::aten::ScalarType::Short &&
+      weight.scalar_type() == ::executorch::aten::ScalarType::Char) {
+    ::impl::generic::native::quantized_linear_out(
+        ctx,
+        in,
+        weight,
+        bias,
+        in_zero_point,
+        weight_zero_point,
+        out_multiplier,
+        out_shift,
+        out_zero_point,
+        offset,
+        out);


[nitpick] Consider adding an explicit return; statement after line 236 to make the control flow clearer and more consistent with the conv2d implementations (e.g., op_quantized_conv2d_nhwc_out.cpp:463). This would also allow changing the subsequent else if statements to simple if statements.

Copilot · 2025-12-01T22:07:42Z

backends/cadence/hifi/operators/op_quantized_linear_out.cpp

+  if (out.scalar_type() == ::executorch::aten::ScalarType::Short &&
+      in.scalar_type() == ::executorch::aten::ScalarType::Short &&
+      weight.scalar_type() == ::executorch::aten::ScalarType::Char) {
+    ::impl::generic::native::quantized_linear_per_tensor_out(
+        ctx,
+        in,
+        weight,
+        bias,
+        in_zero_point,
+        weight_zero_point,
+        out_multiplier,
+        out_shift,
+        out_zero_point,
+        offset,
+        out);


[nitpick] Consider adding an explicit return; statement after line 293 to make the control flow clearer and more consistent with the conv2d implementations (e.g., op_quantized_conv2d_nhwc_out.cpp:463). This would also allow changing the subsequent else if statements to simple if statements.

Copilot · 2025-12-01T22:07:42Z

backends/cadence/hifi/operators/tests/test_op_quantized_linear_out.cpp

+  }
+};
+
+// Test quantized_linear_out with int16 activations (asym8s)


The comment incorrectly mentions "(asym8s)" which typically refers to asymmetric 8-bit signed integers. However, this test uses int16 activations with int8 weights (W8A16). Consider updating the comment to: "Test quantized_linear_out with int16 activations and int8 weights (W8A16)"

Suggested change

// Test quantized_linear_out with int16 activations (asym8s)

// Test quantized_linear_out with int16 activations and int8 weights (W8A16)

… for linear in backends (pytorch#15997) Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests. Reviewed By: hsharma35 Differential Revision: D87946776

…for Conv2D in Backends (pytorch#16007) Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We finish by using the generic implementation for all the backends and adding e2e tests as well as unit tests. Reviewed By: hsharma35 Differential Revision: D87993325

Copilot AI review requested due to automatic review settings November 28, 2025 20:58

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 28, 2025

meta-codesync bot added fb-exported meta-exported labels Nov 28, 2025

Copilot AI reviewed Nov 28, 2025

View reviewed changes

RahulC7 force-pushed the export-D87993325 branch from a8e37d4 to dcc29dd Compare December 1, 2025 21:35

Copilot AI review requested due to automatic review settings December 1, 2025 22:02

RahulC7 force-pushed the export-D87993325 branch from dcc29dd to 7392370 Compare December 1, 2025 22:02

Copilot started reviewing on behalf of RahulC7 December 1, 2025 22:03 View session

Copilot finished reviewing on behalf of RahulC7 December 1, 2025 22:06

Copilot AI reviewed Dec 1, 2025

View reviewed changes

hsharma35 approved these changes Dec 1, 2025

View reviewed changes

RahulC7 added 2 commits December 2, 2025 09:35

RahulC7 force-pushed the export-D87993325 branch from 7392370 to d9f4835 Compare December 2, 2025 17:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Using generic implemntation for 16-bit activations and 8 bit weights for Conv2D in Backends #16007

Using generic implemntation for 16-bit activations and 8 bit weights for Conv2D in Backends #16007

RahulC7 commented Nov 28, 2025

Uh oh!

pytorch-bot bot commented Nov 28, 2025 •

edited

Loading

Uh oh!

meta-codesync bot commented Nov 28, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 28, 2025

Uh oh!

Copilot AI Nov 28, 2025

Uh oh!

github-actions bot commented Dec 1, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 1, 2025

Uh oh!

Copilot AI Dec 1, 2025

Uh oh!

Copilot AI Dec 1, 2025

Uh oh!

Copilot AI Dec 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		}
		else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {

	}
	else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {
	} else if (out.scalar_type() == executorch::aten::ScalarType::Byte) {

	# quantized_conv2d_nchw_out and quantized_conv2d_nhwc_out need additional dependency for int16 support
	# quantized_conv2d_nchw_out and quantized_conv2d_nhwc_out need additional dependency for int16 support

	// Test quantized_linear_out with int16 activations (asym8s)
	// Test quantized_linear_out with int16 activations and int8 weights (W8A16)

Using generic implemntation for 16-bit activations and 8 bit weights for Conv2D in Backends #16007

Are you sure you want to change the base?

Using generic implemntation for 16-bit activations and 8 bit weights for Conv2D in Backends #16007

Conversation

RahulC7 commented Nov 28, 2025

Context

Current Behavior

This Diff

Uh oh!

pytorch-bot bot commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16007

✅ No Failures

Uh oh!

meta-codesync bot commented Nov 28, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 1, 2025

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pytorch-bot bot commented Nov 28, 2025 •

edited

Loading

This PR needs a `release notes:` label