[X86] Improve illegal return type handling in FastISel by LuoYuanke · Pull Request #186723 · llvm/llvm-project

LuoYuanke · 2026-03-16T01:52:02Z

Previously, FastISel would fall back to DAG ISel for any illegal return
type. This change adds a more precise check to determine if the ABI
requires a type conversion that FastISel cannot handle.

For example, bfloat is returned as f16 in XMM0, but FastISel would
assign f32 register type and store it in FuncInfo.ValueMap, causing DAG
to incorrectly perform type conversion from f32 to bfloat later.

However, i1 is promoted to i8 and returned as i8 per the ABI, so FastISel
can safely lower it without switching to DAGISel. This change enables
FastISel to handle such cases properly.

Previously, FastISel would fall back to DAG ISel for any illegal return type. This change adds a more precise check to determine if the ABI requires a type conversion that FastISel cannot handle. For example, bfloat is returned as f16 in XMM0, but FastISel would assign f32 register type and store it in FuncInfo.ValueMap, causing DAG to incorrectly perform type conversion from f32 to bfloat later. However, i1 is promoted to i8 and returned as i8 per the ABI, so FastISel can safely lower it without switching to DAGISel. This change enables FastISel to handle such cases properly.

llvmbot · 2026-03-16T01:52:34Z

@llvm/pr-subscribers-backend-x86

Author: Luo Yuanke (LuoYuanke)

Changes

Previously, FastISel would fall back to DAG ISel for any illegal return
type. This change adds a more precise check to determine if the ABI
requires a type conversion that FastISel cannot handle.

For example, bfloat is returned as f16 in XMM0, but FastISel would
assign f32 register type and store it in FuncInfo.ValueMap, causing DAG
to incorrectly perform type conversion from f32 to bfloat later.

However, i1 is promoted to i8 and returned as i8 per the ABI, so FastISel
can safely lower it without switching to DAGISel. This change enables
FastISel to handle such cases properly.

Full diff: https://github.com/llvm/llvm-project/pull/186723.diff

2 Files Affected:

(modified) llvm/lib/Target/X86/X86FastISel.cpp (+18-4)
(added) llvm/test/CodeGen/X86/i1-fast-isel.ll (+106)

diff --git a/llvm/lib/Target/X86/X86FastISel.cpp b/llvm/lib/Target/X86/X86FastISel.cpp
index 0093a6b29d226..67e5a2ec869c4 100644
--- a/llvm/lib/Target/X86/X86FastISel.cpp
+++ b/llvm/lib/Target/X86/X86FastISel.cpp
@@ -3200,11 +3200,25 @@ bool X86FastISel::fastLowerCall(CallLoweringInfo &CLI) {
   bool Is64Bit        = Subtarget->is64Bit();
   bool IsWin64        = Subtarget->isCallingConvWin64(CC);
 
-  // If the return type is illegal, don't bother to promote it, just fall back
-  // to DAG ISel.
+  // If the return type is illegal, check if the ABI requires a type conversion
+  // that FastISel cannot handle. Fall back to DAG ISel in such cases.
+  // For example, bfloat is returned as f16 in XMM0, however FastISel would
+  // assign f32 register type and store it in FuncInfo.ValueMap. This would
+  // cause DAG incorrectly perform type conversion from f32 to bfloat after get
+  // the value from FuncInfo.ValueMap.
+  // However, i1 is promoted to i8 and return i8 defined by ABI, so FastISel can
+  // lower it without switching to DAGISel.
   MVT RetVT;
-  if (!isTypeLegal(CLI.RetTy, RetVT) && !CLI.RetTy->isVoidTy())
-    return false;
+  if (!isTypeLegal(CLI.RetTy, RetVT) && !CLI.RetTy->isVoidTy()) {
+    EVT RetEVT = EVT::getEVT(CLI.RetTy, /*HandleUnknown=*/true);
+    if (RetEVT == MVT::Other)
+      return false; // Unknown type, let DAG ISel handle it.
+    EVT RetABIEVT = TLI.getRegisterTypeForCallingConv(CLI.RetTy->getContext(),
+                                                      CLI.CallConv, RetEVT);
+    MVT RegVT = TLI.getRegisterType(CLI.RetTy->getContext(), RetEVT);
+    if (RetABIEVT != RegVT)
+      return false;
+  }
 
   // Call / invoke instructions with NoCfCheck attribute require special
   // handling.
diff --git a/llvm/test/CodeGen/X86/i1-fast-isel.ll b/llvm/test/CodeGen/X86/i1-fast-isel.ll
new file mode 100644
index 0000000000000..1f111129de13c
--- /dev/null
+++ b/llvm/test/CodeGen/X86/i1-fast-isel.ll
@@ -0,0 +1,106 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc --fast-isel < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
+
+define i8 @test_direct_call(ptr %f) nounwind {
+; CHECK-LABEL: test_direct_call:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    callq foo@PLT
+; CHECK-NEXT:    movzbl %al, %edi
+; CHECK-NEXT:    callq bar@PLT
+; CHECK-NEXT:    popq %rcx
+; CHECK-NEXT:    retq
+entry:
+  %call = call i1 @foo(ptr %f)
+  %call2 = call zeroext i8 @bar(i1 %call)
+  ret i8 %call2
+}
+
+define i8 @test_fast_direct_call(ptr %f) nounwind {
+; CHECK-LABEL: test_fast_direct_call:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    callq foo_fast@PLT
+; CHECK-NEXT:    movzbl %al, %edi
+; CHECK-NEXT:    callq bar@PLT
+; CHECK-NEXT:    popq %rcx
+; CHECK-NEXT:    retq
+entry:
+  %call = call fastcc i1 @foo_fast(ptr %f)
+  %call2 = call zeroext i8 @bar(i1 %call)
+  ret i8 %call2
+}
+
+define i8 @test_indirect_all(ptr %fptr, ptr %f) nounwind {
+; CHECK-LABEL: test_indirect_all:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rbx
+; CHECK-NEXT:    movq %rdi, %rbx
+; CHECK-NEXT:    movq %rsi, %rdi
+; CHECK-NEXT:    callq foo@PLT
+; CHECK-NEXT:    movzbl %al, %edi
+; CHECK-NEXT:    callq *%rbx
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    retq
+entry:
+  %call = call i1 @foo(ptr %f)
+  %call2 = call zeroext i8 %fptr(i1 %call)
+  ret i8 %call2
+}
+
+define i8 @test_indirect_all2(ptr %fptr, ptr %f, i1 %cond) nounwind {
+; CHECK-LABEL: test_indirect_all2:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    pushq %rbx
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    movl %edx, %ebp
+; CHECK-NEXT:    movq %rdi, %rbx
+; CHECK-NEXT:    movq %rsi, %rdi
+; CHECK-NEXT:    callq foo@PLT
+; CHECK-NEXT:    testb $1, %bpl
+; CHECK-NEXT:    je .LBB3_2
+; CHECK-NEXT:  # %bb.1: # %exit
+; CHECK-NEXT:    movzbl %al, %edi
+; CHECK-NEXT:    callq *%rbx
+; CHECK-NEXT:    jmp .LBB3_3
+; CHECK-NEXT:  .LBB3_2: # %exit2
+; CHECK-NEXT:    movb $3, %al
+; CHECK-NEXT:  .LBB3_3: # %exit2
+; CHECK-NEXT:    addq $8, %rsp
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    retq
+entry:
+  %call = call i1 @foo(ptr %f)
+  br i1 %cond, label %exit, label %exit2
+
+exit:
+  %call2 = call zeroext i8 %fptr(i1 %call)
+  ret i8 %call2
+
+exit2:
+  ret i8 3
+}
+
+
+define i8 @test_fast_indirect_all(ptr %fptr, ptr %f) nounwind {
+; CHECK-LABEL: test_fast_indirect_all:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rbx
+; CHECK-NEXT:    movq %rdi, %rbx
+; CHECK-NEXT:    movq %rsi, %rdi
+; CHECK-NEXT:    callq foo@PLT
+; CHECK-NEXT:    movzbl %al, %edi
+; CHECK-NEXT:    callq *%rbx
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    retq
+entry:
+  %call = call fastcc i1 @foo(ptr %f)
+  %call2 = call zeroext i8 %fptr(i1 %call)
+  ret i8 %call2
+}
+
+declare i1 @foo(ptr %f)
+declare zeroext i8 @bar(i1)
+declare fastcc i1 @foo_fast(ptr %f)

phoebewang · 2026-03-16T08:31:15Z

llvm/lib/Target/X86/X86FastISel.cpp

+      return false; // Unknown type, let DAG ISel handle it.
+    EVT RetABIEVT = TLI.getRegisterTypeForCallingConv(CLI.RetTy->getContext(),
+                                                      CLI.CallConv, RetEVT);
+    MVT RegVT = TLI.getRegisterType(CLI.RetTy->getContext(), RetEVT);


getRegisterType require MVT. Should check isSimple() before it.

Revised. Now I use RetVT from isTypeLegal(). It must be simple type here, otherwise isTypeLegal would assign a type other than MVT::Other to RetVT.

phoebewang · 2026-03-16T08:32:13Z

llvm/lib/Target/X86/X86FastISel.cpp

-  if (!isTypeLegal(CLI.RetTy, RetVT) && !CLI.RetTy->isVoidTy())
-    return false;
+  if (!isTypeLegal(CLI.RetTy, RetVT) && !CLI.RetTy->isVoidTy()) {
+    EVT RetEVT = EVT::getEVT(CLI.RetTy, /*HandleUnknown=*/true);


Can we use RetVT here?

phoebewang · 2026-03-16T08:33:20Z

llvm/lib/Target/X86/X86FastISel.cpp

+    EVT RetEVT = EVT::getEVT(CLI.RetTy, /*HandleUnknown=*/true);
+    if (RetEVT == MVT::Other)
+      return false; // Unknown type, let DAG ISel handle it.
+    EVT RetABIEVT = TLI.getRegisterTypeForCallingConv(CLI.RetTy->getContext(),


getRegisterTypeForCallingConv return MVT too.

nikic · 2026-03-16T10:16:31Z

Confirming that this does fix the -O0 tramp3d-v4 compile-time regression: https://llvm-compile-time-tracker.com/compare.php?from=f894e8e92dd25b42f233810b6d90faa8745de8b2&to=a37d8ecd34569e59b49e4c407f2b6d690dec9f31&stat=instructions:u

phoebewang

LGTM.

Previously, FastISel would fall back to DAG ISel for any illegal return type. This change adds a more precise check to determine if the ABI requires a type conversion that FastISel cannot handle. For example, bfloat is returned as f16 in XMM0, but FastISel would assign f32 register type and store it in FuncInfo.ValueMap, causing DAG to incorrectly perform type conversion from f32 to bfloat later. However, i1 is promoted to i8 and returned as i8 per the ABI, so FastISel can safely lower it without switching to DAGISel. This change enables FastISel to handle such cases properly. --------- Co-authored-by: Yuanke Luo <ykluo@birentech.com>

Previously, FastISel would fall back to DAG ISel for any illegal return type. This change adds a more precise check to determine if the ABI requires a type conversion that FastISel cannot handle. For example, bfloat is returned as f16 in XMM0, but FastISel would assign f32 register type and store it in FuncInfo.ValueMap, causing DAG to incorrectly perform type conversion from f32 to bfloat later. However, i1 is promoted to i8 and returned as i8 per the ABI, so FastISel can safely lower it without switching to DAGISel. This change enables FastISel to handle such cases properly. --------- Co-authored-by: Yuanke Luo <ykluo@birentech.com> (cherry picked from commit 140adc9)

dianqk · 2026-03-28T02:19:04Z

/cherry-pick 140adc9

llvmbot · 2026-03-28T02:25:51Z

/pull-request #189139

[perf] Revert FastISel patch This caused a significant compile-time regression for debug builds. There is another change (llvm/llvm-project#186723) that mitigates that regression, but not fully. Revert it for now.

Previously, FastISel would fall back to DAG ISel for any illegal return type. This change adds a more precise check to determine if the ABI requires a type conversion that FastISel cannot handle. For example, bfloat is returned as f16 in XMM0, but FastISel would assign f32 register type and store it in FuncInfo.ValueMap, causing DAG to incorrectly perform type conversion from f32 to bfloat later. However, i1 is promoted to i8 and returned as i8 per the ABI, so FastISel can safely lower it without switching to DAGISel. This change enables FastISel to handle such cases properly. --------- Co-authored-by: Yuanke Luo <ykluo@birentech.com> (cherry picked from commit 140adc9)

llvmbot added the backend:X86 label Mar 16, 2026

LuoYuanke requested review from RKSimon, arsenm, nikic, phoebewang and topperc March 16, 2026 01:52

LuoYuanke mentioned this pull request Mar 16, 2026

[FastISel] Lower call instruction with illegal type returned #180322

Merged

phoebewang reviewed Mar 16, 2026

View reviewed changes

[FastISel] Address Phoebe's comments

d67ff24

LuoYuanke requested a review from phoebewang March 16, 2026 11:56

phoebewang approved these changes Mar 16, 2026

View reviewed changes

LuoYuanke merged commit 140adc9 into llvm:main Mar 17, 2026
10 checks passed

LuoYuanke deleted the fast-isel-compiling-time branch March 17, 2026 00:20

nikic mentioned this pull request Mar 27, 2026

Revert "Rollup merge of #154344 - dianqk:update-llvm, r=nikic" rust-lang/rust#154470

Closed

dianqk added this to the LLVM 22.x Release milestone Mar 28, 2026

github-project-automation bot added this to LLVM Release Status Mar 28, 2026

github-project-automation bot moved this to Done in LLVM Release Status Mar 28, 2026

nikic mentioned this pull request Mar 28, 2026

[perf] Revert FastISel patch rust-lang/rust#154511

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[X86] Improve illegal return type handling in FastISel#186723

[X86] Improve illegal return type handling in FastISel#186723
LuoYuanke merged 2 commits intollvm:mainfrom
LuoYuanke:fast-isel-compiling-time

LuoYuanke commented Mar 16, 2026

Uh oh!

llvmbot commented Mar 16, 2026

Uh oh!

phoebewang Mar 16, 2026

Uh oh!

LuoYuanke Mar 16, 2026

Uh oh!

phoebewang Mar 16, 2026

Uh oh!

LuoYuanke Mar 16, 2026

Uh oh!

phoebewang Mar 16, 2026

Uh oh!

LuoYuanke Mar 16, 2026

Uh oh!

nikic commented Mar 16, 2026

Uh oh!

phoebewang left a comment

Uh oh!

Uh oh!

dianqk commented Mar 28, 2026

Uh oh!

llvmbot commented Mar 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

LuoYuanke commented Mar 16, 2026

Uh oh!

llvmbot commented Mar 16, 2026

Uh oh!

phoebewang Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

LuoYuanke Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

phoebewang Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

LuoYuanke Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

phoebewang Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

LuoYuanke Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

nikic commented Mar 16, 2026

Uh oh!

phoebewang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dianqk commented Mar 28, 2026

Uh oh!

llvmbot commented Mar 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants