Skip to content

Add test execution gateway for dynamic input operator test #675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
baominghelly wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add test execution gateway for dynamic input operator test #675

baominghelly wants to merge 4 commits into from
+454 −2

Conversation

@baominghelly
Copy link
Contributor

@baominghelly baominghelly commented Nov 27, 2025 •
edited
Loading

Description

Add test execution gateway for dynamic input operator test.

Tests

Test json file example(inplace out = A)

{
    "operator": "mul",
    "device": "cuda",
    "description": "Mul - INPLACE(a) (Complex Strides)",
    
    "torch_op": "torch.mul",
    "infinicore_op": "infinicore.mul",

    "tolerance": {
        "atol": 1e-5,
        "rtol": 1e-3
    },
    "args": {
        "bench": "both",
        "num_prerun": 10,
        "num_iterations": 100,
        "verbose": false
    },
    "inputs": [
        {
            "name": "a",
            "dtype": "float32",
            "shape": [13, 4, 4],
            "strides": [20, 4, 1]
        },
        {
            "name": "b",
            "dtype": "float32",
            "shape": [13, 4, 4],
            "strides": [20, 4, 1]
        }
    ],
    "output_spec": null,
    "kwargs": {
        "out": 0
    },
    "comparison_target": 0
}

Test evidence

Out of place

(py3.13) libaoming@LAPTOP-4206MIBQ:~/workplace/InfiniCore$ python test/infinicore/framework/test_gateway.py test/infinicore/framework/mul1.json --device cpu --n
um_prerun 20 --num_iterations 200
🚀 Gateway: Start processing...
📄 Source: Loading test/infinicore/framework/mul1.json
⚙️  Ready to execute operator: 'mul'
    Targets: Torch -> torch.mul
             Infini -> infinicore.mul
    Target Devices: CPU
    Config: Bench=both, Prerun=20, Iterations=200
    Description: Mul - OUT_OF_PLACE (Complex Strides)

============================================================
Testing mul on CPU
============================================================
TestCase(Mul - OUT_OF_PLACE (Complex Strides) - inputs=[a: tensor(13, 4, 4), strides=(20, 4, 1), float32; b: tensor(13, 4, 4), strides=(20, 4, 1), float32])
    PyTorch    time - Host: 0.003349 ms, Device: 0.000000 ms
    InfiniCore time - Host: 0.210795 ms, Device: 0.000000 ms
✓ Passed

============================================================
TEST SUMMARY
Total tests: 1
Passed: 1
Success rate: 100.0%

All tests passed!
------------------------------------------------------------
BENCHMARK SUMMARY
PyTorch Host Total Time: 0.670 ms
InfiniCore Host Total Time: 42.159 ms
Host Speedup (PyTorch/InfiniCore): 0.02x
============================================================
🏁 Gateway: Process finished.

Inplace C

(py3.13) libaoming@LAPTOP-4206MIBQ:~/workplace/InfiniCore$ python test/infinicore/framework/test_gateway.py test/infinicore/framework/mul2.json --device cpu --n
um_prerun 20 --num_iterations 200
🚀 Gateway: Start processing...
📄 Source: Loading test/infinicore/framework/mul2.json
⚙️  Ready to execute operator: 'mul'
    Targets: Torch -> torch.mul
             Infini -> infinicore.mul
    Target Devices: CPU
    Config: Bench=both, Prerun=20, Iterations=200
    Description: Mul - INPLACE(out) (Complex Strides)

============================================================
Testing mul on CPU
============================================================
TestCase(Mul - INPLACE(out) (Complex Strides) - inputs=[a: tensor(13, 4, 4), strides=(20, 4, 1), float32; b: tensor(13, 4, 4), strides=(20, 4, 1), float32], kwargs={out=c: tensor(13, 4, 4), strides=(20, 4, 1), float32})
    PyTorch    time - Host: 0.002892 ms, Device: 0.000000 ms
    InfiniCore time - Host: 0.009530 ms, Device: 0.000000 ms
✓ Passed

============================================================
TEST SUMMARY
Total tests: 1
Passed: 1
Success rate: 100.0%

All tests passed!
------------------------------------------------------------
BENCHMARK SUMMARY
PyTorch Host Total Time: 0.578 ms
InfiniCore Host Total Time: 1.906 ms
Host Speedup (PyTorch/InfiniCore): 0.30x
============================================================
🏁 Gateway: Process finished.

Inplace A

(py3.13) libaoming@LAPTOP-4206MIBQ:~/workplace/InfiniCore$ python test/infinicore/framework/test_gateway.py test/infinicore/framework/mul3.json --device cpu --n
um_prerun 20 --num_iterations 200
🚀 Gateway: Start processing...
📄 Source: Loading test/infinicore/framework/mul3.json
⚙️  Ready to execute operator: 'mul'
    Targets: Torch -> torch.mul
             Infini -> infinicore.mul
    Target Devices: CPU
    Config: Bench=both, Prerun=20, Iterations=200
    Description: Mul - INPLACE(a) (Complex Strides)

============================================================
Testing mul on CPU
============================================================
TestCase(Mul - INPLACE(a) (Complex Strides) - inputs=[a: tensor(13, 4, 4), strides=(20, 4, 1), float32; b: tensor(13, 4, 4), strides=(20, 4, 1), float32], kwargs={out=a})
    PyTorch    time - Host: 0.002911 ms, Device: 0.000000 ms
    InfiniCore time - Host: 0.006440 ms, Device: 0.000000 ms
✓ Passed

============================================================
TEST SUMMARY
Total tests: 1
Passed: 1
Success rate: 100.0%

All tests passed!
------------------------------------------------------------
BENCHMARK SUMMARY
PyTorch Host Total Time: 0.582 ms
InfiniCore Host Total Time: 1.288 ms
Host Speedup (PyTorch/InfiniCore): 0.45x
============================================================
🏁 Gateway: Process finished.

Inplace B

(py3.13) libaoming@LAPTOP-4206MIBQ:~/workplace/InfiniCore$ python test/infinicore/framework/test_gateway.py test/infinicore/framework/mul4.json --device cpu --n
um_prerun 20 --num_iterations 200
🚀 Gateway: Start processing...
📄 Source: Loading test/infinicore/framework/mul4.json
⚙️  Ready to execute operator: 'mul'
    Targets: Torch -> torch.mul
             Infini -> infinicore.mul
    Target Devices: CPU
    Config: Bench=both, Prerun=20, Iterations=200
    Description: Mul - INPLACE(b) (Complex Strides)

============================================================
Testing mul on CPU
============================================================
TestCase(Mul - INPLACE(b) (Complex Strides) - inputs=[a: tensor(13, 4, 4), strides=(20, 4, 1), float32; b: tensor(13, 4, 4), strides=(20, 4, 1), float32], kwargs={out=b})
    PyTorch    time - Host: 0.004103 ms, Device: 0.000000 ms
    InfiniCore time - Host: 0.008236 ms, Device: 0.000000 ms
✓ Passed

============================================================
TEST SUMMARY
Total tests: 1
Passed: 1
Success rate: 100.0%

All tests passed!
------------------------------------------------------------
BENCHMARK SUMMARY
PyTorch Host Total Time: 0.821 ms
InfiniCore Host Total Time: 1.647 ms
Host Speedup (PyTorch/InfiniCore): 0.50x
============================================================
🏁 Gateway: Process finished.

@baominghelly baominghelly self-assigned this Nov 27, 2025
@baominghelly baominghelly linked an issue Dec 10, 2025 that may be closed by this pull request
[DEV] 测试框架测试信息读写 #707
Open
wooway777
wooway777 requested changes Dec 12, 2025
View reviewed changes
Copy link
Contributor

@wooway777 wooway777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

建议考虑一下老pr里预想的使用方法。

test/infinicore/framework/testcase_manager.py
"execution_results": results,
}

def _save_all_results(self, save_path, total_results):
Copy link
Contributor

@wooway777 wooway777 Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个跟reporter的save是否重复？

test/infinicore/framework/testcase_manager.py

# --- Helpers ---

def _discover_op_path(self, op_name: str, candidates: List[str]) -> str:
Copy link
Contributor

@wooway777 wooway777 Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果反正都要去发现路径的话，是不是不如直接在run.py里加？然后把重复的动作提成函数去调用？

test/infinicore/framework/testcase_manager.py
from framework.devices import InfiniDeviceEnum


class TestCaseManager:
Copy link
Contributor

@wooway777 wooway777 Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个要不还是交loader吧，不然manager应该包括之前的report功能才对。
其实之前那个主要也是save不是在report，考虑把两个合在一起然后叫manager也可以。

test/infinicore/run_external_case.py
parser.add_argument("file_path", type=str, nargs="?", help="Path to JSON config file")

# Overrides
parser.add_argument("--device", type=str, default=None, help="Override target device (e.g. nvidia, cpu)")
Copy link
Contributor

@wooway777 wooway777 Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这些是一定要在这里parse么？能不能复用framework里的parsing？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wooway777 wooway777 wooway777 requested changes

Requested changes must be addressed to merge this pull request.

Assignees

@baominghelly baominghelly

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[DEV] 测试框架测试信息读写

3 participants

@baominghelly @wooway777