Skip to content

Conversation

@baominghelly
Copy link
Contributor

Description

Add a new test tool which allows external test cases

Test

python test/infinicore/run_with_external_test_cases.py --nvidia

--- Scenario A: Single Case ---
🚀 Running 1 externally injected Case(s) for 'add'...

============================================================
Testing Add on NVIDIA
============================================================
TestCase(Custom Small Case - inputs=[in_0: tensor(2, 4), float32; in_0: tensor(2, 4), float32])
✓ Passed

============================================================
TEST SUMMARY
Total tests: 1
Passed: 1
Success rate: 100.0%

All tests passed!
============================================================

--- Scenario B: Multiple Cases List ---
🚀 Running 2 externally injected Case(s) for 'add'...

============================================================
Testing Add on NVIDIA
============================================================
TestCase(Custom Small Case - inputs=[in_0: tensor(2, 4), float32; in_0: tensor(2, 4), float32])
✓ Passed
TestCase(Custom Large Case - inputs=[in_0: tensor(16, 16), float32; in_0: tensor(16, 16), float32])
✓ Passed

============================================================
TEST SUMMARY
Total tests: 2
Passed: 2
Success rate: 100.0%

All tests passed!
============================================================
[1]    2837324 IOT instruction (core dumped)  python test/infinicore/run_with_external_test_cases.py --nvidia

Copy link
Contributor

@wooway777 wooway777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

首先,这个功能应该是集成到framework里的,而不是作为一个外部脚本存在。
同时,必须要能在不修改代码的情况下,通过命令行选项指定文件并执行其中的测例,这点没有体现出来。
现在这个写法和新写一个测试脚本没有本质区别。

考虑这样几种使用场景:

  1. 限定算子:
    1-1. 指定文件 python test/infinicore/ops/matmul.py --load [path-to-file]
    预期行为:用matmul跑指定文件中的所有测试
    如果指定文件不来自matmul则报错
    1-2. 指定文件夹 python test/infinicore/ops/matmul.py --load [path-to-dir]
    预期行为:遍历指定文件中所有符合格式的json文件,跑其中所有matmul测例
  2. 不限定算子
    2-1. 指定文件 python test/infinicore/run.py --load [path-to-file]
    预期行为:用文件中指定的算子跑其中的所有测试
    2-2. 指定文件夹 python test/infinicore/run.py --load [path-to-dir]
    预期行为:遍历指定文件中所有符合格式的json文件,并用每个文件指定的算子跑该文件中的测例
  3. 指定多个文件: --load [file-0] [file-1] [file-2]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants