KiVar testing

[honzakutil]

I have prepared a little testing pcb file for KiVar: errors.zip. This one should test, if KiVar discovers and correctly reports an error. Righ now, I manually open the KiVar gui and read through all the errors. In the future, this could be done by kivar-cli and script which will parse and check the output.

My next idea is to prepare similar board with more tests - changing values of components, changing DNP, case sensitivity for aspects and choices naming ... In this case, KiVar should not report and error, instead it shoul run and perform variant switching. After that, Pick-and-Place file would be generated and compared to the correct one, which was manually checked. This could be also automated by kivar-cli and some scripts.

More ideas and feedback are welcomed, especially:

• it this even right way, how to test KiVar?
• what else should be tested? Where could be bugs?
• way of collaboration on this - Should I fork KiVar repository and create pull request or create another repo? Sending pcb file zipped as attachment to issue is not comfortable ...

[markh-de]

I would prefer unit-testing, i.e. a Python test app that directly calls KiVar functions, provides some test input and checks for the correct test output. For unit-testing there is no need for a board file (although this could be part of an extra test). Instead, you would push rules for fictional components into the module, and check the resulting database.

However, unit-testing requires cleaning up and restructuring parts of the implementation. I am planning to do this, but I'd rather get the new features done first (esp. #22, #26), so that users can use them.

Yes, the syntax will change once again (but as the design progresses, I get the feeling that this will be the "final" syntax format). It will be more flexible, so that users can pick their favorite style of defining rules and choices.

So, there's a lot of WIP, and I'm not sure if we should introduce testing now, as the implementation still needs to settle.

But if you want, feel free to fork the project and add some tests. However, as the main branch has strongly diverged, I would recommend to use the aux-rules branch right now. I just cannot promise to merge it anytime soon.

[honzakutil]

Unit testing sounds good, I am really looking forward to it. Unfortunatelly, I will not be able to help, it is far beyond my programming skills.

For now, to have some tests, I have prepared two PCBs and simple batch file, which calls the kivar-cli and kicad-cli pcb export pos .... Then, their results are compared to reference text files using git diff. Actual reference files are adapted to kivar 0.2.0-dev10 with fix described in #44. Files: testscript.zip

Testing is right now working on Windows 11 via KiCad 8.0 Command Prompt. Later, I will probably adapt it for linux and kicad running in docker. Since a I am little bit lazy to install actual version of kivar to kicad running in docker, I am secretly waiting for #40 to be finished, guessing it will save me some work :-)

[markh-de]

While the code is still using a function-oriented, not object-oriented, style, we now have many more possibilities to do unit-testing. We do not need a real board file, just a footprint-dictionary, which we could feed into the module and check the results. Similar for many more functions.

[honzakutil]

I modified my tests to work with actuall KiVar syntax and tested kivar 0.2.0-dev32. No problems found, but tests does not cover all possible cases - but I guess that some testing is better than nothing :-). Actual tests are here:
testscript.zip

I also used KiVar in one real project (sorry, can not share it). Also, no problems found. Just short summary about project:

• around 20 unique kivar rules, circa 70 components affected
• 12 aspects, each have 2 or 3 choices
• kivar changes value, fitted / not fitted attribute and custom fields (datasheet, description)
• tried multiple syntax styles Var, Var.Aspect, Datasheet.Var(1210)

I guess I used around 60% of KiVar funcionality, everything is fine, logical, I did not found any catches or unexpected features.