Fuzzy testing brainstorming

[eerimoq]

Below is the output from a short brainstorming session on introducing fuzzy testing as part of Simba. Will it be useful or just a time consuming activity nobody want to use? Ideally the fuzzy testing will not only test for memory corruption and security related issues, but also the application logic as a whole.

• Sections data_fuzzer and bss_fuzzer that can be reinitialized for each new fuzzed data. Eliminates the need for cleanup functions clobbering the application. Use objcopy (or other tool) to rename sections. binutils linker creates __start_<section> and __stop_<section> symbols.

• setjmp()/longjump() instead of pthreads in kernel port. Deterministic and no memory leaks. Will it work? Maybe use https://en.wikipedia.org/wiki/Setcontext?

• System tick called based of fuzzed data. Increment 0-255 times to start with. Maybe increment with a number of milliseconds instead for better application compatibility.

• Use libFuzzer.

• New port called libfuzzer.

• Create an example application with LLVMFuzzerTestOneInput().

• Goals:

  • Fast.

  • Deterministic.

  • Minimal user effort.

• Test external interfaces, not individual modules and function calls. Use manually written unit tests to test individual modules and function calls.

• How to input data into the application? Automatic in drivers? What does the user have to do? What does the user want to do?

• Need clang support in build system.

• Enable sanitizers.

• Run fuzzer until the code coverage is high. Then run only the corpus and save output data. Possibly create a testcase validating the output.

• No randomization allowed in the application. The same fuzzed input data must always produce the same output.

• How long will it take to fuzz the average sized application? Minutes? Hours? Years?

• Does any already existing fuzzer tool meet the requirements? honggfuzz? afl?