can't run nyx-based fuzzers

[epi052]

I apologize for the vague title, but I'm not entirely sure where the real issue lies.

The problem

I was trying to get nyx_libxml2_standalone to run, as a precursor to writing my own nyx fuzzer.

when i cargo run, I get the following output (output hangs at the point you see a ctrl+c)

[!] libnyx: spawning qemu with:
 /home/epi/PycharmProjects/LibAFL/libafl_nyx/QEMU-Nyx/x86_64-softmmu/qemu-system-x86_64 -kernel /home/epi/PycharmProjects/LibAFL/libafl_nyx/packer/linux_initramfs/bzImage-linux-4.15-rc7 -initrd /home/epi/PycharmProjects/LibAFL/libafl_nyx/packer/linux_initramfs/init.cpio.gz -append nokaslr oops=panic nopti ignore_rlimit_data -display none -serial none -enable-kvm -net none -k de -m 512 -chardev socket,server,path=/tmp/nyx_libxml2/workdir/interface_0,id=nyx_interface -device nyx,chardev=nyx_interface,bitmap_size=65536,input_buffer_size=1048576,worker_id=0,workdir=/tmp/nyx_libxml2/workdir,sharedir=/tmp/nyx_libxml2/ -machine kAFL64-v1 -cpu kAFL64-Hypervisor-v1,+vmx
[QEMU-Nyx] Could not access KVM-PT kernel module!
[QEMU-Nyx] Trying vanilla KVM...
[QEMU-Nyx] NYX runs in fallback mode (no Intel-PT tracing or nested hypercall support)!
[QEMU-Nyx] Max Dirty Ring Size -> 1048576 (Entries: 65536)
[QEMU-Nyx] Warning: Attempt to use unsupported CPU model (PT) without KVM-PT (Hint: use '-cpu kAFL64-Hypervisor-v2' instead)
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
[QEMU-Nyx] Dirty ring mmap region located at 0x7f39624bb000
^C[qemu-nyx] bye! (pid: 15451 / signal: 2)

Stuff I've looked at/tried

• running cargo run as root
• running the /tmp/nyx... dir with afl-fuzz
• running nyx_libxml2_parallel

Based on the above output, I tried updating the -cpu value to match the hint. I did this by cloning libnyx pointing libafl's cargo.toml at my local copy, and then modifying the cmd variable.

i tried a few combinations:

• -machine kAFL64-v1 -cpu kAFL64-Hypervisor-v2,+vmx
• -machine kAFL64 -cpu kAFL64-Hypervisor-v2,+vmx
• -machine kAFL64 -cpu kAFL64-Hypervisor-v1,+vmx

Environment info

I understand this may be a "me" problem, i.e. something wonky with my setup. Here's some relevant information.

uname -a 
Linux main 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

lsmod | grep kvm
kvm_intel             368640  0
kvm                  1028096  1 kvm_intel

cat /sys/module/kvm/parameters/enable_vmware_backdoor 
Y

ls /sys/devices/intel_pt/format

branch  cyc_thresh  mtc         noretcomp   pt   pwr_evt
cyc     fup_on_ptw  mtc_period  psb_period  ptw  tsc

This may be important? No amount of running sudo modprobe kvm-intel pt_mode=1 will get the parameter below to reflect that value.

cat /sys/module/kvm_intel/parameters/pt_mode 
0

perf with intel-pt support works...?

sudo perf record -e intel_pt// -a sleep 1
[ perf record: Woken up 10 times to write data ]
[ perf record: Captured and wrote 14.937 MB perf.data ]

parting thoughts

is there a chance that qemu-nyx is falsely reporting kvm extension capability status?

I was poking around QEMU-Nyx's kvm checks, and as far as I can tell, my system CAN run intel-pt stuff. So, maybe the ioctl is falsely reporting on KVM_CAP_NYX_PT or KVM_CAP_NYX_FDL?

Idk, I'm at a loss at this point. Any help/clues would be greatly appreciated!

[epi052]

another pt support check

grep intel_pt /proc/cpuinfo -o 
20:intel_pt
48:intel_pt
76:intel_pt
104:intel_pt
132:intel_pt
160:intel_pt
188:intel_pt
216:intel_pt

[tokatoka]

@syheliel

Do you have any idea here?

[marcellomaugeri]

@epi052 same problem, have you managed to solve?

[syheliel]

I checked the CI of libafl and the fuzzer pass the test. Maybe you can use the original fuzzer in https://github.com/nyx-fuzz/spec-fuzzer to identify the system(?) problem.

[schumilo]

Could you please provide some more information regarding your system like CPU model and OS? And are we talking about a libAFL issue or libnyx issue in general?

The libAFL implementation uses a somewhat outdated version of Nyx (Spec-Fuzzer also relies on a pretty outdated version of the Nyx stack). Could you please try AFL++ Nyx-Mode or Nyx-Testing and confirm if the issue still persists? Both use the latest version of libnyx, with AFL++ utilizing the C interface and Nyx-Testing using the Rust interface of libnyx.

In case you use a more recent Intel CPU, there might be an issue with E-Cores I guess. As far as I know, efficiency cores don't fully support Intel PT.

[marcellomaugeri]

@schumilo I am trying to use the AFL++ Nyx-Mode.
I tried both Ubuntu 20.04 and after updated to Ubuntu 22.04 without success.
This is the result of lscpu:

  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         40 bits physical, 57 bits virtual
  Byte Order:            Little Endian
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               GenuineIntel
  Model name:            Intel Xeon Processor (Icelake)
    CPU family:          6
    Model:               134
    Thread(s) per core:  1
    Core(s) per socket:  1
    Socket(s):           8
    Stepping:            0
    BogoMIPS:            5187.76
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdts
                         cp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic m
                         ovbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssb
                         d ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms in
                         vpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 
                         xsaves wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq 
                         la57 rdpid md_clear arch_capabilities
Virtualization features: 
  Virtualization:        VT-x
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):     
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    32 MiB (8 instances)
  L3:                    128 MiB (8 instances)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-7
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected```

[marcellomaugeri]

The target is built with afl-lto, then I am not using KVM-PT mode

[marcellomaugeri]

@schumilo I tried with another computer and now everything works. Do you have any idea what could be the cause? Maybe a different cpu?

[GanbaruTobi]

@marcellomaugeri Could u try to use mitigations=off on the hostkernel?
For me it seems to solve a similar issue on my 13th gen Intel cpu

[GanbaruTobi]

I also tried with a 7th gen Intel CPU and I also can't run the fuzzer on a nyx-kernel. The only time I managed to run the fuzzer was on a normal kernel, which is pointless for blackbox fuzzing I guess...