thread 'main' panicked at 'No rankings in rankings.json', root_cause_analysis/src/rankings.rs:26:5

[c01dkit]

Hi Aurora team,

I came across a panic thread 'main' panicked at 'No rankings in rankings.json', root_cause_analysis/src/rankings.rs:26:5 when running the command cargo run --release --bin rca -- --eval-dir $EVAL_DIR --trace-dir $EVAL_DIR --monitor --rank-predicates in README file.

Here is how I build the system:
In a docker running Ubuntu 18.04, I follow the README instructions for Preparation, Tracing, and Root Cause Analysis sections. For a quick testing, I simply unzip the example.zip, setup the $EVAL_DIR and copy the crashes and non_crashes dirs into it.
At first, the same problem occurs as #8. So I switched to the develop branch. Then a panic of thread 'main' panicked at 'No rankings in rankings.json', root_cause_analysis/src/rankings.rs:26:5 occurs, when running cargo run --release --bin rca -- --eval-dir $EVAL_DIR --trace-dir $EVAL_DIR --monitor --rank-predicates

Here is the full backtrace:

warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /home/user/aurora/root_cause_analysis/predicate_monitoring/Cargo.toml
workspace: /home/user/aurora/root_cause_analysis/Cargo.toml
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /home/user/aurora/root_cause_analysis/trace_analysis/Cargo.toml
workspace: /home/user/aurora/root_cause_analysis/Cargo.toml
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /home/user/aurora/root_cause_analysis/root_cause_analysis/Cargo.toml
workspace: /home/user/aurora/root_cause_analysis/Cargo.toml
warning: use of deprecated associated function `zip::read::ZipFile::<'a>::sanitized_name`: by stripping `..`s from the path, the meaning of paths can change.
                         `mangled_name` can be used if this behaviour is desirable
   --> trace_analysis/src/trace.rs:282:42
    |
282 |         let trace_file_path = trace_file.sanitized_name().to_str().unwrap().to_string();
    |                                          ^^^^^^^^^^^^^^
    |
    = note: `#[warn(deprecated)]` on by default

warning: for loop over an `Option`. This is more readably written as an `if let` statement
   --> trace_analysis/src/control_flow_graph.rs:178:31
    |
178 |             for successors in self.successors.get(&node) {
    |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(for_loops_over_fallibles)]` on by default
help: to check pattern in a loop use `while let`
    |
178 |             while let Some(successors) = self.successors.get(&node) {
    |             ~~~~~~~~~~~~~~~          ~~~
help: consider using `if let` to clear intent
    |
178 |             if let Some(successors) = self.successors.get(&node) {
    |             ~~~~~~~~~~~~          ~~~

warning: `trace_analysis` (lib) generated 2 warnings
    Finished release [optimized] target(s) in 0.09s
warning: the following packages contain code that will be rejected by a future version of Rust: nom v5.1.2
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 64`
     Running `target/release/rca --eval-dir /home/user/aurora/evaluation_origin --monitor --rank-predicates`
thread 'main' panicked at 'No rankings in rankings.json', root_cause_analysis/src/rankings.rs:26:5
stack backtrace:
   0:     0x55555558a6fa - std::backtrace_rs::backtrace::libunwind::trace::ha271a8a7e1f3d4ef
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55555558a6fa - std::backtrace_rs::backtrace::trace_unsynchronized::h85739da0352c791a
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55555558a6fa - std::sys_common::backtrace::_print_fmt::hbc6ebcfb2910b329
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x55555558a6fa - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he1c117e52d53614f
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x5555555ae1fe - core::fmt::write::h25eb51b9526b8e0c
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/fmt/mod.rs:1213:17
   5:     0x555555587c25 - std::io::Write::write_fmt::ha9edec5fb1621933
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/io/mod.rs:1682:15
   6:     0x55555558a4c5 - std::sys_common::backtrace::_print::hf8657cd429fc3452
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x55555558a4c5 - std::sys_common::backtrace::print::h41b9b18ed86f86bd
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x55555558bcaf - std::panicking::default_hook::{{closure}}::h22a91871f4454152
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:267:22
   9:     0x55555558b9eb - std::panicking::default_hook::h21ddc36de0cd4ae7
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:286:9
  10:     0x55555558c3b9 - std::panicking::rust_panic_with_hook::h5059419d6d59b3d0
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:688:13
  11:     0x555555442eea - std::panicking::begin_panic::{{closure}}::hb75ff785e4da62b4
  12:     0x555555442ebc - std::sys_common::backtrace::__rust_end_short_backtrace::hf00f73ef884ccdd2
  13:     0x55555541608a - std::panicking::begin_panic::h192eb58691437ec3
  14:     0x5555554343a8 - root_cause_analysis::rankings::rank_predicates::hd90177f577ea8e2b
  15:     0x555555424890 - rca::main::h97f8ab931b39245d
  16:     0x555555424553 - std::sys_common::backtrace::__rust_begin_short_backtrace::hc386a1fb90cdfa8a
  17:     0x555555424c79 - std::rt::lang_start::{{closure}}::hba7c4263a55a9f7a
  18:     0x55555558317c - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h203afb3af230319a
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/ops/function.rs:287:13
  19:     0x55555558317c - std::panicking::try::do_call::hf68e87013b70f3c5
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:483:40
  20:     0x55555558317c - std::panicking::try::h040ea8f298390ba2
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:447:19
  21:     0x55555558317c - std::panic::catch_unwind::h1e17b198887a05fa
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panic.rs:140:14
  22:     0x55555558317c - std::rt::lang_start_internal::{{closure}}::hfb902d8927e51b86
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/rt.rs:148:48
  23:     0x55555558317c - std::panicking::try::do_call::h354e6eb41f2e7d42
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:483:40
  24:     0x55555558317c - std::panicking::try::h4a39749cd018228c
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:447:19
  25:     0x55555558317c - std::panic::catch_unwind::h30bce83b8de61cca
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panic.rs:140:14
  26:     0x55555558317c - std::rt::lang_start_internal::h8f7e70b1a2558118
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/rt.rs:148:20
  27:     0x555555424aa5 - main
  28:     0x7ffff6e22c87 - __libc_start_main
  29:     0x55555542446a - _start
  30:                0x0 - <unknown>

Finally I found that it is the panic thread 'main' panicked at 'failed to read memory: Sys(EPERM)', predicate_monitoring/src/lib.rs:24:55 in /home/user/aurora/root_cause_analysis/predicate_monitoring/src/lib.rs.

In $EVAL_DIR, the addresses.json, mnemonics.json, predicates.json, scores_linear_serialized.json and scores_linear.csv files are all not empty. The $EVAL_DIR/traces directory is also properly filled with *.zip files in $EVAL_DIR/traces/crashes and $EVAL_DIR/traces/non_crashes.
The content in $EVAL_DIR/traces/stat.txt is:

STATS: traced 5991/5991 files in 1733.535923242569s with 160 cores for /home/user/aurora/evaluation_origin/inputs

However, the ranking.json is simply filled with "[]". It really confuses me a lot.

Could you please give me some advice? We would be grateful for any guidance you can offer.

[mu00d8]

Hi,

I just tried to reproduce this but it works for me without problems. I've used

• a clean docker container with Ubuntu 18.04, not the one in this repository
• followed the steps as outlined in the README
• a non-root user
• and Aurora's develop branch.

Now, what's interesting is that you got a 'failed to read memory: Sys(EPERM)', predicate_monitoring/src/lib.rs:24:55 in /home/user/aurora/root_cause_analysis/predicate_monitoring/src/lib.rs panic. Can you double-check your user has the appropriate permissions for the executable and libraries?

[SonicStark]

As #8 mentioned:

When I disable the dynamic link library, the compiled binary can be successfully analyzed by aurora.

Be careful aurora/docker/Dockerfile would build mruby_trace here:

aurora/docker/Dockerfile

Lines 56 to 60 in 0dfceba

	# clean
	RUN cd mruby && make clean
	# build normal version for tracing/rca
	RUN cd mruby && CFLAGS="-ggdb -O0" make -e -j && mv ./bin/mruby ../mruby_trace

Actually gcc would build PIE output as default.

And 03_rca.sh(or root_cause_analysis/target/release/rca) implicitly finds target executable file *_trace in $EVAL_DIR and sends it to moniter to spawn the subprocess:

aurora/root_cause_analysis/root_cause_analysis/src/monitor.rs

Lines 111 to 117 in 0dfceba

	pub fn executable(config: &Config) -> String {
	let pattern = format!("{}/*_trace", config.eval_dir);
	let mut results = glob_paths(pattern);
	assert_eq!(results.len(), 1);
	results.pop().expect("No trace executable found")
	}

If the one in $EVAL_DIR was still the PIE version rather than your dynamic-link-disabled version tracing on,
when predicate_monitoring/src/lib.rs:24 uses ptracer to fetch instruction from target process at a specified address which comes from $EVAL_DIR/predicates.json

aurora/root_cause_analysis/predicate_monitoring/src/lib.rs

Lines 22 to 24 in 0dfceba

	fn instruction(decoder: &Decoder, pid: Pid, address: usize) -> Option<DecodedInstruction> {
	let mut code = [0u8; 16];
	ptracer::util::read_data(pid, address, &mut code).expect("failed to read memory");

ptracer would indeed access the invalid address in virtual memory, raising EPERM.

BTW you should also pay attention to

aurora/root_cause_analysis/root_cause_analysis/src/config.rs

Lines 38 to 44 in 707b94f

	#[structopt(
	long = "load-offset",
	default_value = "0x0000555555554000",
	parse(try_from_str = parse_hex),
	help = "Load offset of the target"
	)]
	pub load_offset: usize,

which means you should pass correct offset through --load-offset according to your no-pie target, otherwise you would get tons of panicked

root_cause_analysis/src/addr2line.rs:16:9:
attempt to subtract with overflow

when running cargo run --release --bin addr2line -- --eval-dir $EVAL_DIR.

Hope it can be helpful to you :) @c01dkit