Segmentation Fault on Linux when rustls is present

[jonas-schievink]

Reproduction

cargo new --bin segfault

Cargo.toml

[package]
name = "ta-client"
version = "0.1.0"

[dependencies]
rmp-serde = "0.13.7"
failure = "0.1.5"
rustls = "0.14.0"

main.rs

extern crate failure;
extern crate rustls;

use failure::Error;

fn main() {
    rmp_serde::from_slice::<()>(&[128])
        .map_err(Error::from)
        .unwrap();
}

Running it, setting RUST_BACKTRACE=1 to make failure obtain a backtrace:

$ env RUST_BACKTRACE=1 cargo run

    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/ta-client`
fish: 'env RUST_BACKTRACE=1 cargo run' terminated by signal SIGSEGV (Address boundary error)

Removing the extern crate rustls; makes the crash disappear.

Replacing rmp_serde::from_slice::<()>(&[128]) with "abc".parse::<u8>() yields 2 garbage backtraces:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ParseIntError { kind: InvalidDigit }

stack backtrace:
   0:  (0x556643867bd2)
   1: DfU (0x55664386796d)
   2:  (0x556643866289)
   3: (0x5566438661b4)
   4:  (0x5566438668c4)
   5:  (0x556643866586)
   6: .c (0x556643866d3a)
   7:  (0x556643866e7f)
   8: DfU (0x55664389b572)
   9: __rust_maybe_catch_panic (0x55664389d699)
  10:  (0x55664389c0c5)
  11:  (0x556643866e58)
  12: mainA (0x556643866d79)
  13: __libc_start_main (0x7fdc712aa222)
  14:  (0x55664386602d)
  15: <unknown> (0x0)', src/libcore/result.rs:999:5
stack backtrace:
   0: Q
>  1: 
   2: 
   3: <unknown>
   4:y
   5: U
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: 
  11: <unknown>
  12: 
  13: 
  14: 
  15: 
  16: 
  17: __libc_start_main
  18: <unknown>

This also fixes itself when removing rustls.

GDB

Running GDB via env RUST_BACKTRACE=1 gdb target/debug/ta-client

(gdb) r
Starting program: /home/jonas/dev/segfault/target/debug/ta-client 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7ed5715 in __strlen_avx2 () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007ffff7ed5715 in __strlen_avx2 () from /usr/lib/libc.so.6
#1  0x00005555555c2b7a in std::ffi::c_str::CStr::from_ptr ()
#2  0x00005555555a58c3 in backtrace::symbolize::libbacktrace::Symbol::name ()
#3  0x000055555559d65b in backtrace::symbolize::Symbol::name ()
#4  0x0000555555596c34 in backtrace::capture::Backtrace::resolve::{{closure}} ()
#5  0x00005555555a5cec in backtrace::symbolize::libbacktrace::call ()
#6  0x00005555555a5b8f in backtrace::symbolize::libbacktrace::syminfo_cb ()
#7  0x00005555555a6e69 in elf_syminfo ()
#8  0x00005555555a6235 in __rbt_backtrace_syminfo ()
#9  0x00005555555a5e36 in backtrace::symbolize::libbacktrace::resolve ()
#10 0x000055555559d613 in backtrace::symbolize::resolve_unsynchronized ()
#11 0x000055555559d590 in backtrace::symbolize::resolve ()
#12 0x00005555555969f9 in backtrace::capture::Backtrace::resolve ()
#13 0x00005555555948d6 in failure::backtrace::internal::InternalBacktrace::as_backtrace ()
#14 0x00005555555944bb in <failure::backtrace::Backtrace as core::fmt::Debug>::fmt ()
#15 0x0000555555594161 in <&T as core::fmt::Debug>::fmt ()
#16 0x00005555555df722 in core::fmt::write ()
#17 0x00005555555e0884 in <core::fmt::Formatter<'_> as core::fmt::Write>::write_fmt ()
#18 0x0000555555593e8c in <failure::error::Error as core::fmt::Debug>::fmt ()
#19 0x00005555555df722 in core::fmt::write ()
#20 0x00005555555c86d5 in <std::panicking::continue_panic_fmt::PanicPayload<'a> as core::panic::BoxMeUp>::get ()
#21 0x00005555555c898d in std::panicking::rust_panic_with_hook ()
#22 0x00005555555c8522 in std::panicking::continue_panic_fmt ()
#23 0x00005555555c8406 in rust_begin_unwind ()
#24 0x00005555555dc6ed in core::panicking::panic_fmt ()
#25 0x0000555555587c46 in core::result::unwrap_failed ()
#26 0x0000555555586260 in <core::result::Result<T, E>>::unwrap ()
#27 0x000055555558dda0 in ta_client::main ()
#28 0x000055555558de60 in std::rt::lang_start::{{closure}} ()
#29 0x00005555555c83a3 in std::panicking::try::do_call ()
#30 0x00005555555ca61a in __rust_maybe_catch_panic ()
#31 0x00005555555c8ef6 in std::rt::lang_start_internal ()
#32 0x000055555558de39 in std::rt::lang_start ()
#33 0x000055555558ddda in main ()

For some reason an invalid pointer is passed to CStr::from_ptr (no null pointer, though).

This might very well be a bug in rustc/LLVM, but could also be located in (lib)backtrace like the backtrace indicates, so I'm opening the issue here.

Toolchain versions

rustc 1.33.0-nightly (8e2063d02 2019-01-07)
binary: rustc
commit-hash: 8e2063d02062ee9f088274690a97826333847e17
commit-date: 2019-01-07
host: x86_64-unknown-linux-gnu
release: 1.33.0-nightly
LLVM version: 8.0

gcc version 8.2.1 20181127 (GCC) 

glibc 2.28-5

Linux 4.20.0-arch1-1-ARCH

On stable, it produces a garbled stacktrace instead of a crash.

[alexcrichton]

Thanks for the report, and this definitely looks bad! Would it be possible to perhaps make a docker image to reproduce this? I can't reproduce this locally, and it's likely related to system differences like glibc or C compilers perhaps?

[jonas-schievink]

We couldn't reproduce this on an Ubuntu machine either (I'm running Arch Linux). I'll look into it.

However, on my machine this appears to have fixed itself after the newest nightly (rustc 1.33.0-nightly (d22fa2d87 2019-01-08)), while the one before that still reliably crashes (rustc 1.33.0-nightly (8e2063d02 2019-01-07)). The commits between the 2 nightlies suggest that this might've been fixed by you in rust-lang/rust#57429, does that sound possible? The other merges don't look suspicious.

I'll try to pinpoint the Rust version that introduced this behaviour now.

[jonas-schievink]

So after lots of bisecting I found out that this is apparently only happening when linking with LLD (which I had symlinked as ld a while ago to speed up compilation). I guess I'll close this and try to report this upstream somehow.

[alexcrichton]

Interesting! I actually think there's two bugs here, one is probably with LLD but one is definitely still with libbacktrace here because no matter what the debuginfo looks like it shouldn't segfault.

Would you be willing to help me out get a reproduction for this for this repository to still try to fix the segfault?

[jonas-schievink]

Sure, I made a Docker reproducer anyway: https://github.com/jonas-schievink/lldbug/

(additionally this might also be a bug in rustc's/LLVM's debuginfo handling)

[alexcrichton]

I'm gonna close this in favor of #189 since I think that's basically the only viable way to fix this.

[Chronostasys]

Hi @alexcrichton , is this issue actually fixed? I'm experiencing same issue on MacOS even after I set feature "gimli-symbolize".

Here is a minimum reproduce example https://github.com/Chronostasys/backtracebug .

[Chronostasys]

Hi @alexcrichton , is this issue actually fixed? I'm experiencing same issue on MacOS even after I set feature "gimli-symbolize".

Here is a minimum reproduce example https://github.com/Chronostasys/backtracebug .

More info: seems the crash only happens on mac with arm64 target.