Regression with sccache 0.2.4: failed to acquire jobserver token: early EOF on jobserver pipe

[SimonSapin]

Problem

In recent Nightlies, any compilation with sccache 0.2.4 fails with the same panic in rustc, with a message about jobservers. (If there is more than one crate to build).

Steps

curl https://github.com/mozilla/sccache/releases/download/0.2.4/sccache-0.2.4-x86_64-unknown-linux-musl.tar.gz -L | tar xz
cargo new foo
cd foo
echo 'matches = "0.1"' >> Cargo.toml
export RUSTC_WRAPPER=../sccache-0.2.4-x86_64-unknown-linux-musl/sccache
export RUST_BACKTRACE=full
rustup run --install nightly-2018-10-21 cargo build
rustup run --install nightly-2018-10-22 cargo build -v

The first build succeeds, the second one prints:

   Compiling matches v0.1.8                                                                                            
     Running `/home/servo/simon/foo/../sccache-0.2.4-x86_64-unknown-linux-musl/sccache rustc --crate-name matches /home/servo/.cargo/registry/src/github.com-1ecc6299db9ec823/matches-0.1.8/lib.rs --color always --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=6a626e308a29780e -C extra-filename=-6a626e308a29780e --out-dir /home/servo/simon/foo/target/debug/deps -L dependency=/home/servo/simon/foo/target/debug/deps --cap-lints allow`
thread '<unnamed>' panicked at 'failed to acquire jobserver token: early EOF on jobserver pipe', librustc_codegen_llvm/back/write.rs:1980:29
stack backtrace:                                                                                                       
   0:     0x7f0cc2ff172f - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::hb6bbf9c852e8bb81                
                               at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49                                        
   1:     0x7f0cc2fd66b7 - std::sys_common::backtrace::print::h0b05a26fe4434b95                                        
                               at libstd/sys_common/backtrace.rs:71                                                    
                               at libstd/sys_common/backtrace.rs:59                                                    
   2:     0x7f0cc2ffcaff - std::panicking::default_hook::{{closure}}::h137fa5ca768795d3                                
                               at libstd/panicking.rs:211                                                              
   3:     0x7f0cc2ffc864 - std::panicking::default_hook::h52fea0104d6161d8                                             
                               at libstd/panicking.rs:227                                                              
   4:     0x7f0cbf64f131 - rustc::util::common::panic_hook::h596f772bfbae21ad                                          
   5:     0x7f0cc2ffd2f6 - std::panicking::rust_panic_with_hook::h7c51e7d0fdff9bd2                                     
                               at libstd/panicking.rs:480                                                              
   6:     0x7f0cc2ffce41 - std::panicking::continue_panic_fmt::h499943668b519270                                       
                               at libstd/panicking.rs:390                                                              
   7:     0x7f0cc2ffcd8e - std::panicking::begin_panic_fmt::hf8a762b584404a73                                          
                               at libstd/panicking.rs:345                                                              
   8:     0x7f0cb8bdd07b - std::sys_common::backtrace::__rust_begin_short_backtrace::h959b03245837a19f                 
   9:     0x7f0cb8c1585d - std::panicking::try::do_call::hb7dfe518403aa5b1                                             
  10:     0x7f0cc300a619 - __rust_maybe_catch_panic                                                                    
                               at libpanic_unwind/lib.rs:102                                                           
  11:     0x7f0cb8c1a398 - <F as alloc::boxed::FnBox<A>>::call_box::h893756b5f7e5da26                                  
  12:     0x7f0cc2ff7abd - std::sys_common::thread::start_thread::h7c22dec0f118f1ff                                    
                               at liballoc/boxed.rs:682                                                                
                               at libstd/sys_common/thread.rs:24                                                       
  13:     0x7f0cc2fcd455 - std::sys::unix::thread::Thread::new::thread_start::haea81cde2be173fc                        
                               at libstd/sys/unix/thread.rs:90                                                         
  14:     0x7f0cbd03d183 - start_thread                                                                                
  15:     0x7f0cc2cac03c - clone                                                                                       
  16:                0x0 - <unknown>                                                                                   
query stack during panic:                                                                                              
end of query stack                                                                                                     
error: failed to acquire jobserver token: early EOF on jobserver pipe                                                  
                                                                                                                       
error: aborting due to previous error                                                                                  
                                                                                                                       
error: Could not compile `matches`.   

Notes

$ rustup run --install nightly-2018-10-21 rustc -V
rustc 1.31.0-nightly (155510e37 2018-10-20)
$ rustup run --install nightly-2018-10-22 rustc -V
rustc 1.31.0-nightly (424a749a0 2018-10-21)

rust-lang/rust@155510e...424a749

$ rustup run --install nightly-2018-10-21 cargo -V
cargo 1.31.0-nightly (09ce4b519 2018-10-16)
$ rustup run --install nightly-2018-10-22 cargo -V
cargo 1.31.0-nightly (2d0863f65 2018-10-20)

09ce4b5...2d0863f

This occured on Servo’s CI on Ubuntu 14.04. I managed to reproduce on Ubuntu 18.04 but not on Archlinux.

[jdm]

It would be worth seeing if this is reproducible with more recent versions of sccache. 0.2.5 in particular had a jobserver-specific fix included.

[SimonSapin]

sccache 0.2.7 (currently the latest) seems ok with both Rust toolchains tested above, as well as 1.30.0 stable and nigthly-2018-11-06.

Still, it seems something has changed between those two nightlies but the commit ranges don’t show anything that seems suspicious.

[alexcrichton]

It's probably rust-lang/rust#54919 which caused this, although I don't know precisely why. Before sccache 0.2.5 it was basically "UB with file descriptors" so it could just be that updating some dependency is tickling something in just the wrong way. Maybe OpenSSL is now managing file descriptors with CLOEXEC? (unsure!)

In any case the fix here is to upgrade sccache, and I'm not sure if there's much else we can do on Cargo's side of things

[ehuss]

Considering this is pretty old, and it seems like sccache is working now, I'm going to close. If there is still an issue, feel free to re-open or file a new issue.