Building for microcontrollers

[kiranshila]

Hey everyone,

I use crane a lot, but am running into issues trying to get it to build a microcontroller binary. These certainly are weird because of linker nonsense + no_std.

I spent a few hours trying to get crane to build me a binary, but no such luck.

My repo is here. I'm been reading the likes of

• https://crane.dev/examples/cross-rust-overlay.html
• https://github.com/IamTheCarl/ros_assistant/blob/9ecfcda85bf8547a8bcb581f88e5f3d494c27482/packages/firmware/example_projects/demo_setup/firmware/default.nix#L26
• https://discourse.nixos.org/t/cross-compiling-a-rust-application-to-thumbv7em-none-eabihf-no-matter-the-host-platform/65136
  But cannot find a magic combination that makes crane happy. cargo build --release in the dev shell works perfectly fine, though.

I don't think I want to use nix's cross-comp infra here (correct me if I'm wrong), so I was just trying to build a standard-looking crane package. My understanding is I would need nix's cross comp stuff and the callPackage magic if I were linking in code, which I am obviously not for microcontroller firmware.

I needed to modify the src to include the memory.x linker script that is quite standard for MCU builds.

Here is the flake

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    crane.url = "github:ipetkov/crane";
    flake-utils.url = "github:numtide/flake-utils";
    rust-overlay = {
      url = "github:oxalica/rust-overlay";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs = {
    self,
    nixpkgs,
    crane,
    flake-utils,
    rust-overlay,
    ...
  }:
    flake-utils.lib.eachDefaultSystem (
      system: let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [(import rust-overlay)];
        };

        inherit (pkgs) lib;

        target = "thumbv8m.main-none-eabihf";

        rustToolchainFor = p:
          p.rust-bin.nightly.latest.default.override {
            extensions = [ "rust-src" ];
            targets = [target];
          };
        craneLib = (crane.mkLib pkgs).overrideToolchain rustToolchainFor;

        src = lib.fileset.toSource rec {
          root = ./.;
          fileset = lib.fileset.unions [
            # Standard rust stuff
            (craneLib.fileset.commonCargoSources root)
            # Linker script
            ./memory.x
          ];
        };

        commonArgs = {
          inherit src;
          strictDeps = true;
          doCheck = false;
          CARGO_BUILD_TARGET = target;
          nativeBuildInputs = with pkgs; [flip-link];
        };

        cargoArtifacts = craneLib.buildDepsOnly commonArgs;
        asp_link = craneLib.buildPackage (commonArgs // {inherit cargoArtifacts;});
      in {
        checks = {
          inherit asp_link;
        };
        packages.default = asp_link;
        devShells.default = craneLib.devShell {
          # Inherit inputs from checks.
          checks = self.checks.${system};
        };
      }
    );
}

Trying to build gives a linker error, but I think that's because the cargoArtifacts aren't being built for the target I'm setting. So, any help would be appreciated.

[kiranshila]

Ah, success!

Apparently, building deps first breaks a lot of the linking magic that embedded projects need. By explicitly setting cargoArtifacts = null, everything works as expected.

[dpc]

By not using cargoArtifacts to builds deps only, you're missing out on a lot of benefits of using crane.

Trying to build gives a linker error,

Please paste the error.

[kiranshila]

Ok! Ill post once I'm back at my computer. The main bit seemed to be that the panic handler crate seemed to get LTOed out of existence.

[kiranshila]

Ok @dpc, so if I don't set cargoArtifacts to null - I get the error

       > error: linking with `flip-link` failed: exit status: 1
       >   |
       >   = note:  "flip-link" "-flavor" "gnu" "<1 object files omitted>" "--as-needed" "-Bstatic" "<sysroot>/lib/rustlib/thumbv8m.main-none-eabihf/lib/libcompiler_builtins-*.rlib" "-L" "/build/rustcKmFs8j/raw-dylibs" "-Bdynamic" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/build/source/target/thumbv8m.main-none-eabihf/release/build/cortex-m-rt-910b4276b5d19db1/out" "-L" "/build/source/target/thumbv8m.main-none-eabihf/release/build/defmt-9aba2c75104c783c/out" "-L" "/build/source/target/thumbv8m.main-none-eabihf/release/build/cortex-m-b9c40b9e3d0e4008/out" "-L" "/build/source/target/thumbv8m.main-none-eabihf/release/build/rp-pac-992e6260543d1fa6/out" "-o" "/build/source/target/thumbv8m.main-none-eabihf/release/deps/asp_link-3296cf259f505f46" "--gc-sections" "-Tlink.x" "-Tdefmt.x" "--nmagic"
       >   = note: some arguments are omitted. use `--verbose` to show all linker arguments
       >   = note: rust-lld: error: /build/source/target/thumbv8m.main-none-eabihf/release/build/cortex-m-rt-910b4276b5d19db1/out/link.x:23: cannot find linker script memory.x
       >           >>> INCLUDE memory.x
       >           >>>         ^
       >           
       >           flip-link: the native linker failed to link the program normally; please check your project configuration and linker scripts

Which has to do with that linker script not being accessible to the linker during the deps build. This makes sense, I suppose, as the deps don't include my source which contains memory.x. However, linking won't work because the partial linker script provided by the cortex-m-rt crate need a memory map. I can hack this a bit by adding

preBuild = ''
   cp ${src}/memory.x memory.x
 '';

to the buildPackage call, but then I get the error

       > error: linking with `flip-link` failed: exit status: 1
       >   |
       >   = note:  "flip-link" "-flavor" "gnu" "<1 object files omitted>" "--as-needed" "-Bstatic" "<sysroot>/lib/rustlib/thumbv8m.main-none-eabihf/lib/libcompiler_builtins-*.rlib" "-L" "/build/rustcTJMKti/raw-dylibs" "-Bdynamic" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/build/source/target/thumbv8m.main-none-eabihf/release/build/cortex-m-rt-910b4276b5d19db1/out" "-L" "/build/source/target/thumbv8m.main-none-eabihf/release/build/defmt-9aba2c75104c783c/out" "-L" "/build/source/target/thumbv8m.main-none-eabihf/release/build/cortex-m-b9c40b9e3d0e4008/out" "-L" "/build/source/target/thumbv8m.main-none-eabihf/release/build/rp-pac-992e6260543d1fa6/out" "-o" "/build/source/target/thumbv8m.main-none-eabihf/release/deps/asp_link-3296cf259f505f46" "--gc-sections" "-Tlink.x" "-Tdefmt.x" "--nmagic"
       >   = note: some arguments are omitted. use `--verbose` to show all linker arguments
       >   = note: rust-lld: error: /build/source/target/thumbv8m.main-none-eabihf/release/build/cortex-m-rt-910b4276b5d19db1/out/link.x:49: symbol not found: DefaultHandler_
       >           rust-lld: error: /build/source/target/thumbv8m.main-none-eabihf/release/build/cortex-m-rt-910b4276b5d19db1/out/link.x:49: symbol not found: DefaultHandler_
       >           rust-lld: error: /build/source/target/thumbv8m.main-none-eabihf/release/build/cortex-m-rt-910b4276b5d19db1/out/link.x:49: symbol not found: DefaultHandler_
       >           
       >           flip-link: the native linker failed to link the program normally; please check your project configuration and linker scripts

which if I'm understanding this correctly is due to missing the panic handler. This is set up in the application with

use panic_probe as _;

but of course, that's not in the deps-only build. So, I'm not surprised the deps fail to build when they don't have all the information. Embedded, no_std stuff seems like it needs the whole context.

For the rest of the details, here's my .cargo/config.toml for the linker args

[target.'cfg(all(target_arch = "arm", target_os = "none"))']
runner = "probe-rs run --chip RP235x"
rustflags = [
  "-C", "linker=flip-link",
  "-C", "link-arg=-Tlink.x",
  "-C", "link-arg=-Tdefmt.x",
  "-Z", "trap-unreachable=no",
  "-C", "link-arg=--nmagic",
]

[kiranshila]

Thanks for the suggestion, @ipetkov. I'm actually still confused a bit on the error I'm seeing. The docs for cortex-m-rt say that DefaultHandler should have a default infinite loop fallback and yet I still get

link.x:49: symbol not found: DefaultHandler_

Curiously, if I add

cargoArtifacts = craneLib.buildDepsOnly (commonArgs
      // {
        dummySrc = craneLib.mkDummySrc {
          inherit src;
          dummyrs = ./dummy.rs;
        };
      });

where dummy.rs contains basically what's in your example except I take out the cfg_attrs and just leave

#![allow(clippy::all)]
#![allow(dead_code)]
#![no_std]
#![no_main]

#[allow(unused_extern_crates)]
extern crate core;

#[panic_handler]
fn panic(_info: &::core::panic::PanicInfo<'_>) -> ! {
    loop {}
}

pub fn main() {}

I get

       > error: unwinding panics are not supported without std
       >   |
       >   = help: using nightly cargo, use -Zbuild-std with panic="abort" to avoid unwinding
       >   = note: since the core library is usually precompiled with panic="unwind", rebuilding your crate with panic="abort" may not be enough to fix the problem

But I'm confused then why the target os isn't none if in .cargo/config.toml I have

[build]
target = "thumbv8m.main-none-eabihf"

and I was assuming this was being applied to the deps.

[dpc]

I might be off about some details, but in general crane will strip a lot of things from Cargo.toml and strip unrelated files, which might include .cargo/config.toml.

You can use mkDummySrc to add more files that you want during deps-only build.

[kiranshila]

The docs say the .cargo/config.toml is kept as is.

Any cargo configuration files (i.e. files name config or config.toml whose parent directory is named .cargo) are kept as-is.
Any changes to these files will invalidate the build cache

[dpc]

help: using nightly cargo, use -Zbuild-std with panic="abort" to avoid unwinding

I would expect thumbv8m.main-none-eabihf to already use panic abort, which seems like maybe wrong target is used or something.

[kiranshila]

Yeah exactly my thought. Again, I have target = "thumbv8m.main-none-eabihf" in the config.toml. I tried setting CARGO_BUILD_TARGET but no change.

[ipetkov]

@kiranshila I dug into this and I think I figured out the issue and a path forward:

• I noticed you are using a custom linker in .cargo/config.toml, and these configuration files are left as is during source dummification for buildDepsOnly so we respect that setting
• This linker appears to require that certain symbols are always defined, so the default dummy source (which defines an empty main and a panic handler) isn't enough: thus a custom dummyrs file definition will be required
• What's also interesting is the build.rs script appears to exist to copy memory.x into $OUT_DIR (maybe how/where the linker looks for this file means it can't just be at the source root), so we might actually want to use the "real" build script even in buildDepsOnly (or if this build script ever becomes more complicated, you'd want buildDepsOnly to run with a simplified build script which only handles copying the memory.x file in place). I just made this a bit easier to customize in mkDummySrc: also accept dummyBuildrs #910

So here's how to make your project build:

1. Update your crane input
2. Apply the following diff:

diff --git a/flake.nix b/flake.nix
index a824345b1a..49807c1a46 100644
--- a/flake.nix
+++ b/flake.nix
@@ -69,7 +69,31 @@
 
         asp_link = craneLib.buildPackage (commonArgs
           // {
-            cargoArtifacts = null;
+            cargoArtifacts = craneLib.buildDepsOnly (commonArgs // {
+              preBuild = ''
+                cp ${./memory.x} ./memory.x
+                cp ${./build.rs} ./build.rs
+              '';
+              dummyBuildrs = "./build.rs";
+              dummyrs = pkgs.writeText "myDummy.rs" ''
+                #![no_std]
+                #![no_main]
+                #![feature(impl_trait_in_assoc_type)]
+
+                use defmt_rtt as _;
+                use panic_probe as _;
+
+                #[embassy_executor::task]
+                async fn feed_watchdog(_wd: embassy_rp::watchdog::Watchdog) {
+                    loop {}
+                }
+
+                #[embassy_executor::main]
+                async fn main(_s: embassy_executor::Spawner) -> ! {
+                  loop {}
+                }
+              '';
+            });
             postInstall = ''
               elf2flash convert --board rp2350 $out/bin/asp_link $out/bin/asp_link.uf2
             '';

---

Hope this helps!

[kiranshila]

Thanks for looking into this @ipetkov!

The custom linker shouldn't be doing a whole lot, it's essentially a wrapper around the normal rust lld that flips around the memory layout of the binary for stack overflow protection. It's not super critical here, but I still get issues without it (and it's pretty ubiquitous in embedded rust).

I'm not sure why memory.x can't just be at the source root. The file that I copied this from seems to say that having it at the source root is usually fine?

Maybe I'm misunderstanding something, but why would I need to bring in embassy's executor and the watchdog timer into the deps build? It seemed like I only needed the panic handler.

In any case, with your changes I still get the following error:

error: Cannot build '/nix/store/6v5bfy98w1kdg38wk1wdi5iqs1v183sp-asp_link-deps-0.0.0.drv'.
       Reason: builder failed with exit code 101.
       Output paths:
         /nix/store/fdiiadb7sj95504d7qjmklrm0xylh8kh-asp_link-deps-0.0.0
       Last 25 log lines:
       >   |
       > 9 | async fn feed_watchdog(_wd: embassy_rp::watchdog::Watchdog) {
       >   |                             ^^^^^^^^^^ use of unresolved module or unlinked crate `embassy_rp`
       >   |
       >   = help: if you wanted to use a crate named `embassy_rp`, use `cargo add embassy_rp` to add it to your `Cargo.toml`
       >
       > error: `#[panic_handler]` function required, but not found
       >
       > error: unwinding panics are not supported without std
       >   |
       >   = help: using nightly cargo, use -Zbuild-std with panic="abort" to avoid unwinding
       >   = note: since the core library is usually precompiled with panic="unwind", rebuilding your crate with panic="abort" may not be enough to fix the problem
       >
       > error[E0433]: failed to resolve: use of unresolved module or unlinked crate `embassy_executor`
       >   --> /nix/store/wmz6g42lj5xb82r6ab8yndnzrcc8cmy5-myDummy.rs:14:19
       >    |
       > 14 | async fn main(_s: embassy_executor::Spawner) -> ! {
       >    |                   ^^^^^^^^^^^^^^^^ use of unresolved module or unlinked crate `embassy_executor`
       >    |
       >    = help: if you wanted to use a crate named `embassy_executor`, use `cargo add embassy_executor` to add it to your `Cargo.toml`
       >
       > Some errors have detailed explanations: E0432, E0433.
       > For more information about an error, try `rustc --explain E0432`.
       > error: could not compile `asp_link` (build script) due to 8 previous errors
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run:
         nix log /nix/store/6v5bfy98w1kdg38wk1wdi5iqs1v183sp-asp_link-deps-0.0.0.drv
error: Cannot build '/nix/store/4v35vfzvmb7q0wz42gj18367868jcvmw-asp_link-0.0.0.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/fh2rvl5k8h8qsb072b31p3z2fqrd6asj-asp_link-0.0.0

[ipetkov]

I'm not sure why memory.x can't just be at the source root

I saw that the build script was moving it to $OUT_DIR and then instructing cargo to cargo:rustc-link-search=$OUT_DIR. Maybe there's a way to instruct the linker to look at the root of the directory for memory.x but I didn't dig into it further!

Maybe I'm misunderstanding something, but why would I need to bring in embassy's executor and the watchdog timer into the deps build?

My guess is the custom linker will fail if it doesn't find whatever symbols the embassy macros happen to export. I know its not required for more vanilla setups (which is why the default dummy file doesn't define anything except for a panic handler). Ultimately the buildDepsOnly script will inherit your linker settings, and if the linker fails when invoked, the derivation won't build (maybe there's an easier way to convince your linker to accept an empty file, but this was the simplest thing I could come up with conceptually)

In any case, with your changes I still get the following error:

Make sure your flake.lock points to 07c296e or later. That commit is what allows injecting a custom build script and seems to make things work, otherwise I remember seeing similar issues when trying to get your reproduction to build.