Simple `fill_bytes` function for platform provided secure random generation

[ChrisDenton]

Proposal

Problem statement

Provide a simple to use way to fill a fixed-sized buffer with unpredictable data using whatever platform provided API is most appropriate for the target.

Also the current tracking issues for secure random number generation (rust-lang/rust#130703) mixes together both secure generation and generic RNG trait(s), leading to a somewhat chaotic discussion. I.e. the currently proposed random::Distribution trait is in no way specific to secure random number generation and could be (arguable more) useful for other types of RNGs.

Motivating examples or use cases

Often it is useful to completely fill a small fixed-size buffer with cryptographically secure random data. Example use cases:

• cryptographic key material
• to use as a token
• creating a nonce/IV

Solution sketch

Tracking issues

Divide the tracking issues differently:

• A general purpose rand-like traits and possibly non-secure implementation(s)
• Implementation of guaranteed secure random generation from the system

This will hopefully focus the discussion more effectively.

fill_bytes

A standalone, infallible, fill_bytes function that returns system provided cryptographically secure random data.

// In std::random 

fn fill_bytes(dest: &mut [u8]);

This is independent of any random traits that may (or may not) be added to the standard library. So I propose splitting them from one another to keep discussion focused.

The name is of course amenable to bikeshedding. E.g. getrandom, system_random, cprng, etc.

Alternatives

• Fully separate the OS CRNG from the other rand implementations and traits to make it super clear that these are special. E.g. std::crng::fill_bytes or std::secure_random::fill_bytes.
• Or have a SecureRandom type in the std::random module which would be a bit clearer than DefaultRandomSource.
• Instead of doing any of this, encourage people to use traits only so that the CRNG can be overridden by injecting an alternative CRNG.

Future possibilities

There could be types that wrap this interface to implement std traits. But this should exist independently of those.

fill_bytes requires the bytes to be pre-initialised. While I don't think this make much practical impact (at least judging from the RustCrypto experience and the lack of use of getrandom::uninit),, even that initialisation can be optimised out by LLVM if we can tell it that the bytes are writeonly. Rust doesn't currently have a way to do this but it could. Even an unstable attribute could be used purely for optimisation purposes internally by std.

This ACP does not rule out having some variation on a fallible/uninit function in the future (e.g. try_fill_bytes). But that is explicitly not proposed here. Even with that, the simpler function should be available.

Links and related work

• Tracking Issue for secure random data generation in std rust#130703
• Tracking Issue for deterministic random number generation rust#131606

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[kennytm]

from the few comments I see from rust-lang/rust#130703 i don't think the answer to "&mut [u8] vs &mut [MaybeUninit<u8>] vs BorrowedBuf" is decided.

[ChrisDenton]

That's why I created this ACP.

I also don't think it's an either/or situation but I do think it's worth having the simpler interface regardless.

[ChrisDenton]

For completeness I should add that it's possible to have our cake and eat it if the size of an array is statically known, though I'm not really convinced the advantages are worth it. E.g.:

fn fill_bytes<const N: usize>(buf: &mut [u8; N]) {
    let mut array = const { [MaybeUninit::<u8>::uninit(); N] };
    fill_bytes_internal(&mut array);
    let array: [u8; N] = unsafe { transmute(array) };
    buf.copy_from_slice(&array);
}

The optimiser will realise the stack buffer and copy are superfluous thus eliminate them. But it will retain the knowledge that buf is never read from and is fully overwritten. The downside is it's more awkward to call if you have an &mut [u8] rather than a &mut [u8; N] (the former is more common even when using static sized arrays).

Really it'd be ideal if we could annotate FFI functions in some way so this optimisation can be done regardless. But I don't think that's possible today. And again, I don't see zero initialising a small buffer as a major concern for the use cases in this ACP.

[the8472]

LLVM already has a writeonly attribute for function parameters and people have been discussing in-place init / &out references for ages, that'd fit nicely here.

[kennytm]

I wonder how bad would it be if the interface was

fn make_bytes<const N: usize>() -> [u8; N];

[abgros]

After playing around with defining an Rng trait I believe that a function returning an [u8; N] is the best primitive. The problem with a fill_bytes function that takes a &mut buf is that it either:

• Requires the buffer to be pre-initialized, adding overhead
• Doesn't require the buffer to be pre-initialized, because it is trusted to initialize everything - this makes fill_bytes unsafe to implement

Here's what I came up with:

Definition of the Rng and IoRng traits

use std::{io, mem::MaybeUninit};

/// Trait for fallible RNGs.
/// # Safety
/// If this method returns Ok(n), the first `n` bytes at `ptr` have been initialized.
unsafe trait IoRng {
	unsafe fn fill_bytes_raw(&mut self, ptr: *mut u8, len: usize) -> Result<usize, io::Error>;
}

/// Trait for infallible RNGs.
trait Rng {
	fn fill_bytes(&mut self, buf: &mut [u8]) {
		// this default implementation might be kind of slow but can be safely overridden
		const CHUNK: usize = 128;
		let mut chunks = buf.chunks_exact_mut(CHUNK);
		for chunk in &mut chunks {
			chunk.copy_from_slice(&self.gen_bytes::<CHUNK>());
		}
		let rem = chunks.into_remainder();
		rem.copy_from_slice(&self.gen_bytes::<CHUNK>()[..rem.len()]);
	}

	fn gen_u32(&mut self) -> u32 {
		u32::from_ne_bytes(self.gen_bytes())
	}

	fn gen_u64(&mut self) -> u64 {
		u64::from_ne_bytes(self.gen_bytes())
	}

	fn gen_bytes<const L: usize>(&mut self) -> [u8; L]; // always fast
}

impl<T: IoRng> Rng for T {
	#[inline]
	fn gen_bytes<const L: usize>(&mut self) -> [u8; L] {
		let mut buf = MaybeUninit::<[u8; L]>::uninit();

		match unsafe { self.fill_bytes_raw(buf.as_mut_ptr().cast(), L) } {
			Ok(n) if n == L => unsafe { buf.assume_init() },
			Ok(n) => panic!("failed to write {L} bytes; only wrote {n}"),
			Err(e) => panic!("IO error: {e}"),
		}
	}
}

// Example impls

struct SystemRng;

// SAFETY: ProcessPrng is infallible
unsafe impl IoRng for SystemRng {
	unsafe fn fill_bytes_raw(&mut self, ptr: *mut u8, len: usize) -> Result<usize, io::Error> {
		// windows impl
		unsafe extern "system" {
			unsafe fn ProcessPrng(pbData: *mut u8, cbData: usize) -> i32;
		}
		_ = unsafe { ProcessPrng(ptr, len) };
		Ok(len)
	}
}

pub struct Wyrand(u64);

impl Rng for Wyrand {
	#[inline]
	fn gen_bytes<const L: usize>(&mut self) -> [u8; L] {
		let mut buf = [0u8; L];
		let mut chunks = buf.chunks_exact_mut(8);
		for chunk in &mut chunks {
			chunk.copy_from_slice(&self.gen_u64().to_ne_bytes());
		}
		let rem = chunks.into_remainder();
		rem.copy_from_slice(&self.gen_u64().to_ne_bytes()[..rem.len()]);
		buf
	}

	#[inline]
	fn gen_u64(&mut self) -> u64 {
		self.0 = self.0.wrapping_add(0x2d358dccaa6c78a5);
		let t = u128::from(self.0) * u128::from(self.0 ^ 0x8bb84b93962eacc9);
		(t as u64) ^ (t >> 64) as u64
	}
}

[hanna-kruppe]

Worth noting that the getrandom crate has both a &mut [u8] version and a &mut [MaybeUninit<u8>] version, which seems like strong evidence for @ChrisDenton's "it's worth having the simpler interface regardless" position (which I share, FWIW).

Also worth noting that getrandom seriously considered removed the uninit version because it had no measurable performance benefit. The main reason why they didn't is because of an interaction with MemorySanitizer. Thus, objections or alternative proposals based on the assumption that avoiding &mut [u8] will be faster appear to be on very shaky ground. This is another reason to start with the simple interface first: additional or more complex APIs that avoid initialization can then be benchmarked against the simple one to test whether the claimed benefit is real or not.

[conradludgate]

For the sake of (avoiding) bikeshedding, I recall mentions of some platforms that have faster APIs for generating 4 or 8 bytes only - hence why getrandom crate also has the u32/u64 methods.

Since this impl would be in the std library, we can make use of the https://doc.rust-lang.org/std/intrinsics/fn.is_val_statically_known.html function to choose a more optimal implementation for each platform depending on the requested size without needing a different API.

If the caller only wants a u32, and the platform has a faster u32 impl, they can provide an array of 4 bytes and the std impl can check if that length is statically known and <= 4 accordingly.

This would also work with the const generic version. This though has the benefit of not needing to monomorphise a lot of different implementations of a getrandom() function, but it does rely on inlining for the optimisation to take place afaiu.

[newpavlov]

To add to @hanna-kruppe's comment, searching code on GitHub shows only 5 uses of getrandom::fill_uninit in the wild (after removing instances of vendored code). In other words, the practice shows that such function is virtually unused. Personally, I am sympathetic to the desire to eliminate the unnecessary buffer initialization, but we certainly can postpone it as a potential later addition. Hopefully, we will also get stabilized safe helper types for working with uninit buffers, which would help to make the function more ergonomic and eliminate unsafe in user code.

As for the OP proposal, I am mildly in support of it (though I think the function docs should explicitly mention that it may panic, and adding functions for generating uniform u32/u64 could also be worthwhile), but only as the initial step. As I argued in the other issues, I believe std should introduce rand_core-like RNG traits and RNG source types (SystemRng, ThreadRng, etc.) which would implement it. And there should be at least a roadmap draft of how this feature could be eventually extended to no_std.

[ChrisDenton]

In this ACP I really want to separate out the different concerns because I think there's a lot of cross-discussion confusing the situation and making it harder to tease out competing ideas. I don't think this has to be a everything or nothing situation, we can focus on certain things we'd like even if we had a all the traits and helpers fully designed. For example. adding functions for uniform u32/u64 would not change the utility of a fill_bytes function specifically for cryptographically secure random generation.

I would definitely like traits to be considered beyond the scope of this particular ACP. Which aims to answer these questions:

• Should we have a std::random::fill_bytes function that produces cryptographically secure random bytes regardless of other RNG things that may be added to std?
• If so, should that function be shaped as fn fill_bytes(&mut [u8]);?

Obviously my answer to those two questions is "yes". But if we can settle this question now (one way or another) we can hopefully move the discussion on instead of getting bogged down on this specific point.

[traviscross]

In the past, I know I had said there was some appeal to having a bare low-level function for this, but the more we've talked about this, the more I really just want to start by adding a type that implements Read in the normal way.

Regarding the small number of uses of fill_uninit, I'd mention that getrandom in vDSO is relatively new, and that it greatly expands the number of use cases where it is appropriate and desirable to call getrandom, in particular for bulk reading of pseudorandom bytes rather than just for collecting 32 or 64 bytes to use as a seed for an in-process KDF. So I think the existing uses of fill_uninit represent a backward-looking rather than forward-looking indication of likely use cases.