Security implications of tofu-as-a-service

[ghrltu]

I'm kicking around an idea which boils down to:

• running tofu on behalf of 3rd parties
• using untrusted .tf files

There seems like a bunch of potential for abuse from malicious tofu configurations, so I'd like to explore whether it can be done safely.

I'd be running my tofu binary and my provider binary against my API, using configurations sourced from untrusted 3rd parties.

Mitigations and controls I'm thinking about:

• I only need to support my provider (I can reject configurations which require other providers)
• I can firewall the tofu service so that it can only reach my provider registry and the intended API service
• I don't anticipate a need for any modules (I can reject configurations which use modules)
• I don't anticipate a need for any file-based functions (can disable these within tofu, so no sniffing around on the local filesystem)
• I don't anticipate a need for any provisioners (not sure what to do about these exactly)

Am I off my rocker? Can this be done safely?

[apparentlymart]

Hi @ghrltu,

OpenTofu is not designed to be used in the way you've described, so before we get into the details I want to just give a general caution that we're going to be discussing current implementation details rather than committed constraints we will definitely preserve in future releases of the project, and any guidance I'm going to give below is not necessarily complete nor aligned with your specific threat model. Ultimately it will be your responsibility to study each release of OpenTofu carefully yourself to ensure that it meets your needs.

With that said...

• Restricting to only a specific provider does eliminate a number of potential abuse vectors, since providers are the main example of "arbitrary code execution" when working with an untrusted OpenTofu configuration.

  Of course, you'd need to guarantee that only that provider can be used. One way to achieve that would be to configure OpenTofu CLI with an explicit provider installation method configuration which includes only a single mirror source (either filesystem_mirror or network_mirror in today's OpenTofu) referring to a location which includes only the specific provider you intend to allow use of. tofu init will therefore be unable to discover and install any other providers, or to discover and install alternative builds of your own provider from other locations.

• Ensuring that the execution environment where OpenTofu is running cannot access anything other than the designated remote API is also a good idea for keeping things constrained.

  The way I'd characterize the benefit of this mitigation is in decreasing the potential value of compromise: if arbitrary code running in this environment can only interact with the local system and with one specific remote API then it's not an attractive target for someone who wants to use your system to indirectly abuse another third-party system, or to run other software that needs to access remote services to be useful such as cryptocurrency "miners". This hopefully means that if thare are any gaps left by your other mitigations then they would be of only very specialized use anyway.

• Your remaining points about limiting which language features you'll allow seem to imply that you'd be implementing something that analyzes the provided configuration and rejects anything which uses language features outside of your narrow expected bounds.

  That is a reasonable idea in principle, but I think also the trickiest thing to nail down in practice because the OpenTofu language is going to continue to evolve and so you will need to carefully track additions and make sure that new features don't present any new opportunities to abuse your system.

  If you successfully implement logic that can analyze one OpenTofu module and reject unwanted language features then you could presumably also run that same logic against other modules in the configuration, and so rejecting configurations containing module blocks might not be strictly required, although ruling that out entirely does seem like it would simplify the problem by avoiding the need for you to match OpenTofu's behavior for resolving child modules and making sure that there aren't any loopholes where someone can introduce a module in a way that your logic can't "see".

  With the other mitigations I discussed in the other bullet points I think the main remaining concern is that the execution environment where OpenTofu is running will presumably have access to some credentials for interacting with your API, and you'd presumably like to prevent an attacker from extracting those credentials. If that is a correct understanding of your threat model then indeed prohibiting the use of functions that access the filesystem and the use of any provisioners does seem to cut of the main ways that someone might extract arbitrary configuration files or environment variables from the execution environment, but I can't promise you that those are the only possibilities.

If I were implementing something like what you've described, with constraints similar to those you have listed, I expect I'd go a little further and not let third-parties directly provide OpenTofu modules at all, and instead require them to provide a bespoke configuration format tailored to your system and then translate that carefully to OpenTofu configuration purely as an implementation detail so that the potential scope of that configuration is limited only to what the transpiler allows. Of course there could still be bugs in the transpiler that would leave gaps, but this alternative is effectively an allowlist-based approach rather than a denylist-based approach and that tends to be easier to secure because there's less exposure to "unknown unknowns".

Ultimately you're going to need to conduct your own careful security review to be sure that your system meets all of the guarantees you want it to meet, but I hope the notes above are a helpful starting point.

[ghrltu]

I really appreciate you taking the time to share your thoughts on this, @apparentlymart!

I'll be digesting it all slowly and carefully.

[ghrltu]

Hi @apparentlymart,

OpenTofu is not designed to be used in the way you've described...

Yep. I get it.

Contemplating this mess has got me thinking back to the origins of OpenTofu. Surely there are folks doing something like this or we might not have gotten OpenTofu in the first place? I wonder what they're doing, how they're doing it safely, and what I don't understand about it.

configure OpenTofu CLI with an explicit provider installation method configuration

I'd assumed I'd be scraping the configuration (possibly with terraform-config-inspect?) but making other providers unavailable is even better. I'd already been planning to include a registry in the project, so I feel silly for not having put this together on my own. Thank you for the suggestion.

Ensuring that the execution environment where OpenTofu is running cannot access anything other than the designated remote API

The way I'd characterize the benefit of this mitigation is in decreasing the potential value of compromise

Yep, I get it. We're on the same page regarding these details.

remaining points about limiting which language features you'll allow seem to imply that you'd be implementing something that analyzes the provided configuration and rejects anything which uses language features outside of your narrow expected bounds

I've been thinking that I would comment out this, this and this before building my tofu CLI application.

Doing so kills off the features which seem most dangerous to me.

the OpenTofu language is going to continue to evolve and so you will need to carefully track additions

Yes. Whenever I bring a new release of tofu into the environment, I'll have to take a close look at its capabilities.

rejecting configurations containing module blocks might not be strictly required ... although ruling that out entirely does seem like it would simplify the problem

It's looking like my strategy is converging on mitigating risks within tofu, rather than looking through the configuration for dangerous elements. My intent in redlining modules, as you surmised, was to simplify the config scanning problem, but I'm beginning to think I might not need to do much (any?) of that.

the execution environment where OpenTofu is running will presumably have access to some credentials for interacting with your API, and you'd presumably like to prevent an attacker from extracting those credentials

The users will be using their own credentials against "my" API in this scenario, so I'm not super worried about that, but thank you for bringing it up. I still don't want them poking around in the local filesystem though :)

instead require them to provide a bespoke configuration format ... this alternative is effectively an allowlist-based approach

Ooh, that's a really interesting idea.

I hope the notes above are a helpful starting point.

Did it ever! Thank you again.

[apparentlymart]

Thanks for that extra context! I was assuming you were intending to use the official OpenTofu releases, but if you are intending to maintain your own fork then of course you'll have some additional options! If you're doing that anyway, you could potentially go further with some other changes, though of course the more you change the harder it would be to upgrade so I'm only mentioning these in case they are interesting, not necessarily recommending any of them:

• Disable all of the provider installation behavior completely and only support loading your special provider from a fixed location.

  To do this, you could patch command.Meta.providerFactories by deleting everything it currently does and instead just returning a hard-coded result with a single entry for your special provider.

      return map[addrs.Provider]providers.Factory{
          yourProviderSourceAddress: providerFactory(&providercache.CachedProvider{
              Provider: yourProviderSourceAddress,
              Version: placeholderVersion, // doesn't matter if you aren't letting users choose a provider release anyway
              PackageDir: pathToYourProviderPackageDir,
          }),
      }, nil

  pathToYourProviderPackageDir should be a filesystem path to the directory containing the executable file named something like terraform-plugin-whatever, which OpenTofu would normally have constructed automatically as something under .terraform/providers but can be anywhere in the filesystem you like in this patched version.

• What I described above would also disable the special terraform.io/builtin/terraform provider that offers the terraform_remote_state data source, which is something I forgot about in my previous comment.

  If left enabled, that would allow someone to use a data "terraform_remote_state" "example" block to cause any of the available remote state backends to fetch a state snapshot from somewhere. The local backend could potentially be used to read files from local disk, though admittedly only if they are sufficiently like OpenTofu's state snapshot format for them to be decoded successfully.

• Make it illegal to declare module blocks, if you decide you want to pursue that. (I know we've both been discussing whether that is truly necessary if you also make the other changes we've been discussing.)

  To do this, you could delete the entire body of configs.decodeModuleBlock and replace it with just immediately returning an error diagnostic reporting that module blocks are not allowed in your proprietary subset of the OpenTofu language.

• You were already on the right track for disabling provisioners, but similar to what I said about providers above it's actually command.Meta.provisionerFactories that you'd need to modify.

  If you don't do that then someone could include a provisioner executable in the configuration they send you, which would be found by the discovery.FindPlugins call in that function because m.pluginDirs returns a list of search paths that includes the current working directory and a specific named subdirectory of it.

• Removing the function declarations you indicated ought to get rid of the filesystem-related functions as you suspected, though I might suggest leaving the functions declared but replacing them all with an implementation that immediately returns an error explaining that those functions are not available in your proprietary subset of the OpenTofu language.

  abspath and pathexpand both actually interact with the filesystem too, but in much more limited ways that might not be concerning to you.

With all of that said: if the only truly-sensitive information available in the execution environment is credentials that the author of the configuration already knows anyway then I'd be tempted to instead just make sure there's nothing else that's sensitive available in the filesystem accessible to the OpenTofu process and the plugin child processes -- for example, by running them in separate namespaces including a separate mount namespace whose root filesystem contains only the OpenTofu executable, the provider plugin package directory, and the configuration that your user provided.

Then you could use an unmodified OpenTofu and rely on the isolation facilities in the host OS to keep things confined... I guess this is a generalization of your original idea of using a network firewall to minimize what external network resources these processes can access, extended to also constrain what filesystem and other local OS resources the processes can access.

Of course there's still some risk of there being bugs in this OS-level isolation mechanisms, but far more organizations are relying on Linux's namespace features for process isolation than are relying on a hypothetical locally-patched OpenTofu CLI executable and so as long as you keep your host OS up-to-date it's likely that any such problems will be found and fixed by someone else before they become a problem for you. 😀

[ghrltu]

Thank you for the detailed reply.

Provider Guardrails

I've decided use the filesystem_mirrror approach to limit access to available providers because:

• Allowing the author of the configuration to specify the version of my provider sounds useful.
• A filesystem-based mirror means there's one less outbound network service I need to account for.
• Allowing other providers into the environment (say, one of the utility providers) is straightforward.

So I've modified command.Meta.providerFactores() to return nil, nil.

edit: It looks like that's not the correct strategy. I've got more fiddling around in here to do.

I think this covers provider-related risks: Configurations will only be able to make use of the specific providers I include in the mirror. Even built-in providers will not function.

Provisioner Guardrails

As you suggested, I've amended command.Meta.provisionerFactores() to return nil.

Function Guardrails

Thank you for mentioning the *path* functions. I've removed those as well. Though I'm tinkering with self-sandboxing via the landlock Linux syscall to prevent OpenTofu from sniffing around outside the module root. If that works out (I don't know the kernel version I'll have available in production), I may not need to kill the filesystem-related functions at all.

[ghrltu]

Okay, I figured out the provider guardrails mentioned previously.

I'd failed to grok the distinction between tofu init, in which providers are cached, and the rest of the lifecycle of a given project.

tofu init is fenced with CLI configuration so that it can only retrieve permitted providers from a local filesystem provider mirror, and network policies will ensure that no network-based registry is available.

tofu apply is when we run command.Meta.providerFactories(). This function has been amended so that it returns anything cached by tofu init, but zeros out the other sources of provider factories: built in, dev overrides, etc...

Regarding namespaces: Yep, I get it. Ideally we'd be running tofu someplace where it can't do any damage. I'm working in a big team and can only control so much. Recommendations about ephemeral runtime environments with limited visibility and limited reach have been made, and I hope will be implemented. I still want to minimize the holes in my slice of cheese to the extent it is possible for me to do so.

Thank you again, @apparentlymart for so generously providing guidance around this.