Add an option to keep plans parallel but run only inits sequentially

[nikolay]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

• I'd be willing to implement this feature (contributing guide)

Describe the user story

When using a plugin cache, doing parallel inits, which download or upgrade plugins, leads to errors.

Describe the solution you'd like

It's a known issue that running terraform init in parallel leads to errors. The only clean solution is to disable parallel plans with parallel_plan set to false. But the issue is with init, not with plan, so, without being necessary, we also disable the parallel plans.

Describe the drawbacks of your solution

Parallel plans with sequential inits may cause the same errors.

Describe alternatives you've considered

I am currently setting parallel_plan to false.

[jseiser]

@nikolay

We are running this in production, and doing single workspace parallel plans/applies.

#5264

Really hoping it gets merged in at some point, since its reduced our prod deployments from an hour to like 15 minutes.

[ricardbejarano]

@nikolay You might be able to avoid serializing inits with this.

This is how we made Terraform's plugin cache concurrent-safe (for parallel inits to work) while this gets fixed upstream.

Hope it helps!

[nikolay]

@jseiser @ricardbejarano Honestly, without this being fixed, Atlantis is a huge pain in the neck. We have a monorepo and I need to run a bunch of atlantis plan in order for them to eventually pass. Until then, the comments get filled up with lots of weird errors. And this is terrible. All this in addition to no persistence of atlantis apply output, silent failures when there is no approval, undiverged actually not working. Coming from Terraform Cloud (now HPC Terraform),

[jseiser]

@nikolay

I'm right there with you, we have had to remove Atlantis from our production workflow because it turns 10 minute deployments to an hour.

We have an open MR to fix some of the parallel, so hopefully we can get it back in place.

[ricardbejarano]

@nikolay @jseiser I wonder what are you doing that Atlantis feels so sluggish. We felt it was a bit slow before the performance improvements I link in the other comment, but nothing too crazy. Terraform is resource-intensive so you need to give it solid hardware to run on, this includes CPU, RAM and disk, all of them.

After enabling concurrent-safe caching we've seen PRs plan 200 projects in 3 minutes, 1500 projects in 19min. So either you're running a bigger scale deployment than I am, or there are some other settings to be tuned.

[jseiser]

A prod deployment to one of our environments is around 200 states, 90% of which could run in parallel and have no depends on each other, this process takes over an hour.

Atlantis is running in a large cluster, but since it's not actually built for k8s we are just on a massive single pod, with the redis backend. Atlantis ideally would spin up a pod for every deployment but that's a large lift.

As for your link, it's not clear to me how that's supposed to be integrated into a normal k8s flow.

I can definitely mount another volume to the prod, I'm sure I can override the entry point to create those dir's if they don't exist and mount the volume

Running your script post init, and how to enable parallel init without parallel plan/apply I don't know.

Any help would be massive, sober can get Atlantis back into the prod workflow

[ricardbejarano]

Atlantis ideally would spin up a pod for every deployment but that's a large lift.

Yup, that's something I'd like to see too.

You should run everything in the post by creating a custom workflow, that's what we do. It runs on every plan/apply, allowing for parallel planning to be enabled but writes to init are sequential. We run on ECS, it doesn't matter if you run this in Kubernetes, the setup should be basically the same.

Are you in the #atlantis channel of the CNCF slack? Find me there and I'll help you set this up.

[jseiser]

I am, once I get my kids off to school I'll be in.

Thanks,

[nikolay]

@ricardbejarano This should work outside of the box as it's a basic use case to have more than one project affected by a PR. Turning off plugin caching does not fix this. Turning off parallel planning does not either. Basically, all we must do is keep commenting with atlantis plan until all projects pass! I won't even mention how annoying are all the useless comments with errors such as weird file system errors, complaints that a provider is missing, etc. Why do we need custom workflows for the most basic scenario supported in the documentation, i.e. a Terraform monorepo - I have no idea!

[ricardbejarano]

If all you want is serialize inits and keep parallel plans, you can just modify your workflow to do:

flock /path/to/lock terraform${ATLANTIS_TERRAFORM_VERSION} init

I'm not saying Atlantis should not do this out of the box, but I wonder how big of an issue this is if you can't be bothered to do a one-line workaround.

[nikolay]

@ricardbejarano I know how I can do this, but I shouldn't have to, given it's a well-known fact that terraform init cannot be executed in parallel. It is a big deal given the cryptic errors and the need to run multiple plans without a custom workflow. The least you have to do is document this and the suggestion above should be in the documentation until you fix this in the core.

[matthewmrichter]

I've submitted a fix in #6376 that addresses the root cause of this issue.

The problem is that Clone() acquires an exclusive write lock on the shared clone directory for every project, even when just verifying the commit is correct (read-only git rev-parse). Since runSteps() holds a read lock during plan/apply, RWMutex.Lock() in Clone() blocks until the previous project's steps finish — serializing all parallel operations to effectively 1 at a time.

The fix adds an optimistic read-lock fast path: when the directory exists and is at the correct commit, Clone() returns immediately under a read lock without ever acquiring the write lock. This allows true parallel execution for the common autoplan case.