Changes in recloning for merge strategy leads to disappearing plans

[grimm26]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Using Github Enterprise with a custom workflow and checkout-strategy set to merge.

Since

• fix(checkout-strategy): On merge strategy, reclone when necessary on base update #3187

I get plans that "disappear." I'll do a plan. Then a bit later try the apply. This shows in the log "base branch has been updated, using merge checkout-strategy and will clone again", then ""creating dir "/home/atlantis/.atlantis/repos/terraforms/production/10605/default"". The apply runs and comments on the PR "Ran Apply for 0 projects:".

Plan again, then apply immediately and it works.

It seems like atlantis is just recloning the PR and blowing away my plan instead of just merging in the base branch again.

Reproduction Steps

Have checkout-strategy set to merge. Plan a PR. Merge in changes to base branch. Apply PR.

Logs

relevant log shown above. Lemme know if more detailed log is needed.

Environment details

• Atlantis version: I've experienced this problem on 0.23.5 and 0.24.2
• Deployment method: docker container on AWS fargate

[nitrocode]

cc @finnag have you noticed this issue too? Could you look into this when you get a chance?

[finnag]

We run with autoplanning enabled, so if the base branch gets updated it will immediately be replanned, and have therefore not hit this issue.

right now I can think of two ways to remedy it:

• reclone on upstream update only when planning, not on policy-check or apply
• reclone on upstream update only immediately after grabbing a new lock (if you already had the lock, don't check master again).

Will look more into it tomorrow to see what is least intrusive, or if I get any better ideas.

[finnag]

@grimm26 I'm trying to reproduce this now, but need some clarification about the workflow.

You write that this is your workflow:

• Plan a PR
• Merge in changes to base branch
• Apply PR.

If this is correct, how do you merge your changes without closing the PR? If it is closed, how do you then apply it...

The normal workflow is plan, apply, merge.

[grimm26]

@finnag My normal workflow is

1. open a PR. This plans automatically.
2. Wait for review/approval.
3. Apply PR with automerge.

Between steps 1 and 3 another PR for a different project in the same repo may have been applied and merged. This seems to cause (with checkout-strategy = merge) the apply for my PR to reclone and blow away the plan before it tries to apply.

[finnag]

@grimm26 this is similar to the workflow we use all the time, and we do not hit this issue. PRs can be applied/merged in any order, and one PR being merged does not affect any other plans, they can still be applied afterwards.

Do you use the automerge functionality in atlantis, or in github?

The code path that handles the atlantis apply command should not call Clone at all, so if you are able to get it to log "base branch has been updated, using merge strategy and will clone again" as a result of an atlantis apply command, it would be very interesting to see what the log output from atlantis is around when that happens. Especially if you can enable debug logging ( --log-level debug)

[finnag]

Do you have any workflow hooks defined?

[grimm26]

@grimm26 this is similar to the workflow we use all the time, and we do not hit this issue. PRs can be applied/merged in any order, and one PR being merged does not affect any other plans, they can still be applied afterwards.

Do you use the automerge functionality in atlantis, or in github?

The code path that handles the atlantis apply command should not call Clone at all, so if you are able to get it to log "base branch has been updated, using merge strategy and will clone again" as a result of an atlantis apply command, it would be very interesting to see what the log output from atlantis is around when that happens. Especially if you can enable debug logging ( --log-level debug)

Well, https://github.com/runatlantis/atlantis/blob/main/server/events/working_dir.go#L120-L122 indicates that I'll get a forced clone. I can set up my test atlantis instance with debug logs to get some more detailed info for you.

[grimm26]

Do you have any workflow hooks defined?

yes, I use a pre-workflow hook to build my atlantis.yaml file at the top of the repo.

[finnag]

Right. the pre-workflow hook will call Clone(), so that's probably the difference in your setup. I'll try to reproduce and see what the best fix is.

[nitrocode]

@grimm26 please include your configuration too, it may help reproduce the issue.

[edbighead]

We're experiencing the same issue after migrating from 0.22.3 to 0.24.2.

Errors look like this

dir ".tmp/dev/my-service/infra" does not exist

files in this directory are rendered during pre_workflow_hook step and atlantis.yaml looks like

projects:
  - dir: .tmp/dev/my-service/infra
...

Error usually happens when PR has 20+ plans and works with under 10 plans, perhaps because stuff gets merged to main branch when the long plans are running