diff --git a/agent-tasks/github-webhook-repo-sync-plan.md b/agent-tasks/github-webhook-repo-sync-plan.md new file mode 100644 index 000000000..7869788a2 --- /dev/null +++ b/agent-tasks/github-webhook-repo-sync-plan.md @@ -0,0 +1,43 @@ +# GitHub repo sync via webhooks + +## Goals +- Move repo sync away from the OAuth callback and drive updates from GitHub webhooks. +- Keep Digger’s repo list accurate when repos are added/removed from the app scope. +- On uninstall, soft-delete repos (and their installation records) so they disappear from the UI/API. + +## Current behavior (source of truth today) +- OAuth callback (`backend/controllers/github_callback.go`) validates the install, links/creates org, then lists all repos via `Apps.ListRepos`, soft-deletes existing `github_app_installations` and repos for the org, and recreates them via `GithubRepoAdded` + `createOrGetDiggerRepoForGithubRepo`. +- Webhook handler (`backend/controllers/github.go`) only uses `installation` events with action `deleted` to mark installation links inactive and set `github_app_installations` status deleted for the repos in the payload. It does not touch `repos`. There is no handling for `installation_repositories` add/remove. +- Runtime lookups (`GetGithubService` / `GetGithubClient`) require an active record in `github_app_installations` for the repo. + +## Target design +- Keep OAuth callback minimal: verify installation, create/link org, store the install id/app id, but do **not** list or mutate repos. It should return immediately and rely on webhooks for repo population. +- Webhook-driven reconciliation: + - `installation` event (`created`, `unsuspended`, `new_permissions_accepted`): ensure installation link exists/active; reconcile repos using the payload’s `installation.repositories` list as authoritative. If the link is missing, log an error and return (no auto-create). + - Soft-delete existing `github_app_installations` for that installation id, and soft-delete repos for the linked org (scoped to that installation) before re-adding. + - Upsert each repo: mark/install via `GithubRepoAdded` and create/restore the Digger repo record (store app id, installation id, default branch, clone URL when available). + - `installation_repositories` event: incrementally apply scope changes. + - For `repositories_added`: fetch repo details (to get default branch + clone URL), then call `GithubRepoAdded` and create/restore the repo record. + - For `repositories_removed`: mark `GithubRepoRemoved`, soft-delete the repo **and its projects**, and handle absence gracefully. + - `installation` event (`deleted`): mark installation link inactive, mark installation records deleted, and soft-delete repos **and projects** for that installation’s org so they no longer appear in APIs/UI. +- Shared helpers: + - `syncReposForInstallation(installationId, appId, reposPayload)` to wrap the add/remove logic and reuse between `installation` and `installation_repositories` handlers. + - `softDeleteRepoAndProjects(orgId, repoFullName)` to encapsulate repo + project soft-deletion. +- Observability: structured logs per action, and possibly a metric for sync success/failure per installation. + +## Migration plan +1) Add webhook handling for `installation_repositories` in `GithubAppWebHook` switch and wire to a new handler. +2) Extend `installation` handling to cover `created`/`unsuspended` (not just `deleted`) and call `syncReposForInstallation`. +3) Update uninstall handling to also soft-delete repos and projects. +4) Strip repo enumeration/deletion from the OAuth callback; leave only installation/org linking. +5) Add tests using existing payload fixtures (`installationRepositoriesAddedPayload`, `installationRepositoriesDeletedPayload`, `installationCreatedEvent`) to verify DB state changes (installation records + repos soft-delete/restore). +6) Backfill existing installations: one-off job/command or admin endpoint to resync repos via `Apps.ListRepos` and `syncReposForInstallation` to align data after deploying (manual trigger, no cron yet). + +## Testing / validation +- Unit tests for add/remove/uninstall flows verifying: + - `github_app_installations` status transitions. + - Repos are created/restored with correct installation/app ids. + - Repos and projects are soft-deleted on removal/uninstall. + +## Open questions +- None right now (decided: log missing-link errors only; soft-delete repos and projects on removal/uninstall; add manual resync endpoint, no cron yet). diff --git a/backend/bootstrap/main.go b/backend/bootstrap/main.go index 99ce5c9b3..aed837454 100644 --- a/backend/bootstrap/main.go +++ b/backend/bootstrap/main.go @@ -242,6 +242,7 @@ func Bootstrap(templates embed.FS, diggerController controllers.DiggerController githubApiGroup := apiGroup.Group("/github") githubApiGroup.POST("/link", controllers.LinkGithubInstallationToOrgApi) + githubApiGroup.POST("/resync", controllers.ResyncGithubInstallationApi) vcsApiGroup := apiGroup.Group("/connections") vcsApiGroup.GET("/:id", controllers.GetVCSConnection) diff --git a/backend/controllers/github.go b/backend/controllers/github.go index 6accb0c34..cc5355dd7 100644 --- a/backend/controllers/github.go +++ b/backend/controllers/github.go @@ -73,6 +73,24 @@ func (d DiggerController) GithubAppWebHook(c *gin.Context) { c.String(http.StatusAccepted, "Failed to handle webhook event.") return } + } else if *event.Action == "created" || *event.Action == "unsuspended" || *event.Action == "new_permissions_accepted" { + if err := handleInstallationUpsertEvent(c.Request.Context(), gh, event, appId64); err != nil { + slog.Error("Failed to handle installation upsert event", "error", err) + c.String(http.StatusAccepted, "Failed to handle webhook event.") + return + } + } + case *github.InstallationRepositoriesEvent: + slog.Info("Processing InstallationRepositoriesEvent", + "action", event.GetAction(), + "installationId", event.Installation.GetID(), + "added", len(event.RepositoriesAdded), + "removed", len(event.RepositoriesRemoved), + ) + if err := handleInstallationRepositoriesEvent(c.Request.Context(), gh, event, appId64); err != nil { + slog.Error("Failed to handle installation repositories event", "error", err) + c.String(http.StatusAccepted, "Failed to handle webhook event.") + return } case *github.PushEvent: slog.Info("Processing PushEvent", diff --git a/backend/controllers/github_api.go b/backend/controllers/github_api.go index ea58ea056..dc0ceed31 100644 --- a/backend/controllers/github_api.go +++ b/backend/controllers/github_api.go @@ -8,7 +8,10 @@ import ( "github.com/diggerhq/digger/backend/middleware" "github.com/diggerhq/digger/backend/models" + "github.com/diggerhq/digger/backend/utils" + ci_github "github.com/diggerhq/digger/libs/ci/github" "github.com/gin-gonic/gin" + "github.com/google/go-github/v61/github" "gorm.io/gorm" ) @@ -85,3 +88,82 @@ func LinkGithubInstallationToOrgApi(c *gin.Context) { c.JSON(http.StatusOK, gin.H{"status": "Successfully created Github installation link"}) return } + +func ResyncGithubInstallationApi(c *gin.Context) { + type ResyncInstallationRequest struct { + InstallationId string `json:"installation_id"` + } + + var request ResyncInstallationRequest + if err := c.BindJSON(&request); err != nil { + slog.Error("Error binding JSON for resync", "error", err) + c.JSON(http.StatusBadRequest, gin.H{"status": "Invalid request format"}) + return + } + + installationId, err := strconv.ParseInt(request.InstallationId, 10, 64) + if err != nil { + slog.Error("Failed to convert InstallationId to int64", "installationId", request.InstallationId, "error", err) + c.JSON(http.StatusBadRequest, gin.H{"status": "installationID should be a valid integer"}) + return + } + + link, err := models.DB.GetGithubAppInstallationLink(installationId) + if err != nil { + slog.Error("Could not get installation link for resync", "installationId", installationId, "error", err) + c.JSON(http.StatusInternalServerError, gin.H{"status": "Could not get installation link"}) + return + } + if link == nil { + slog.Warn("Installation link not found for resync", "installationId", installationId) + c.JSON(http.StatusNotFound, gin.H{"status": "Installation link not found"}) + return + } + + var installationRecord models.GithubAppInstallation + if err := models.DB.GormDB.Where("github_installation_id = ?", installationId).Order("updated_at desc").First(&installationRecord).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + slog.Warn("No installation records found for resync", "installationId", installationId) + c.JSON(http.StatusNotFound, gin.H{"status": "No installation records found"}) + return + } + slog.Error("Failed to fetch installation record for resync", "installationId", installationId, "error", err) + c.JSON(http.StatusInternalServerError, gin.H{"status": "Could not fetch installation records"}) + return + } + + appId := installationRecord.GithubAppId + ghProvider := utils.DiggerGithubRealClientProvider{} + + client, _, err := ghProvider.Get(appId, installationId) + if err != nil { + slog.Error("Failed to create GitHub client for resync", "installationId", installationId, "appId", appId, "error", err) + c.JSON(http.StatusInternalServerError, gin.H{"status": "Failed to create GitHub client"}) + return + } + + repos, err := ci_github.ListGithubRepos(client) + if err != nil { + slog.Error("Failed to list repos for resync", "installationId", installationId, "error", err) + c.JSON(http.StatusInternalServerError, gin.H{"status": "Failed to list repos for resync"}) + return + } + + installationPayload := &github.Installation{ + ID: github.Int64(installationId), + AppID: github.Int64(appId), + } + resyncEvent := &github.InstallationEvent{ + Installation: installationPayload, + Repositories: repos, + } + + if err := handleInstallationUpsertEvent(c.Request.Context(), ghProvider, resyncEvent, appId); err != nil { + slog.Error("Resync failed", "installationId", installationId, "error", err) + c.JSON(http.StatusInternalServerError, gin.H{"status": "Resync failed"}) + return + } + + slog.Info("Resync completed", "installationId", installationId, "repoCount", len(repos)) + c.JSON(http.StatusOK, gin.H{"status": "Resync completed", "repoCount": len(repos)}) +} diff --git a/backend/controllers/github_callback.go b/backend/controllers/github_callback.go index 7b6d34ee3..28b145bb6 100644 --- a/backend/controllers/github_callback.go +++ b/backend/controllers/github_callback.go @@ -5,12 +5,9 @@ import ( "log/slog" "net/http" "strconv" - "strings" "github.com/diggerhq/digger/backend/models" "github.com/diggerhq/digger/backend/segment" - "github.com/diggerhq/digger/backend/utils" - "github.com/diggerhq/digger/libs/ci/github" "github.com/gin-gonic/gin" "github.com/google/uuid" ) @@ -30,27 +27,13 @@ func (d DiggerController) GithubAppCallbackPage(c *gin.Context) { } //setupAction := c.Request.URL.Query()["setup_action"][0] codeParams, codeExists := c.Request.URL.Query()["code"] - if !codeExists || len(codeParams) == 0 { - slog.Error("There was no code in the url query parameters") - c.String(http.StatusBadRequest, "could not find the code query parameter for github app") - return - } - code := codeParams[0] - if len(code) < 1 { - slog.Error("Code parameter is empty") - c.String(http.StatusBadRequest, "code parameter for github app is empty") - return + code := "" + if codeExists && len(codeParams) > 0 && len(codeParams[0]) > 0 { + code = codeParams[0] } appId := c.Request.URL.Query().Get("state") - slog.Info("Processing GitHub app callback", "installationId", installationId, "appId", appId) - - clientId, clientSecret, _, _, err := d.GithubClientProvider.FetchCredentials(appId) - if err != nil { - slog.Error("Could not fetch credentials for GitHub app", "appId", appId, "error", err) - c.String(http.StatusInternalServerError, "could not find credentials for github app") - return - } + slog.Info("Processing GitHub app callback", "installationId", installationId, "appId", appId, "hasCode", code != "") installationId64, err := strconv.ParseInt(installationId, 10, 64) if err != nil { @@ -62,30 +45,42 @@ func (d DiggerController) GithubAppCallbackPage(c *gin.Context) { return } - slog.Debug("Validating GitHub callback", "installationId", installationId64, "clientId", clientId) + // vcsOwner is used for analytics; we'll populate it if we can validate via OAuth + var vcsOwner string + + // If we have a code parameter, validate the callback via OAuth + // This provides additional security by confirming the user authorized the installation + if code != "" { + clientId, clientSecret, _, _, err := d.GithubClientProvider.FetchCredentials(appId) + if err != nil { + slog.Error("Could not fetch credentials for GitHub app", "appId", appId, "error", err) + c.String(http.StatusInternalServerError, "could not find credentials for github app") + return + } + + slog.Debug("Validating GitHub callback", "installationId", installationId64, "clientId", clientId) - result, installation, err := validateGithubCallback(d.GithubClientProvider, clientId, clientSecret, code, installationId64) - if !result { - slog.Error("Failed to validate installation ID", + result, installation, err := validateGithubCallback(d.GithubClientProvider, clientId, clientSecret, code, installationId64) + if !result { + slog.Error("Failed to validate installation ID", + "installationId", installationId64, + "error", err, + ) + c.String(http.StatusInternalServerError, "Failed to validate installation_id.") + return + } + + if installation != nil && installation.Account != nil && installation.Account.Login != nil { + vcsOwner = *installation.Account.Login + } + } else { + slog.Info("No code parameter provided, skipping OAuth validation (repos will sync via webhook)", "installationId", installationId64, - "error", err, ) - c.String(http.StatusInternalServerError, "Failed to validate installation_id.") - return } - // TODO: Lookup org in GithubAppInstallation by installationID if found use that installationID otherwise - // create a new org for this installationID - // retrieve org for current orgID - installationIdInt64, err := strconv.ParseInt(installationId, 10, 64) - if err != nil { - slog.Error("Failed to parse installation ID as int64", - "installationId", installationId, - "error", err, - ) - c.JSON(http.StatusInternalServerError, gin.H{"error": "installationId could not be parsed"}) - return - } + // Lookup or create org for this installation + installationIdInt64 := installationId64 slog.Debug("Looking up GitHub app installation link", "installationId", installationIdInt64) @@ -153,11 +148,7 @@ func (d DiggerController) GithubAppCallbackPage(c *gin.Context) { org := link.Organisation orgId := link.OrganisationId - var vcsOwner string = "" - if installation.Account.Login != nil { - vcsOwner = *installation.Account.Login - } - // we have multiple repos here, we don't really want to send an track event for each repo, so we just send the vcs owner + // vcsOwner was populated earlier if we had a code parameter for OAuth validation segment.Track(*org, vcsOwner, "", "github", "vcs_repo_installed", map[string]string{}) // create a github installation link (org ID matched to installation ID) @@ -172,120 +163,9 @@ func (d DiggerController) GithubAppCallbackPage(c *gin.Context) { return } - slog.Debug("Getting GitHub client", - "appId", *installation.AppID, - "installationId", installationId64, - ) - - client, _, err := d.GithubClientProvider.Get(*installation.AppID, installationId64) - if err != nil { - slog.Error("Error retrieving GitHub client", - "appId", *installation.AppID, - "installationId", installationId64, - "error", err, - ) - c.JSON(http.StatusInternalServerError, gin.H{"error": "Error fetching organisation"}) - return - } - - // we get repos accessible to this installation - slog.Debug("Listing repositories for installation", "installationId", installationId64) - - repos, err := github.ListGithubRepos(client) - if err != nil { - slog.Error("Failed to list existing repositories", - "installationId", installationId64, - "error", err, - ) - c.String(http.StatusInternalServerError, "Failed to list existing repos: %v", err) - return - } - - // resets all existing installations (soft delete) - slog.Debug("Resetting existing GitHub installations", - "installationId", installationId, - ) - - var AppInstallation models.GithubAppInstallation - err = models.DB.GormDB.Model(&AppInstallation).Where("github_installation_id=?", installationId).Update("status", models.GithubAppInstallDeleted).Error - if err != nil { - slog.Error("Failed to update GitHub installations", - "installationId", installationId, - "error", err, - ) - c.String(http.StatusInternalServerError, "Failed to update github installations: %v", err) - return - } - - // reset all existing repos (soft delete) - slog.Debug("Soft deleting existing repositories", - "orgId", orgId, - ) - - var ExistingRepos []models.Repo - err = models.DB.GormDB.Delete(ExistingRepos, "organisation_id=?", orgId).Error - if err != nil { - slog.Error("Could not delete repositories", - "orgId", orgId, - "error", err, - ) - c.String(http.StatusInternalServerError, "could not delete repos: %v", err) - return - } - - // here we mark repos that are available one by one - slog.Info("Adding repositories to organization", - "orgId", orgId, - "repoCount", len(repos), - ) - - for i, repo := range repos { - repoFullName := *repo.FullName - repoOwner := strings.Split(*repo.FullName, "/")[0] - repoName := *repo.Name - repoUrl := fmt.Sprintf("https://%v/%v", utils.GetGithubHostname(), repoFullName) - - slog.Debug("Processing repository", - "index", i+1, - "repoFullName", repoFullName, - "repoOwner", repoOwner, - "repoName", repoName, - ) - - _, err := models.DB.GithubRepoAdded( - installationId64, - *installation.AppID, - *installation.Account.Login, - *installation.Account.ID, - repoFullName, - ) - if err != nil { - slog.Error("Error recording GitHub repository", - "repoFullName", repoFullName, - "error", err, - ) - c.String(http.StatusInternalServerError, "github repos added error: %v", err) - return - } - - cloneUrl := *repo.CloneURL - defaultBranch := *repo.DefaultBranch - - _, _, err = createOrGetDiggerRepoForGithubRepo(repoFullName, repoOwner, repoName, repoUrl, installationId64, *installation.AppID, defaultBranch, cloneUrl) - if err != nil { - slog.Error("Error creating or getting Digger repo", - "repoFullName", repoFullName, - "error", err, - ) - c.String(http.StatusInternalServerError, "createOrGetDiggerRepoForGithubRepo error: %v", err) - return - } - } - - slog.Info("GitHub app callback processed successfully", + slog.Info("GitHub app callback processed", "installationId", installationId64, "orgId", orgId, - "repoCount", len(repos), ) c.HTML(http.StatusOK, "github_success.tmpl", gin.H{}) diff --git a/backend/controllers/github_installation.go b/backend/controllers/github_installation.go index 1db95b13a..9c6abea9c 100644 --- a/backend/controllers/github_installation.go +++ b/backend/controllers/github_installation.go @@ -1,13 +1,124 @@ package controllers import ( + "context" + "errors" + "fmt" + "log/slog" + "time" + "github.com/diggerhq/digger/backend/models" + "github.com/diggerhq/digger/backend/utils" "github.com/google/go-github/v61/github" - "log/slog" + "github.com/sethvargo/go-retry" ) +func getAccountDetails(account *github.User) (string, int64) { + if account == nil { + return "", 0 + } + return account.GetLogin(), int64(account.GetID()) +} + +// fetchRepoIdentifiers returns repo identifiers and fills missing branch/clone URL by calling GitHub if needed. +func fetchRepoIdentifiers(ctx context.Context, client *github.Client, repo *github.Repository, installationId int64) (repoFullName, owner, name, defaultBranch, cloneURL string, err error) { + repoFullName = repo.GetFullName() + if repo.Owner != nil { + owner = repo.Owner.GetLogin() + } + name = repo.GetName() + defaultBranch = repo.GetDefaultBranch() + cloneURL = repo.GetCloneURL() + + if repoFullName == "" && owner != "" && name != "" { + repoFullName = fmt.Sprintf("%s/%s", owner, name) + } + + if (defaultBranch == "" || cloneURL == "") && owner != "" && name != "" { + repoDetails, _, fetchErr := client.Repositories.Get(ctx, owner, name) + if fetchErr != nil { + slog.Error("Error fetching repo details", + "installationId", installationId, + "repoOwner", owner, + "repoName", name, + "error", fetchErr) + return repoFullName, owner, name, defaultBranch, cloneURL, fetchErr + } + if defaultBranch == "" { + defaultBranch = repoDetails.GetDefaultBranch() + } + if cloneURL == "" { + cloneURL = repoDetails.GetCloneURL() + } + } + + return repoFullName, owner, name, defaultBranch, cloneURL, nil +} + +func upsertRepo(ctx context.Context, ghClient *github.Client, repo *github.Repository, installationId int64, appId int64, accountLogin string, accountId int64) error { + repoFullName, owner, name, defaultBranch, cloneURL, err := fetchRepoIdentifiers(ctx, ghClient, repo, installationId) + if err != nil { + return err + } + if repoFullName == "" || owner == "" || name == "" { + slog.Warn("Skipping repo with missing identifiers", + "installationId", installationId, + "repoFullName", repoFullName, + "owner", owner, + "name", name, + ) + return nil + } + + if _, err := models.DB.GithubRepoAdded(installationId, appId, accountLogin, accountId, repoFullName); err != nil { + slog.Error("Error recording GitHub repository", + "installationId", installationId, + "repoFullName", repoFullName, + "error", err) + return err + } + + repoUrl := fmt.Sprintf("https://%s/%s", utils.GetGithubHostname(), repoFullName) + if _, _, err := createOrGetDiggerRepoForGithubRepo(repoFullName, owner, name, repoUrl, installationId, appId, defaultBranch, cloneURL); err != nil { + slog.Error("Error creating or getting Digger repo", + "installationId", installationId, + "repoFullName", repoFullName, + "error", err) + return err + } + + return nil +} + +func removeRepo(ctx context.Context, repo *github.Repository, installationId int64, appId int64, orgId uint) error { + repoFullName := repo.GetFullName() + if repoFullName == "" { + slog.Warn("Skipping repo removal with empty full name", "installationId", installationId) + return nil + } + + if _, err := models.DB.GithubRepoRemoved(installationId, appId, repoFullName); err != nil { + slog.Error("Error marking GitHub repo removed", + "installationId", installationId, + "repoFullName", repoFullName, + "error", err) + return err + } + + if err := models.DB.SoftDeleteRepoAndProjects(orgId, repoFullName); err != nil { + slog.Error("Error soft deleting repo and projects on remove", + "installationId", installationId, + "repoFullName", repoFullName, + "orgId", orgId, + "error", err) + return err + } + + return nil +} + func handleInstallationDeletedEvent(installation *github.InstallationEvent, appId int64) error { - installationId := *installation.Installation.ID + installationId := installation.Installation.GetID() slog.Info("Handling installation deleted event", "installationId", installationId, @@ -20,26 +131,28 @@ func handleInstallationDeletedEvent(installation *github.InstallationEvent, appI return err } - _, err = models.DB.MakeGithubAppInstallationLinkInactive(link) - if err != nil { + if link == nil { + slog.Error("Installation link not found for deletion", "installationId", installationId) + return nil + } + + if _, err = models.DB.MakeGithubAppInstallationLinkInactive(link); err != nil { slog.Error("Error making installation link inactive", "installationId", installationId, "error", err) return err } - for _, repo := range installation.Repositories { - repoFullName := *repo.FullName - slog.Info("Removing installation for repo", - "installationId", installationId, - "repoFullName", repoFullName, - ) + if err := models.DB.GormDB.Model(&models.GithubAppInstallation{}).Where("github_installation_id = ?", installationId).Update("status", models.GithubAppInstallDeleted).Error; err != nil { + slog.Error("Error marking installations deleted", "installationId", installationId, "error", err) + return err + } - _, err := models.DB.GithubRepoRemoved(installationId, appId, repoFullName) - if err != nil { - slog.Error("Error removing GitHub repo", - "installationId", installationId, - "repoFullName", repoFullName, - "error", err, - ) + if err := models.DB.SoftDeleteReposAndProjectsByInstallation(link.OrganisationId, installationId); err != nil { + slog.Error("Error soft deleting repos/projects for installation", "installationId", installationId, "orgId", link.OrganisationId, "error", err) + return err + } + + for _, repo := range installation.Repositories { + if err := removeRepo(context.Background(), repo, installationId, appId, link.OrganisationId); err != nil { return err } } @@ -47,3 +160,127 @@ func handleInstallationDeletedEvent(installation *github.InstallationEvent, appI slog.Info("Successfully handled installation deleted event", "installationId", installationId) return nil } + +func handleInstallationUpsertEvent(ctx context.Context, gh utils.GithubClientProvider, installation *github.InstallationEvent, appId int64) error { + installationId := installation.Installation.GetID() + appIdFromPayload := appId + if installation.Installation.AppID != nil { + appIdFromPayload = installation.Installation.GetAppID() + } + + accountLogin, accountId := getAccountDetails(installation.Installation.Account) + + // Retry fetching the link since webhook may arrive before OAuth callback creates it + var link *models.GithubAppInstallationLink + backoff := retry.WithMaxRetries(5, retry.NewConstant(2*time.Second)) + err := retry.Do(ctx, backoff, func(ctx context.Context) error { + var dbErr error + link, dbErr = models.DB.GetGithubInstallationLinkForInstallationId(installationId) + if dbErr != nil { + return dbErr // permanent error, stop retrying + } + if link == nil { + return retry.RetryableError(errors.New("installation link not found")) + } + return nil + }) + if err != nil { + slog.Error("Installation link not found after retries", "installationId", installationId, "error", err) + return fmt.Errorf("installation link not found for installation %d after retries: %w", installationId, err) + } + + repoList := installation.Repositories + if len(repoList) == 0 { + slog.Warn("No repositories found to sync for installation", "installationId", installationId) + return nil + } + + slog.Info("Syncing repositories for installation", + "installationId", installationId, + "appId", appIdFromPayload, + "repoCount", len(repoList), + ) + + if err := models.DB.GormDB.Model(&models.GithubAppInstallation{}).Where("github_installation_id = ?", installationId).Update("status", models.GithubAppInstallDeleted).Error; err != nil { + slog.Error("Error marking installations deleted prior to resync", "installationId", installationId, "error", err) + return err + } + + if err := models.DB.SoftDeleteReposAndProjectsByInstallation(link.OrganisationId, installationId); err != nil { + slog.Error("Error soft deleting existing repos/projects prior to resync", "installationId", installationId, "orgId", link.OrganisationId, "error", err) + return err + } + + ghClient, _, err := gh.Get(appIdFromPayload, installationId) + if err != nil { + slog.Error("Error creating GitHub client for repo sync", "installationId", installationId, "error", err) + return err + } + + for _, repo := range repoList { + if err := upsertRepo(ctx, ghClient, repo, installationId, appIdFromPayload, accountLogin, accountId); err != nil { + return err + } + } + + slog.Info("Successfully synced repositories for installation", "installationId", installationId) + return nil +} + +func handleInstallationRepositoriesEvent(ctx context.Context, gh utils.GithubClientProvider, event *github.InstallationRepositoriesEvent, appId int64) error { + installationId := event.Installation.GetID() + appIdFromPayload := appId + if event.Installation.AppID != nil { + appIdFromPayload = event.Installation.GetAppID() + } + + accountLogin, accountId := getAccountDetails(event.Installation.Account) + + // Retry fetching the link since webhook may arrive before OAuth callback creates it + var link *models.GithubAppInstallationLink + backoff := retry.WithMaxRetries(5, retry.NewConstant(2*time.Second)) + err := retry.Do(ctx, backoff, func(ctx context.Context) error { + var dbErr error + link, dbErr = models.DB.GetGithubInstallationLinkForInstallationId(installationId) + if dbErr != nil { + return dbErr // permanent error, stop retrying + } + if link == nil { + return retry.RetryableError(errors.New("installation link not found")) + } + return nil + }) + if err != nil { + slog.Error("Installation link not found after retries", "installationId", installationId, "error", err) + return fmt.Errorf("installation link not found for installation %d after retries: %w", installationId, err) + } + + client, _, err := gh.Get(appIdFromPayload, installationId) + if err != nil { + slog.Error("Error creating GitHub client for installation_repositories event", "installationId", installationId, "error", err) + return err + } + + var errs []error + for _, repo := range event.RepositoriesAdded { + if err := upsertRepo(ctx, client, repo, installationId, appIdFromPayload, accountLogin, accountId); err != nil { + errs = append(errs, err) + } + } + + for _, repo := range event.RepositoriesRemoved { + if err := removeRepo(ctx, repo, installationId, appIdFromPayload, link.OrganisationId); err != nil { + errs = append(errs, err) + } + } + + slog.Info("Handled installation_repositories event", + "installationId", installationId, + "addedCount", len(event.RepositoriesAdded), + "removedCount", len(event.RepositoriesRemoved), + ) + if len(errs) > 0 { + return fmt.Errorf("one or more errors during installation_repositories handling: %v", errs) + } + return nil +} diff --git a/backend/go.mod b/backend/go.mod index 7e067fb46..22975c5dc 100644 --- a/backend/go.mod +++ b/backend/go.mod @@ -52,6 +52,7 @@ require ( require ( github.com/diggerhq/digger/libs v0.0.0-00010101000000-000000000000 + github.com/sethvargo/go-retry v0.3.0 gorm.io/datatypes v1.2.7 ) diff --git a/backend/go.sum b/backend/go.sum index 35f00cf4b..031a86e0b 100644 --- a/backend/go.sum +++ b/backend/go.sum @@ -1,7 +1,5 @@ ariga.io/atlas-go-sdk v0.7.2 h1:pvS8tKVeRQuqdETBqj5qAQtVbQE88Gya6bOfY8YF3vU= ariga.io/atlas-go-sdk v0.7.2/go.mod h1:cFq7bnvHgKTWHCsU46mtkGxdl41rx2o7SjaLoh6cO8M= -ariga.io/atlas-provider-gorm v0.5.0 h1:DqYNWroKUiXmx2N6nf/I9lIWu6fpgB6OQx/JoelCTes= -ariga.io/atlas-provider-gorm v0.5.0/go.mod h1:8m6+N6+IgWMzPcR63c9sNOBoxfNk6yV6txBZBrgLg1o= ariga.io/atlas-provider-gorm v0.5.4 h1:64xboUDrP+JHdZOy4juPydHT5UP1kY152b5Gh/xNzmM= ariga.io/atlas-provider-gorm v0.5.4/go.mod h1:cXt4kxq8KIldPXHoWXC0HvSr8dVI0dIykZt3MZ4AmqE= c2sp.org/CCTV/age v0.0.0-20240306222714-3ec4d716e805 h1:u2qwJeEvnypw+OCPUHmoZE3IqwfuN5kgDfo5MLzpNM0= @@ -759,10 +757,6 @@ github.com/ajstarks/deck v0.0.0-20200831202436-30c9fc6549a9/go.mod h1:JynElWSGnm github.com/ajstarks/deck/generate v0.0.0-20210309230005-c3f852c02e19/go.mod h1:T13YZdzov6OU0A1+RfKZiZN9ca6VeKdBdyDV+BY97Tk= github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af/go.mod h1:K08gAheRH3/J6wwsYMMT4xOr94bZjxIelGM0+d/wbFw= github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b/go.mod h1:1KcenG0jGWcpt8ov532z81sp/kMMUG485J2InIOyADM= -github.com/alecthomas/kong v0.7.1 h1:azoTh0IOfwlAX3qN9sHWTxACE2oV8Bg2gAwBsMwDQY4= -github.com/alecthomas/kong v0.7.1/go.mod h1:n1iCIO2xS46oE8ZfYCNDqdR0b0wZNrXAIAqro/2132U= -github.com/alecthomas/kong v1.9.0 h1:Wgg0ll5Ys7xDnpgYBuBn/wPeLGAuK0NvYmEcisJgrIs= -github.com/alecthomas/kong v1.9.0/go.mod h1:p2vqieVMeTAnaC83txKtXe8FLke2X07aruPWXyMPQrU= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= @@ -1807,6 +1801,8 @@ github.com/segmentio/backo-go v1.0.0/go.mod h1:kJ9mm9YmoWSkk+oQ+5Cj8DEoRCX2JT6As github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo= github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 h1:n661drycOFuPLCN3Uc8sB6B/s6Z4t2xvBgU1htSHuq8= github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4= +github.com/sethvargo/go-retry v0.3.0 h1:EEt31A35QhrcRZtrYFDTBg91cqZVnFL2navjDrah2SE= +github.com/sethvargo/go-retry v0.3.0/go.mod h1:mNX17F0C/HguQMyMyJxcnU471gOZGxCLyYaFyAZraas= github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24/go.mod h1:M+9NzErvs504Cn4c5DxATwIqPbtswREoFCre64PpcG4= github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o= github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= @@ -2845,7 +2841,6 @@ gorm.io/gorm v1.23.7/go.mod h1:l2lP/RyAtc1ynaTjFksBde/O8v9oOGIApu2/xRitmZk= gorm.io/gorm v1.23.8/go.mod h1:l2lP/RyAtc1ynaTjFksBde/O8v9oOGIApu2/xRitmZk= gorm.io/gorm v1.23.10/go.mod h1:DVrVomtaYTbqs7gB/x2uVvqnXzv0nqjB396B8cG4dBA= gorm.io/gorm v1.24.0/go.mod h1:DVrVomtaYTbqs7gB/x2uVvqnXzv0nqjB396B8cG4dBA= -gorm.io/gorm v1.30.0 h1:qbT5aPv1UH8gI99OsRlvDToLxW5zR7FzS9acZDOZcgs= gorm.io/gorm v1.30.0/go.mod h1:8Z33v652h4//uMA76KjeDH8mJXPm1QNCYrMeatR0DOE= gorm.io/gorm v1.30.1 h1:lSHg33jJTBxs2mgJRfRZeLDG+WZaHYCk3Wtfl6Ngzo4= gorm.io/gorm v1.30.1/go.mod h1:8Z33v652h4//uMA76KjeDH8mJXPm1QNCYrMeatR0DOE= diff --git a/backend/models/storage.go b/backend/models/storage.go index 066636c57..0ded34fd4 100644 --- a/backend/models/storage.go +++ b/backend/models/storage.go @@ -511,6 +511,38 @@ func (db *Database) GithubRepoRemoved(installationId int64, appId int64, repoFul return item, nil } +// SoftDeleteRepoAndProjects soft deletes a repo and all its projects for the given org and repo full name. +func (db *Database) SoftDeleteRepoAndProjects(orgId uint, repoFullName string) error { + return db.GormDB.Transaction(func(tx *gorm.DB) error { + if err := tx.Where("organisation_id = ? AND repo_full_name = ?", orgId, repoFullName).Delete(&Project{}).Error; err != nil { + slog.Error("failed to soft delete projects for repo", "orgId", orgId, "repoFullName", repoFullName, "error", err) + return err + } + if err := tx.Where("organisation_id = ? AND repo_full_name = ?", orgId, repoFullName).Delete(&Repo{}).Error; err != nil { + slog.Error("failed to soft delete repo", "orgId", orgId, "repoFullName", repoFullName, "error", err) + return err + } + return nil + }) +} + +// SoftDeleteReposAndProjectsByInstallation soft deletes all repos and projects for a specific installation in an org. +func (db *Database) SoftDeleteReposAndProjectsByInstallation(orgId uint, installationId int64) error { + var repos []Repo + if err := db.GormDB.Where("organisation_id = ? AND github_app_installation_id = ?", orgId, installationId).Find(&repos).Error; err != nil { + slog.Error("failed to fetch repos for soft delete", "orgId", orgId, "installationId", installationId, "error", err) + return err + } + + for _, repo := range repos { + if err := db.SoftDeleteRepoAndProjects(orgId, repo.RepoFullName); err != nil { + return err + } + } + + return nil +} + func (db *Database) GetGithubAppInstallationByOrgAndRepo(orgId any, repo string, status GithubAppInstallStatus) (*GithubAppInstallation, error) { link, err := db.GetGithubInstallationLinkForOrg(orgId) if err != nil { @@ -707,16 +739,16 @@ func (db *Database) MakeGithubAppInstallationLinkInactive(link *GithubAppInstall return link, nil } -func (db *Database) CreateImpactedProject(repoFullName string, commitSha string, projcectName string, branch *string, prNumber *int) (*ImpactedProject,error) { +func (db *Database) CreateImpactedProject(repoFullName string, commitSha string, projcectName string, branch *string, prNumber *int) (*ImpactedProject, error) { ip := ImpactedProject{ - ID: uuid.New(), + ID: uuid.New(), RepoFullName: repoFullName, CommitSha: commitSha, ProjectName: projcectName, - Branch: branch, - PrNumber: prNumber, - Planned: false, - Applied: false, + Branch: branch, + PrNumber: prNumber, + Planned: false, + Applied: false, } err := db.GormDB.Create(&ip).Error if err != nil { diff --git a/backend/models/storage_test.go b/backend/models/storage_test.go index c2e1f8301..0ecfdeda8 100644 --- a/backend/models/storage_test.go +++ b/backend/models/storage_test.go @@ -125,6 +125,88 @@ func TestGithubRepoRemoved(t *testing.T) { assert.Equal(t, GithubAppInstallDeleted, i.Status) } +func TestSoftDeleteRepoAndProjects(t *testing.T) { + teardownSuite, db, org := setupSuite(t) + defer teardownSuite(t) + + installationId := int64(1) + appId := int64(1) + repoFullName := "test/test" + + repo, err := db.CreateRepo("test-test", repoFullName, "test", "test", "", org, "", installationId, appId, "main", "") + assert.NoError(t, err) + assert.NotNil(t, repo) + + project := Project{ + Name: "proj", + OrganisationID: org.ID, + Organisation: org, + RepoFullName: repoFullName, + Status: ProjectActive, + } + err = db.GormDB.Create(&project).Error + assert.NoError(t, err) + + err = db.SoftDeleteRepoAndProjects(org.ID, repoFullName) + assert.NoError(t, err) + + var repoRecord Repo + err = db.GormDB.Unscoped().Where("id = ?", repo.ID).First(&repoRecord).Error + assert.NoError(t, err) + assert.True(t, repoRecord.DeletedAt.Valid) + + var projectRecord Project + err = db.GormDB.Unscoped().Where("id = ?", project.ID).First(&projectRecord).Error + assert.NoError(t, err) + assert.True(t, projectRecord.DeletedAt.Valid) +} + +func TestSoftDeleteReposAndProjectsByInstallation(t *testing.T) { + teardownSuite, db, org := setupSuite(t) + defer teardownSuite(t) + + appId := int64(1) + installA := int64(1) + installB := int64(2) + + repoA, err := db.CreateRepo("org-repo-a", "org/repo-a", "org", "repo-a", "", org, "", installA, appId, "main", "") + assert.NoError(t, err) + repoB, err := db.CreateRepo("org-repo-b", "org/repo-b", "org", "repo-b", "", org, "", installB, appId, "main", "") + assert.NoError(t, err) + + projectA := Project{ + Name: "proj-a", + OrganisationID: org.ID, + Organisation: org, + RepoFullName: repoA.RepoFullName, + Status: ProjectActive, + } + projectB := Project{ + Name: "proj-b", + OrganisationID: org.ID, + Organisation: org, + RepoFullName: repoB.RepoFullName, + Status: ProjectActive, + } + assert.NoError(t, db.GormDB.Create(&projectA).Error) + assert.NoError(t, db.GormDB.Create(&projectB).Error) + + err = db.SoftDeleteReposAndProjectsByInstallation(org.ID, installA) + assert.NoError(t, err) + + var repoARecord, repoBRecord Repo + assert.NoError(t, db.GormDB.Unscoped().Where("id = ?", repoA.ID).First(&repoARecord).Error) + assert.NoError(t, db.GormDB.Unscoped().Where("id = ?", repoB.ID).First(&repoBRecord).Error) + assert.True(t, repoARecord.DeletedAt.Valid) + assert.False(t, repoBRecord.DeletedAt.Valid) + + var projectARecord, projectBRecord Project + assert.NoError(t, db.GormDB.Unscoped().Where("id = ?", projectA.ID).First(&projectARecord).Error) + assert.NoError(t, db.GormDB.Unscoped().Where("id = ?", projectB.ID).First(&projectBRecord).Error) + assert.True(t, projectARecord.DeletedAt.Valid) + assert.False(t, projectBRecord.DeletedAt.Valid) +} + func TestGetDiggerJobsForBatchPreloadsSummary(t *testing.T) { teardownSuite, _, _ := setupSuite(t) defer teardownSuite(t) diff --git a/docs/ce/local-development/backend.mdx b/docs/ce/local-development/backend.mdx index 21895892b..1e7a93a07 100644 --- a/docs/ce/local-development/backend.mdx +++ b/docs/ce/local-development/backend.mdx @@ -1,5 +1,5 @@ --- -title: Backend (orchestrator) local setup +title: Orchestrator local setup --- The backend serves orchestration APIs, GitHub app endpoints, and internal APIs the UI relies on. diff --git a/docs/ce/local-development/github-app.mdx b/docs/ce/local-development/github-app.mdx new file mode 100644 index 000000000..82c577d64 --- /dev/null +++ b/docs/ce/local-development/github-app.mdx @@ -0,0 +1,33 @@ +--- +title: GitHub App settings for local dev +--- + +Use these settings when connecting a GitHub App to your local stack (tunneled via the UI domain). + +## Required URLs + +- **Callback URL** (OAuth/web): `https:///orchestrator/github/callback` +- **Webhook URL**: `https:///orchestrator/github/webhook` +- **Setup URL (optional)**: `https:///dashboard/onboarding?step=github` + +> The UI forwards these to the backend. Ensure `ORCHESTRATOR_BACKEND_URL`/`SECRET` are set in UI and the backend is reachable from the UI host. + +## Permissions & events (recommended) + +- Permissions: `contents:read`, `pull_requests:write`, `issues:write`, `statuses:write`, `checks:write`, `metadata:read`, `administration:read`, `workflows:write`, `repository_hooks:write`, `members:read`. +- Events: `issue_comment`, `pull_request`, `pull_request_review`, `pull_request_review_comment`, `push`, `check_run`, `status`. + +## Install URL + +After creating the app, use its install URL (e.g., `https://github.com/apps//installations/new`) as `ORCHESTRATOR_GITHUB_APP_URL` in `ui/.env.local`. This drives the "Connect with GitHub" button. + +## Creating an app via the backend wizard + +- Open `http://localhost:3000/github/setup` (or the backend’s public URL) to generate a manifest and create the app in GitHub. Needed envs on backend: `HOSTNAME` set to a reachable URL, and optional `GITHUB_ORG` if you want to scope to an org. +- Once created, copy the install URL into `ORCHESTRATOR_GITHUB_APP_URL` and restart the UI. + +## Troubleshooting + +- **404 on connect**: `ORCHESTRATOR_GITHUB_APP_URL` not set or points to a non-existent path. +- **Callbacks fail**: UI not exposed publicly; tunnel the UI port and update callback/webhook URLs to that domain. +- **Backend rejects /api/github/link**: ensure `DIGGER_ENABLE_API_ENDPOINTS=true` and `DIGGER_INTERNAL_SECRET` matches the UI `ORCHESTRATOR_BACKEND_SECRET`. diff --git a/docs/ce/local-development/overview.mdx b/docs/ce/local-development/overview.mdx index d1ab2912a..1234b896d 100644 --- a/docs/ce/local-development/overview.mdx +++ b/docs/ce/local-development/overview.mdx @@ -6,7 +6,7 @@ This section explains how to run the three core services locally: - **Backend** (`backend/`, port 3000 by default) – orchestrator + REST APIs for repos/orgs/jobs. - **Statesman** (`taco/cmd/statesman`, port 8080) – state storage API and Terraform Cloud-compatible endpoints. -- **UI** (`ui/`, port 3030) – TanStack Start frontend that talks to both services and WorkOS. +- **UI** (`ui/`, port 3030) – TanStack Start frontend that talks to both services and WorkOS. When tunneling (e.g., ngrok), expose the UI host; WorkOS and GitHub callbacks should point to the UI domain. ## Prerequisites @@ -27,7 +27,7 @@ This section explains how to run the three core services locally: 2) **Start statesman** with internal endpoints enabled; use memory storage for quick start. 3) **Configure UI** `.env.local` with URLs + secrets + WorkOS creds; run `pnpm dev --host --port 3030`. 4) **Sync org/user** into backend and statesman (WorkOS org id and user id/email) via the provided curl snippets in each page. -5) (Optional) **GitHub App**: set `ORCHESTRATOR_GITHUB_APP_URL` to your install URL or `http://localhost:3000/github/setup` to create one via the backend. +5) (Optional) **GitHub App**: set `ORCHESTRATOR_GITHUB_APP_URL` to your install URL or `http://localhost:3000/github/setup` to create one via the backend. Use the UI domain for app callback/webhook URLs (see GitHub App settings page). ## Troubleshooting cheatsheet diff --git a/docs/ce/local-development/ui.mdx b/docs/ce/local-development/ui.mdx index b5e5768e8..efb83022a 100644 --- a/docs/ce/local-development/ui.mdx +++ b/docs/ce/local-development/ui.mdx @@ -2,21 +2,21 @@ title: UI local setup --- -The UI is a TanStack Start app that authenticates via WorkOS and calls both backend and statesman. +The UI is a TanStack Start app that authenticates via WorkOS and calls both backend and statesman. It also acts as the public gateway when tunneling (e.g., ngrok): expose the UI port, and point external callbacks to the UI domain. ## Quick start 1. Copy `.env.example` to `.env.local` in `ui/` and fill the essentials: ```bash # URLs - PUBLIC_URL=http://localhost:3030 - ALLOWED_HOSTS=localhost,127.0.0.1 + PUBLIC_URL=http://localhost:3030 # replace host with your public tunnel when exposing UI + ALLOWED_HOSTS=localhost,127.0.0.1 # include your public tunnel host here # WorkOS (User Management) WORKOS_CLIENT_ID= WORKOS_API_KEY= WORKOS_COOKIE_PASSWORD=<32+ random chars> - WORKOS_REDIRECT_URI=http://localhost:3030/api/auth/callback + WORKOS_REDIRECT_URI=http://localhost:3030/api/auth/callback # replace host with your public tunnel; must match WorkOS config WORKOS_WEBHOOK_SECRET= # Backend @@ -40,7 +40,7 @@ The UI is a TanStack Start app that authenticates via WorkOS and calls both back pnpm install # or npm install pnpm dev --host --port 3030 ``` - Open `http://localhost:3030` and sign in with a WorkOS user that belongs to at least one org. + Open `http://localhost:3030` (or your tunnel URL) and sign in with a WorkOS user that belongs to at least one org. Ensure the WorkOS redirect URI matches the public URL you configured. 3. Ensure backend + statesman were started and the same secrets are in place (see [Backend](./backend) and [Statesman](./statesman)). 4. Sync the WorkOS org/user to both services using the curl snippets on those pages (required for repos/units to load). @@ -49,3 +49,4 @@ The UI is a TanStack Start app that authenticates via WorkOS and calls both back - **NotFound/Forbidden listing units**: statesman org/user not synced or webhook secret mismatch. - **404 on repos or GitHub connect**: backend missing org/user, `DIGGER_ENABLE_API_ENDPOINTS` not set, or `ORCHESTRATOR_GITHUB_APP_URL` points to a non-existent path. - **WorkOS login succeeds but dashboard redirects to / or errors**: the signed-in user has no WorkOS org membership; add to an org and resync to services. +- **WorkOS redirect blocked**: public URL not whitelisted; add your tunnel host to `ALLOWED_HOSTS` and to the WorkOS redirect URI list. diff --git a/docs/docs.json b/docs/docs.json index d839592a8..c4c354811 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -172,7 +172,8 @@ "ce/local-development/overview", "ce/local-development/backend", "ce/local-development/statesman", - "ce/local-development/ui" + "ce/local-development/ui", + "ce/local-development/github-app" ] }, {