Skip to content

Stabilize tests#1450

Merged
carterworks merged 23 commits intomainfrom
fix-some-flakey-tests
Mar 6, 2026
Merged

Stabilize tests#1450
carterworks merged 23 commits intomainfrom
fix-some-flakey-tests

Conversation

@carterworks
Copy link
Collaborator

@carterworks carterworks commented Feb 25, 2026

Description

During the creation of the create-browser-package branch, I discovered a couple of patterns that were causing a number of instability in our integration and unit tests.

This includes

  • Mock's on important globals like fetch or querySelector That weren't getting cleaned up.
  • Cookies that were being written and not deleted.
  • URL parameters and URL changes that were happening and not getting returned to the original URL.
  • Dom parameters and nonces that were getting set and not cleaned up.

Because we run our tests in a non isolated and nonparallel manner on our continuous integration actions runners, this was causing the effects of one test to leak into others, and they would fail sometimes because the order was non deterministic.

Related Issue

Motivation and Context

Screenshots (if appropriate):

Checklist:

  • I have signed the Adobe Open Source CLA or I'm an Adobe employee.
  • I have made any necessary test changes and all tests pass.
  • I have run the Sandbox successfully.
  • I have added a Changeset (pnpm changeset) or it is not necessary because this PR is not consumer-facing.

@changeset-bot
Copy link

changeset-bot bot commented Feb 25, 2026

🦋 Changeset detected

Latest commit: d831b02

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@carterworks carterworks requested a review from dompuiu February 25, 2026 21:55
@carterworks carterworks self-assigned this Feb 25, 2026
@carterworks carterworks added the ignore-for-release Do not include this PR in release notes label Feb 25, 2026
};
};

const makeSendServiceWorkerTrackingData =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In serviceWorker.js you already have a makeSendServiceWorkerTrackingData. Can you pass that to this module? When you need to create a new instance?

@dompuiu
Copy link
Member

dompuiu commented Feb 25, 2026

The tests are still failing.

@dompuiu
Copy link
Member

dompuiu commented Feb 26, 2026

I was thinking at this, and the refactoring seems to circumvent the errors but we still not know why is causing the tests to fail. We should probably still try to find out why the tests are flaky.

@carterworks
Copy link
Collaborator Author

I was thinking at this, and the refactoring seems to circumvent the errors but we still not know why is causing the tests to fail. We should probably still try to find out why the tests are flaky.

I think they are failing because two or more tests import the same file, but one test mocks the file and the other does not. Our tests are run all in the same environment to use fewer resources on the CI environment. So the tests are order-dependent, I think. This PR removes the vi.mock(modulePath) usage and the tests pass on CI now.

@carterworks carterworks marked this pull request as draft March 3, 2026 20:49
@carterworks carterworks changed the title fixed flaky tests by restructuring DI boundaries Fixed flaky unit and integration tests Mar 3, 2026
@carterworks carterworks closed this Mar 4, 2026
@carterworks carterworks reopened this Mar 4, 2026
@carterworks carterworks marked this pull request as ready for review March 4, 2026 18:08
@carterworks carterworks changed the title Fixed flaky unit and integration tests Fixed flaky tests by injecting advertising and service-worker dependencies Mar 4, 2026
@carterworks carterworks changed the title Fixed flaky tests by injecting advertising and service-worker dependencies Fixed some flaky tests Mar 4, 2026
@carterworks carterworks changed the title Fixed some flaky tests Stabilize tests Mar 4, 2026
Copy link
Collaborator

@jonsnyder jonsnyder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we get the Ad-Tech contributors to look at your changes to the advertising component?

@carterworks
Copy link
Collaborator Author

Should we get the Ad-Tech contributors to look at your changes to the advertising component?

I don't think that's necessary, since the changes are organizational, not functional

@Spencer-Smith
Copy link
Collaborator

What's up with the "fixme" notes? Is that things that were fixed or need to be fixed?

@carterworks
Copy link
Collaborator Author

What's up with the "fixme" notes? Is that things that were fixed or need to be fixed?

Forgot about those. Fixed

@carterworks carterworks merged commit 78e2f5b into main Mar 6, 2026
6 checks passed
@carterworks carterworks deleted the fix-some-flakey-tests branch March 6, 2026 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ignore-for-release Do not include this PR in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants