Add readiness probes to deployments in functional tests

[sarthyparty]

Proposed changes

Problem: Functional tests failed inconsistently, due to deployments and pods not be ready.

Solution: Added readiness probes to all deployments.

Testing: N/A

Closes #1866

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.88%. Comparing base (2eaffeb) to head (a5de465).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3740      +/-   ##
==========================================
+ Coverage   86.86%   86.88%   +0.01%     
==========================================
  Files         128      128              
  Lines       16187    16187              
  Branches       62       62              
==========================================
+ Hits        14061    14064       +3     
+ Misses       1953     1950       -3     
  Partials      173      173              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bjee19]

For testing can you run the functional tests locally a few times (i know the pipeline runs it 4 times) to make sure there aren't any flakey situations? Also can you verify the readiness probes are applied in the deployments when the tests are running?

[sarthyparty]

@bjee19 After running them several times, the tracing test failed once. I'm not sure how to fix it, as there are readiness probes and we are waiting for them to be ready. It's mentioned in a comment here

[bjee19]

@sarthyparty Yea that looks like its probably not an issue with the example apps not being ready in time (something that wouldn't get fixed with the readiness probes) so the other tests passing consistently is good. Were you also able to verify the readiness probes in the deployment yaml as the apps were rolling out?

[sarthyparty]

I saw the readiness probe on the pod and deployment when I described them

[bjee19]

Technically this updates the probes in the NFRs too, but i think thats fine/better. nice