[SVLS-8054] add integration testing

[jchrostek-dd]

Overview

Setup integration tests for the lambda extension. These integration tests will run on every PR.

Details

This PR includes:

• CDK stacks for deploying lambda integration tests. This is for the lambda, and any related resources, we want to test against.
• Integration tests, setup with Jest. These invoke lambda functions, wait, then get Datadog telemetry data to test/verify against.
• Gitlab Integration Test Step (info below).
• README.md for how to run tests locally.

Note:

• For simplicity, this is setup to just test against the ARM variant (not AMD). This also doesn't include FIPS or AppSec builds. I think this should be a reasonable starting point for our integration tests and we can evaluate adding additional configuration support as needed.

Gitlab Integration Test Step

The integration tests step in Gitlab will:

1. Publish the lambda extension.
2. Deploy CDK stacks, using the newly published lambda extension.
3. Run a test suite.
4. Destroy the CDK stacks.
5. Delete the lambda extension.

Executing the integration tests

The integration tests will automatically run on every PR. Developers can also run the integration tests locally by running npm run test. Full information is included in README.md.

Example Integration Tests

I added a 2 basic tests, one for node and one for python. These lambda function logs 'Hello World' and are setup with the extension and tracer library. The integration test gets the logs and traces from Datadog. It confirms that we have a log with the message 'Hello World!'. It also confirms we have spans with names aws.lambda.cold_start, aws.lambda.load and aws.lambda. Note that this isn't actually working correct for python for aws.lambda.load and aws.lambda.cold_start. Those spans are created, but with a different traceId so they aren't getting linked to aws.lambda. I will follow up and investigate.

I plan on having a follow up PR with other runtimes.

Testing

This PR triggered the integration tests, can see the corresponding gitlab pipeline with the newly added step 'integration-tests'. (Or see the 'dd-gitlab/integration-test' in the checks for this PR)

The results from the integration test can be obtained by going to integration step and downloading the artifacts. Screenshot attached below.

[apiarian-datadog]

might be worth adding some notes here about the sorts of tests we should be including in this suite (vs the e2e test suite.)

[apiarian-datadog]

i'd think that if this is intended to test the extension, then making sure that it is built correctly for arm vs. amd runtimes becomes important. or have we never seen architecture-dependent differences in our extension builds?