Improve flaky and errored tests in report

[ysfaran]

Issues

While starting to work with CTRF and playwright-ctrf-json-reporter I noticed some ways to improve the errors. Resolving mentioned issues below would generally help to debug errors more easily.

1. Include snippet for errors

Currently the playwright-ctrf-json-reporter only takes the message and stack from the testResult.error object:

playwright-ctrf-json-reporter/src/generate-report.ts

Lines 370 to 387 in 31d6f32

	extractFailureDetails(testResult: TestResult): Partial<CtrfTest> {
	if (
	(testResult.status === 'failed' ||
	testResult.status === 'timedOut' ||
	testResult.status === 'interrupted') &&
	testResult.error !== undefined
	) {
	const failureDetails: Partial<CtrfTest> = {}
	if (testResult.error.message !== undefined) {
	failureDetails.message = testResult.error.message
	}
	if (testResult.error.stack !== undefined) {
	failureDetails.trace = testResult.error.stack
	}
	return failureDetails
	}
	return {}
	}

What would be helpful is to also include the testResult.error.snippet in the report, which contains the code snippet, that caused the error. It's a string value that looks like this when logged:

2. Include retry attempts for flaky tests

Currently the code only takes the latestResult into consideration:

playwright-ctrf-json-reporter/src/generate-report.ts

Line 154 in 31d6f32

const latestResult = testCase.results[testCase.results.length - 1]

This is enough to get a summary for the test, but the CTRF report won't contain any details about the failed retry attempts in case of a flaky test. I would like to at least include the errors of the failed attempts, but if attempts are added anyway, it might just make sense to include all details.

Resolution

First of all I want to hear your opinion about these changes. I find them very useful, but just started with CTRF so I don't know about the architectural plans here.

If you find the suggestions useful and you are in need of assistance, I would also gladly work on this issue.
There are just few things I would need to know as the changes themselves are quite straightforward:

• If we decide to add the snippet field, which field should I use: extra or just a field snippet next to stack and message
• If we decide to add retry attempt details, which field should I use: extra or a new field retryAttempts which would be an Array<CtrfTest> without retryAttempts.

(P.S. I want to build an internal dashboard for our team to monitor our tests (especially the flaky ones) from different sources like jest and playwright. CTRF is super helpful in this case. I really appreciate all the work done, thank you a lot!)

[Ma11hewThomas]

I agree that adding a snippet property to the schema is a great idea. While it might be specific to Playwright at the moment - other frameworks I’ve worked with usually include snippets within the failure message or stack trace - having a dedicated snippet property like Playwright offers clear benefits. Since we’re still at beta 0.0.0 of the schema, I’m happy to add this property directly to the test object. Please feel free to submit a PR to add the snippet to the Playwright reporter in this repo.

Regarding previousAttempts, this has also been raised here, and I’m also keen to add it, but it requires further design consideration. Modeling it as an Array<CtrfTest> could lead to recursive nesting of the previousAttempts property, and also some test properties would remain unchanged across attempts.

To move forward:

I will add a snippet property of type string to the test object, so you can proceed with a PR for the reporter here.

A previousAttempts property of type object will be added to the test object. However, its design is still being finalised - although it will likely include at least status, duration, message, trace, and snippet. You’re welcome to prepare a PR for this change if you’d like.

P.S. Great to hear you’re building a dashboard on CTRF!

[ysfaran]

@Ma11hewThomas I added a PR #21 for the snippet now.
I will add a seperate PR for the previousAttempts PR hopefully soon, with the minimum properties you mentioned.

It would then suffice my needs :)

[ysfaran]

@Ma11hewThomas I finally had the time to add another PR for the previousAttempts: #24

[Ma11hewThomas]

Thanks @ysfaran! v0.0.22