Intermittent failures in data upload tests (e.g., B.3.16)

[kileyyeaman]

I noticed that many of the data import tests fail intermittently (e.g., B.3.16). I am working off the 15.5.7 branch of redcap_cypress right now for a 15.5.20 revalidation.

I am not sure if the intermittent failures are still a problem, but if it is, I think it might be related to these lines in commands/data_imports.js:

    cy.top_layer(label_selector).within(() => {
        if(nearest_text.length > 0) {
            upload_element = cy.get_labeled_element(upload_selector, nearest_text).first()
        } else {
            upload_element = cy.get(selector)
        }

        upload_element.selectFile(filePath)
        
        if(button_label !== '') {
            upload_element.closestIncludingChildren(submit_button_selector).click()
        }
    })

I spent some time trying to debug and from what I have discovered, Cypress documentation notes that you cannot assign return values of any Cypress command (including cy.get()), to constants/variables due to it's asynchronous nature. (See the first code chunk near the top of the page for an explanation). So I think lines such as the one below could potentially be causing intermittent failures:

upload_element = cy.get_labeled_element(upload_selector, nearest_text).first()

Unfortunately I don't have the bandwidth this week to refactor this and make a PR, as I see it relies on some functions in commands/base.js. If it continues to be an issue I can possibly contribute in the future. I am new to the Cypress framework, so if I am way off base on this, feel free to ignore. Just wanted to give a heads up! :)

[mmcev106]

Hmm, I'm not seeing intermittent failures for any tests on the cypress cloud, and at first glance am not seeing any changes that mention B.3.16 tests in rctf or redcap_rsvc since 15.5.7. While it's true that return values work a little differently in cypress, I'm not seeing any issues with those lines. Most cypress calls return a "Chainable" object. As long as that return value is used as a chainable rather than a raw value, things will work as expected.

If you share the specifics of the intermittent failure you're seeing, I may be able to help.

[kileyyeaman]

Thank @mmcev106 for the reply. This was helpful and sent me down a bit of a rabbit hole. Looking at your Cypress cloud results, I am having significantly more failures in my environment. After debugging, it appears that REDCap's AJAX requests are very slow in my environment. Often times when a click is executed in Cypress (usually on the data import, data dictionary import, and user rights pages), the request takes longer than the 2000ms to execute, resulting in a failed test. These failures are pretty consistent when I test across REDCap versions, so I think it's definitely an environment issue.

Our long term goal is to run this in a CI/CD environment, but as of right now, I am running this locally on my macOS machine using docker. My guess is I don't have enough resources to run these efficiently and there's probably some institutional firewalls messing with the network speeds.

What I am wondering now if is if we should just move forward with setting up a CI/CD pipeline on our end. It seems like we might get results that are more consistent with your test results if we bite the bullet and move forward with setting up CircleCI rather than running these locally.

We have institutional support for DevOps tasks like this, but before I reach out to our engineering teams, I just wanted to ask a few clarifying questions:

1. Is all the environmental information we need for setting up a CI/CD pipeline located in the CircleCI config.yml file?
2. Are your VMs self hosted or running in CircleCI cloud? From what I can tell it looks like it's running the in the cloud?
3. Do you have a Performance or Enterprise plan through CircleCI?

[mmcev106]

1. Yes, all of our CircleCI configuration is contained within redcap_cypress/.circleci/config.yml.
2. We run our tests in the Cypress cloud, which uses CircleCI as a backend and captures the results in a nice Cypress specific UI.
3. We have a special open source team plan issued by Cypress that includes so many CircleCI credits per year. We were able to get special approval for it since our test runs benefit the entire consortium.

I should probably take this opportunity to mention that it is recommended to rely on Vanderbilt's validation of REDCap's core functionality so that you don't have to re-run those tests at your institution. I understand some institutions want to re-run them anyway in their own specific environment. This page provides a quick checklist on how you might go about that (if you haven't come across it already):

https://github.com/vanderbilt-redcap/redcap_cypress_docker/blob/main/creating-an-automated-testing-environment.md

Hopefully that's helpful. Please let me know if you have any ideas for improving the docs to clarify anything in ways that you would find helpful. I'm happy to update them.