Make reporting/logging for Python integration test infrastructure a bit more sane

[absurdfarce]

Feature Request

Description

#104 introduced a new framework for integration tests implemented entirely in Python. In the rush to get this done some shortcuts were taken around reporting and logging that should be remedied when we have a bit more time. These shortcuts include:

• Location of the log file
  • The framework makes vigorous use of Python's facilities for logging, but this leads to some inconsistency between Python versions. Python2 creates the log file "adelphi.log" in what appears to be the current working directory while Python3 winds up creating it in the temp directory we create while running the tests. This introduces at least two distinct problems:
    • You have to remember which Python version you're using to remember where to look
    • Temp directories are deleted by default after the tests complete so unless you pass the KEEP_LOGS env var you're out of luck
• Reporting of differences between reference schemas and generated schemas
  • Comparison of CQL statements is done by comparing sha256 hashes of the statements but some effort was put in to preserving the raw text statements themselves for logging purposes
  • Unfortunately these statements are only available in the log
  • The test as currently implemented does an unittest.assertEqual() on two sets of sha256 digests... which means when differences exist they fail the test but only report the digests
  • Users must then correlate these digests to statements by looking in adelphi.log
  • This should be amended to directly report CQL statements when assertions fail
• _compareToReference assumes it's dealing with CQL files [ADDRESSED IN Export to output directory not properly constrained by keyspace #106]
  • This is wrong in the abstract
  • Also wrong in practice since it won't work with the nosqlbench test
• test_diff is poorly named [ADDRESSED IN Export to output directory not properly constrained by keyspace #106]
  • It made sense originally but we have other diff tests now
  • The test should be renamed to identify what's unique about it now, specifically that it deal with CQL statements

┆Issue is synchronized with this Jira Task by Unito
┆Issue Number: AD-35

[absurdfarce]

Note that the PR for #106 addresses the CQL-specific _compareToReference stuff mentioned above.

[absurdfarce]

The bit about test_diff and it's naming is also likely to get addressed as part of the PR for #106

[absurdfarce]

Yup, #106 did wind up addressing the test_diff naming question.

[absurdfarce]

The log bits of this ticket are addressed via the use of tox and it's log capture facilities. Pretty sure all that's left here is a better hash comparison of CQL files... and even that at least consistently shows up in the tox output IIRC