Blacklist & whitelist validation requires further documentation

[atilantera]

Long story short: some validation features of Graderutils are scarcely documented. Either document these further and/or make an user survey related to Graderutils features.

Problem description

Python-grader-utils can perform elementary program structure checks on students' submissions. This is known as validation in Python-grader-utils' manual. In particular, there are blacklist and whitelist validation functionalities which allow checking whether student's program has certain elements of Python syntax.

The problem is that these features appear insufficiently documented from the user's point of view. Sure, there is a minimal configuration file example, but the documentation implicitly assumes that the reader is familiar with the ast module in the Python standard library. In constrast, consider that the user here might be a second-year student working at summer. Have they heard of syntax trees, or understand that ast.parse in the Python-grader-utils documentation refers to the aforementioned module? Furthermore, the documentation of the ast module is a verbose technical manual, not a tutorial.

The concept of a syntax tree itself should be understandable for anyone taken a data structures course, i.e., not advanced algorithmics incomprehensible to undergraduate summer workers. However, this ast-utilizing feature of Python-grader-utils might still require a tutorial for the aforementioned reasons.

Solution proposal

Short tutorial of ast:

• What is an abstract syntax tree,
• Refer to the ast module in the Python standard library with a hyperlink
• Example program and its abstract syntax tree
• Provide a tool to easily dump the abstract syntax tree of a given Python file

Further documentation of functionality:

• For option node_names, list all options, or indicate clearly how to apply Python standard library documentation in this case
• For option node_dumps, refer to the aforementioned tutorial, but give further examples: input code and corresponding node dump
• For option node_dump_regexp: refer to the re module of Python standard library
• For option node_dump_regexp: explain how Python-grader-utils walks through the abstract syntax tree (see also Usability of AST traversal in white-/blacklist #43)

Additional information

Here is an example of printing the abstract syntax tree of a Python program.

import ast
import sys

if len(sys.argv) == 1:
    print(f"Usage: {sys.argv[0]} file.py")
else:
    filename = sys.argv[1]
    with open(filename, encoding="utf-8") as submitted_file:
        source = submitted_file.read()
        print(ast.dump(ast.parse(source), indent=4))

Example input:

def sieve_of_eratosthenes(n):
    """ Return the number of prime numbers between the range (2, n) """
    raise NotImplementedError("sieve_of_eratosthenes function is missing!")

Example output:

Module(
    body=[
        FunctionDef(
            name='sieve_of_eratosthenes',
            args=arguments(
                posonlyargs=[],
                args=[
                    arg(arg='n')],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]),
            body=[
                Expr(
                    value=Constant(value=' Return the number of prime numbers between the range (2, n) ')),
                Raise(
                    exc=Call(
                        func=Name(id='NotImplementedError', ctx=Load()),
                        args=[
                            Constant(value='sieve_of_eratosthenes function is missing!')],
                        keywords=[]))],
            decorator_list=[]),