Instructions and examples of randomized questionnaires

[markkuriekkinen]

Under the questionnaires modules, there could be a new chapter that describes randomized questionnaires in detail with examples.

There are two different types of randomization for questionnaires:

• the pick-randomly option of the questionnaire RST directive, and
• the randomized option of the pick-any RST directive used under questionnaires.

A-plus-rst-tools README desribes the options and the additional related options too.
https://github.com/apluslms/a-plus-rst-tools/blob/master/README.md#2-graded-questionnaire

In the new Aplus manual chapter, it would be important to also explain how the system makes random sampling and what affects it. Teachers may break the sampling for existing student submissions if they modify the question after releasing it. This may be particularly problematic if the existing submissions are automatically regraded after the question has been changed.

[markkuriekkinen]

Concerning the random sampling and how it changes when the teacher modifies the question afterwards, here are comments adapted from an email to a teacher. The teacher had modified the correct choices in a randomized pick-any question after some students had already submitted. This led to confusion when the old submissions were automatically regraded and some of them were moved to the "rejected" status.

In a randomized checkbox question, the random sampling is affected by the number of choices (checkboxes) and also by the correct checkboxes. If you change the set of correct checkboxes after releasing the questionnaire to the students, it is possible that the sampling changes for the existing submissions (that were submitted before the correct choices were changed). Thus, the student answered the question with one sampling (a set of the choices) and when the submission is regraded after modifying the correct choices, the grading uses a different sampling than what the student was originally given. Therefore, the grading results after regrading may look very confusing. After regrading, the rendered feedback will show choices that match the new sampling, but that is not what the student originally saw.

Regrading (after modifying the question) may set the submission to the "rejected" status if the submitted answer contains choices that are not included in the new sample. The submission becomes rejected because the input is considered invalid against the new sample.

In the submission details in the inspect submission page, you can see the original sample used when the student answered the questionnaire.

If you set "preserve-questions-between-attempts" in the RST questionnaire, the sampling does not change after submissions. But the sampling may still change if you modify the question choices and correct choices.

At any rate, if the student reloads the assignment page after the teacher has made the changes in the question, then the student sees the new version of the assignment and can answer normally. In randomized questionnaires, the questionnaire is always loaded "fresh". You may have seen that in normal questionnaires, the feedback of the previous submission is shown in the place of the questionnaire for students, but in randomized questionnaires, the assignment page always loads the "clean, fresh" assignment without signs of what was previously submitted. This is good since the student gets a questionnaire with the new choice sampling. (Randomized questionnaires include hidden checksums, which is why the previous rendered feedback HTML may not be used as a form for the new submissions; a new, "clean" form is needed.)

Internal ticket from the teacher:
https://rt.cs.aalto.fi/Ticket/Display.html?id=24616

[markkuriekkinen]

This function in the MOOC-Grader takes care of the random sampling for questionnaires.

https://github.com/apluslms/mooc-grader/blob/10051cd7f9c021e2b5b05ef541891eb7e851ad66/access/types/forms.py#L386