The current implementation may not give the requested number of ratings per hit because hits may fall into the "accept but not use". In a recent run 34% of hits fell into this bucket. This adds inconsistency to the ratings (not a constant number) and is also inefficient (wasted $). There are at least two solutions:
- Add a real-time failure for the pair comparison environment check which was 72% of failures in a recent run.
- Use the MTurk API to create a controller that monitors and updates the hits to meet the target goals.