Providing high-quality feedback on writing is cognitively demanding, requiring reviewers to identify issues, suggest fixes, and ensure consistency. We introduce AnnotateGPT, a system that uses pen-based annotations as an input modality for AI agents to assist with essay feedback. AnnotateGPT enhances feedback by interpreting handwritten annotations and extending them throughout the document. One AI agent classifies the purpose of each annotation, which is confirmed or corrected by the user. A second AI agent uses the confirmed purpose to generate contextually relevant feedback for other parts of the essay. In a study with 12 novice teachers annotating essays, we compared AnnotateGPT with a baseline pen-based tool without AI support. Our findings demonstrate how reviewers used annotations to regulate AI feedback generation, refine AI suggestions, and incorporate AI-generated feedback into their review process. We highlight design implications for AI-augmented feedback systems, including balanced human-AI collaboration and using pen annotations as subtle interaction.
In .env, please provide the following information from OpenAI API:
NEXT_PUBLIC_OPEN_AI_KEY: OpenAI API keyNEXT_PUBLIC_ANNOTATE_VECTOR_STORE: Vector store ID to store the document
Optional:
NEXT_PUBLIC_PURPOSE_VECTOR_STORE: Vector store ID to store annotation historyNEXT_PUBLIC_DOCUMENT_ONE_ID: First document ID for studyNEXT_PUBLIC_DOCUMENT_TWO_ID: Second document ID for studyNEXT_PUBLIC_PRACTICE_DOCUMENT_ID: Practice document ID for study
- Install Node.js
- Next, install all dependencies
npm install
- Supported browsers:
- Google Chrome
- Opera
- Microsoft Edge
To run locally
npm run build
npm start
Or in development
npm run dev
Once you access http://localhost:3000/, you can use Ctrl+y to open settings and upload your document to annotate or use the default documents provided.