agent-study-ui

A Unity app that lets the user play a hidden-object-game while interacting with a virtual agent that will give hints and feedback along the way. The agent's behavior will be either a non-responsive animation, a simple mimicry of face and gaze, or a complex model of behavior.

The user will interact with the agent through speech input; however, recognition will be done Wizard-of-Oz to avoid latency and errors.

Project is developed using Unity/C# (for the game and agent display) and a separate Python app (for the head and face tracking). The agent model and movements is driven by FACSvatar.

System Requirements

• PC to run the game and agent
  • Windows 10 operating system
  • Needs GPU! Recommended GTX 1060 or higher for computer vision processing
• Touchscreen for interacting with the game
• Keyboard to control WoZ interactions

Dependencies

• Unity 2020.3.19 LTS (any version in the 2020 LTS release line should work)
  • Use Unity Hub to manage installations and projects
• Python 3.6
  • See agent-mimicry-python for package requirements
• NuGet Unity package
  • Should be included in project already
  • If needed, a copy is available on GitHub: NuGetForUnity
  • Packages used:
    • ZeroMQ
    • Newtonsoft.JSON
• Azure Speech SDK for Unity (Azure Cognitive Services)
  • The Azure private key followed by the server region needs to be in StreamingAssets in the 'azure' file
  • Keys can be obtained from the Azure portal

Cloning the Project

git clone https://github.com/Ruiz-HCI-Lab/agent-study-ui.git
cd agent-mimicry-system
git submodule update --init --recursive

Project Structure

The project is split up into two parts: Unity and Python The root directory is the actual Unity project (can open this in Unity Hub).

Python files are put under Assets/StreamingAssets/Python This allows Unity to launch the Python scripts directly instead of needing to run it manually. These should automatically be retrieved by git when doing the submodule update.

Game Level Configuration

The individual puzzles are stored under Assets/Resources/. Each level is represented as an XML file, with images and keys (where the hidden object is) in the images/ folder. The full list of levels is configured in the PrepScene, which randomly orders the levels per agent.

Key Unity Scenes

• PrepScene: first scene in the game. Allows researcher to configure the study and contains the ordering of the game levels and study groups
• ControlAgent: game with agent displaying random head movements
• MimicAgent: game with simple delayed agent mimicry (launches Python backend)
• ComplexAgent: game with complex model-driven agent behaviors (launches Python backend)
• GameTemplate: scene without agent, for testing the game only