Official repository for the Real Engine (official name pending) hands-free, movement controlled multiplayer platform. Presentation: https://docs.google.com/presentation/d/1DVUhVgrW4Rvt4cCYu1NlqKjjue3QQUlD5qqZLg43hDs/edit?usp=sharing
04/20/2024: Basic upper-body game control via the camera is established. Real-world landmarkers for hands still need to be applied correctly (MediaPipe's "holistic" system is outdated and will not support this, will require an independent HandLandmarkerResult to work). Scaling the coordinates and correctly positioning them according to each body part's origin in the game and in real life needs to be improved (e.g wrist and elbow positioning may be good, but shoulder motion should be more sensitive to changes in body position).
05/03/2024: Temporarily removed pose estimation alone to focus on hand tracking. Hand landmarkers are perfectly translated into the world and bone/flesh is simulated with the use of cylindrical/ellipsoidical wedges connecting the joints of the hands. Using coordinates of the hands (XYZ) that are not calibrated for real-world dimensions seem to be far more accurate. Socket communication of hand coordinates is optimized now to send all 30ish joints at the same time to the Unity TCP listener. Avatar functionality currently removed. Next steps are gesture recognition, first person movement, re-adding pose estimation for holistic character tracking and movement.
05/04/2024: Full body pose rendering in 3D is functional. Holistic landmarking --> 3D game space coordinate system conversion is working at high speeds. Joint connections (muscle, bone) between arms, shoulders, digits on hands, and wrists working, need rotation and scaling improvement + material. Binaries produced from this code will provide a demo environment for working full body motion.
06/09/2024: First draft finished.