Development
The goal for this project was to create pong where players controlled the paddle with their hands using camera based handtracking, and the play field was displayed using a projector to create an AR pong experience.
Technical goals were to get oriented in TouchDesigner, and experiment with using a webcam as an input device using MediaPipe, a plugin featuring a variety of machine learning computer vision tools for face, body and hand tracking.
Early on I decided to project the play surface horizontally, and players could use their hands to control the paddles. This allowed for players to be facing each other, and removed their bodies from the tracking area which helped with hand tracking clarity.
The camera and projector faced the same surface, and were arranged to allow a 1:1 position for the paddles. Calibration was done very roughly by hand, by aligning the camera view with the projected screen, but in a small projection surface this was plenty precise.
The logic for pong was written in python, and was mostly self contained. The graphics were created with SOP components in TouchDesigner (paddle, ball) and referenced from the python script.
I opted for a python core as it seemed that this was more intuitive than creating a component network for collision, game events etc. Although I will look into if there are better TouchDesigner-specific solutions.
Early testing was done by yours truly - with a single player board (bouncing against the back wall).
Lag/Interpolation was added to the input network to reduce jitter from the camera tracking.
I added a colored smokey effect to the ball using a feedback network and displacing the residual images with noise.
The result was an ambient, AR “air hockey” like experience.
Feedback and Improvements
There is lag and dropouts in the camera tracking. I assume this is to to lighting conditions or general finicky nature of the mediapipe tracking. This does not allow for fast-paced play.
Ideally I can fix the lag, but if not, lean into a slower game experience and add more interpolation for a softer game feel.
I would also like to add sound effects, and play with ways to make an interactive/ambient soundscape influenced by the gameplay.
On a technical note which could help future projects - I’d like to figure out a more comprehensive calibration system for projection and camera view, so I can be more precise than the manual alignment and allow for variations in projection and camera positions.