JazzHandsXR: A Mixed Reality Application for Visualising Hand Structure in Piano Playing

This thesis introduces JazzHandsXR, a mixed reality application designed to support self-taught piano learners by providing real-time visualization of hand positioning and movement.

Description

Traditional piano instruction offers immediate feedback on ergonomic hand posture, which self-learners often miss when using conventional resources like sheet music or apps such as Synthesia. These tools typically focus on note sequences, lacking guidance on hand technique —an essential component for preventing strain and promoting efficient playing. JazzHandsXR addresses this gap by projecting virtual ”ghost hands” that model correct hand shapes and finger transitions, enabling users to mirror these movements in real time. By providing this interactive feedback, JazzHandsXR gives self-learners valuable insight into effective hand positioning without the immediate need for a teacher’s supervision, encouraging healthier and more productive practice habits. In evaluating this system, a user study was designed to assess its usability, effectiveness, and overall learning support. Initial observations reveal that while JazzHandsXR offers substantial benefits, it has certain limitations: the lack of haptic feedback and the simplified animation may affect the precision of technique training. Nonetheless, the flexibility it offers—allowing learners to practice at their own pace and repeat exercises as needed—highlights its value for users without consistent access to in-person guidance. Ultimately, while JazzHandsXR cannot fully replace in-person instruction, the study’s findings suggest it is an accessible and promising aid for self-learners aiming to develop foundational hand techniques. I used Unity3d, Blender and the Meta Interaction SDK to bring this project to life.

Results

We examined the state of the art and analyzed existing applications. We found that there is a wealth of research on this topic and that certain apps already target specific issues. For example, some use machine learning to detect and classify poor hand posture based on images, while others rely on VR applications to generate optimal hand movements for specific passages. A significant amount of research has been conducted on how pianists move their hands, utilizing tracking technologies to collect data and provide feedback. However, we found that there are still few applications that use overlaid models specifically to train hand posture, drawing the attention away from playing the correct notes. Therefore, I aimed to develop a method to create exercises that address this issue and make them more intuitive through the ghost hand. To develop such an app, we had to explore the capabilities of modern technologies and their potential limitations. I learned that with the relatively new color passthrough feature of the Quest 3, it is possible to build powerful mixed reality applications that blend the real world with virtual objects, allowing learning to go beyond traditional boundaries. However, there are certain limitations in terms of accuracy. It is not yet very reliable to test detailed fingerings or analyze movements at the finger level, as the hand tracking through the cameras is not capable of detecting the smallest details or handling occlusion optimally. Therefore, the focus was placed more on the hand as a whole rather than on fine-grained interaction with individual notes. After studying the Taubman Approach and the concepts of body mapping, it became clear that a method was needed to manage tension—ensuring that playing remains as relaxed as possible and that the hand structure adapts to each played note in a way that aligns optimally with anatomy. This allows for the minimal necessary effort while still producing the same sound quality. As a result, the exercises focused on recognizing poor hand posture and shaking out tension, as well as carefully positioning the hands for each note. By removing time and rhythm from the equation, players could fully concentrate on practicing proper hand positioning. To support these ideas with data, I designed the user study to provide a clear comparison by evaluating hand similarities between playing with and without the Ghost Hand. Additionally, a questionnaire was included to better assess the app’s qualities and gather detailed user feedback on how the new feature is perceived and whether it effectively serves its purpose. The results showed statistical evidence that the Ghost Hand positively influences hand posture toward the given target posture and that the app is easy to use, helpful, and enjoyable. However, it is important to note that this evidence does not yet conclusively prove that the hand posture will improve in the long term or that this method provides a robust learning approach. Overall, this provides a solid foundation for further research. Future studies could focus on generating a theoretically optimal hand posture based on expert data and applying this methodology to refine the approach. This could eventually lead to the development of a tool that not only assists in hand posture training but also ensures that players adopt better techniques long term.

Files

Full version of the bachelor's thesis (English only)

License

This original work is copyright by University of Bremen.
Any software of this work is covered by the European Union Public Licence v1.2. To view a copy of this license, visit eur-lex.europa.eu.
The Thesis provided above (as PDF file) is licensed under Attribution-NonCommercial-NoDerivatives 4.0 International.
Any other assets (3D models, movies, documents, etc.) are covered by the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit creativecommons.org.
If you use any of the assets or software to produce a publication, then you must give credit and put a reference in your publication.
If you would like to use our software in proprietary software, you can obtain an exception from the above license (aka. dual licensing). Please contact zach at cs.uni-bremen dot de.