This project was carried out as part of the IGR205 course at Télécom Paris, with Alix Larroque and Charles Pombet, under the supervision of Jan Gugenheimer.

The goal of this project was to get familiar with the development of applications in Virtual Reality (VR), which we had never experienced before. The instruction was to imagine and implement novel forms of hand interaction in VR. We implemented a mechanism where user’s finger are swapped in VR. You can find a demo video here.

Brainstorming

We first started a process of idea generation by brainstorming. We came up with different ideas. An idea was to implement a grappling hook, a locomotion technique where the user could grab a far-away object by extending their arm, and being teleported when bringing their arm back to their chest. An other idea was to implement a detaching arm, where the user could be able to detach their arm by grabbing their shoulder, to extend it’s reach.

Finger Swapper

The idea that we ultimately decided to implement is what we call “finger swapper”. The idea is that the movement of the real fingers is mapped to different virtual fingers. For example in the figure below, the finger swapping means that moving your middle finger in real life will move the pinky finger in the virtual world (see color correspondance in blue).

Implementation

To implement this mechanism, we used the Unity Editor and the Oculus assets. In order to allow hand tracking, we used the OVRCustomHand prefab of the Oculus assets. The hands are rendered using the combination of a skeleton and a mesh. The skeleton is made of a list of bones: there are 4 bones for the thumb, and 3 bones for each other finger.

Each bone has a relative position and orientation. To change the order of the fingers, we just have to change the order of the bones in this list. For example, suppose you reverse the relative position of the index bones in the list with the relative position of the middle finger. When the user moves their index finger in the real world, their middle finger will move in the virtual world, and vice versa. So we’re almost there.

However, this simple implementation poses some problems. Indeed, by inverting the fingers we are left with unnatural fingers positions.

This is due to the fact that each finger has different degrees of freedom, and moves in a different direction. For example, when you close your fist, your fingers tend to meet towards the center of your hand.

To avoid this problem, we thought of several solutions.

The first solution supposes that the first two fingers of the hand have a behavior symmetrical to the last two fingers. The idea then consists in symmetrizing the direction of the fingers with respect to a central vertical axis.

Another idea, the one we chose to implement, consists in constraining the possible movement of the fingers to a single direction, unique to all the fingers. This direction being unique, it will be the same whether the fingers are reversed or not. This solution makes us lose some realism of the hand (as the user can’t move their fingers in any direction), but we assume that this level of realism is sufficient for our purpose so that it does not break the immersion of the user.

From the finger swapper mechanism, we created a game based on gesture recognition. The goal of this game is to imitate the gesture that is displayed, to pass levels. The player must make the gesture in a certain time to progress in the level, otherwise they lose points. The first level is very easy since the fingers are not inverted. Once the player succeeds at this level, the fingers are inverted. As we are not used to do certain hand positions, it becomes more and more difficult to do the requested gestures. The more the levels advance, the more there are inverted fingers and the less time the player has to perform the gestures. See a demo video here.

Conclusion

The idea that we implemented, which consists of inverting the fingers, is a feature that allows to imagine new game mechanics, and to push the limits of reality. We had to make certain compromises, in particular by constraining the movements of the fingers in order to limit the aberrations of rendering of the hand, but the result is satisfactory. In the games that we have implemented, the player is led to have to do gymnastics between their real perceptions, and their virtual perceptions to reach certain goals. This duality between these two contradictory perceptions is something that is worth studying, to answer certain questions like: Is our brain able to adapt to a body which is not its own? How long does it take to adapt to this new perception of the body? How far can the brain adapt? This kind of study is already carried out with tools such as mirroring goggles to prove how fast the brain can adapt to a new perception, but our mechanism could be used to deepen or supplement these studies. In addition to being a fun and quite disturbing feature for games, the VR finger inversion mechanics that we implemented could be used as an interesting tool to conduct neurology studies.