Immersive Learning with VR

In collaboration with Inez Moutarde of Luchs Technology Group

VRfree® System Description

The Sensoryx VRfree® system uses a fusion of different sensor types, the data from which is processed using our proprietary software and algorithms to provide highly accurate and precise hand and finger tracking with multiple training use cases in VR/AR applications.  VRfree® consists of Gloves, Wrist modules and a Head Module (see Figure 1.) that can be easily mounted on almost all VR HMDs and also has the potential to be attached to AR headsets or, with additional development, to be integrated completely into the headset form factor.  Check out videos here!

Figure 1. – VRfree® System

System Components

Gloves

The VRfree® gloves can be produced to multiple fabric and material specifications, in accordance with customer requirements. 

Each glove is fitted with thirteen (13) Inertial Measurement Units (IMUs); two (2) for each finger, three (3) for the thumb, one (1) for the center-back of the hand and (1) inside the Wrist Module.  The IMUs allow for the precise measurement of finger movements, including bending, spreading and rotation.  The IMUs are connected to the center of the hand and then to the Wrist Module, from where the data is communicated to the Head Module. 

Wrist Module

Each glove has a removable Wrist Module attached. The Wrist Module contains a rechargeable, commercially available, Li-On battery with a running time of approximately six (6) hours per charge.  Each Wrist Module contains an LED in the IR frequency band and an ultrasonic transmitter.  It also contains an RF transmitter that currently operates in the Industrial, Scientific and Medical (ISM) band to communicate data back to the Head Module.

Head Module

The VRfree® Head Module is the brain of the system.  As well as containing a camera (to detect the IR LEDs), receivers (for the ultrasound) and an RF receiver (to receive IMU data from the gloves), it is also a computation device that contains the required electronic hardware and software/firmware  to enable processing of the wrist, hand and finger data using our proprietary software and algorithms.     

Tracking Solution

IR and Ultrasound

The fusion of optical data with ultrasonic range measurements allows for submillimeter accurate positioning of the Wrist Module relative to the Head Module. The field of view of more than 190 degrees of the fused position reconstruction removes any constraints on the interactions of a VR/AR user, i.e. with VRfree® users can enjoy realistic and intuitive AR and VR interactions.

IMUs

Positioned in the fingers, thumbs, hands and Wrist Modules, the IMUs allow very accurate tracking of finger, hand and wrist movements.  Due to the measurement principle applied, hand configuration can be measured even when the glove is outside the field of view of the Head Module or even completely covered by real world objects. For setups not requiring accurate positioning the system can be modified to transmit data reliably to 20 meters.  VRfree® incorporates proprietary algorithms, which calculate the attitude (orientation in space) of the more than 25 IMUs. The algorithms also compensate for environmental influences due to metallic objects, so that interacting with real-world ferromagnetic objects is feasible without deterioration of the attitude estimation.

Software and Algorithms

VRfree® is a complex system, as it relies on low-latency communication between Trackers (e.g. Wrist Modules etc.) and the Head Module, features a high number of IMUs to be processed and filtered in real time and needs to be robust against external influences, e.g. changing light conditions and external electro-magnetic fields.  A unique hardware design combined with proprietary algorithms unleash the full potential of the system. The Head Module is an autonomous unit not requiring external infrastructure.  It will calculate the measurements and communicate a stream of position and orientation entities to an attached display device, e.g. computer, VR system etc.  The API on the back-end side is intentionally very lightweight as we focus on limiting the computational load on any attached device to a minimum.

System Features

  • Up to 120Hz sampling rate.
  • High refresh rate (~10ms) ensures low latency and real-time tracking.
  • Beyond Field of View (FOV) tracking.  Current FOV is 190°, but the IMUs ensure we can continue to track position and movement outside the FOV.  Measuring the configuration of the hand is independent of the positional tracking.  Thus, outside the field of view the positional tracking of the hand position has to fall back to inverse kinematic approaches. Since the system does not measure the hand/finger configuration optically, occlusions do not impact the reconstruction of hand poses at all.
  • Full degrees of freedom of hands and fingers.
  • Multi-user capability currently allows seven (7) users to train simultaneously in the same physical space.  This number can be increased if required. 
  • Small size and lightweight design ensure the system is unobtrusive and comfortable to wear.

Tracking Unit (Object Tracker & 3D Stylus)

Using the same baseline tracking technology, Sensoryx is now conducting prototype testing of a small, lightweight, accurate and cost-effective Tracking Unit that can be easily integrated into the VRfree® ecosystem.  The Tracking Unit can be used to accurately and precisely track real-world objects (including weapons, other equipment and body parts) and represent them in VR/AR applications.  The Tracking Unit design is fully modular, meaning that if specific sensor types are not required for certain use cases, they can be easily removed to further reduce the size of the Tracking Unit. 

Figure 2. – Object Tracker and 3D Stylus Advanced Prototypes

The data from the Tracking Unit is communicated to the Head Module of the individual interacting with the tracked object/body part and processed in the same way as the hand and finger data.  Each Head Module can communicate with multiple Tracking Units, in addition to the hands of the individual trainee.  As with the hands, the position of the Tracking Unit can be accurately reconstructed within the field of view of the Head Module.  Outside that or beyond the maximal distance and up to 30 meters, only orientation data of the Tracking Unit will be reported back to the Head Module. Therefore, a simulation environment should take measures to update the position of a Tracking Unit whilst it is in the range of a Head Module and it should fall back on relative positioning to a mobile headset while outside the field of view.

Specifically for weapons, trigger pulls could also be detected and reported, if some additional development work was completed.  This could be achieved using small pressure sensors, either located on the trigger finger of the VRfree® gloves or on the trigger itself, whichever might best meet the requirements.

**Are you an educator interested in writing a guest blog for Rachelle’s site? Would love to share your ideas! Submit your post here. Looking for a new book to read? Find these available at bit.ly/Pothbooks

************ Also check out my THRIVEinEDU Podcast Here!

Join my weekly show on Mondays and Fridays at 6pm EST THRIVEinEDU on Facebook, Twitter, LinkedIn and YouTube. Join the group here

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s