"The bloodiest experience on the Gear VR "

We are building an app for surgeons and surgeons in training to learn or review quickly surgical procedures.

Quick review

  • This is a very graphic experience, "Walking dead" level
  • Stereoscopic footage of a real reconstructive surgery, captured through two cameras on the surgeon's head

Controls : This app uses the trackpad. With the trackpad you can :

  • at the beginning of the experience, launch it (tap)
  • when looking at the chapters "banner", swipe horizontally to go through the chapters (if don't release your finger, you're in "continous mode" ; tap to select the right chapter afterwards) or just tap to select a chapter
  • when looking at the "screen" , fast forward (15s) by sliding on the trackpad or backwards (15s)
  • When looking at the 2D sequences (CT scans) : swipe forward to scroll the images one-way, tap to stop, backwards to scroll the other way
  • When looking at the 3D CT of the patient (the skull) : swipe to rotate the skull

Mechanics : enjoy the show and navigate in the content, linearly or not.

VR Innovations :

  • UX : the trackpad + chapter banner is an intuitive way to interact with video content that you won't necessarily watch linearly
  • video : we think we have found a good technical trade-off with our video capture/experience that fits our bill. It has a limited FOV (120°), that allows for good immersion, good stereoscopy at the center, good performances on the Note with high resolution, no motion sickness when stabilized and a small camera rig on the surgeon's head.
  • Surgery : yep, that's new to VR :-)

The context

  • ** 5 billion** people don't have access to safe surgical care. ((source : The Lancet ) The n°1 reason is not a lack of equipment, but a lack of trained surgeons. The traditional way of training surgeons, mentoring, is not scalable and more and more difficult due to increasing time pressure for experienced surgeons. The Lancet estimates that the world need an additional 2.2 million people in the surgical force to cover 80% of the needs. The ability to train new surgeons quickly and in a cost-effective way could save millions of lives.
  • There should 4 billion smartphones circulating in 2020. Imagine if everyone of them could potential be a surgical training center....
  • Surgeons skills are deeply linked to the surgeon ability to appreciate complex 3D spatial relationships and have a proper understanding of the 3D structure of the human body. Yet nearly all the learning materials a surgeon encounters are in 2D : books, anatomical drawings, videos, etc
  • Video is quickly becoming an integral tool in a surgeon's training that's proven to be efficient. The mantra of surgery learning "see one, do one, teach one" is replaced by a safer "see many, do one"

For these reasons, we think mobile VR will be in the next year a very affordable and very efficient way to train new surgeons at a global scale.

Who we are

We are four VR lovers ; two developers and two designers, with a foot in the medical world (Remi's wife is a physician). We designed last year the first surgical Oculus Rift experience, by putting two Gopros on a surgeon's head so that students would be able to see the surgery in stereoscopic 3D with a DK1/DK2. This allowed us to gather lots of positive feedback from surgeons, researchers and students on how they would want to use such a system. We wanted to take things to the next level, and the Gear VR allowed us to do exactly that. We teamed up for this experience with Pr Sébastien Albert, an ENT surgeon in a parisian hospital.

The experience

Our experience is built around a surgery. We’re focusing here on a facial reconstruction after a tongue tumor extraction, known as a "free-flap" procedure, performed by Pr Albert. The surgeon extracts some tissue from the patient's arm to reconstruct his face following the tumor removal. We should note that we had of course the signed consent of the patient before filming any footage. He was actually enthusiastic about our idea and got out of the hospital safely last week.

Here's how it was filmed

This experience puts you in the shoes of the surgeon, by viewing the content filmed through two gopros put on Pr Albert's head.

This experience is not a simulation : you don't influence the results of the surgery. It is more related with a process cognitive sciences researchers call "mental practice" : observation with a minimal "cognitive load" allows for very efficient learning. Traditional surgical videos are often very low FOV, because they're meant to be seen on small screens, and they lack precious information on the surgeon’s posture, and are captured for a different position than the surgeon's.

UX

We put a lot of work in the UX. When we were discussing with surgeons, we understood that they remember surgeries as a series of instructions, so we implemented chaptering in the experience. By sliding on the trackpad, you move to the next chapter. By keeping you finger on the trackpad, you browse through the different chapters. We had first a parabola displaying the different chapters but we found out that the circular metaphor was more coherent with our approach.

The trackpad is our swiss army knife to interact with the different objects, as we didn’t want to have another control device You can interact with the additional content that we put on the experience with that, rotate 3D scan, navigate through 2D CT-scans, etc.

The engineering

We encountered several engineering challenges in the process of designing our app.

  • resolution and FOV There's currently a trade-off on the Gear VR between easiness of capture, resolution, stereoscopy and FOV. We opted for a FOV of 120° that allows to see in details the operating field, with a high resolution and correct stereoscopy. We're benefitting from the 360° interface offered by the Gear VR (thank you swivel chair) to display additional contextual content useful to understand more precisely what's happening during the surgery (CT-scan of the patient for example).

  • distorsion The high FOV requires the use of wide-angle lenses. To display the distorted images it produces and keep as much content and quality as possible, we used generated quasi-spherical display geometries in our app, centered on the user's viewpoint. It makes the videos very natural to look at for the viewer.

  • Stabilization We have made our own video stabilization on the recordings, making the video safer to look at in terms of motion-sickness. That is a VERY good thing the surgeon's head is always moving through a surgery. So if you're sick, it should be the blood, not the shaky camera.

  • Additional content Since our videos are not 360°, we have a lot of room to put additional content that will help for the comprehension of the surgery. That's why you should try our app with a swivel chair! CT scans are quite huge in terms of polys and difficult for the Note 4 to manage. We're working on that!

Milestone II Update

We're working hard on enhancing the app. Stereoscopy is such a complex matter... We're thinking of dropping the parabola for a more classic "sphere" UX, because of visibility problems with the Gear on.

Milestone III Update

Work on the engineering part ; stabilization now works! Refactored the text of the project presentation to show the stakes in more detail.

Milestone IV Update

We had a lot of difficulties in the last steps, especially to make a build that works! But we're submitting a (hopefully) working version, and we just need to sleep right now...

Built With

Share this project:

Updates