The Experience

Summary

This experience brings a German medieval castle ruin to VR with elaborated Photogrammetry Technique made with low altitude UAV photos and photos from the ground to create a photo realistic sightseeing experience in Unreal Engine 4, where you can freely move around in the scene.

What to expect

This is primarily an exploration experience where you visit (parts of) the Castle Ruin Hohenrechberg. The intention is to give you an accurate photo-realistic and immersive impression of the real place.

In addition, you will find information on the castle itself, about its history and about the Legend of Barbarossa ("Red Beard") from the "House of Hohenstaufen," under whom this castle was built to garrison troops.

Listening to classical music performed on guitar while watching the beautiful landscape surrounding the castle can also be a relaxing and meditative experience.

Controls

The experience makes use of "gaze" mechanics, to give some short information about what you are looking at. Occasionally, you will find "Infosigns" with additional information, that can be activated by tapping the touchpad or pressing "Button 1" on the Controller.

For Navigation through the experience you can either use the Touchpad or the Gamepad (see picture). Turning with buttons will happen in 30° Steps to avoid motion sickness.

CastleVRuincontrols
(You will also see this information graphic in the expirience)

Innovations

Making use of Photogrammetry in games is still an innovation on its own (e.g., "The Vanishing of Ethan Carter" or the "Unreal Engine Kite Trailer") and I am not aware of any use in Virtual Tourism and would thus count that as innovation on its own. To give a good impression of the whole scene, I captured a 360° Panorama from 30 m (100 ft) above the castle by combining UAV and photos from the ground and stitching them above the horizon and then using this 360° Panorama to surround the scene. Although it is not an accurate representation of the landscape close to the castle, it works amazingly well in VR to give you an impression of the actual horizon and really helps to immerse yourself into the scene.

The 360° Panorama surrounding the scene: 360 Panoram (You can also download and view this in Gear VRs 360 Photo App)

Known Issues

Android 5 (lollipop) update broke UE4 MSAA support on Adreno GPU (and crushed performance). To assure broad compatibility I disabled MSAA.
Occasional frame drops can cause some "timewarp tearing". I couldn't fix this, but I also experienced this even in "Oculus home," so my guess is that this might be a broader issue with Android 5.
I had some rare crashes when too much textures were streamed in (although I'm below the 500mb limit).

The Development Process

Some additional in depth information on the project.

Motivation

While 360° (Stereo) Video can give you a vivid experience for Virtual Tourism, it is still a passive experience. By creating a photogrammetric reconstruction you can freely walk around as though being there. As these reconstructions are static it lacks the vividness of video of crowded places, but is awesome for things that don't move, like monuments or castles, and is thus ideal for these kinds of projects.

Challenges

Working with Photogrammetry for real-time rendering is tedious work, especially with the limitation of a mobile device (even with unlit models). The short time is also a factor (I signed up one day before milestone one, completely unprepared). Also, my equipment is limited (camera, UAV and PC are "consumer grade"). I have some experience in photogrammetry from working in archaeology, but I only reconstructed smaller objects (e.g., graves) before on my own. This is my first (learning) project where I reconstructed a larger scene and the first use of my UAV (apart from learning how to fly). I realized that 40 minutes flight time with the two batteries I own is too little time and I have to get more to not be forced to fly on too many different days (thankfully I had luck with the weather). I also realized that I lack a good ultra wide angle (fisheye) lens for narrow areas and couldn't compensate by simply taking more pictures, because I lack RAM (and the time needed for the computations). Working around these limitations is not an easy task and to reconstruct the complete castle with entrance and rooms takes more time. Maybe I will continue this project after the VR Jam, but for the contest submission the visit-able areas of the castle will be limited to the outer ward, the cortile and the courtyard.

Another challenge was scaling (or "georeferencing") the castle without hiring a surveyor (and I don't own a total station theodolite yet). I tried to make enough overlap with ground and UAV photos so that I can photogrammetrically align the detailed chunks with the large model of the whole castle. For this I simply scaled the model based on a Google Maps screenshot with measurements:

georef (Scaling the reconstroction in Blender with a screenshot of a measurement from Google Maps)

This is, of course, not very precise (photogrammetry can be used for measurements too and in some cases can even compete with the precision of LiDAR Scans). However, for a VR Visualization Project it is precise enough, as scaling errors from IPD and even sense of scale from "floor dragging" changes the sensation of the real scale more than the measurement error.

Conclusion

I think photogrammetry based experiences have much potential for VR, especially when recreating the real world. While, in my opinion, in the future Lightfields is the way to go for achieving Presence in "Real World scenarios," this tech is still in its infancy. Thus, photogrammetry is, for now, the way to go for scenarios with mostly static scenes. While mobile VR isn't the platform for "photorealism," still much can be done with static scenes. The limitations of photogrammetry for gaming assets ("baked in" lighting in photos) can even be an advantage for those kind of projects (as real time lighting is very limited on mobile anyway). This is just the beginning for me and I intend to do more in this area. I enjoyed working on this project and I am quite happy with some parts of the reconstruction. Since this my first large photogrammetric reconstruction and my first mobile App, I learned a lot from this project and especially from all the flaws of the reconstruction. I may continue this Project after the VR Jam and add the missing parts and fix/rework flaws.

For more info about my services visit realities.io (German) or send an email to david@realities.io

Special Thanks to: