Virtual Reality and Physically-Based Simulation WS 17/18
Virtual Reality (VR) is a research area at the intersection of computer graphics, physically-based simulation, and 3D human-computer interaction (HCI). VR and 3D realtime computer games share a lot of algorithmic challenges: novel interaction technologies and interaction metaphors in virtual environments (in particular, intuitive and direct metaphors), immersion and presence, and real-time rendering. Another important topic is physically-based simulation in real-time, which tries to simulate real-world phenomena such as fire, cloth, the behavior of rigid objects when colliding with each other, fluids, or objects made of deformable material.
Over the past two decades, VR has established itself as an important tool in several industries, such as manufacturing (e.g., automotive, airspace, ship building), architecture, and pharmaceutical industries. During the past few years, we have been witnessing the second "wave" of VR, this time in the consumer, in particular, entertainment markets.
In this course, we will first look at the fundamental methods, and then go on to more advanced algorithms that are needed to build complex and full-fledged VR systems or real-time computer games. Example topics are 3D interaction, physically-based behavior of objects, acoustic rendering, haptics, and collision detection.
The assignments will be mostly practical ones, based on the cross-platform
game engine
Unreal.
Participants will start developing with "visual programming", and later use C++ to solve the assignments.
You are encouraged to work on assignments in small teams.
Some of the topics to be covered (tentatively):
- Introduction, basic notions of VR, a few applications
- VR technologies: displays, tracking, input devices, software design
- Stereo rendering
- Error correction: tracking correction, filtering
- Techniques for real-time rendering
- Fundamental immersive interaction techniques: gesture recognition, navigation, selection, grasping, menues in 3D
- Complex immersive interaction techniques: world-in-miniature, action-at-a-distance, magic lens, etc.
- Particle systems
- Spring-mass systems
- Force feedback: rendering forces
- Collision detection
- Acoustic rendering
Note: this list is just tentative and subject to change during the semester.
News
Slides
The following table will, eventually, contain all the topics that were covered in this class, the accompanying slides, exercise sheets, and frameworks for solving the programming exeercises. (This table will be filled week by week.)
Week | Topics |
---|---|
1. |
Orga stuff Introduction: definitions, applications, immersion, fidelity, presence, |
2. |
Intro 2: body ownership illusion,
history, Milgram's continuum Scenegraphs: immediate / retained mode, semantics of nodes and edges, special issues with light sources, |
3. | Scenegraphs 2: shared geometry (instancing), thread-safe scenegraphs, distributed rendering, fields & routes concept, types of nodes, the Phong model for specification of the material, indexed face set, OBJ file format, FBX file format, |
4. |
Scenegraphs 3:
hierarchical transformations,
routes & events, behavior graph, KV pool. Displays and stereo rendering 1: depth cues, human visual system and stereopsis, horopter and fusion area, multiplexing techniques for stereo images (polarization, shuttering, color filtering), immersive didplays / immersive projection technologies |
5. | Stereo rendering 2: correct stereo projection, hypo- and hyper-stereo, issues with stereo rendering (depth aliasing, convergence-focus incongruity, incorrect viewpoint, stereo violation, depth blur, too much parallax), model of user's head, coherent virtual workspace, multiplexing for multiple users, automultiscopic display, pre-distortion for HMDs, |
6. | Dies Academicus |
7. | Real-time rendering 1: simulator sickness, latency and its sources, view-independent rendering, prioritized rendering, level-of-detail techniques, static/dynamic/psychophysiological LOD selection, predictive LOD selection |
8. |
Real-time rendering 2: dynamic LODs, progressive meshes,
view-dependent LODs,
portal culling,
state sorting using online sorting buffer algorithms,
stereoscopic image warping,
latencies with conventional rendering,
reducing latency using time warping
Input Devices: degrees of freedom, multimodality, virtual trackball, directness continuum, isotonic vs isometric, tracking, data gloves, locomotion devices. |
9. | Interaction techniques 1: Universal Interaction Tasks, design of user interfaces, gesture recognition, the navigation task (wayfinding & locomotion), taxonomies as a design tool, abstract representation of the user for navigation, navigation metaphors (point-and-fly, scene-in-hand, two-handed, hands-free, walking in place, et al.), 3D navidget, user models (power law of practice, Hick's law, Fitts's law), |
10. | Interaction techniques 2: iso-/non-isomorphic techniques, control-display ratio, go-go technique, task decomposition of selection task, selection techniques: ray-based & cone-based techniques, flexible pointer, friction surface (bent ray), eye-hand mismatch, cone-based technique with ranking, balloon selection, object manipulation (grasping and moving), taxonomy of grasping, Natural User Interaction, principles for interaction design: action-at-a-distance, image plane interaction, proprioceptive interaction, world-in-miniature, magic lenses, redirected walking and related techniques, |
11. | Particle systems: Newton's Laws, dynamics/kinematics, Euler integration, phase space, definition of particle systems, forces from physical effects, non-physical effects, collision handling, rendering of particle systems, alpha blending, flames & fire, procedural modeling of plants, the concept of streaming architectures |
12. | Mass-spring systems: definition, single spring-damper element, explicit Euler integration, instability and error accumulation with explicit Euler integration, Runge-Kutta, Verlet integration, constraints, implicit integration, tangent stiffness matrix, comparison to explicit integration, mesh creation for volumetric objects, |
13. | Haptics: applications, devices, the haptic loop, human haptic sense and human factors, simulation factors, haptic textures, buzzing, intermediate representations, software architecture, impedance and admittance, the surface contact point method, voxmap-pointshell method, friction in one contact point, |
14. |
Collision detection:
motivation, definitions, collision detection pipeline, broad/narrow phase,
plane-sweep technique,
separating planes algorithm (for convex objects),
hierarchical coll.det.,
bounding volume hierarchies, types of BV's,
separating axis lemma for convex polyhedra,
inner sphere trees, sphere packings, proximity computation using ISTs,
penalty forces using ISTs. |
Literature
- Ralf Dörner, Wolfgang Broll, Paul Grimm, Bernhard Jung (Hrsg.): Virtual und Augmented Reality (VR/AR), Grundlagen und Methoden der Virtuellen und Augmentierten Realität. Springer 2013. You can read the e-book from within the university's network.
- Kay M. Stanney (Ed.): Handbook of Virtual Environments. Lawrence Erlbaum Associates, 2002
- William R. Sherman, Alan B. Craig: Understanding Virtual Reality. Morgan Kaufmann.
- Don Brutzman, Leonard Daly: X3D: Extensible 3D Graphics for Web Authors. Morgan Kaufmann, 2007.
- Daniel Fleisch: A Student's Guide to Vectors and Tensors. Cambridge University Press
- Kenny Erleben et al.: Physics Based Animation. Charles River Media, 2005.
- Mario Gutiérrez, Frédéric Vexo, Daniel Thalmann: Stepping into Virtual Reality. Springer, 2008. You can read the e-book from within the university's network.
- Anthony Steed, Manuel Oliveira: Networked Graphics: Building Networked Games and Virtual Environments. Morgan Kaufman, 2009. From this book, only chapters 7, 10, and 11 are relevant to this course.
Warning: these text books can only give you a general introduction to the field of VR! Most of the topics taught in class will not be covered by any of these text books directly -- in fact, AFAIK there are no text books that cover these topics. Therefore, I recommend to attend class.
If you are thinking of buying some of these books, then I suggest to consider buying a used copy of them -- very often, you can find them at a fraction of the price of a new copy. The following are two very good internet sites for finding inexpensive used copies of books: Abebooks and BookButler.
Assignments
In order to acquire the "Schein", you will have to do a number of assignments. These will be mostly small practical programming assignments (based on a freely available VR system).
Help and documentation on the Unreal Engine:
- For general help, take a look at the Unreal Engine 4 Documentation and in case of problems also the UE4 AnswerHub
- For quick help you can visit the IRC channels #unrealengine and #ue4linux on Freenode (web client)
- Compiling and running Unreal on Linux is explained in the wiki
- To setup c++ debugging this Youtube video is highly recommended
- If you have experience with the Unity engine take a look at Unreal for Unity developers
- YouTube Tutorials by EpicGames
Online Literature and Resources on the Internet
- Some general readings (and listening) about VR, that do not talk so much about the
technical details, but more about the visionary, or historic, aspects of VR -- or both.
- An interview with Jaron Lanier from 2017, where he talks about VR, music, some of the milestones of VR history, AI, consciousness, what should, and what shouldn't be done with VR, and a lot more. (Source)
- A review of Jaron Lanier's new book, which is, actually, msotly about Jaron Lanier, which is quite as fascinating. (Source)
- If you are interested in what is currently going on in VR, then this podcast, called Voices of VR by Kent Bye, is right for you. It does not require a lot of computer scientific knowledge, and it covers a huge breadth of topics.
- The original article praising the panorama in Blackwood's Edinburgh Magazine (vol. 15, 1824)
- A visionary article by Vannevar Bush, that shows that the idea of virtual reality is not that new: As We May Think (1945)
- Interview with Jaron Lanier and David Eggers (it was part of the program "Druckfrisch" and aired on 5.10.2014 on the German ARD, H.265 codec)
- Literature on stereoscopic ("3D") rendering:
- Implementing Stereoscopic 3D in Your Applications by Samuel Gateau and Steve Nash from NVidia at GPU Technology Conference 2010 (Source)
- Rendering 3D Anaglyph in OpenGL by Animesh Mishra, 2011 (Source)
- A very nice chapter on Binocular Vision and Space Perception. A pretty accessible read.
- A very good conference tutorial talk on factors and limits of human stereoscopic vision from a perception and neuroscientific perspective, by Martin Banks, 2013.
- For people interested in modern real time rendering pipelines, take a look at this GTA V - Graphics Study by Adrian Courrèges, 2015. (Source)
- More literature on the topic of user interface design:
- The Siggraph 2001 tutorial Advanced Topics in 3D User Interface Design
- An entertaining and very insightful Quiz on Fitts' Law
- Giving You Fitts: an article by Jensen Harris describing where Fitts' law has been used in the UI design of Office 2007 ( Source).
- Here are the -- not quite serious -- Cartoon Laws of Physics ;-) .
- Literature on particle and on spring-mass systems:
- William T. Reeves: Particle Systems - A Technique for Modeling a Class of Fuzzy Objects;
- More advanced particle systems (e.g., n-body systems) are explained in Real-Time Particle Systems on the GPU in Dynamic Environments by Shannon Drone (Source)
- Georgii, Westermann: Mass-Spring Systems on the GPU, Simulation Practice and Theory 2005. (Source)
- Literature on physically-based simulation in general:
- Introductory Vector Calculus by Norman Wittels (Source)
- Siggraph 2008 course notes: Real Time Physics
- Siggraph 2001 course notes: Physically Based Modeling
- Survey paper over the field of deformation simulation methods: Physically Based Deformable Models in Computer Graphics by Andrew Nealen, Mathias Muller, Richard Keiser, Eddy Boxerman and Mark Carlson, EG 2005
- Here is a survey article on collision detection. You probably need some basic knowledge on collision detection.
- A bit on the history of Virtual and Mixed Reality:
- An article from The Verge: The Rise and Fall and Rise of Virtual Reality by Adi Robertson and Michael Zelenko (Source, in particular here)
- Highlights from an interview with Jaron Lanier and Kevin Kelly from The Verge: Digital Natives, A conversation between virtual reality visionaries Jaron Lanier and Kevin Kelly by Casey Newton (Source, in particular here)
- The Ultimate Display by Donald Sutherland
- The short novel
Gegen den Strich
by Joris-Karl Huysmans.
(Sorry, I have it in German only)
Denkanregung: was hat das mit VR zu tun?
Literature and Resources on X3D/VRML
Since X3D/VRML is no longer the platform for the practical exercises in this course, I have demoted the links to X3D/VRML to this place.- Tools that can "play" X3D- / VRML97 files:
- InstantReality; on their homepage, you can also find a lot of examples and tutorials.
- FreeWRL (for Linux and OSX).
- Cortona; this is a commercial browser plugin.
- Introduction to VRML / X3D:
- A "Cheat Sheet" for VRML.
- SIGGRAPH 2008 Class Notes: Don't be a WIMP; These course notes not only discuss some post-WIMP interaction techniques, but also explain how to use some of the more advanced features of InstantReality, such as different stereo rendering modi, special interaction devices, clustering, scripting, animations, etc. (Source)
- The Annotated VRML97 Reference Manual (Source)
- The VRML Primer and Tutorial; those chapters that deal with HTML are not relevant for this course. In addition, the chapter on tools is outdated; but otherwise it is still a veritable introduction for VRML novices. (Source)
-
The most important documents on the X3D standard:
- Architecture and base components (i.e., the specification of nodes and profiles)
- Scene access interface (SAI)
- X3D language bindings for Javascript (ECMAScript)
- X3D language bindings for Java
- Examples for X3D / VRML:
- The examples from the lecture (and many more!)
- More examples are on the homepage of InstantReality;
- And on the homepage of the X3D book by Don Brutzman and Leonard Daly;
- A meta web page with lots of links to archives containing more examples.
- A large collection of materials,
both in VRML, as well as in XML encoding.
And here it is again as a ZIP archive. - Links on socalled authoring tools; for VRML/X3D; if you want, you are welcome to use them. But if you have a decent programming editor (preferrably in ASCII), then you are probably more efficient with your editor.
- A handy little tool for
Calculating VRML Viewpoints.
Takes as input viewpoint, look-at, and up vector, outputs rotation as axis + angle to achieve the rotation from world coordinate frame into (viewpoint, look-at, up). From there it is trivial to compute the quaternion.
Readings That Have Nothing to do With VR, but are Still Highly Recomended
- An article about the modern tools to influence voters via so-called social media: Wahlmanipulation mittels Psychometrik und Social Media (Quelle: Das Magazin). The article is in German only, sorry to the non-German speakers. Of course, the techniques and tools described in the article can also be used for other purposes, which makes them even more dangerous to society.
- One of the last interviews with Joseph Weizenbaum, a very well-known pioneer of AI and critic of too naive belief in technology (German only, sorry).
Last modified: Wed Jan 31 14:24:46 CET 2018