Virtual Reality and Physically-Based Simulation WS 16/17
Virtual Reality (VR) is a research area at the intersection of computer graphics, physically-based simulation, and 3D human-computer interaction (HCI). VR and 3D realtime computer games share a lot of algorithmic challenges: novel interaction technologies and interaction metaphors in virtual environments (in particular intuitive and direct metaphors), immersion and presence, and real-time rendering. Another important topic is physically-based simulation in real-time, which tries to simulate real-world phenomena such as fire, cloth, the behavior of rigid objects when colliding with each other, fluids, or objects made of deformable material.
Over the past one or two decades, VR has established itself as an important tool in several industries, such as manufacturing (e.g., automotive, airspace, ship building), architecture, and pharmaceutical industries. In the past few years, we are witnessing the second "wave" of VR, this time in the consumer, i.e., entertainment markets. (Some people might argue that it is actually the third wave.)
In this course, we will first look at the fundamental methods, and then go on to more advanced algorithms that are needed to build complex and full-fledged VR systems or real-time computer games. Example topics are object behavior, acoustic rendering, haptics, and collision detection.
The assignments will be mostly practical ones, based on the cross-platform
VR system
InstantReality.
Participants can choose the programming language from
Java, Javascript, and C++.
You are encouraged to work on assignments in small teams.
Some of the topics to be covered (tentatively):
- Introduction, basic notions of VR, a few applications
- VR technologies: displays, tracking, input devices, software design
- Stereo rendering
- Error correction: tracking correction, filtering
- Techniques for real-time rendering
- Fundamental immersive interaction techniques: gesture recognition, navigation, selection, grasping, menues in 3D
- Complex immersive interaction techniques: world-in-miniature, action-at-a-distance, magic lens, etc.
- Particle systems
- Spring-mass systems
- Force feedback: rendering forces
- Collision detection
- Acoustic rendering
Note: this list is just tentative and subject to change during the semester.
News
Slides
The following table will, eventually, contain all the topics that were covered in this class, the accompanying slides, exercise sheets, and frameworks for solving the programming exeercises. (This table will be filled week by week.)
Week | Topics |
---|---|
1. |
Orga stuff Introduction: definitions, immersion, fidelity, presence, history, Milgram's continuum |
2. | Scenegraphs: immediate / retained mode, semantics of nodes and edges, special issues with light sources, shared geometry (instancing), thread-safe scenegraphs, distributed rendering, fields & routes concept, types of nodes, specification of the material using the Phong model, indexed face set, hierarchical transformations, routes & events, behavior graph, KV pool. |
3. | Displays and Stereo rendering 1: depth cues, human visual system and stereopsis, horopter and fusion area, multiplexing stereo images, multiplexing techniques for stereo images (polarization, shuttering, color filtering), immersive projection technologies |
4. | Dies Academicus |
5. | Stereo rendering 2: correct stereo projection, issues with stereo rendering (depth aliasing, convergence-focus incongruity, in-correct viewpoint, stereo violation), hypo- and hyper-stereo, model of user's head, projetion issues for multiple co-located users, multiplexing for multiple users, automultiscopic display, pre-distortion for HMDs, low and full persistence, space-time diagrams, smearing and judder and color fringes. |
6. | Real-time rendering 1: simulator sickness, latency and its sources, view-independent rendering, prioritized rendering, level-of-detail techniques, static/dynamic/psychophysiological LOD selection, predictive LOD selection |
7. |
Real-time rendering 2: dynamic LODs, progressive meshes,
view-dependent LODs,
portal culling,
state sorting using online sorting buffer algorithms,
stereoscopic image warping,
|
8. |
Input Devices: degrees of freedom, multimodality, virtual trackball,
directness continuum, isotonic vs isometric,
tracking, data gloves, locomotion devices |
9. | Interaction techniques 1: Universal Interaction Tasks, design of user interfaces, gesture recognition, the navigation task (wayfinding & locomotion), taxonomies as a design tool, abstract representation of the user for navigation, navigation metaphors (point-and-fly, scene-in-hand, two-handed, walking in place, et al.), user models (power law of practice, Hick's law, Fitts's law), |
10. | Interaction techniques 2: iso-/non-isomorphic techniques, control-display ratio, go-go technique, task decomposition of selection task, selection techniques: ray-based & cone-based techniques, friction surface (bent ray), eye-hand mismatch, cone-based technique with ranking, balloon selection, object manipulation (grasping and moving), DOF separation/reduction by widgets, principles for interaction design: action-at-a-distance, image plane interaction, proprioceptive interaction, world-in-miniature, voodoo dolls, magic lenses, redirected walking and related techniques, |
11. | Particle systems: dynamics/kinematics, Euler integration, phase space, definition of particle systems, physical effects, non-physical effects, collision handling, rendering, rendering transparent objects, flames & fire, procedural modeling of plants, |
12. | Mass-spring systems: Newton's Laws, single spring-damper element, explicit Euler integration, instability and error accumulation with explicit Euler integration, Runge-Kutta, implicit integration, tangent stiffness matrix, comparison to explicit integration, mesh creation for volumetric objects, |
13. | Haptics: applications, devices, the haptic loop, human haptic sense and factors, simulation factors, haptic textures, buzzing, intermediate representations, impedance and admittance, surface contact point approach, voxmap-pointshell method, friction in one contact point, |
14. |
Collision detection:
motivation, definitions, collision detection pipeline, broad/narrow phase,
plane-sweep technique,
Mikowski sums, intersection test for convex objects based on Minkowski sums,
hierarchical coll.det.,
bounding volume hierarchies, types of BV's,
separating axis lemma for convex polyhedra,
overlap test for k-DOP's,
construction of BVH's,
inner sphere trees, sphere packings, proximity computation using ISTs,
penalty forces using ISTs. |
Literature
- Ralf Dörner, Wolfgang Broll, Paul Grimm, Bernhard Jung (Hrsg.): Virtual und Augmented Reality (VR/AR), Grundlagen und Methoden der Virtuellen und Augmentierten Realität. Springer 2013. You can read the e-book from within the university's network.
- Kay M. Stanney (Ed.): Handbook of Virtual Environments. Lawrence Erlbaum Associates, 2002
- William R. Sherman, Alan B. Craig: Understanding Virtual Reality. Morgan Kaufmann.
- Don Brutzman, Leonard Daly: X3D: Extensible 3D Graphics for Web Authors. Morgan Kaufmann, 2007.
- Daniel Fleisch: A Student's Guide to Vectors and Tensors. Cambridge University Press
- Kenny Erleben et al.: Physics Based Animation. Charles River Media, 2005.
- Mario Gutiérrez, Frédéric Vexo, Daniel Thalmann: Stepping into Virtual Reality. Springer, 2008. You can read the e-book from within the university's network.
- Anthony Steed, Manuel Oliveira: Networked Graphics: Building Networked Games and Virtual Environments. Morgan Kaufman, 2009. From this book, only chapters 7, 10, and 11 are relevant to this course.
Warning: these text books can only give you a general introduction to the field of VR! Most of the topics taught in class will not be covered by any of these text books directly -- in fact, AFAIK there are no text books that cover these topics. Therefore, I recommend to attend class.
If you are thinking of buying some of these books, then I suggest to consider buying a used copy of them -- very often, you can find them at a fraction of the price of a new copy. The following are two very good internet sites for finding inexpensive used copies of books: Abebooks and BookButler.
Assignments
In order to acquire the "Schein", you will have to do a number of assignments. These will be mostly small practical programming assignments (based on a freely available VR system).
Online Literature and Resources on the Internet
- The original article praising the panorama in Blackwood's Edinburgh Magazine (vol. 15, 1824)
- Another visionary article by Vannevar Bush: As We May Think (1945)
- Interview with Jaron Lanier (it was part of the program "Druckfrisch" and aired on 5.10.2014 on the German ARD)
- Literature on stereoscopic ("3D") rendering:
- Implementing Stereoscopic 3D in Your Applications by Samuel Gateau and Steve Nash from NVidia at GPU Technology Conference 2010 (Source)
- Rendering 3D Anaglyph in OpenGL by Animesh Mishra, 2011 (Source)
- A very nice chapter on Binocular Vision and Space Perception. A pretty accessible read.
- A very good conference tutorial talk on factors and limits of human stereoscopic vision from a perception and neuroscientific perspective, by Martin Banks, 2013.
- More literature on the topic of user interface design:
- The Siggraph 2001 tutorial Advanced Topics in 3D User Interface Design
- An entertaining and very insightful Quiz on Fitts' Law
- Giving You Fitts: an article by Jensen Harris describing where Fitts' law has been used in the UI design of Office 2007 ( Source).
- Here are the -- not quite serious -- Cartoon Laws of Physics ;-) .
- Literature on particle and on spring-mass systems:
- William T. Reeves: Particle Systems - A Technique for Modeling a Class of Fuzzy Objects;
- More advanced particle systems (e.g., n-body systems) are explained in Real-Time Particle Systems on the GPU in Dynamic Environments by Shannon Drone (Source)
- Georgii, Westermann: Mass-Spring Systems on the GPU, Simulation Practice and Theory 2005. (Source)
- Literature on physically-based simulation in general:
- Introductory Vector Calculus by Norman Wittels (Source)
- Siggraph 2008 course notes: Real Time Physics
- Siggraph 2001 course notes: Physically Based Modeling
- Survey paper over the field of deformation simulation methods: Physically Based Deformable Models in Computer Graphics by Andrew Nealen, Mathias Muller, Richard Keiser, Eddy Boxerman and Mark Carlson, EG 2005
- A bit on the history of Virtual and Mixed Reality:
- An article from The Verge: The Rise and Fall and Rise of Virtual Reality by Adi Robertson and Michael Zelenko (Source, in particular here)
- Highlights from an interview with from The Verge: Digital Natives, A conversation between virtual reality visionaries Jaron Lanier and Kevin Kelly by Casey Newton (Source, in particular here)
- The Ultimate Display by Donald Sutherland
- The short novel
Gegen den Strich
by Joris-Karl Huysmans.
(Sorry, I have it in German only)
Denkanregung: was hat das mit VR zu tun? - Help and documentation for the Unreal Engine:
- For general help, take a look at the Unreal Engine 4 Documentation and in case of problems also the UE4 AnswerHub
- For quick help you can visit the IRC channels #unrealengine and #ue4linux on Freenode (web client)
- Compiling and running Unreal on Linux is explained in the wiki
- To setup c++ debugging this Youtube video is highly recommended
- If you have experience with the Unity engine take a look at Unreal for Unity developers
- YouTube Tutorials by EpicGames
- For people interested in modern real time rendering pipelines, take a look at this GTA V - Graphics Study
Literature and Resources on X3D/VRML
Since X3D/VRML is no longer the platform for the practical exercises in this course, I have demoted the links to X3D/VRML to this place.- Tools that can "play" X3D- / VRML97 files:
- InstantReality; on their homepage, you can also find a lot of examples and tutorials.
- FreeWRL (for Linux and OSX).
- Cortona; this is a commercial browser plugin.
- Introduction to VRML / X3D:
- A "Cheat Sheet" for VRML.
- SIGGRAPH 2008 Class Notes: Don't be a WIMP; These course notes not only discuss some post-WIMP interaction techniques, but also explain how to use some of the more advanced features of InstantReality, such as different stereo rendering modi, special interaction devices, clustering, scripting, animations, etc. (Source)
- The Annotated VRML97 Reference Manual (Source)
- The VRML Primer and Tutorial; those chapters that deal with HTML are not relevant for this course. In addition, the chapter on tools is outdated; but otherwise it is still a veritable introduction for VRML novices. (Source)
-
The most important documents on the X3D standard:
- Architecture and base components (i.e., the specification of nodes and profiles)
- Scene access interface (SAI)
- X3D language bindings for Javascript (ECMAScript)
- X3D language bindings for Java
- Examples for X3D / VRML:
- The examples from the lecture (and many more!)
- More examples are on the homepage of InstantReality;
- And on the homepage of the X3D book by Don Brutzman and Leonard Daly;
- A meta web page with lots of links to archives containing more examples.
- A large collection of materials,
both in VRML, as well as in XML encoding.
And here it is again as a ZIP archive. - Links on socalled authoring tools; for VRML/X3D; if you want, you are welcome to use them. But if you have a decent programming editor (preferrably in ASCII), then you are probably more efficient with your editor.
- A handy little tool for
Calculating VRML Viewpoints.
Takes as input viewpoint, look-at, and up vector, outputs rotation as axis + angle to achieve the rotation from world coordinate frame into (viewpoint, look-at, up). From there it is trivial to compute the quaternion.
Readings That Have Nothing to do With VR, but are Still Highly Recomended
- An article about the modern tools to influence voters via so-called social media: Wahlmanipulation mittels Psychometrik und Social Media (Quelle: Das Magazin). The article is in German only, sorry to the non-German speakers. Of course, the techniques and tools described in the article can also be used for other purposes, which makes them even more dangerous to society.
- One of the last interviews with Joseph Weizenbaum, a very well-known pioneer of AI and critic of too naive belief in technology (German only, sorry).
Last modified: Wed Feb 01 23:01:41 CET 2017