Virtual Reality and Physically-Based Simulation WS 18/19


Virtual Reality (VR) is a research area at the intersection of computer graphics, physically-based simulation, and 3D human-computer interaction (HCI). VR and 3D realtime computer games share a lot of algorithmic challenges: novel interaction technologies and interaction metaphors in virtual environments (in particular, intuitive and direct metaphors), immersion and presence, and real-time rendering. Another important topic is physically-based simulation in real-time, which tries to simulate real-world phenomena such as fire, cloth, the behavior of rigid objects when colliding with each other, fluids, or objects made of deformable material.

Over the past two decades, VR has established itself as an important tool in several industries, such as manufacturing (e.g., automotive, airspace, ship building), architecture, and pharmaceutical industries. During the past few years, we have been witnessing the second "wave" of VR, this time in the consumer, in particular, in the entertainment markets.

In this course, we will first look at the fundamental methods, and then go on to more advanced algorithms that are needed to build complex and full-fledged VR systems or real-time computer games. Example topics are 3D interaction, physically-based behavior of objects, acoustic rendering, haptics, and collision detection.

The assignments will be mostly practical ones, based on the cross-platform game engine Unreal. Participants will start developing with "visual programming", and later use C++ to solve the assignments.
You are encouraged to work on assignments in small teams.

Some of the topics to be covered (tentatively):

  1. Introduction, basic notions of VR, several example applications
  2. VR technologies: displays, tracking, input devices, scene graphs, game engines
  3. The human visual system and Stereo rendering
  4. Techniques for real-time rendering
  5. Fundamental immersive interaction techniques: fundamentals and principles, 3D navigation, user models, 3D selection, redirected walking, system control
  6. Complex immersive interaction techniques: world-in-miniature, action-at-a-distance, magic lens, etc.
  7. Particle systems
  8. Spring-mass systems
  9. Haptics and force feedback
  10. Collision detection
  11. Acoustic rendering

Note: this list is just tentative and subject to change during the semester.

News

Change of exam date!
Exams are on March 26, March 27, and April 10.
Please sign up using StudIP.

Slides

The following table will, eventually, contain all the topics that were covered in this class, the accompanying slides, exercise sheets, and frameworks for solving the programming exeercises. (This table will be filled week by week.)

Week Topics Slides Exercise
sheets 		Frameworks
1. Orga stuff
Introduction: definitions, applications, immersion, fidelity, presence, 		PDF0 PDF1 Assignment 1
2. Intro 2: body ownership illusion, history, Milgram's continuum, augmented reality, the hype cycle.
Scenegraphs: immediate / retained mode, semantics of nodes and edges, special issues with light sources, shared geometry (instancing), 		PDF1 PDF2
3. Scenegraphs 2: thread-safe scenegraphs, distributed scenegraphs, fields & routes concept, types of nodes, the Phong model for specification of the material, indexed face set, OBJ file format, FBX file format, PDF Assignment 2
4. Scenegraphs 3: transformations, hierarchical transformations, billboards, behavior graph, routes & events, KV pool, distributed scenegraphs (again).
PDF
5. Displays and stereo rendering 1: depth cues, human visual system and stereopsis, horopter and fusion area, multiplexing techniques for stereo images (polarization, shuttering, color filtering), PDF Assignment 3 UE Project
6. Displays and stereo rendering 2: immersive didplays / immersive projection technologies correct stereo projection, hypo- and hyper-stereo, issues with stereo rendering (depth aliasing, convergence-focus incongruity, incorrect viewpoint, stereo violation, depth blur, too much parallax), model of user's head, coherent virtual workspaces, multiplexing for multiple users, automultiscopic display, pre-distortion for HMDs, low and full persistence, space-time diagrams, smearing and judder and color fringes. PDF
7. Real-time rendering 1: simulator sickness, latency and its sources, view-independent rendering, prioritized rendering, level-of-detail techniques, static/dynamic/psychophysiological LOD selection, predictive LOD selection, progressive meshes, PDF Assignment 4 UE Project
8. Lab meeting; Dies Academicus
9. Real-time rendering 2: view-dependent LODs, portal culling, state sorting using online sorting buffer algorithms, latencies with conventional rendering, stereoscopic image warping, 3D warping
PDF1
10. Input Devices: intrusiveness, degrees of freedom, multimodality, virtual trackball, isotonic vs isometric, tracking technologies, including data gloves, directness continuum, locomotion devices, logische Geräte.
Interaction techniques 1: Universal Interaction Tasks, design of user interfaces, gesture recognition, the navigation task (wayfinding & locomotion), taxonomies as a design tool, abstract representation of the user for navigation, navigation metaphors (point-and-fly, scene-in-hand, two-handed, hands-free, walking in place, et al.), 3D navidget, 		PDF1 PDF2
11. Interaction techniques 2: user models: power law of practice, Hick's law, Fitts's law; 3D selection techniques, iso-/non-isomorphic techniques, control-display ratio, go-go technique, task decomposition of selection task, ray-based & cone-based techniques, flexible pointer, friction surface (bent ray), balloon selection, selection by progressive refinement: SQUAD, Disambiguation Canvas, Point-and-Zoom, Expand, HorizontalDragger, PDF Assignment 5
12. Interaction techniques 2: object manipulation (grasping and moving), taxonomy of grasping, Natural User Interaction (NUI), DOF separation/reduction: PRISM metaphor and 3D widgets, action-at-a-distance principle, image plane interaction, proprioceptive interaction principle, world-in-miniature, voodoo dolls magic lenses, redirected walking and related techniques,
Haptics: applications, devices, the haptic loop, the human haptic sense and human factors, simulation factors, haptic textures, the problems of buzzing and latency, intermediate representations, software architecture, impedance and admittance, the surface contact point method, 		PDF1 PDF2
13. Haptics 2: voxmap-pointshell method, friction in one contact point, haptic illusions.
Lab meeting: presentation of interaction assigments. 		PDF
14. Collision detection: motivation, definitions, collision detection pipeline, broad/narrow phase, 3D grid for potentially colliding pairs, separating planes algorithm (for convex objects), Minkowski sums, intersection test for convex objects based on Minkowski sums, hierarchical collision detection, bounding volume hierarchies, types of BV's, construction of BVH's using the volume heuristic, inner sphere trees, sphere packings, proximity computation using ISTs, penalty forces using ISTs.
Sound rendering: motivation, human factors, mixing sound sources, image source method, 		PDF PDF

Literature

Warning: these text books can only give you a general introduction to the field of VR! Most of the topics taught in class are not covered by any of these text books directly -- in fact, AFAIK there are some topics that are not covereed by any text book! Therefore, I recommend to attend class.

If you are thinking of buying some of these books, then I suggest to consider buying a used copy of them -- very often, you can find them at a fraction of the price of a new copy. The following are two very good internet sites for finding inexpensive used copies of books: Abebooks and BookButler.

Assignments

Help and documentation on the Unreal Engine:

Online Literature and Resources on the Internet

Literature and Resources on X3D/VRML
Since X3D/VRML is no longer the platform for the practical exercises in this course, I have demoted the links to X3D/VRML to this place.

Readings That Have Nothing to do With VR, but are Still Highly Recomended

Gabriel Zachmann
Last modified: Wed Sep 11 16:01:21 CEST 2019