Virtual Reality and Physically-Based Simulation, WS 22/23


Virtual Reality (VR) is a research area at the intersection of computer graphics, physically-based simulation, and 3D human-computer interaction (HCI). novel interaction technologies and interaction metaphors VR and 3D realtime computer games share a lot of algorithmic challenges: in virtual environments (in particular, intuitive and direct metaphors), immersion and presence, and real-time rendering. Another important topic is physically-based simulation in real-time, which tries to simulate real-world phenomena such as fire, cloth, the behavior of rigid objects when colliding with each other, fluids, or objects made of deformable material.

Over the past two decades, VR has established itself as an important tool in several industries, such as manufacturing (e.g., automotive, airspace, ship building), architecture, and pharmaceutical industries. During the past few years, we have been witnessing the second "wave" of VR, this time in the consumer, in particular, in the entertainment markets.

In this course, we will first look at the fundamental methods, and then go on to more advanced algorithms that are needed to build complex and full-fledged VR systems or real-time computer games. Example topics are 3D interaction, physically-based behavior of objects, acoustic rendering, haptics, and collision detection.

The assignments will be mostly practical ones, based on the cross-platform game engine Unreal. Participants will start developing with "visual programming", and later use C++ to solve the assignments.
You are encouraged to work on assignments in small teams.

Some of the topics to be covered (tentatively):

  1. Introduction, basic notions of VR, several example applications
  2. VR technologies: displays, tracking, input devices, scene graphs, game engines
  3. The human visual system and Stereo rendering
  4. Techniques for real-time rendering
  5. Fundamental immersive interaction techniques: fundamentals and principles, 3D navigation, user models, 3D selection, redirected walking, system control
  6. Complex immersive interaction techniques: world-in-miniature, action-at-a-distance, magic lens, etc.
  7. Particle systems
  8. Spring-mass systems
  9. Haptics and force feedback
  10. Collision detection
  11. Acoustic rendering

Note: this list is just tentative and subject to change during the semester.

For Students from Other Institutions

If you are from an educational institution other than University of Bremen, you can still join my course, if you fill out this form and apply as a guest student. When successfull, you will get an account with the Universtity of Bremen, which will allow you to access StudIP. There, you can sign up for the course.

News

Teaching mode with physical attendance this year! yay :-) (fingers crossed)

You can use this Discord channel to discuss or coordinate amongst yourself. Occasionally, the tutors will answer questions, too, should you have any for them.

Exams will be on: February28, and March 1; please register on StudIP! First come, first serve :-)

Slides, Assignments, Recordings

The following table will, eventually, contain all the topics that were covered in this class, the accompanying slides, exercise sheets, and frameworks for solving the programming exercises. This table will be filled week by week.
Occasionally, I will post short videos here, too, which I would like to ask you to watch in preparation for the respective week's lecture.
Videos encoded with H.265/HEVC (mp4 container) play fine in Safari, encodings with VP9 (webm container) play in all other browsers. (H265 only appears to play under Windows in Edge, but it is buggy.) Otherwise, just download them, then play them with VLC.

Week Topics Slides Video Assignments Frameworks
1. Orga stuff
Introduction and Psychological aspects of VR: classification of vritual environments (VEs), definiton of virtualization, immersion, benefits of immersion, fidelity, presence, components and types of presence, potential explanation and factors for the occurence of presence, measuring presence, 		PDF0 PDF1 Assignment 1 Unreal Inroduction
2. Introduction and Psychological aspects of VR: behavior change (virtual coral reef application), avatars and virtual body ownership, measuring IVBO and importance of realism, cybersickness, technical causes and neurophysiological hypotheses, mitigation, measuring cybersickness, Milgram's continuum
Lab meeting with demos 		PDF
3. Scenegraphs: immediate / retained mode, semantics of nodes and edges, inheritance, special issues with light sources, shared geometry (instancing), thread-safe scenegraphs, distributed scenegraphs, fields & routes concept, types of nodes, nodes and fields (entities and components), the Phong model for specification of the material, superficial intro to BRDF's, indexed face set, OBJ file format, FBX file format, transformations, hierarchical transformations, behavior graph, data flow paradigm, execution model & event cascades, PDF
4. Displays and stereo rendering: depth cues, human visual system and stereopsis, horopter and fusion area, VR displays / immersive displays / immersive projection technologies, multiplexing techniques for stereo images (polarization, shuttering, color filtering), automultiscopic display, correct stereo projection, hypo- and hyper-stereo, issues with stereo rendering (depth aliasing, convergence-focus conflict, incorrect viewpoints, stereo violation, blur divergence, too much parallax), coherent virtual workspaces, pre-distortion for HMDs, stuttering with low and full persistence HMDs, temporal aliasing,
Lab meeting 		PDF H256 VP9 Assignment 2
5. Displays and stereo rendering: optimization of stereo rendering performance, model of user's head, issues with stereo rendering (depth aliasing, convergence-focus conflict, blur divergence, stereo violation, incorrect viewpoints, too much parallax), coherent virtual workspaces, pre-distortion for HMDs, stuttering with low and full persistence HMDs, temporal aliasing,
Real-time rendering: latency and its sources, latency and its sources, view-independent rendering, prioritized rendering, efficient memory layout for fast rendering, level-of-detail techniques, static/adaptive/psychophysiological LOD selection, predictive LOD selection, progressive meshes, view-dependent LODs, Foveated rendering 		PDF1 PDF2 Assignment 3 Laggy Jump LOD Meshes
6. (Principles of input devices: self-study)
Interaction techniques: Universal Interaction Tasks, approaches to design of user interfaces, the navigation task (wayfinding & locomotion), taxonomies as a design tool, abstract representation of the user for navigation, navigation metaphors (point-and-fly, scene-in-hand, two-handed, hands-free, walking in place, et al.), magic mirror, 3D navidget, 		PDF1 PDF2 Assignment 4 Blocks
7. Interaction techniques: user models: power law of practice, Hick's law, Fitts's law; 3D selection techniques, iso-/non-isomorphic techniques, control-display ratio, go-go technique, task decomposition of selection task, ray-based & cone-based techniques, flexible pointer, friction surface (bent ray), bubble cursor, depth ray, balloon selection, selection by progressive refinement: SQUAD, Disambiguation Canvas, Point-and-Zoom, Expand, HorizontalDragger; object manipulation (grasping and moving), taxonomy of grasping, the manifold of hand postures, Natural User Interaction (NUI), PDF
8. Dies Academicus
Assignment class
9. Interaction techniques (cont'd): general 3D interaction design principles: two-handed and multimodal interaction, physical props / tangible UIs, DOF separation/reduction: PRISM metaphor and 3D widgets, action-at-a-distance (examples: image plane interaction, voodoo dolls), proprioceptive interaction, world-in-miniature, magic lenses, redirecting the user, redirected walking and related techniques, body retargeting, amplified head rotation, spatial illusions, system control, menus, 2D marking menus, PDF
10. Particle systems: Newton's Laws, dynamics vs kinematics, Euler integration, phase space, definition of particle systems, forces from physical effects, non-physical effects, collision handling, PDF
11. Mass-spring systems: definition, single spring-damper element, explicit Euler integration, instability and error accumulation with explicit Euler integration, Runge-Kutta, Verlet integration, constraints, implicit integration, tangent stiffness matrix, comparison to explicit integration, mesh creation for volumetric objects, consistent collision response for volumetric mass-spring systems. PDF
12. Haptics: definitions, applications, devices, the haptic loop(s), the human haptic sense and human factors, simulation factors, haptic textures, the problems of buzzing and latency, intermediate representations, impedance vs. admittance-based approach, the surface contact point method, PDF H256 VP9 Assignment 5 Spring
13. Haptics (cont'd): voxmap-pointshell method, friction in one contact point,
Collision detection: motivation, definitions, collision detection pipeline, broad/narrow phase, 3D grid for filtering potentially colliding pairs, plane-sweep technique, separating planes algorithm (for convex objects) with perceptron learning rule, 		PDF
14. Coll.Det. (cont'd): Minkowski sums, intersection test for convex objects based on Minkowski sums, hierarchical collision detection, bounding volume hierarchies, types of BV's, separating axis lemma for convex polyhedra, separating axis test (SAT), overlap test for k-DOP's, construction of BVH's using the volume heuristic, inner sphere trees, sphere packings, proximity computation using ISTs, penalty forces using ISTs. PDF
Recommendations towards achieving the profile area "visual and medical computing" within your master's program (only for computer scientists)
Call for theses at the CGVR lab 		PDF1 PDF2

Screen Recordings of the Lecture

The following video recordings are only there in an effort to help with your preparatoins for the exam. The definitive table of topics is the one above!

The following table contains recordings of the lectures as of WS 20/21. They encoded with H.265/HEVC and should play fine with Safari and IE. With Chrome and Firefox, you might need to download the videos (right-click) then play them with VLC.

Chapter Videos
Introduction and Psychological Aspects of VR (Presence et al.) H265 VP9
Scene graphs and Game engines
Tutorial on Unreal Engine 		H265 VP9
H256, VP9
Display Technologies, Stereopsis, Stereo Rendering H256 VP9
Techniques for Real-Time Rendering H256 VP9
Principles of Input Devices H256 VP9
Interaction Techniques H256 VP9
Haptics, devices excerpt H256 VP9
Particle Systems H256 VP9
Mass-Spring Systems H256 VP9
Collision Detection H256 VP9
Sound Rendering H256 VP9
Call for Theses H256 VP9

You can find video recordings of the class taught in WS 19/20 here . The password is virtual2020.

Literature

Warning: these text books can only give you a general introduction to the field of VR! Most of the topics taught in class are not covered by any of these text books directly -- in fact, AFAIK there are some topics that are not covereed by any text book! Therefore, I recommend to attend class.

If you are thinking of buying some of these books, then I suggest to consider buying a used copy of them -- very often, you can find them at a fraction of the price of a new copy. The following are three very good internet sites for finding inexpensive used copies of books: abebooks, Booklooker, and ZVAB.

Help and Documentation on the Unreal Engine:

Online Literature and Resources from the Internet

Literature and Resources on X3D/VRML

Since X3D/VRML is no longer the platform for the practical exercises in this course, I have demoted the links to X3D/VRML to this place.

Readings That Have Nothing to do With VR, but are Still Highly Recomended

Gabriel Zachmann
Last modified: Wed Feb 08 19:27:53 CET 2023