Virtual Reality and Physically-Based Simulation WS 20/21
Virtual Reality (VR) is a research area at the intersection of computer graphics, physically-based simulation, and 3D human-computer interaction (HCI). novel interaction technologies and interaction metaphors VR and 3D realtime computer games share a lot of algorithmic challenges: in virtual environments (in particular, intuitive and direct metaphors), immersion and presence, and real-time rendering. Another important topic is physically-based simulation in real-time, which tries to simulate real-world phenomena such as fire, cloth, the behavior of rigid objects when colliding with each other, fluids, or objects made of deformable material.
Over the past two decades, VR has established itself as an important tool in several industries, such as manufacturing (e.g., automotive, airspace, ship building), architecture, and pharmaceutical industries. During the past few years, we have been witnessing the second "wave" of VR, this time in the consumer, in particular, in the entertainment markets.
In this course, we will first look at the fundamental methods, and then go on to more advanced algorithms that are needed to build complex and full-fledged VR systems or real-time computer games. Example topics are 3D interaction, physically-based behavior of objects, acoustic rendering, haptics, and collision detection.
The assignments will be mostly practical ones, based on the cross-platform
game engine
Unreal.
Participants will start developing with "visual programming", and later use C++ to solve the assignments.
You are encouraged to work on assignments in small teams.
Some of the topics to be covered (tentatively):
- Introduction, basic notions of VR, several example applications
- VR technologies: displays, tracking, input devices, scene graphs, game engines
- The human visual system and Stereo rendering
- Techniques for real-time rendering
- Fundamental immersive interaction techniques: fundamentals and principles, 3D navigation, user models, 3D selection, redirected walking, system control
- Complex immersive interaction techniques: world-in-miniature, action-at-a-distance, magic lens, etc.
- Particle systems
- Spring-mass systems
- Haptics and force feedback
- Collision detection
- Acoustic rendering
Note: this list is just tentative and subject to change during the semester.
For Students from Other Institutions
If you are from an educational institution other than University of Bremen, you can still join my course, if you fill out this form and apply as a guest student. When successfull, you will get an account with the Universtity of Bremen, which will allow you to access StudIP. There, you can sign up for the course.
News
Hybrid teaching mode this year!
Students can choose to attend physically in the class room, or join via Zoom or Twitch.
Zoom: the link will be announced in an email via Studip.
Twitch: join the channel https://www.twitch.tv/gabzach .
Exams will be on: March 29 through 31. Please register via Studip.
Slides
The following table will, eventually, contain all the topics that were covered in this class, the accompanying slides, exercise sheets, and frameworks for solving the programming exercises. (This table will be filled week by week.)
Week | Topics |
---|---|
1. |
Orga stuff Introduction: definitions, applications, immersion, fidelity, presence, |
2. |
Introduction:
examples of the importance of fidelity and presence,
measuring presence, behavior change using VR,
Lab meetings and assignments |
3. |
Introduction:
body ownership illusion,
history, Milgram's continuum, augmented reality, the hype cycle.
Scenegraphs: immediate / retained mode, semantics of nodes and edges, inheritance, special issues with light sources, shared geometry (instancing), thread-safe scenegraphs, distributed scenegraphs, fields & routes concept, types of nodes, nodes and fields (entities and components), the Phong model for specification of the material, indexed face set, OBJ file format, FBX file format, |
4. | Scenegraphs: transformations, hierarchical transformations, behavior graph, data flow paradigm, execution model & event cascades, KV pool, distributed scenegraphs (again). |
5. | Displays and stereo rendering: depth cues, human visual system and stereopsis, horopter and fusion area, VR displays / immersive displays / immersive projection technologies, multiplexing techniques for stereo images (polarization, shuttering, color filtering), |
6. | Displays and stereo rendering: correct stereo projection, hypo- and hyper-stereo, issues with stereo rendering (depth aliasing, convergence-focus conflict, incorrect viewpoints, stereo violation, blur divergence, too much parallax), coherent virtual workspaces, automultiscopic display, |
7. | Real-time rendering: latency and its sources, view-independent rendering, prioritized rendering, efficient memory layout for fast rendering, level-of-detail techniques, static/adaptive/psychophysiological LOD selection, predictive LOD selection, progressive meshes, view-dependent LODs, |
8. |
Real-time rendering:
Foveated rendering
Input Devices: intrusiveness, degrees of freedom, multimodality, virtual trackball, isotonic vs isometric, tracking technologies, data gloves, locomotion devices, methods to classify input devices, logic devices. |
9. | Particle systems: Newton's Laws, dynamics vs kinematics, Euler integration, phase space, definition of particle systems, forces from physical effects, non-physical effects, collision handling, rendering of particle systems, alpha blending, |
10. | Mass-spring systems: definition, single spring-damper element, explicit Euler integration, instability and error accumulation with explicit Euler integration, Runge-Kutta, Verlet integration, constraints, implicit integration, tangent stiffness matrix, comparison to explicit integration, mesh creation for volumetric objects, consistent collision response for volumetric mass-spring systems. |
11. | Interaction techniques: Universal Interaction Tasks, approaches to design of user interfaces, the navigation task (wayfinding & locomotion), taxonomies as a design tool, abstract representation of the user for navigation, navigation metaphors (point-and-fly, scene-in-hand, two-handed, hands-free, walking in place, et al.), magic mirror, 3D navidget, user models: power law of practice, Hick's law, Fitts's law; 3D selection techniques, iso-/non-isomorphic techniques, control-display ratio, go-go technique, task decomposition of selection task, ray-based & cone-based techniques, flexible pointer, friction surface (bent ray), bubble cursor, depth ray, balloon selection, selection by progressive refinement: SQUAD, Disambiguation Canvas, Point-and-Zoom, Expand, object manipulation (grasping and moving), taxonomy of grasping, the manifold of hand postures, Natural User Interaction (NUI), |
12. |
Interaction techniques:
general 3D interaction design principles,
two-handed and multimodal interaction,
physical props / tangible UIs,
DOF separation/reduction: PRISM metaphor and 3D widgets,
action-at-a-distance
(examples: image plane interaction, voodoo dolls),
proprioceptive interaction,
world-in-miniature,
magic lenses,
redirecting the user,
redirected walking and related techniques, body retargeting,
amplified head rotation, spatial illusions,
system control,
menus,
Nielsen's usability heuristics
Haptics: definitions, applications, devices, the haptic loop(s), the human haptic sense and human factors, simulation factors, haptic textures, the problems of buzzing and latency, |
13. |
Haptics:
intermediate representations,
impedance/admittance-based approach,
the surface contact point method,
voxmap-pointshell method,
friction in one contact point,
Collision detection: motivation, definitions, collision detection pipeline, broad/narrow phase, 3D grid for filtering potentially colliding pairs, plane-sweep technique, separating planes algorithm (for convex objects), Minkowski sums, intersection test for convex objects based on Minkowski sums, |
14. |
Collision detection:
hierarchical collision detection,
bounding volume hierarchies, types of BV's,
separating axis lemma for convex polyhedra,
separating axis test (SAT),
overlap test for k-DOP's,
restricted boxtrees,
construction of BVH's using the volume heuristic,
inner sphere trees, sphere packings, proximity computation using ISTs,
penalty forces using ISTs.
Sound rendering: motivation, human factors, mixing sound sources, image source method, beam tracing method, cell partitioning, |
15. |
Screen Recordings of the Lecture as of WS 20/21
The videos are encoded with H.265/HEVC and should play fine with Safari and IE. With Chrome and Firefox, you might need to download the videos, or install a plugin/extension.
Chapter |
---|
Introduction (340 MB) |
Scene graphs and Game engines
(250 MB),
Tutorial on Unreal Engine, (270 MB) |
Display Technologies, Stereopsis, Stereo Rendering (250 MB) |
Techniques for Real-Time Rendering (320 MB) |
Principles of Input Devices (120 MB) |
Particle Systems (140 MB) |
Mass-Spring Systems (200 MB) |
Interaction Techniques (450 MB) |
Haptics (250 MB) |
Collision Detection (250 MB) |
Sound Rendering (90 MB) |
Call for Theses |
Note: all videos use the codec H265 (HEVC). (Saves about 50% disk space and bandwidth on average.) They work directly in the browsers Safari and IE (Edge), but not Chrome, and probably not Firefox (as of 2021). But you can always download the videos and play them using VLC.
You can find video recordings of the class taught in WS 19/20 here . The password is virtual2020.Literature
- Ralf Dörner, Wolfgang Broll, Paul Grimm, Bernhard Jung (Hrsg.): Virtual und Augmented Reality (VR/AR), Grundlagen und Methoden der Virtuellen und Augmentierten Realität. Springer 2013. You can read the e-book from within the university's network.
- Kay M. Stanney (Ed.): Handbook of Virtual Environments. Lawrence Erlbaum Associates, 2002
- William R. Sherman, Alan B. Craig: Understanding Virtual Reality. Morgan Kaufmann.
- Don Brutzman, Leonard Daly: X3D: Extensible 3D Graphics for Web Authors. Morgan Kaufmann, 2007.
- Daniel Fleisch: A Student's Guide to Vectors and Tensors. Cambridge University Press
- Kenny Erleben et al.: Physics Based Animation. Charles River Media, 2005.
- Mario Gutiérrez, Frédéric Vexo, Daniel Thalmann: Stepping into Virtual Reality. Springer, 2008. You can read the e-book from within the university's network.
- Anthony Steed, Manuel Oliveira: Networked Graphics: Building Networked Games and Virtual Environments. Morgan Kaufman, 2009. From this book, only chapters 7, 10, and 11 are relevant to this course.
Warning: these text books can only give you a general introduction to the field of VR! Most of the topics taught in class are not covered by any of these text books directly -- in fact, AFAIK there are some topics that are not covereed by any text book! Therefore, I recommend to attend class.
If you are thinking of buying some of these books, then I suggest to consider buying a used copy of them -- very often, you can find them at a fraction of the price of a new copy. The following are three very good internet sites for finding inexpensive used copies of books: abebooks, Booklooker, and ZVAB.
Help and documentation on the Unreal Engine:
- Unreal Online Learning Courses
- Material provided and curated by Epic: I suggest to take the "Getting Started" Learning Path first, then, depending on your tasks and goals, "Unreal Engine Kickstart for Developers", "Introducing the Principles of Real-time", "Exploring Blueprints", "Constructing Believable Environments", "An In-Depth Look at Real-Time Rendering", "Balancing Blueprint and C++ in Game Development"; but I encourage to to explore by yourself, they keep adding stuff!
- For general help, take a look at the Unreal Engine 4 Documentation and in case of problems also the UE4 AnswerHub
- For quick help you can visit the IRC channels #unrealengine and #ue4linux on Freenode (web client)
- Compiling and running Unreal on Linux is explained in the wiki
- To setup c++ debugging this Youtube video is highly recommended
- If you have experience with the Unity engine take a look at Unreal for Unity developers
- YouTube Tutorials by EpicGames
Online Literature and Resources on the Internet
- Some general readings (and listening) about VR, that do not talk so much about the
technical details, but more about the visionary, or historic, aspects of VR -- or both.
- A paper by a large group of people working in VR: The Ethics of Realism in Virtual and Augmented Reality.
- The short novel
Gegen den Strich
by Joris-Karl Huysmans.
(Sorry, I have it in German only)
Denkanregung: was hat das mit VR zu tun? - An interview with Jaron Lanier from 2017, where he talks about VR, music, some of the milestones of VR history, AI, consciousness, what should, and what shouldn't be done with VR, and a lot more. (Source)
- A review of Jaron Lanier's new book, which is, actually, msotly about Jaron Lanier, which is quite as fascinating. (Source)
- If you are interested in what is currently going on in VR, then this podcast, ResearchVR, by Azad Balabanian & Petr Legkov, is right for you. It does not require a lot of computer scientific knowledge, and it covers a huge breadth of topics.
- The original article praising the panorama in Blackwood's Edinburgh Magazine (vol. 15, 1824)
- A visionary article by Vannevar Bush, that shows that the idea of virtual reality is not that new: As We May Think (1945)
- Interview with Jaron Lanier and David Eggers (it was part of the program "Druckfrisch" and aired on 5.10.2014 on the German ARD, H.265 codec)
- Howard Rheingold's Tools For Thought is a good and easy read on the evolution of futuristic ideas in the computer space. Recommended if you are interested in the history of VR and extended reality in general.
- Literature on stereoscopic ("3D") rendering:
- Implementing Stereoscopic 3D in Your Applications by Samuel Gateau and Steve Nash from NVidia at GPU Technology Conference 2010 (Source)
- Rendering 3D Anaglyph in OpenGL by Animesh Mishra, 2011 (Source)
- A very nice chapter on Binocular Vision and Space Perception. A pretty accessible read.
- A very good conference tutorial talk on factors and limits of human stereoscopic vision from a perception and neuroscientific perspective, by Martin Banks, 2013.
- A good survey paper of the state of the knowledge on stereopsis by Robert Patterson, 1992.
- For people interested in modern real time rendering pipelines, take a look at this GTA V - Graphics Study by Adrian Courreges, 2015. (Source)
- More literature on the topic of user interface design:
- The Siggraph 2001 tutorial Advanced Topics in 3D User Interface Design
- An entertaining and very insightful Quiz on Fitts' Law
- A Lecture on Fitts Law by Heiko Drewes, LMU, 2013: Nice survey with a critical eye on the value of some of the research that has been done Fitts' law (or, rather, over-research) ( Source).
- Giving You Fitts: an article by Jensen Harris describing where Fitts' law has been used in the UI design of Office 2007 ( Source).
- If you feel familiar with at least the material of the first half of this course, then this set of episodes on advanced topics in XR might be very instructive for you. Each episode is delivered by one of the world-renowned experts on the topic. Topics include 3DUI in VR, cybersickness, navigation, illusions in VR, spatial perception, and cognitive maps, among others.
- Here are the -- not quite serious -- Cartoon Laws of Physics ;-) .
- Literature on particle and on spring-mass systems:
- William T. Reeves: Particle Systems - A Technique for Modeling a Class of Fuzzy Objects;
- More advanced particle systems (e.g., n-body systems) are explained in Real-Time Particle Systems on the GPU in Dynamic Environments by Shannon Drone (Source)
- Georgii, Westermann: Mass-Spring Systems on the GPU, Simulation Practice and Theory 2005. (Source)
- Literature on physically-based simulation in general:
- Introductory Vector Calculus by Norman Wittels (Source)
- Siggraph 2008 course notes: Real Time Physics
- Siggraph 2001 course notes: Physically Based Modeling
- Survey paper over the field of deformation simulation methods: Physically Based Deformable Models in Computer Graphics by Andrew Nealen, Mathias Muller, Richard Keiser, Eddy Boxerman and Mark Carlson, EG 2005
- Here is a survey article on collision detection. You probably need some basic knowledge on collision detection.
- A bit on the history of Virtual and Mixed Reality:
- An article from The Verge: The Rise and Fall and Rise of Virtual Reality by Adi Robertson and Michael Zelenko (Source, in particular here)
- Highlights from an interview with Jaron Lanier and Kevin Kelly from The Verge: Digital Natives, A conversation between virtual reality visionaries Jaron Lanier and Kevin Kelly by Casey Newton (Source, in particular here)
- The Ultimate Display by Donald Sutherland
Literature and Resources on X3D/VRML
Since X3D/VRML is no longer the platform for the practical exercises in this course, I have demoted the links to X3D/VRML to this place.- Tools that can "play" X3D- / VRML97 files:
- InstantReality; on their homepage, you can also find a lot of examples and tutorials.
- FreeWRL (for Linux and OSX).
- Cortona; this is a commercial browser plugin.
- Introduction to VRML / X3D:
- A "Cheat Sheet" for VRML.
- SIGGRAPH 2008 Class Notes: Don't be a WIMP; These course notes not only discuss some post-WIMP interaction techniques, but also explain how to use some of the more advanced features of InstantReality, such as different stereo rendering modi, special interaction devices, clustering, scripting, animations, etc. (Source)
- The Annotated VRML97 Reference Manual (Source)
- The VRML Primer and Tutorial; those chapters that deal with HTML are not relevant for this course. In addition, the chapter on tools is outdated; but otherwise it is still a veritable introduction for VRML novices. (Source)
-
The most important documents on the X3D standard:
- Architecture and base components (i.e., the specification of nodes and profiles)
- Scene access interface (SAI)
- X3D language bindings for Javascript (ECMAScript)
- X3D language bindings for Java
- Examples for X3D / VRML:
- The examples from the lecture (and many more!)
- More examples are on the homepage of InstantReality;
- And on the homepage of the X3D book by Don Brutzman and Leonard Daly;
- A meta web page with lots of links to archives containing more examples.
- A large collection of materials,
both in VRML, as well as in XML encoding.
And here it is again as a ZIP archive. - Links on socalled authoring tools; for VRML/X3D; if you want, you are welcome to use them. But if you have a decent programming editor (preferrably in ASCII), then you are probably more efficient with your editor.
- A handy little tool for
Calculating VRML Viewpoints.
Takes as input viewpoint, look-at, and up vector, outputs rotation as axis + angle to achieve the rotation from world coordinate frame into (viewpoint, look-at, up). From there it is trivial to compute the quaternion.
Readings That Have Nothing to do With VR, but are Still Highly Recomended
- An article about the modern tools to influence voters via so-called social media: Wahlmanipulation mittels Psychometrik und Social Media (Quelle: Das Magazin). The article is in German only, sorry to the non-German speakers. Of course, the techniques and tools described in the article can also be used for other purposes, which makes them even more dangerous to society.
- One of the last interviews with Joseph Weizenbaum, a very well-known pioneer of AI and critic of too naive belief in technology (German only, sorry).
Last modified: Wed Aug 04 15:57:25 MDT 2021