Nvidia Researchers Make Real-Time Rendering a Virtual Reality


Silicon Valley chip designer Nvidia is on the way to making real-time rendering of 3D virtual environments from 2D video a reality for a broad spectrum of industry. Using machine learning, their technique promises to expand the market for the company’s GPUs, too.

By synthesizing images with neural networksOpens a new window , a team of the company’s researchers said their work significantly reduces the time it takes to create three-dimensional worlds in cyberspace. The technique relies on generative modeling to describe observed objects in data, which computers then use to build their virtual counterparts.

The models train machines to recognize objects from the flows of unsupervised data generated by digital video cameras. Replicating them based on learned characteristics eliminates the need for individual rendering, thus sparing time and expense over current techniques.

The models run on multiple GPUs that work in tandem to parse information. Neural networks improve pattern recognition, while their adaptive ability lets them learn from the unstructured datasets they process.

Real-Time Video-to-Video Language

The Nvidia research team set about solving problems with video-to-video synthesis, aiming to map precisely depictions of objects in source video for their accurate reproduction at high resolution in subsequent outputs, such as virtual reality and computer simulations.

At present, each object must be modeled separately when creating a virtual environment. Teaching machines to do that work from video images enables automatic execution of those processes and lets users augment those environments with relative ease.

Another team of researchers at Nvidia is using neural networks to make the insertion of objects into rendered environments content-awareOpens a new window . Generative models in those applications speed the placement and posing of objects in so-called spatial transformations by automatically working out location and geometry based on the dynamics of the virtual environment.

While the company has built its business in gaming since it launched the first GPUs in 1999, the technique has applications in transport and architecture. Real-time rendering also promises to boost both robotics and virtual reality applications in the workplace.

Streamlined Rendering Dramatically Speeds Up Reproduction

The technique was demonstrated by Nvidia researchers last month at NeurlIPS 2018Opens a new window , the 32nd conference on neural information processing systems held annually in Montreal. Players of an interactive game could drive through an urban environment rendered in real time from video images.

Generative models let the networks render scenes without the need for digitized descriptions of materials, lighting and geometry. The reduced data requirements make for faster reproduction of virtual environments than standard graphics engines that need more information to accomplish the task.

The advances build on the company’s GPU pioneering, where architectures and chipsets are ever more customized for machine-learning workflows. In summer, Nvidia added a pair of deep-learning acceleratorsOpens a new window to the CPUs at work on its Issac platform, enabling the chipset to perform 30 trillion operations per second.

Those are the speeds at which robots take stock of their environment and execute functions autonomously based on the sensory information. Improved perception means the robot can self-correct based on task performance and on changes in the surrounding environment.

Light Ray Advances

Nvidia also launched a line of GPUsOpens a new window designed to deliver ray tracing technology for more accurate rendering of images in 3D environments. The company’s RTX technology uses algorithms for real-time shading as light meets solid, transparent and translucent objects in virtual environments.

The chipset’s artificial intelligence engine and graphics accelerator permit rendering of the reflection, refraction, scattering and dispersion that occurs when light rays encounter geometric objects. Greater precision in rendering the shifting bounds of light and shading better simulates the look and feel of 3D in two-dimensional space.

So far, newness has yet to produce documented use-cases for RTX beyond entertainment-industry targets like gaming and cinematography. However, the company is expressing hope that lower costs from real-time video rendering will speed its uptake across a wider spectrum for simulations that train machines to interact more effectively with their environments.