Tech
AI method reconstructs 3D scene details from simulated images using inverse rendering
Over the past decades, computer scientists have developed many computational tools that can analyze and interpret images. These tools have proved useful for a broad range of applications, including robotics, autonomous driving, health care, manufacturing and even entertainment.
Most of the best performing computer vision approaches employed to date rely on so-called feed-forward neural networks. These are computational models that process input images step by step, ultimately making predictions about them.
While some of these models were found to perform well when tested on the data they analyzed during training, they often do not generalize well across new images and in different scenarios. In addition, their predictions and the patterns they extract from images can be difficult to interpret.
Researchers at Princeton University recently developed a new inverse rendering approach that is more transparent and could also interpret a wide range of images more reliably. The new approach, introduced in a paper published in Nature Machine Intelligence, relies on a generative artificial intelligence (AI)-based method to simulate the process of image creation, while also optimizing it by gradually adjusting a model’s internal parameters.
“Generative AI and neural rendering have transformed the field in recent years for creating novel content: producing images or videos from scene descriptions,” Felix Heide, senior author of the paper, told Tech Xplore. “We investigate whether we can flip this around and use these generative models for extracting the scene descriptions from images.”
The new approach developed by Heide and his colleagues relies on a so-called differentiable rendering pipeline. This is a process for the simulation of image creation, relying on compressed representations of images created by generative AI models.
“We developed an analysis-by-synthesis approach that allows us to solve vision tasks, such as tracking, as test-time optimization problems,” explained Heide. “We found that this method generalizes across datasets, and in contrast to existing supervised learning methods, does not need to be trained on new datasets.”
Essentially, the method developed by the researchers works by placing models of 3D objects in a virtual scene depicting real world settings. These models of objects are generated by a generative AI based on random sample of 3D scene parameters.
“We then render all these objects back together into a 2D image,” said Heide. “Next, we compare this rendered image with the real observed image. Based on how different they are, we backpropagate the difference through both the differentiable rendering function and the 3D generation model to update its inputs. In just a few steps, we optimize these inputs to make the rendered match the observed images better.”
-
Optimizing 3D models through inverse neural rendering. From left to right: the observed image, initial random 3D generations, and three optimization steps that refine these to better match the observed image. The observed images are faded to show the rendered objects clearly. The method effectively refines object appearance and position, all done at test time with inverse neural rendering. Credit: Ost et al.
-
Generalization of 3D multi-object tracking with Inverse Neural Rendering. The method directly generalizes across datasets such as the nuScenes and Waymo Open Dataset benchmarks without additional fine-tuning and is trained on synthetic 3D models only. The observed images are overlaid with the closest generated object and tracked 3D bounding boxes. Credit: Ost et al.
A notable advantage of the team’s newly proposed approach is that it allows very generic 3D object generation models trained on synthetic data to perform well across a wide range of datasets containing images captured in real-world settings. In addition, the renderings produced by the models are far more explainable than those produced by conventional rendering tools based on feed-forward machine learning models.
“Our inverse rendering approach for tracking works just as well as learned feed-forward approaches, but it provides us with explicit 3D explanations of its perceived world,” said Heide.
“The other interesting aspect is the generalization capabilities. Without changing the 3D generation model or training it on new data, our 3D multi-object tracking through Inverse Neural Rendering works well across different autonomous driving datasets and object types. This can significantly reduce the cost of fine-tuning on new data or at least work as an auto-labeling pipeline.”
This recent study could soon help to advance AI models for computer vision, improving their performance in real-world settings while also increasing their transparency. The researchers now plan to continue improving their method and start testing it on more computer vision-related tasks.
“A logical next step is the expansion of the proposed approach to other perception tasks, such as 3D detection and 3D segmentation,” added Heide. “Ultimately, we want to explore if inverse rendering can even be used to infer the whole 3D scene, and not just individual objects. This would allow our future robots to reason and continuously optimize a three-dimensional model of the world, which comes with built-in explainability.”
Written for you by our author Ingrid Fadelli,
edited by Gaby Clark, and fact-checked and reviewed by Robert Egan—this article is the result of careful human work. We rely on readers like you to keep independent science journalism alive.
If this reporting matters to you,
please consider a donation (especially monthly).
You’ll get an ad-free account as a thank-you.
More information:
Julian Ost et al, Towards generalizable and interpretable three-dimensional tracking with inverse neural rendering, Nature Machine Intelligence (2025). DOI: 10.1038/s42256-025-01083-x.
© 2025 Science X Network
Citation:
AI method reconstructs 3D scene details from simulated images using inverse rendering (2025, August 23)
retrieved 23 August 2025
from https://techxplore.com/news/2025-08-ai-method-reconstructs-3d-scene.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.
The screen is brighter now, reaching a peak brightness of 3,000 nits, and I haven’t had any trouble reading it in sunny conditions (though it hasn’t been as sunny as I’d like it to be these past few weeks). I appreciate the glass upgrade from Gorilla Glass 3 to Gorilla Glass 7i. It should be more protective, and anecdotally, I don’t see a single scratch on the Pixel 10a’s screen after two weeks of use. (I’d still snag a screen protector to be safe.)
Photograph: Julian Chokkattu
Another notable upgrade is in charging speeds—30-watt wired charging and 10-watt wireless charging. I’ll admit I haven’t noticed the benefits of this yet, since I’m often recharging the phone overnight. You can get up to 50 percent in 30 minutes of charging with a compatible adapter, and that has lined up with my testing.
My biggest gripe? Google should have taken this opportunity to add its Pixelsnap wireless charging magnets to the back of this phone. That would help align the Pixel 10a even more with the Pixel 10 series and bring Qi2 wireless charging into a more affordable realm—actually raising the bar, which wouldn’t be a first for the A-series. After all, Apple did exactly that with the new iPhone 17e, adding MagSafe to the table. Or heck, at least make the Pixel 10a Qi2 Ready like Samsung’s smartphones, so people who use a magnetic case can take advantage of faster wireless charging speeds.
Battery life has been OK. With average use, the Pixel 10a comfortably lasts me a full day, but it still requires daily charging. With heavier use, like when I’m traveling, I’ve had to charge the phone in the afternoon a few times to make sure it didn’t die before I got into bed. This is a fairly big battery for its size, but I think there’s more Google could do to extend juice, akin to Motorola’s Moto G Power 2026.
After a week of product announcements—starting with the iPhone 17e, a refreshed iPad Air, and more powerful MacBook Pro models—Apple has unveiled a new category in its laptop lineup for the first time in a while: the “MacBook Neo.”
Photograph: Julian Chokkattu
Positioned below the MacBook Air as an entry-level machine, this new MacBook is the most affordable laptop the company has ever made, with a starting price of $599. While it’s been possible to buy a new MacBook Air at lower prices—like the 2020 M1 MacBook Air Apple sold for several years for $699 exclusively through Walmart—this is officially the cheapest MacBook out the gate.
Aside from the price, its approach to color also makes it unique among the other MacBooks in Apple’s lineup. You have several color options, including Silver, Indigo, Blush, and Citrus. The colors harken back a bit to the iBook G3 of yesteryear and are akin to the current iMac design. In person, the colors aren’t a bright and bold as expected, still exhibiting a more subtle hue. Apple says the aluminum device weighs 2.7 pounds, which is the same as the 13-inch MacBook Air. We’re still waiting on official measurements on the thickness.
Despite its price, Apple doesn’t appear to be cutting corners on the quality of the screen. With a resolution of 2408 by 1506 and up to 500 nits of brightness, Apple boasts that it is “both brighter and higher in resolution than most PC laptops in this price range.” The display doesn’t use a notch for the webcam like the MacBook Air or MacBook Pro. There’s a 1080p camera, a Touch ID sensor, and side-firing speakers with Dolby Atmos. Unfortunately, the Touch ID sensor is only available on the $699 model, which comes with 512 GB of storage.
The MacBook Neo does make plenty of other concessions to hit its aggressive price though. It’s powered by the A18 Pro chip—the same processor inside the iPhone 16 Pro and 16 Pro Max. Yup—you read that right. iPads have used Mac chips for years, but now a MacBook is using an iPhone chip. Still, this processor should deliver more power than the original M1 chip in the MacBook Air. Apple claims the chip gives the MacBook Neo up to 16 hours of battery life. That’s less than the MacBook Air or MacBook Pro. Apple also says the chip is up to 50 percent faster in daily tasks like web browsing than “the bestselling PC with the latest chipping Intel Core Ultra 5.” According to the liner notes, this was based on a Speedometer test, a popular browser-based benchmark.
Other compromises to the device are the use of a mechanical multi-touch trackpad (rather than one that uses haptic feedback), a non-backlit keyboard, and the more limited port selection. The use of the iPhone chip means this MacBook only supports one external monitor through one if its two USB-C ports. Either port can be used for charging. There’s also a headphone jack, located in an odd position next to the side-firing speakers near the front of the device. While technically this is the same amount of USB-C ports as the MacBook Air, it’s missing the magnetic MagSafe 3 charging port, which frees up one of the USB-C ports.
Initially, Gorham used his brain-computer interface for single clicks, Oxley says. Then he moved on to multi-clicks and eventually sliding control, which is akin to turning up a volume knob. Now he can move a computer cursor, an example of 2D control—horizontal and vertical movements within a two-dimensional plane.
Over the years, Gorham has gotten to try out different devices using his implant. Zafar Faraz, a field clinical engineer for Synchron, says Gorham directly contributed to the development of Switch Control, a new accessibility feature Apple announced last year that allows brain-computer interface users the ability to control iPhones, iPads, and the Vision Pro with their thoughts.
In a video demonstration shown at an Nvidia conference last year in San Jose, California, Gorham demonstrates using his implant to play music from a smart speaker, turn on a fan, adjust his lights, activate an automatic pet feeder, and run a robotic vacuum in his home in Melbourne, Australia.
“Rodney has been pushing the boundaries of what is possible,” Faraz says.
As a field clinical engineer, Faraz visits Gorham in his home twice a week to lead sessions on his brain-computer interface. It’s Faraz’s job to monitor the performance of the device, troubleshoot problems, and also learn the range of things that Gorham can and can’t do with it. Synchron relies on this data to improve the reliability and user-friendliness of its system.
In the years he’s been working with Gorham, the two have done a lot of experimenting to see what’s possible with the implant. Once, Faraz says, he had Gorham using two iPads side by side, switching between playing a game on one and listening to music on the other. Another time, Gorham played a computer game in which he had to grab blocks on a shelf. The game was tied to an actual robotic arm at the University of Melbourne, about six miles from Gorham’s home, that remotely moved real blocks in a lab.
Gorham, who was an IBM software salesman before he was diagnosed with ALS in 2016, has relished being such a key part of the development of the technology, his wife Caroline says.
“It fits Rodney’s set of life skills,” she says. “He spent 30 years in IT, talking to customers, finding out what they needed from their software, and then going back to the techos to actually develop what the customer needed. Now it’s sort of flipped around the other way.” After a session with Faraz, Gorham will often be smiling ear to ear.
Through field visits, the Synchron team realized it needed to change the setup of its system. Currently, a wire cable with a paddle on one end needs to sit on top of the user’s chest. The paddle collects the brain signals that are beamed through the chest and transmits them via the wire to an external unit that translates those signals into commands. In its second generation system, Synchron is removing that wire.
“If you have a wearable component where there’s a delicate communication layer, we learned that that’s a problem,” Oxley says. “With a paralyzed population, you have to depend on someone to come and modify the wearable components and make sure the link is working. That was a huge learning piece for us.”
How many countries has US bombed since 9/11, and what has it cost?
Google’s Pixel 10a May Not Be Exciting, but It’s Still an Unbeatable Value
Rupee falls to all-time low of 92.05 against dollar as oil surge, foreign outflows rattle markets – The Times of India
India Us Trade Deal: Fresh look at India-US trade deal? May be ‘rebalanced’ if circumstances change, says Piyush Goyal – The Times of India
Households set for lower energy bills amid price cap shake-up
US arrests ex-Air Force pilot for ‘training’ Chinese military
-
Business5 days ago
India Us Trade Deal: Fresh look at India-US trade deal? May be ‘rebalanced’ if circumstances change, says Piyush Goyal – The Times of India
-
Business1 week ago
Households set for lower energy bills amid price cap shake-up
-
Politics7 days agoUS arrests ex-Air Force pilot for ‘training’ Chinese military
-
Politics6 days agoWhat are Iran’s ballistic missile capabilities?
-
Business6 days agoAttock Cement’s acquisition approved | The Express Tribune
-
Fashion6 days agoPolicy easing drives Argentina’s garment import surge in 2025
-
Sports1 week agoTop 50 USMNT players of 2026, ranked by club form: USMNT Player Performance Index returns
-
Sports6 days agoSri Lanka’s Shanaka says constant criticism has affected players’ mental health