NVIDIA and Siemens Healthineers Are Teaching Ultrasound to Actually Listen

Ultrasound has always been a compromise. You get real-time imaging, it’s safe, portable, and cheap—but the picture you see on the screen is a heavily processed approximation. The raw sensor data, the actual sound waves bouncing through tissue, gets thrown away during reconstruction. Everyone just accepted that trade-off because the traditional beamforming pipeline was the only game in town.

NVIDIA and Siemens Healthineers just called that bluff. They released a model called NV-Raw2Insights-US that skips the reconstruction entirely and learns directly from raw ultrasound channel data. The idea isn’t new—researchers have been poking at this for years—but the scale and deployment approach here are different.

What Raw2Insights Actually Does

Ultrasound probes don’t capture images. They capture sound echoes. The reconstruction pipeline makes assumptions—like a constant speed of sound throughout the body—that are flat-out wrong for most patients. Fat, muscle, bone, tumors: sound moves through each at different speeds. The traditional approach ignores that and produces a uniform image regardless of who’s on the table.

NV-Raw2Insights-US estimates a personalized map of sound speed for each patient in a single AI pass. That map gets fed back to the scanner to correct focus in real time. What used to require complex, iterative computation now happens faster than the frame rate of the ultrasound display. That’s impressive, but more importantly, it’s practical.

This is the first application in what NVIDIA calls the Raw2Insights class of models. The vision is end-to-end AI for ultrasound—skipping the hand-engineered pipeline entirely and learning from the rawest form of sensor data available. I’ve seen too many AI papers that look great in simulation but fall apart on actual hardware. This one seems built for real deployment from day one.

The Deployment Story Matters More Than You’d Think

Raw ultrasound channel data is notoriously hard to access on clinical scanners. It’s high-bandwidth, and most systems aren’t designed to export it. NVIDIA’s solution is Holoscan Sensor Bridge (HSB), an open-source FPGA IP that streams data from the scanner’s DisplayPort outputs over Ethernet to an NVIDIA IGX system for inference.

They’re calling this Data over DisplayPort, and honestly, it’s a clever hack. Instead of redesigning the scanner hardware, they’re piggybacking on existing outputs. The FPGA development kit is an Altera Agilex-7 paired with HSB, connected to an ACUSON Sequoia scanner. The data goes to GPU memory on a Blackwell-class GPU, inference runs, and the sound-speed estimate streams back to the scanner to adjust focus.

This is higher bandwidth than I expected for a prototype. It’s one thing to demonstrate AI inference on raw data in a lab. It’s another to make it work with clinical hardware in a way that doesn’t require ripping out the entire imaging chain.

What This Unlocks

The architecture is software-defined, which means improvements don’t require hardware swaps. New models can be integrated as long as raw channel data is already in GPU memory. That’s a modular foundation for building diagnostic AI that doesn’t depend on reconstructed images—which are lossy by design.

NVIDIA is releasing the model weights, dataset, and GitHub repository. If you want to tinker, you can start developing on top of it today. The paper references IEEE TMI work on common midpoint phase error optimization and a 2023 arXiv paper on pulse-echo sound speed estimation in breast ultrasound. The research lineage is solid.

The Catch

This technology is under investigational development. It’s not cleared for clinical use yet. The demo architecture works, but real-world validation across diverse patient populations and scanner models is still ahead. Also, the hardware requirements are significant: Holoscan Sensor Bridge, IGX, Blackwell GPU. That’s not cheap. For now, this is aimed at research institutions and advanced clinical development labs, not community hospitals.

Still, this is the first time I’ve seen a major player commit to raw-channel AI for ultrasound at this scale. The approach is sound—literally. Learning from how sound actually moves through each patient instead of assuming a uniform medium is the right direction. The question is whether the clinical infrastructure can catch up to the research.