A bittersweet lesson in the learning

(Highly subjective) AI4Science highlights of July 2026

Unearthing the foundations of continuous data

GenCeption from Google DeepMind recently proposed a new approach to building foundation models for visual data. The key idea is to take a pretrained diffusion generative model (for videos), freeze its autoencoder weights (if it’s a latent diffusion model), and finetune the diffusion model’s Diffusion Transformer (DiT) architecture for both dense and sparse visual representation tasks using learnable task and video tokens. Importantly, dense downstream tasks can reuse the model’s frozen decoder weights (if present), treating new representation learning/prediction tasks as a visual reconstruction objective.

Diffusion models can both generate and represent continuous data well.

This approach currently provides both high performance and wide coverage of tasks spanning computer vision, suggesting that, though a denoising objective, its pretrained DiT architecture has learned highly informative and broadly applicable representations of continuous (visual) data. This suggests that computer vision, perhaps alongside other (non-discrete) data domains, is beginning to identify the axes of neural scaling that are most effective beyond the next-token prediction paradigm prevalent in natural language processing. Scientific applications based on this idea are likely to follow suit. However, the nuances of what it takes to train such an informative generative model in different scientific domains will almost certainly require extra consideration.

Discovering physical laws (through language)

Can large language models (LLMs) deduce the physical laws of novel simulated worlds through text-based interaction alone? If so, would this convince us that there is something inherently informative that LLMs are learning beyond distributions for next-token prediction? With the new DiscoverPhysics benchmark from Princeton, we can begin to (partially) address these questions.

In the DiscoverPhysics benchmark, LLMs have to uncover (new) physical laws.

DiscoverPhysics has a convenient leaderboard to track community progress on this task of uncovering novel physical laws, along with open-source code accompanying the benchmark. Interestingly, with the release of Anthropic’s Claude 5 Fable model, the leaderboard appears to have a new resolute leader in terms of per-world accuracy as well as explainability. Does this suggest that larger models are simply more likely to find better (and more interpretable) solutions to physical phenomena than smaller models (re: epiplexity)?

Reflections

What are the implications of neural scaling for different scientific applications? Do we simply need larger models pretrained on whatever data, be it text, images, or the like? Or do we first need to find the right architecture to scale (as for DeepMind’s GenCeption model)?