Mikael Johansson (KTH - Royal Institute of Technology, Stockholm, Sweden)

Controlled disagreement: using consensus errors to shape curvature in decentralized training

Wednesday, October 14, 09:30-10:30

Abstract:

Decentralized training replaces global all-reduce with local communication between workers, trading exact synchronization for scalability. The resulting consensus errors are usually treated as a nuisance to be minimized. In this talk, we argue that they can instead be a useful resource for finding better solutions. We first describe what it takes to make decentralized training competitive with all-reduce in practice: overlapping computation with communication, adapting the optimizer to small local batches, and careful design of the communication graph. We then show that deliberately maintaining disagreement late in training leads to flatter solutions and better generalization, and propose a simple adaptive consensus scheme that ties the consensus step size to the learning rate. In the final part of the talk, we present ongoing work on explaining why this happens. We illustrate how the spread of workers around the average lets them sense how curvature varies, and how this nudges the average model toward flatter regions. How strongly this happens, and which directions are affected, depends on a delicate interplay between the communication graph, the loss landscape, and the consensus schedule. We close with open questions on adaptive optimizers, consensus schedules, and graph design for implicit regularization. The presentation is based on joint work with Zesen Wang, Jiaojiao Zhang, and Xuyang Wu.

Or Ordentlich (The Hebrew University)

Quantized Matrix Multiplication - Theory and Practice

Wednesday, October 14, 11:00-11:45

Abstract:

Matrix multiplication is the main computational building block of large language models, and is often bottlenecked by moving data from memory. Quantization reduces this cost by representing matrices with fewer bits, but introduces errors that can degrade model accuracy. This talk explores how information theory can guide the design of accurate and efficient quantization schemes. We begin with the rate-distortion function for Gaussian matrix multiplication and a nested-lattice construction that asymptotically achieves this limit. We then discuss NestQuant, which brings these ideas to LLM quantization, and hierarchical lattice schemes that enable matrix multiplication using small lookup tables. Finally, we turn to activation statistics and hardware constraints: WaterSIC combines rate allocation with successive interference cancellation to approach information-theoretic limits for weight quantization, while OSFP4 jointly optimizes diagonal smoothing and block scales for the NVFP4 format. These results connect fundamental limits with practical methods for preserving LLM accuracy at low precision.

Roberto Morabito (DTU)

Running AI at the Edge: Systems Lessons from Models to Distributed Intelligence

Wednesday, October 14, 11:45-12:30

Abstract:

In this talk, I will discuss the evolution of Edge AI from running individual models on resource-constrained devices towards increasingly distributed and adaptive AI systems. I will present some of our recent work on generative AI at the edge and discuss how model capabilities, resource constraints, and system conditions influence where and how inference should be performed. I will then focus on distributed AI service provisioning across heterogeneous edge environments, where AI services are deployed and continuously adapted as application requirements, workloads, and available resources change. Drawing on these experiences, I will discuss some of the systems challenges that arise as AI workloads become more complex, distributed, and dynamic.

Petros Elia (EURECOM)

Fundamental Limits of Hypergraph Edge Partitioning

Wednesday, October 14, 14:00-14:45

Abstract:

Hypergraph edge partitioning is a central problem in computer science, with broad impact in distributed computing and learning. It addresses the following question: given a sparse pattern of computations/interactions, how should these computations be distributed across machines while minimizing the amount of data/model state that must be replicated? In this setting, after given a collection of hyperedges (jobs) -- each consisting of up to d vertices from a ground set of size n (n files) -- one must assign these edges across N partitions (servers/shards) so as to minimize, for example, the vertex footprint, i.e., the maximum number of vertices that appear in any partition. This problem has attracted substantial attention, primarily from an algorithmic perspective, yet its fundamental performance limits remain largely unexplored. We here identify the fundamental limits of hypergraph edge partitioning -- optimized over all algorithms -- for a broad class of well-established probabilistic hypergraph models, and for hypergraphs that range from the fully dense, to the very sparse (with edge-set sizes down to linear in n). Rather surprisingly, optimality is achieved by a blind construction, whose vertex (file) placement is fixed a priori and is completely independent of the realized hypergraph (independent of the assigned tasks).

Linus Thrybom (ABB Corporate Research, Västerås, Sweden)

Industrial Electrification – with AI Optimization – an ABB perspective

Wednesday, October 14, 14:45-15:30

Abstract:

Industrial electrification is crucial for reducing CO2 emissions, but today's energy landscape differs fundamentally from a decade ago. Energy prices now swing in 15 min blocks daily and seasonally, causing significant cost saving opportunities that our customers must master through intelligent control systems. Successful electrification requires more than access to power and energy – it demands integrated power management that balances process demands with grid dynamics. This is where control with AI optimization becomes essential, yet significant research and engineering questions remain unanswered. This presentation explores the emerging discipline of industrial electrification, together with its necessary control with AI optimization. Drawing on real-world use cases, we will discuss the energy transition landscape and identify key research challenges for making this transition economically viable and technically feasible.

Yvonne-Anne Pignolet (SigNetwork)

A Field Guide to Distributed Protocols in the Wild

Thursday, October 15, 09:30-10:30

Abstract:

Textbooks describe distributed protocols in captivity, with clean models and well-defined adversaries and objectives. Outside the lab, these protocols meet open networks, non-determinism, live upgrades and real money at stake.

This talk draws on three systems built for such habitats. The first is the Internet Computer, a decentralized cloud hosting nearly a million apps. The second is StableClear's infrastructure for moving stablecoins between chains. The third is Signet, an MPC network for general cross-chain interactions with threshold signatures.

The protocol populations that survive in these environments show strikingly similar traits. I will discuss where our theoretical models had to be adapted, composed or rethought, which design patterns held up from one system to the next, and which trade-offs turned out to be surprising. You'll get field notes from close encounters in production, tips on safe handling, and pointers to where previously unknown species are still waiting to be discovered.

Malcolm Egan (Inria)

Distributed Learning with Markovian Sampling

Thursday, October 15, 11:00-11:45

Abstract:

A central design decision for distributed learning systems is whether updates should be synchronous or asynchronous. It has recently been shown that the runtime of asynchronous stochastic gradient descent (SGD) can significantly outperform synchronous minibatch SGD with i.i.d. sampling. However, modern machine learning with massive datasets often requires advanced sampling strategies to extract informative samples for specific tasks. As a consequence, sampling is often based on Markov chain Monte Carlo, which induces non-i.i.d. Markovian samples. In this talk, I will overview our recent results on the convergence theory of synchronous and asynchronous SGD with Markovian sampling, which suggest sample complexity gains of asynchronous over synchronous methods. The theory will be illustrated in the context of dictionary learning on real-world large-scale graphs.

Panayotis Mertikopoulos (French National Center for Scientific Research (CNRS))

A large-deviations analysis of stochastic gradient descent

Thursday, October 15, 11:45-12:30

Abstract:

Even though stochastic gradient descent (SGD) was introduced more than 75 years ago, it remains the method of choice for training modern machine learning models and deep learning architectures. Still, despite the method's phenomenal success, we know surprisingly little about its long-run behavior—for example, which minimizers are more likely to be observed in the long run, or how long until it reaches a global minimum (if at all). In this talk, I will outline an approach to study the long-run behavior of SGD based on the theory of large deviations and randomly perturbed dynamical systems. Using this approach, we show that the limiting distribution of SGD follows the Boltzmann-Gibbs law of equilibrium thermodynamics with temperature equal to the method's step-size and energy levels determined by the problem's objective and the statistics of the noise. In particular, we show that, in the long run, (a) the iterates of SGD spend an exponentially small amount of time away from the problem's critical region; (b) any given critical point (or manifold thereof) is visited with probability that is exponentially proportional to its energy; (c) minimizers are visited exponentially more often than non-minimizers; and (d) SGD becomes exponentially concentrated around the problem's "ground state" (which does not always coincide with the minimum of the objective). If there is time, I will also describe a tight characterization of the global convergence time of SGD via matching upper and lower bounds which quantify the most “costly” set of obstacles that SGD may need to overcome to reach a global minimizer from a given initialization.

Fotios Stavrou (EURECOM, Sophia Antipolis, France)

Rate–Distortion–Perception Theory: Redefining the Fundamental Limits of Information Representation

Thursday, October 15, 14:00-14:45

Abstract:

Classical rate–distortion theory asks how many bits are needed to represent a source at a given level of reconstruction error. Yet a reconstruction with low numerical error can still look unnatural, while one with higher error may better preserve the source’s perceptually important features. Rate–distortion–perception (RDP) theory adds a third dimension to this fundamental tradeoff: the statistical similarity between the source and its reconstruction. In this talk, I will introduce the rate–distortion–perception function and explain the coding principles behind it, including how shared randomness affects achievable rates. I will then discuss methods for characterizing and computing RDP tradeoffs for discrete and continuous sources under perceptual constraints based on statistical divergences and Wasserstein distance. Gaussian sources and the perfect-realism setting will provide concrete examples of the theory. I will conclude with open questions connecting RDP theory to neural compression, robust source coding, and communication for networked control.

Nikolaos Pappas (LIU)

Goal-oriented Semantic Communication

Thursday, October 15, 14:45-15:30

Abstract:

Abstract of the talk: This talk presents recent advances in goal-oriented and semantics-aware communications, an emerging paradigm that shifts the focus of wireless networks from reliably delivering bits to delivering information that is most useful for accomplishing a task. As communication systems increasingly support autonomous agents, digital twins, and real-time control applications, classical performance metrics such as throughput, latency, and reliability become necessary but insufficient to capture task effectiveness. The talk is organized into two parts. The first introduces the motivation for semantics-aware and goal-oriented communications and reviews representative frameworks, including the Age of Information (AoI) and Value of Information (VoI). Building on these foundations, we discuss how communication design evolves when the objective is accurate remote tracking, timely decision making, and actuation. The second part presents recent research results demonstrating how semantic and task-aware metrics can guide sampling, scheduling, transmission, and resource allocation decisions to significantly improve system performance. The talk concludes by outlining open research challenges and future directions toward intelligent wireless networks.

Emanuele Natale (CNRS)

Opinion Dynamics: From a Zoo of Results to an Organic Theory

Friday, October 16, 09:30-10:15

Abstract:

Opinion dynamics are extremely simple distributed algorithms: every agent repeatedly updates its state by looking at a few random neighbors, as in the voter, majority, undecided-state and averaging dynamics. They model how populations of very simple agents, from animal groups to chemical reactions and computer networks, coordinate without any central control, and they raise precise questions about what such systems can compute, how fast, and how robustly. I will give an overview of the field through my own path in it, from plurality consensus to community detection by averaging and the metastability of label propagation. I will then argue that, so far, the area has produced a zoo of results, each with its own bespoke proof, rather than a theory. This is starting to change. Several groups have recently closed long-standing gaps, and frontier reasoning models are beginning to prove lemmas that resisted us for years. When proofs become cheap, the bottleneck moves to trusting, organizing and reusing them. I will conclude by presenting Leanamycs, an open library of machine-checked (Lean) formalizations of opinion dynamics, and invite the community to build it together.

Christoforos Hadjicostis (University of Cyprus)

Resilient Distributed Average Consensus via Local Invariant Conditions

Friday, October 16, 10:15-11:00

Abstract:

Recent developments in digital systems and networking technologies have led to the emergence of complex systems that are managed/controlled over cyber infrastructures, such as wireless and wired broadband networks. The emergence of this type of networked control systems, from smart grids and transportation networks to embedded electronic devices and robotic networks, has sparked huge interest in distributed control and coordination problems. In this talk, we present recent progress in this area by focusing on the resiliency aspects of an operation that is key for several such tasks: the distributed computation of the average (more generally, a weighted linear combination) of various parameters held at different agents of a multi-agent system. We discuss challenges pertaining to adversarial network conditions, such as packet drops on the communication links and/or the presence of faulty/malicious agents. More specifically, we identify a key local invariant that can be exploited to perform any-time error detection/correction as well as to collectively remove the effects of previous communication exchanges with agents that are subsequently determined to be faulty/malicious.