Workshop on Optimization for Learning and Control

TBD

In this work, we study optimization methods that leverage the linear minimization oracle (LMO) over a norm-ball. We propose a new stochastic family of algorithms that uses the LMO to adapt to the geometry of the problem and, perhaps surprisingly, show that they can be applied to unconstrained problems. The resulting update rule unifies several existing optimization methods under a single framework. Furthermore, we propose an explicit choice of norm for deep architectures, which, as a side benefit, leads to the transferability of hyperparameters across model sizes. Experimentally, we demonstrate significant speedups on nanoGPT training without any reliance on Adam. The proposed method is memory-efficient, requiring only one set of model weights and one set of gradients, which can be stored in half-precision.

This talk presents recent progress in the field of optimal control of nonsmooth dynamic systems, motivated by applications in robotics. The considered systems are characterized by a special type of ordinary differential equations (differential inclusions) that exhibit not only state dependent switches but even state dependent state jumps due to contacts and friction. Crucially, the nonsmoothness can be encoded in the form of complementarity systems that result from the solution of convex optimization problems that depend on the system state. We discuss several techniques - with either fixed or variable time-steps - to transcribe the continuous time problem into a finite dimensional mathematical program with complementarity constraints (MPCC). Finally, we present tailored numerical methods to efficiently solve the resulting nonlinear MPCC problems that include problem functions based on lower level convex solvers, and application results in simulations and real-world experiments with an assembly robot at Siemens research in Munich.
This talk presents joint work with Armin Nurkanovic, Anton Pozharskiy, Christian Dietz and Sebastian Albrecht.

Download slides (PDF)

Optimization algorithms power applications from embedded control systems to large-scale machine learning. However, the rapid diversification of hardware requires algorithm designs that move beyond abstract complexity measures to an explicit hardware-awareness. This talk presents our efforts toward hardware-aware optimization methods tailored specifically to the architectural constraints and opportunities inherent in embedded systems, GPU accelerators, and decentralized compute clusters. For compute accelerators, we develop a GPU-optimized Douglas-Rachford splitting algorithm for optimal transport problems. By carefully reorganizing the computational workflow to align with GPU warp execution patterns and optimize memory coalescence, our method attains order-of-magnitude speedups compared to traditional implementations, all while maintaining rigorous convergence guarantees. For distributed environments, we address the fundamental challenge of making decentralized training outperform AllReduce-based methods in practical high-performance computing settings. Our approach introduces a novel communication-computation interleaving strategy combined with a specialized Adam optimizer that mitigates the effects of small local batch sizes. By carefully analyzing the interplay between topology design, communication patterns, and optimizer behavior, we demonstrate both reduced wall-clock training times and improved generalization performance on DNN training. Across these computing paradigms, we show how optimization algorithms can preserve theoretical rigor while exploiting hardware-specific characteristics to achieve significant performance gains.

Download slides (PDF)

We consider (nonsmooth) nonconvex optimization problems with nonlinear equality constraints. We assume that (some part of) the objective function and the functional constraints are locally smooth. For solving this problem, we propose linearized augmented Lagrangian methods, i.e., we linearize the objective function and the functional constraints in a Gauss-Newton fashion at the current iterate within the augmented Lagrangian function and add a quadratic regularization, yielding a (convex) subproblem that is easy to solve, and whose solution is the next primal iterate. The update of the dual multipliers is also based on the linearization of functional constraints. Under some specific dynamic regularization parameter choices, we prove boundedness and global asymptotic convergence of the iterates to a first-order solution of the problem. We also derive convergence guarantees for the iterates to an ϵ-first-order solution.

We discuss a new way of development of asymmetric Interior-Point Methods for solving primal-dual problems of Conic Optimization. It is very efficient for problems, where the dual formulation is simpler than the primal one. The problems of this type arise often in Semidefinite Optimization (SDO), for which we propose a new primal-dual method with very attractive computational cost. In this approach, we do not need sophisticated Linear Algebra, restricting ourselves by standard Cholesky factorization. However, our complexity bounds correspond to the best-known polynomial-time results. Moreover, for symmetric cones the bounds automatically depend on the minimal barrier parameter between the primal and the dual feasible sets. We show by SDO-examples that the corresponding gain can be very big. We discuss some classes of SDO-problems, where the complexity bounds are proportional to the square root of the number of linear equality constraints and the computational cost of one iteration is as in Linear Optimization. Our theoretical developments are supported by encouraging numerical testing.

Optimization is a fundamental tool to solve several control and learning tasks, but optimization algorithms are often considered as building blocks with their convergence guarantees. In this talk I will present a different perspective in which optimization algorithms are considered as discrete-time dynamical systems that can be combined or integrated as suitable subsystems into a closed-loop scheme. This opens up the possibility to use system theory tools to analyze their (convergence) properties and also gain key insights for the design of novel and possibly advanced schemes. This is particularly interesting for complex scenarios involving large-scale and distributed systems or online schemes involving simultaneously optimization, learning and control. In particular, timescale separation will be proposed as a tool to frame accelerated and model-free optimization algorithms as well as to design schemes for distributed computing. Finally, scenarios combining optimization with learning and control schemes will be shown.

We analyze nonlinearly preconditioned gradient methods for solving smooth minimiza- tion problems. We introduce a generalized smoothness property, based on the notion of abstract convexity, that is broader than Lipschitz smoothness and provide sufficient first- and second-order conditions. Notably, our framework encapsulates algorithms associated with the clipping gradient method and brings out novel insights for the class of (L0,L1)- smooth functions that has received widespread interest recently, thus allowing us to go beyond already established methods. We investigate the convergence of the proposed method in both the convex and nonconvex setting.