Spring 2026

  • David Shirokoff, New Jersey Institute of Technology (Joint RTG Seminar and ACM Seminar)

    Date and time: March 06 at 2:00 pm (Parker 328)

    Title: Convergence of Markov Chains for Stochastic Gradient Descent and the Failure of the Diffusion Approximation over Long Times

    Abstract: Stochastic gradient descent (SGD) is a popular algorithm for minimizing objective functions that arise in machine learning. While convergence theory is well established for SGD with vanishing step-sizes, less is known about the constant step-size setting.  A common approach is to approximate the SGD iterates by a diffusion approximation stochastic ODE.  However, this approximation is, in general, only valid for finite times.  Focusing on a class of nonconvex objective functions, we establish a “Doeblin-type decomposition,” for the SGD Markov chain in that the state space decomposes into a uniformly transient set and a disjoint union of absorbing sets. Each absorbing set contains a unique (ergodic) invariant measure, with the set of invariant measures being a global attractor of the Markov chain. The theory is highlighted with examples that show (1) the failure of the diffusion approximation to characterize the long-time dynamics of SGD; (2) the global minimum of an objective function may lie outside the support of the invariant measures (i.e., even if initialized at the global minimum, SGD iterates will leave); and (3) bifurcations may enable the SGD iterates to transition between local minima.  This is joint work with Philip Zaleski.

 
  • Shuoyang Wang, University of Louisville (Joint RTG Seminar and SDS Seminar)

    Date and time: March 04 at 2:00 pm (Parker 358)

    Title: Deep Learning for Complex Functional Data Analysis

    Abstract: Functional data are realizations of random functions observed over a continuum, such as signals and images. In many modern applications, including neuroscience and biomedical research, observations are more naturally represented as random functions rather than finite dimensional vectors. The intrinsic complexity of such data stems from high dimensional functional domains, cross cohort heterogeneity, and unknown data generating distributions, which together complicate principled modeling and performance guarantees. Although deep learning has shown strong empirical performance in biomedical studies, its methodological and theoretical foundations for complex functional data settings remain limited. In this talk, I will present two methodological contributions that develop principled deep learning frameworks for complex functional data. First, I will introduce a federated deep learning approach for functional data classification across multiple heterogeneous cohorts. The learner visits each cohort once, performs local updates, and transmits only compressed model weights, thereby preserving privacy and reducing communication and computational costs. To address cross cohort heterogeneity, we develop an adaptive sequential weight updating strategy that progressively corrects distributional shifts and improves performance on a target cohort. We establish minimax optimal excess risk bounds and characterize a sharp sampling threshold governing learnability under both densely and sparsely observed functional data. Second, I will present a deep learning based functional graphical modeling framework for learning conditional independence structures in multivariate functional data. Each node’s neighborhood is estimated via flexible functional regression with embedded feature selection, allowing a fully nonparametric specification, and the overall graph is recovered by aggregating the neighborhood estimates. The method avoids restrictive distributional assumptions and does not rely on a well-defined functional precision operator. We prove global model selection consistency and establish convergence rates that attain the classical nonparametric regression rate up to a logarithmic factor, with a fundamental sampling threshold determining the estimator’s convergence behavior. Empirical performance is demonstrated through simulations and real data applications, including analyses of ADNI dataset and the ADHD-200 Consortium.

 
  • Dr. Eric Cyr, Sandia National Laboratories (Joint RTG Seminar and ACM Seminar)

    Date and time: March 20 at 2:00 pm (Parker 328)

    Title: Towards multilevel training algorithms: Applying scientific computing perspectives to neural networks

    Abstract:

    The recent explosion of large language models (LLMs) in the commercial space has created unprecedented energy demands driven by the growth of data centers. One aspect of this is the need to train large-scale neural network models on massive amounts of data. Recent work has demonstrated that pre-training a LLM on the Frontier supercomputer would require two years with ideal parallelism [1]. Yet despite advances in optimizers, the training algorithms remain largely the same even as the neural networks scale to trillions of parameters and suffer from quadratic scaling in the number parameters. This motivates our aspirational hypothesis that multilevel (or hierarchical) methodologies can dramatically accelerate training algorithms.
     
    This talk proceeds in three parts. In the first part, we discuss an adaptive basis perspective that has proved fruitful in Scientific Machine Learning (SciML) [2]. With this perspective we develop efficient “operator-split” training algorithms, and new initialization strategies motivated by stability concerns. The second part of the talk considers the impact of using quasi-second-order methods to address the “grokking” phenomenon [3]. Through the lens of the spectral bias, we show how Levenberg–Marquardt reduces generalization gap in our experiments and allow training to proceed quickly through the lazy learning regime towards the rich one. The third part presents a view of Kolmogorov-Arnold networks (KANs) that reformulate the activation function as a spline that can be naturally adapted [4]. We explore how this approach can be related to a multi-channel ReLU network and present a multilevel training algorithm based on the relaxation properties of gradient descent using KANs.
     
    Holistically, the goal of this talk is to show how we applied techniques developed for scientific computing to understand and improve neural network training.
    1. Dash, et al., Optimizing distributed training on frontier for large language models. In ISC High Performance 2024 Research Paper Proceedings (39th Intl. Conference), 2024.
    2. Cyr, et al., Robust training and initialization of deep neural networks: An adaptive basis viewpoint, Mathematical and Scientific Machine Learning, 2020.
    3. Jiang, et al., On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regime, ICLR, 2026.
    4. Southworth, et al. Multilevel Training for Kolmogorov Arnold Networks, arXiv:2603.04827, 2026 (submitted to SISC)

Fall 2025

  • Wenjing Liao, Georgia Tech (Joint RTG Seminar and ACM Seminar)

    Date and time: Sep 05 at 2:00 pm (Parker 328)

    Title: Exploiting Low-Dimensional Data Structures and Understanding Neural Scaling Laws of Transformers

    Abstract: When training deep neural networks, a model’s generalization error is often observed to follow a power scaling law dependent on the model size and the data size. A prominent example is transformer-based large language models (LLMs), where networks with billions of parameters are trained on trillions of tokens. A theoretical interest in LLMs is to understand why transformer scaling laws emerge. In this talk, we exploit low-dimensional structures in language datasets by estimating its intrinsic dimension and establish statistical estimation and mathematical approximation theories for transformers to predict the scaling laws. This perspective shows that transformer scaling laws can be explained in a manner consistent with the underlying data geometry. We further validate our theory with empirical observations of LLMs and find strong agreement between the observed empirical scaling laws and our theoretical predictions. Finally, we turn to in-context learning, analyzing its scaling behavior by uncovering a connection between the attention mechanism in transformers and classical kernel methods in machine learning.

     
  •  Wuchen Li, University of South Carolina (Joint RTG Seminar and ACM Seminar)

    Date and time: Sep 26 at 2:00 pm (Parker 328)

    Title: Information Gamma Calculus: Convexity Analysis for Stochastic Differential Equations

    Abstract: We study the Lyapunov convergence analysis for degenerate and non-reversible stochastic differential equations (SDEs). We apply the Lyapunov method to the Fokker–Planck equation, in which the Lyapunov functional is chosen as a weighted relative Fisher information functional. We derive a structure condition and formulate the Lyapunov constant explicitly. Given the positive Lyapunov constant, we prove the exponential convergence result for the probability density function towards its invariant distribution in the L1 norm. Several examples are presented: underdamped Langevin dynamics with variable diffusion matrices, quantum SDEs in Lie groups (Heisenberg group, displacement group, and Martinet sub-Riemannian structure), three oscillator chain models with nearest-neighbor couplings, and underdamped mean field Langevin dynamics (weakly self-consistent Vlasov–Fokker–Planck equations). If time is allowable, some extensions will be discussed on the time-inhomogeneous SDEs.