MATRIX Spring Seminar Series – Dr. Roland Green

Training of Giant Neural Networks with Weight Streaming

Roland Green

Cerebras Systems

 https://utsa.zoom.us/j/92387759081

Friday, March 4, 2022

11 AM – 12 PM CST

State-of-the-art language models have grown in parameter count by three orders of magnitude over the last two years. This growth has presented challenges for training both in terms of storage and compute requirements. In this talk, we survey existing approaches used to scale training to clusters of compute units and explore the limitations of each in the face of giant models. One thing that all these approaches have in common is the storage of model parameters on the compute units, which we find to be a primary driver of complexity and communication overhead. We present a new paradigm for giant model training, called weight streaming, based on the disaggregation of model storage and compute, and describe an implementation of this paradigm using Cerebras wafer-scale systems. Our weight streaming architecture enables the training of models two orders of magnitude larger than the current state-of-the-art, with a simple scaling model. Combined with built-in support for weight sparsity, our solution can make training giant networks tractable for the first time.

LINK

MATRIX Spring Seminar Series – Nicholas Soures

TACOS: Task Agnostic Continual Learning in Spiking Neural Networks

Nicholas Soures

Rochester Institute of Technology

 https://utsa.zoom.us/j/92387759081

Friday, March 25, 2022

11 AM – 12 PM CST

Catastrophic interference, the loss of previously learned information when learning new information, remains a major challenge in machine learning. Since living organisms do not seem to suffer from this problem, researchers have taken inspiration from biology to improve memory retention in artificial intelligence systems. However, previous attempts to use bio-inspired mechanisms have typically resulted in systems that rely on task boundary information during training and/or explicit task identification during inference, information that is not available in real-world scenarios. Here, we show that neuro-inspired mechanisms such as synaptic consolidation and metaplasticity can mitigate catastrophic interference in a spiking neural network, using only synapse-local information, with no need for task awareness, and with a fixed memory size that does not need to be increased when training on new tasks. Our model, TACOS, combines neuromodulation with complex synaptic dynamics to enable new learning while protecting previous information. We evaluate TACOS on sequential image recognition tasks and demonstrate its effectiveness in reducing catastrophic interference. Our results show that TACOS outperforms existing regularization techniques in domain-incremental learning scenarios. We also report the results of an ablation study to elucidate the contribution of each neuro-inspired mechanism separately.

MATRIX Spring Seminar Series – Dr. John Basl

An Ethics Ecosystem for AI and Big Data

John Basl

Northeastern University

https://utsa.zoom.us/j/92387759081

Friday, April 1, 2022

11 AM – 12 PM CST

Realizing the promise of AI while mitigating against ethical pitfalls requires an ethics ecosystem, an ecosystem that includes robust, interdisciplinary research programs, curricula, and training programs, informed regulatory and oversight infrastructure, broad stakeholder engagement, shared conceptual and linguistic space for overcoming challenges, and AI and data ethicists that can translate across disciplinary divides. This talk motivates the need for such an ecosystem and highlights some work attempting to contribute to it.