Efficient Continual Deep Learning

Dr. Christopher Kanan

University of Rochester

10/27/2023

11AM – 12PM CST

Location: SAN PEDRO 1 YOTTA ROOM 430

506 Dolorosa St, San Antonio, TX 78204

Zoom: https://utsa.zoom.us/j/91049981814

Continual learning methods enable progressive knowledge accumulation in deep neural networks, and it has the potential to make training much more efficient and has many exciting applications such as on-device learning. Whereas conventional deep learning can suffer from catastrophic forgetting of previously acquired abilities when updating a network, over the past five years, continual learning has largely overcome this limitation. Despite this achievement, I argue the vast majority of continual learning algorithms have little practical value. In this talk, I show that methods are highly inefficient in terms of compute, memory, and storage. Some methods even require more computation than re-training from scratch as a dataset grows! For continual learning to have real-world applicability, the research community cannot ignore the resources used by these algorithms. Subsequently, I demonstrate that three new neuroscience-inspired algorithms created by my lab can vastly improve computational efficiency on large-scale vision and text understanding datasets. Our work has the potential to make continual learning more practical and widely adopted as well as advancing the goals of GreenAI.