Open Foundation Models: Reproducible Science of Strongly Scalable Transferable Learning

Open foundation models: reproducible science of strongly scalable transferable learning

Abstract

Obtaining models that generalize well and show transfer across various tasks and conditions following generalist pre-training is one of the most important recent breakthroughs in machine learning.

Numerious works provided evidence for advantages of generalist pre-training for transfer to various domains and tasks when comparing to directly training a domain or task special, narrow purpose model. Such generalist foundation models exhibit scaling laws, showing generalization improvement with increasing model, data and compute scales in the pre-training. Scaling laws can predict model function and properties such as generalization on unseen larger scales from experiments on smaller scales. To search for foundation models that scale better with compute and achieve stronger generalization and transfer, scaling laws can also be used to perform learning procedure comparison. The speaker shows how scaling law based comparison can accurately predict which learning procedure results in stronger foundation models at various scales.

Jenia Jitsev will discuss the importance of open foundation models that ensure full reproducibility of the entire research pipeline - data, training, evaluation - for scaling law studies and collaborative improvement of existing learning procedures. Further, properly measuring generalization is crucial part of establishing scaling laws, and he shows that this task is far from being solved. He highlights failures of standardized benchmarks to detect severe deficits in generalization still existing in current frontier foundation models and propose new measurement tools based on controlled variations of simple problems, aiming for evaluation that can detect generalization breakdowns and provide proper assessment of model generalization and transferability.

BIO: Jenia Jitsev is co-founder and scientific lead of LAION e.V, a non-profit research organization committed to research on open large-scale foundation models and datasets. He also leads Scalable Learning & Multi-Purpose AI (SLAMPAI) lab at Juelich Supercomputer Center and is a member of ELLIS