Millions of everyday users are interacting with technologies built with generative AI, such as voice assistants, search engines, and chatbots. While these AI-based systems are being increasingly integrated into modern life, they can also magnify risks, inequities, and dissatisfaction when providers deploy unreliable systems. A primary obstacle to having more reliable systems is the opacity of the underlying large language models— we lack a systematic understanding of how models work, where critical vulnerabilities may arise, why they are happening, and how models must be redesigned to address them. In this talk, I will first describe my work in investigating large language models to illuminate when models acquire knowledge and capabilities. Then, I will describe my work on building methods to enable data transparency for large language models, that allows practitioners to make sense of the information available to models. Finally, I will describe work on understanding why large language models produce incorrect knowledge, and implications for building the next generation of responsible AI systems. 

Hilbert’s dream of mechanising all of mathematics was dealt fatal blows by Gödel, Church, and Turing in the 1930s, almost a hundred years ago. Paradoxically, assisted and automated theorem proving have never been as popular as they are today! Motivated by algorithmic problems in discrete dynamics, non-linear arithmetic, and program analysis, we examine the decidability of various logical theories over the natural numbers, and discuss a range of open questions at the intersection of logic, automata theory, and number theory. No prior knowledge of any of these fields will be assumed.