It is well-known among practitioners of machine learning that algorithms often find unexpected ways to satisfy an objective you supply to them. For reinforcement learning algorithms, where agents are trained to perform complicated tasks through rewards and punishments, specifying the correct rewards is in particular surprisingly challenging.

I first review many examples of (often entertaining) machine creativity resulting from unanticipated flaws in reward functions and simulators for reinforcement learning. I will then introduce the field of AI safety, which aims to alleviate such unexpected machine behavior, by posing and solving the technical challenges that cause learning algorithms’ behavior to deviate from our intentions for them.