Generative Adversarial Imitation Learning

The paper considers learning a policy from example expert behavior without interacting with the expert or accessing any reinforcement signal. One standard approach is indirect: recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function using reinforcement learning, but this is indirect and can be slow. The authors propose a new general framework for directly extracting a policy from data, as if the policy were obtained by running reinforcement learning after inverse reinforcement learning, thereby collapsing the two-stage pipeline.

They show that a certain instantiation of their framework draws an analogy between imitation learning and generative adversarial networks. From this connection they derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments. Known as Generative Adversarial Imitation Learning (GAIL), the method became a foundational technique for scalable imitation learning.