 In the following years he became very close to Prince Ferdinand, the heir to the Spanish throne.This led him into conflict with the weak King Charles IV and his ambitious chief minister Godoy.Infantado was banished from Madrid in 1805.