most commonly you will hear "don't overdo it". "It's essential not to overdo it when trying to please her"

Overdo takes a noun. You can change the verb into a noun by using the gerund form (-ing). You could say: "It's essential not to overdo trying to please her" But it sounds more natural to say it the way that John wrote it: "It's essential not to overdo it when trying to please her". That is how I would prefer to say it in English.