For some reason English speakers tend to say "on the edge" for certain things like, "He is living on the edge" or "The boat is on the edge of disaster," or the "bank is on the edge of collapse," and "she is on the edge of her seat," but "at the edge" for other things like "at the edge of the sea" and the "at the edge of uncertainty." I can't figure out the rule if there is one and so I think that "on" works better in most cases. In the case of your cliff, either works. You could say "precipice" but that sounds a little high minded and used more figuratively than for a purpose actually being at or on the edge of an actual cliff.

At the edge