There isn't really any difference, to me. Ghost is more common to hear. Ghost sounds more colloquial while phantom sounds more literary or fancy. Phantom is also sometimes used as an adjective: "a phantom arm" or "a phantom sensation".