What do you mean? asks for a complete explanation. The speaker did not understand any of what was said, or understood very little of it. How do you mean? asks in what way what was said is true. The speaker understood the main idea of what was said, but wants clarification of the sense in which it was intended or how it applies to specific situations.

I've also heard the "How do you mean"?" used as an informal way to say the other. That is, in the case when you really don't get what they mean. Maybe the distinction is more in the tone than anything.