There are are three possible ways of saying this: 1. With whom did you speak? 2. Whom did you speak with? 3. Who did you speak with? The third form 'Who did you speak with?' is by far the most usual way of asking this question nowadays. This sounds normal to all native speakers, and this is the one which you will hear most in everyday speech. The first 'With whom did you speak?' is considered the most grammatically correct, as it obeys two rules: the use of 'whom' for the object of the sentence, and the (now almost obsolete) avoidance of the preposition 'with' at the end of the sentence. However, you will rarely hear this form, other than in formal situations. It sounds rather awkward and outdated, especially in everyday conversation. The second version is theoretically possible, but is rather odd, because it's a mixture of formal and informal. I wouldn't use that, if I were you. I'd advise you to use the first construction if you want to sound natural. By the way, it's more common to use 'speak to' than 'speak with'. 'Speak with' suggests a more formal kind of conversation than 'speak to'. The commonest way of asking this question is in fact 'Who did you speak to?' I hope that helps.

To whom did you speak? ---Not good .> With whom did you speak to? > With whom were you speaking( to)? >Who ere you speaking with? To whom it may concern Whom did you speak with? = With whom did you speak?"

Both are correct, as "whom" is the accusative form of "who" (used with prepositions, or as the object of a verb) and both of those sentences are with prepositions. "whom" is very quickly falling out of use, though, so you wouldn't be as likely to hear it in regular speech (it's possibly to just use "who" instead).