it is natural it means the person is thinking about your question. most often it means they are thinking about how to lie to you. policeman: did you steal this man's wallet? he says you did? mr italki: no officer hmmm; i didn't hmm: Policeman: you don't seem so sure? mr italki: hmm what do you mean Policeman: you know perfectly well what I mean. Mr italki: hmm hmm no i don't. Policeman: are you going to answer honestly. Mr italki: yes sir i hmmmmmmmm m hymm. Mr italki: hmmmm (runs away) Policeman: stop your under arrest.

The way you wrote it is not correct. It sounds natural to say: 1) To answer your question, yes. 2) The answer to your question is yes.