This doesn't really make sense in English. Do you mean 1) "I hate how most people leave when I am talking" or 2) "Most of the people I hate leave (the room) when I am talking" or 3) "I hate people who leave the room when I am talking"? -- In 1), most of the people in the room leave when you talk. In 2) the people who you already hate leave the room when you talk. In 3), you start to hate people when they leave the room when you talk.

私が嫌いな人たちは、私が話していると、たいてい離れていく。Many people who I hate leave me when I'm talking.