Daniel, Annoy is more intense of a word while bother is less intense. So if the clothing is just uncomfortable and you don't like it then it would bother you. If it is starting to make you angry then it would be annoying. That is just a very simple look at it. I use them as synonyms so usually it is just which word comes to mind at the time. James