I would use it is really a hassle instead of saying its bothering me. You can say its bothering, irritating or annoying to express your feelings. You can say it is annoying when someone asks you to do something you don't want to do. But I would tell the person in a nicer manner than saying any of these. I would say Im a bit busy and cant do whatever you are being asked to do. But If you were talking about the matter to a close friend or family you can be more upfront and say that what you are asking me to do is bothering me because I dont want to do it. Basically they are interchangeable when expressing emotions, Someone can bother, annoy, irritate you all at the same time for the same reason. But irritate can also mean something that is causing discomfort, physically and emotionally. Annoying would be used to express how you feel about a certain situation. If someone keeps prank calling you it can be annoying that someone is bothering you. and bother can be used to say Sorry To bother you, or you can tell someone to stop bothering you because you are busy and need to complete your schoolwork. And that it is annoying that they keep bothering you and it is to the point that it is irritating.