It means you have decided not to deal with a person because the person is an idiot, not because that person scares you. You avoid feces because it's dirty not because it scares you.

It's two separate sentences almost, so you need the context to fully translate it. Someone was most likely speaking when they said this. 더러워서 피하지 --> You avoid something because it's gross/dirty/disgusting. 무서워서 피하겠어요 ---> I'll avoid it because it's scary.