Because while we can say, "It's rather/quite/fairly cold in here"... (here, they mean 'to a certain or significant extent or degree') .. when we use the 'rather too...' the meaning of 'rather' changes. it is then being used to soften the assertiveness/bluntness of the statement as now suggested by the addition of 'too':