This is a surprisingly complicated question and somewhat depends on context. In general, though, "you didn't feel" is more of a one-time thing (in the past), along the lines of "You didn't feel happy when we threw you a surprise party," while "you hadn't felt" is more of a state of being (also in the past), like "You hadn't felt scared for a long time."