'Whom' is very formal and rarely used in spoken English. The following 4 sentences are all correct.
A. He’s a man who we’re proud of. (less formal)
B. He’s a man whom we’re proud of. (formal)
C. He’s a man that we’re proud of. (less formal)
D. He’s a man we’re proud of. (less formal, the relative pronoun 'That' is omitted in this sentence)