you can just use "It's time TO go" or "it's time for us TO go" /как в русском, "Пора идти" идти-неопределенная форма глагола, как и "TO go"/. But the test doesn't give you that option. So, the only correct variant left is "it's time we WENT". Yes, it's a past tense verb. It's used to emphasize that it's urgent. Просто нужно запомнить, что помимо первых двух форм, в "it's time..." можно использовать глагол во 2-ой форме (past simple).

Well, 'd' sounds correct to me. Along with, "Yes, it's time we get going.", or "It's time we should go." To me, at least, none of the other three choices sound natural at all.

In spoken English, you would hear one of the three, probably the third, but the other two are also common: "It's time that we went." "It's time we went." "It's time for us to go."