you can just use "It's time TO go" or "it's time for us TO go" /как в русском, "Пора идти" идти-неопределенная форма глагола, как и "TO go"/. But the test doesn't give you that option. So, the only correct variant left is "it's time we WENT". Yes, it's a past tense verb. It's used to emphasize that it's urgent. Просто нужно запомнить, что помимо первых двух форм, в "it's time..." можно использовать глагол во 2-ой форме (past simple).