It would depend on the time period you’re referring to. If we all lost our family quite recently, you would use “we all have” (present perfect tense) If we all lost our family a long time ago in the past, and it is finished and over with, you might say “we all did” (past simple tense). Here I personally think “we all have” sounds better because it is referring to something relevant now, not a finished action in past.