Your question is right.Our present is the sum total of our karmas of the past.If there is some account left between ur grand father, grandmother and ur self you will surely meet again in one form or the other.Regarding rebirth and after life we have circumstantial evidence that why one child is born in a remote corner of Africa wher he may die of hunger and another in a state where he will be looked after.Also why some kids at the age of 3 or 4 are masters either in music,science,litterature,computers etc.Also many kids when they grow up say that actually they wre so and so in their past birth.
When we have no proof we have to believe the best answer.