It means you didn't give your family enough value when you should have. Another instance: if a man doesn't pay enough attention to his wife's needs and feelings, and she ends up divorcing him, then he might eventually regret it and think: "I took my wife for granted."