Your instinct here is most likely correct. "You ask me to leave my family" is standard grammar. I have not seen that movie (World War Z). Did you hear this sentence in the movie's trailer or in the actual movie? At what time in the movie does Brad Pitt say this? And I know the movie is a work of fiction, and that it is about Zombies, and that it had several screenplays. So I have no idea why or if Brad Pitt says "You ask me leave my family." Maybe you heard him wrong, or if you were using English subtitles added by a non-native speaker, maybe the subtitles were wrong (a common occurrence). But, yes, in real-world English, "You ask me to leave my family" is standard grammar.

Based on the answer provided above by Mr.Alan, I came to know that this is actually a dialogue from "World War Z". Even I haven't watched the movie. But usually, you can take the movie dialogues with a pinch of salt, especially when the scene is emotional, melodramatic, in the midst of an action sequence etc because in such scenes, the main aim of the character is to convey the meaning and thus hardly gives any priority to grammar.