People usually say this when they're fed up with someone, or frustrated with their behavior etc. Here's an example. Mom: "Please clean your room. We'll be having guests over tonight." Son: "No, I don't feel like it." (petulantly) Mom: "You're impossible!" In this case, the mother is fed up with her son's behavior, since he is being disobedient. So she goes ahead and says that he is "impossible" or "impossible to deal with".