"What were you saying" implies that either the speaker or the listener was interrupted by some event and the statement was therefore not finished. It is usually used to resume a conversation after the distraction is finished. "What did you say" refers to a completed statement.

It depends on how much of the other side of the conversation was missed. If someone spoke to me for a before I realized that he/she was speaking, and I wasn't listening, I'd say, "I'm sorry, what were you saying?" However, if I only missed the last sentence, I'm more likely to say, "I'm sorry, what did you say?" There's no measurable difference in meaning really, and in most cases you can use them interchangeably. For example, it's okay to say "What were you saying?" even if only one sentence was said. I wouldn't suggest using it in response to very short sentences phrases though.