According to strict grammar rules, the correct answer for the question ' What did you do yesterday?' is ' I studied English'. (Past Simple). But the correct answer for the question 'What did you do yesterday at 5 p.m. (or from 4 till 7)?' is 'I was studying English'. (Past Continuous)

The people that already gave answers are correct. If the question is "What did you do yesterday?" the answer should be "I studied English." However, many native English speakers don't strictly follow this rule, or even know it, so you'll hear both "I was studying English" or 'I studied English." I know that I would use either of those interchangeably and not even think about whether or not it's correct. You'll often hear the question worded both ways as well, "What did you do yesterday?" or "What were you doing yesterday?" Technically there is a difference, but a lot of people will just choose one of them without thinking about the specifics.

I wouldn't use the past continuous. It sounds strange to me. When a question is asked with the past tense simple, the answer should be in the past tense simple as well. What did you do yesterday (definite time in the past)? I studied English?

Both are correct.