First, your original answer of “watched some dogs were playing” is wrong and was corrected correctly by the other teacher. You can’t use “were playing” because it gives you two verbs in one clause.
In your two examples, both constructions can be used because in the second, “watched some dog playing”, “playing” is used as an adjective modifying “dogs”. You could also write “watched some playing dogs”.
The “ing” form is the present participle, but in addition to being used in the progressive tenses, can be used as a noun or as an adjective.
The adjective usage is given above. “Playing is an important activity for children “ is an example of noun usage.