You can definitely say "I wish you were coming back", especially if the other person is not coming back! In fact, I think that sounds slightly more natural than "I wish you would come back" but both are correct.

They have different meanings. If you say 'I wish you would come back', it implies that the person doesn't want or doesn't intend to come back. You are are wishing that the person would change their mind, or you are trying to persuade them to come back. This is the subjunctive of 'will come', referring to the person's 'will' (as in want they want). With 'I wish you were coming back', there is no such implication. This is the subjunctive form of the present continuous ( 'are coming back') , which refers to a fixed arrangement. Here, the reason that the person isn't coming back may be due to work, study or family obligations and may be outside their control.