Yeah that is correct, if the specific directions of 'backwards' and 'forwards' were important, you would say them, using that to emphasize them. But if you are just pointing out that they are moving you would use 'back and forth'

They can both be used (if I'm not mistaken). However, "back and forth" would be my first choice, because it is a phrase used specifically for describing a motion. "The buildings were swaying" (swaying is a motion). "Backwards and forwards" is very literal and too wordy. They are two opposite directions! Unless you say "backwards and then forwards" it does not sound right.