'As if he were under her spell' is correct. Using 'had been' places this as a completed action before 'followed' meaning that he is no longer under her spell. And this is not true.

I’d go with “were” here: He followed the scent, as if he were under a spell. The fact that the whole event happened in the past doesn’t mean you need “had been.” In this sentence, “as if he were under a spell” describes his condition at the same time that he was following the scent. In other words, he followed it in a way that made it seem like he was under a spell. “Had been under a spell” would normally push the spell further back in time: He followed the scent, as if he had been under a spell. That sounds more like he was under a spell before he followed the scent, or that you’re looking back at an earlier condition. So the difference is basically: were under a spell = the apparent spell and the following happen at the same time had been under a spell = the spell happened before the following Also, “were” here is the traditional subjunctive used after “as if” when describing something imagined or not necessarily true. So for the meaning you probably want, “He followed the scent, as if he were under a spell” is the more natural choice.