If you want to use "tremendous" in this sentence, the best ways to do it would be "there was a tremendous clapping of hands by the audience." or "the audience offered a tremendous clapping of hands.". If you want to use "tremendously" you could say "the audience tremendously clapped their hands.". This last sentence would be highly unusual to hear or read. As Zoe said, tremendously in your original sentence would better be replaced with loudly or even thunderously.

Tremendous means: Extremely large in amount, extent, or degree; enormous. So if you want to stress the way the audience clapped their hands, the use of the adverb "tremendously" could work just fine.

It is grammatically correct but it would sound better if you used loudly or expressively. ex. The audience clapped their hands loudly.