The second one (I insisted that he apologized) is completely wrong... because after " insisted that" we must use the base form of the verb. Please see the link below for more information.
https://englishhelponline.me/2011/06/28/grammatical-word-insist/