Try "actually."

'In fact' isn't especially formal : we use it all the time in normal speech. But - as Brad C suggested - the word 'actually' is very good alternative.

This is where I would suggest you consult your local thesaurus... But try the following: Truth be told indeed truly really literally as a matter of fact in reality or any of these that I got from a thesaurus (context dependent but words that might help you)... adv. without a doubt absolutely actually admittedly as a matter of fact assuredly authentically beyond doubt categorically certainly de facto easily for real genuinely honestly in actuality in effect in fact in point of fact in reality indeed indubitably legitimately literally no ifs ands or buts nothing else but of course positively precisely surely truly undoubtedly unmistakably unquestionably verily well Nick

"Too formal" is a popular mythology, associated with "dumbing down". There is no such thing as a "too formal". You can substitute a term like "objectively" for "in fact".