This is a really good question! "Swear to it,"usually means something like making a promise. ex: I swear to take the trash out. "Swear by it," usually means that you are 100% positive. ex: I swear by my life that I won't be late this time. In this case, you can also say :I swear on my life..." It has the same meaning. "I swear by this book! It will teach you everything!" I think she left early but I couldn't swear on it. I think this is the most natural way. I hope you that helped you.

I think "swear to it" is correct, if I could pick only one. But I guess "swear by it" and "swear on it" are also okay. I imagine it is a regional or other difference in usage from person to person in these cases, but I couldn't swear to it. ;)