"Just let it go" is a direct command. I think this is the one you want. "Just to let it go" has the infinitive form: to (verb). This sounds incomplete, because no-one is actually acting on the verb. I'd only use it in a longer sentence: "She caught the fish, just(only) to let it go".

the first one

Example: Stop arguing, just let it go!