In some cases they are interchangeable, but not all. 'Give it a try' implies that whatever it is may not be possible. This is an example of a phrase containing a 'delexicalised verb', like 'have a look', where the verb (give) carries very little meaning in itself, and the meaning of the sentence depends on a noun phrase ( a try) . And just as 'have a look' basically means 'look', 'give it a try' simply means 'try'. If you try something, you may not succeed. So, let's say that you've never been windsurfing before, your friend offers you his windsurfer and you say 'OK, I'll give it a try'. You might totally fail, but you'll at least 'give it a try'. 'Try it out' suggests a test of some kind. The 'it' might be a piece of new equipment, or a new method of doing something. 'Try it out' may mean that you'll use this thing or this new approach for a while to see if it works.

I think they can be used interchangeably. For instance, if you're trying a new recipe you could say, "I'm giving it a try" or "I'm going to try it out" Or if you're trying to encourage someone to do something. "Give it a try!" "Try it out!" The only difference I can see is for "try outs" That's when you compete to make a team. "She wants to try out for cheerleading." "He tried out for basketball."