It kind of means "because I felt like it" or "for fun." I would say it's pretty informal. You could probably use it when talking to friends or in a casual group setting.

Generally speaking, it means "for the sake of it", "for fun". You wouldn't want to answer a serious question with it. For example, Boss: "Why did you leave early yesterday." You: "For the hell of it". <<< Do NOT say that!! I most often hear it when someone makes a suggestion. For example, "Just for the hell of it, let's try (insert potential solution to an engineering problem) and see if it works".