This is a very broad question, and would make a good topic in the italki discussion section. In general you have the context right, people talk more formally in business situations, particularly with a client, at church, in academic settings, and with older people. Informal speaking is standard among friends, many families, and some work environments. Older people feel students should use more formal language with teachers, but students are often very informal with teachers and professors now. It seems that formal speech is declining some in general. Sir or ma’am can be used as you know. Please can be added to polite requests to make them more formal or there are other ways to be even more formal with requests such as Would you be so kind as to... or simply Would you please... Your example "Sorry, sir, I'm kinda busy. My friends are waiting for me and I've gotta go." has a mix of formal and informal. Sir makes it more formal, but kinda is informal slang, the contraction makes it more informal, and gotta is rather informal slang. A more formal way to say that is: “Sorry, sir, I'm busy. My friends are waiting for me and I need to go." More formal: “Sorry, sir, I'm rather busy. My friends are waiting for me and I need to go." Not using the contraction I’m there would be even more formal, usually to the point of overdoing it. Both of those would still be reasonably formal without the sir, but I would probably say I’m sorry, I’m... In English formality is expressed more by word choice and proper grammar than by pronouns.

Is any difference between "You can come in" and "Come in"? Can I use both in formal conversation? Or I need to add sir/madam in the end of the sentences?

Yes your sentence is polite and the use of sir makes it formal. You is used both formally and informally with everyone

If I say, "Sorry, sir, I'm kinda busy. My friends are waiting for me and I've gotta go.", is it formal and polite?

In English you use placers such as sir, madam.