Yes, there are differences between the two. First of all "nul" is either a noun or an adjective and means something like "totally rubbish". For example, if you can't play football, you could say "Je suis nul au foot", which would mean "I am totally rubbish playing soccer". You may know these books "Cooking for dummies", "French for dummies" and so on, well, in French, they are called "pour les nuls" (here used as a noun), so for people who are rubbish at doing something. Mal is most of the time an adverb (in so very specific cases a noun), so it used with a verb. For example : Je joue mal au foot (I play football badly or I am bad at playing football).

Nul could signify "zero" or that worth nothing. Mal could signify "badly"