別 is an imperative. Essentially you are ordering people not to speak. 不 on the other hand is a declarative, sort of like a statement. Since Chinese is a pro-drop language, meaning that subjects aren't obligatory, 不說話 could mean 'someone does not speak' depending on the context.

不 = not, no, negation. 别 = short for 不要, so 不说话 = no talking 别说话 = don't talk.

[不说话] You (or he, she, they)say no word. [别说话] You (or he, she, they)not allowed others to say any words.(Be quiet)

“不说话” just like you can speak but you don't want to and “别说话” is you want to speak but not allowed

（某某人）不说话 Usually, “不说话” means somebody don't speak/talk. （你）别说话 “别说话” means let somebody do not speak/talk.