Both the words are used to describe negative intentions. Vicious is used more for tricks used against some one with the intention of harming physically or substantially whereas malicious is the word used when tricks are played to destroy someone's reputation, name like we say malicious propaganda, malicious gossip etc

What separates the two words is the intent to do harm. Vicious can mean aggressive. Malicious would then mean aggressive with an intent to harm. For example, a pitbull dog can be vicious. It can cause considerable damage to a person but it doesn't know any better and is just acting out of instinct. A malicious act would be say I was playing football, I tackle the opponent but I also try to break his wrist while we are falling to the ground so that he can't play anymore.