It really depends. Number 2 is correct most of the time (except it would be "Saying lovely things..."). Customs normally deal with cultural mores and wide scale or societal actions, habits are more personal and denote an individual's actions.

Each has a range of meanings and they overlap. In the most common U.S. usage, a "habit" is something we do so often that it has become automatic and we don't think about it. The most common meaning of the word "custom" is "the traditional polite practices of a culture." "Be aware that the Spanish custom is to eat supper very late, at about 9 p.m." When we use "custom" in reference to personal or family routines, we use it mostly in explaining them to other people. "Our custom is to have our apple pie with cheese on it, but we have vanilla ice cream if you like it a la mode." By "telling lovely things," do mean "paying somebody a compliment," such as saying "I like the way you look in that shirt?" I would say, "Paying compliments to people is a very good habit."