This is a difficult question, because there is no precise answer. You would generally use "town" for places that have a population of roughly 50,000 or less but there is really no set rule. Some places with a smaller population like to call themselves "cities" and sometimes, in casual conversation, even large cities are referred to as "towns". Especially if you live in the countryside, you might say to your friend either "I'm going into town" or "I'm going into the city".