"Town" refers to all the town. "Downtown" refers only to the town's center. There is almost never any official boundary for "downtown". You start from the exact center of activity and go out from there.

When referring to a city, "Downtown" means the spot where the skyscrapers are located. Downtown is the opposite of the suburbs and can be considered the heart of a city. Downtown is where many people work, but also come to enjoy nightlife because of the atmosphere. Town just means the entire area of a populated location.