In the last century, as cars became more common, people who could afford it moved from downtown areas to the suburbs. Because poor people weren't able to move, a lot of inner city areas became poor and run down. This is why there is an association between "inner-city" and poverty. This isn't true in all cities, though. Some cities have thriving downtown areas. In the past decade, many cities have had downtown revitalization projects. Furthermore, as people have become more environmentally conscious, more wealthy people are choosing to live in downtown areas where they can use their cars less.

I would refer to the busiest part of a city as the "downtown" area.