To quote John Steinbeck...
“When I was a child growing up in Salinas we called San Francisco “the City”. Of course it was the only city we knew, but I still think of it as the City, and so does everyone else who has ever associated with it. A strange and exclusive work is “city”. Besides San Francisco, only small sections of London and Rome stay in the mind as the City. New Yorkers say they are going to town. Paris has no title but Paris. Mexico City is the Capital".
Not on my block...we say we're going into "The City", meaning Manhattan (even though we are in the actual city limits)...never heard anyone use the term, "going to town".
Now if we are already in Manhattan, we might say we are going "uptown" or "downtown", instead of North or South....