If you have no luggage, that's honestly a little early, for me anyway. I'd personally aim for 7am, unless I was planning to get there early and find breakfast nearby.
I don't know the area downtown too well, but unless there are breakfast spots nearby, you may end up just waiting at the station...