I would vote Westbound for many reasons...
First - You cannot really count on an Empire Builder to Cardinal connection. I have tried that before and failed. We arrived 8 hours late on the Eastbound Empire Builder. Got moved to the Capital Limited, on the next day with a family Bedroom. Missed the Cardinal by 2 hours. They did provide us with a hotel room. But the poor coach passengers, they were only given cash to purchase bus tickets from Greyhound (Ouch). No hotels for them...
Second... Like you say, going westbound on the train is just more fun. Once you fly out west, - at that point, you really are just coming back home, which to me does not feel the same - as it does when you depart your home area on a grand train journey.... And on the Cardinal, if going eastbound, you actually hit the very best scenery in Virginia, and it's often, just before I exit the train in Culpepper, VA, so like you said - go westbound, and you start will then with a bang in VA and then follow that with "Wild and Wonderful West Virginia" - it makes for a good "one-two punch".
Third - The Cardinal has the best westbound schedule of any east coast train - not even close... You depart Alexandria VA at like 11:30 AM. The Capital Limited does not leave until 4:05 PM. That is like 4 1/2 more hours of daylight on the train. And the best part for me is from Culpepper to White Sulfur Springs, WV (about 12:30 PM to about 5:15 PM). The New River gorge is great but very much the same old thing - you follow the New River through the gorge (nice), it is great, but I prefer the open plains of VA, the rugged crossing of the Allegheny Mountains at both Allegheny VA and Afton, VA (just before Blue Ridge Tunnel), and the horse country of Culpepper and Charlottesville, VA, with distant Blue Ridge Mountains looming in the background. The very range you will later cross - after Charlottesville (westbound).
Fourth - Westbound on the Cardinal you get to enjoy the first day on the train while you are fresh - not at the end of a trip.