The cars were also a lot harder to drive in the 1980's than they are today. Schumacher stated this not so long ago and admitted he hasn't struggled with the physical side of things because the cars handle much better and its less strain on the body.
I think that´s entirely different topics. A car harder to drive is not the same than a car that drains the driver´s energy.
2004/2005 cars were really, really easy compared to the 80´s but required much, much more fitness to stand a race distance. You got that quote wrong, Schumacher said in 2010 was that "this F1 is not that physically demanding compared to when he left". Despite 2010 cars being even easier to handle than mid 2000´s ones, they had much slower race pace mainly due to new fuel rules. The most physical punishment comes mostly from the passive effort of resisting the G forces, not that much from the active efforts behind the wheel. If you or me got a chance in a F1 our neck would give up way earlier than we started to feel any sort of unconfortable feeling from the effort of the driving itself.