I think there have to be more nuanced answers than x seconds. I have just written an intranet application where I have put a limit of 5 secs, and a goal of 1 sec. In this specific environment, the limits are more variable, as long as the busiest pages are quick - closer to the 1 sec. And there are more important demands than just page load times.
For an e-commerce type site, when I was producing them, we aimed at 1 second for the home page, 1-2 seconds for the ( cached ) catalogue pages, and only exceeded this for the basket process, which was often not timed, but would be up to 5-10 seconds. All of this based on a company network and internet connection.
The problem with arguing for "as fast as possible" is that you need to balance this against other factors - the cost of development, the timeliness of the data. Producing a page in .1 seconds that has out of date data is worse than producing a page in 2 seconds that is up to date usually. And is it worth the cost of improving from 1 second to 0.95 seconds?
And, as others have pointed out, you really need to compare sites, not produce absolute values, or produce values specifying the infrastructure. 1 second on a high-end broadband connection is not the same as 1 second on dial up.