Of course the ideal time is as fast as possible all of the time. The first part, being as fast as possible, is important. But the second qualification, being fast all of the time, perhaps deserves more attention. A more interesting question is not that of minimising the average page load speed, but of minimising the variance in page load speed. Facebook has said publicly that their focus is not the average response time, but in minimising the variance:
They don't care about average response times, instead, they want to minimize variance. Every click must be responded to quickly. The quality of service for each request matters.
It's OK if a query is slow as long as it is always slow. [...] What is important is that the edge cases are [not bad].
Source: High Scalability.
This makes some sense. If your website loads in 0.1s 90% of the time, and takes 10s the other time 10%, then the average load time is about 1s. Now consider the experience of a website loading in 1s 99.9% of the time. I would argue that the second experience is better for the user since it is consistent and at no point is the user left waiting too long.