Andrew Chen calls the “Local Maximum” a point in which you’ve hit the limit of the current design…it is as effective as its ever going to be in its current incarnation. How do you know, as a UX designer, that you've optimized a design as much as possible?
I think Andrew is aligning with the mathematical idea of finding a maximum value of a function. In UX world, the function would be some combination of factors such as 'speed', 'general layout' down to 'colour of button X'. The output of such function is the UX measure. Suppose such measure depends only on one continuous factor, so we can visualise the idea:
Now, there are a variety of mathematical algorithms for finding the maximum of such a function. If we could evaluate the function at every point, we would. But, to make the problem computable, we have to use less-than-optimal algorithms. They typically involve choosing a starting point and gradually 'working towards' the maximum. The downside to this approach is that the algorithm can't see the 'bigger picture'. So if it works towards the first maximum at ~(0.5, 1.5), it may think it has solved the problem.
In UX world, you -- the UX designer -- are the optimisation algorithm. It is your job to find the maximum. Just like the mathematical algorithm, you can't see the 'bigger picture'. So if you start at x=0, your UX measure is 0. Making tiny changes to this some factor, your UX measure increases until you get to (0.5, 1.5). Then, it starts decreasing. Naively, you think you've optimised the problem. In reality, you've just found the local maximum, like the mathematical algorithm.
That's how you know you've reached a local maximum: a tiny change in any direction doesn't produce a noticeable improvement in your UX measure. At this point, any change you make does not result in a statistically significant improvement.
The solution to this problem? The same as the mathematical algorithm: you choose a new starting point and run the optimisation again. In UX world, this means testing against bigger changes, rather than incy-wincy ones.