Genetic algorithms have been used to successfully find solutions to complex optimization problems. The algorithms rely on the same basic principles of "survival of the fittest" to improve the solution over time (many hundreds or thousands of iterations). On a basic level, many candidate solutions are generated, then scored, and the worst scoring ones are discarded while the better scoring ones are combined in some way (called cross-over, to simulate "breeding") to form new solutions. The better solutions get to survive to the next iteration (generation), and everything is performed again and again.
The challenge here, is to frame interface design as an optimization problem. Firefox performed an experiment recently with exactly this in mind. Some highlights:
Creating the perfect Firefox interface is a challenging problem. Every
user has their own set of common tasks. The challenge is in defining a
common “default” interface that maximizes task success for the most
people, on the most tasks. Using designer insight alone, it’s not
always easy to know the impact of each design element!
Within 700 generations, good candidates started emerging, and by
generation 1100, convergence was achieved. We then chose the most
visually appealing one as our candidate.
There have been other experiments with generating visual art and music (among others). You should look up "interactive evolutionary computation" for some more papers and background material. The major challenge usually involves scoring the solution quality. For problems that can be framed in a mathematical way, this is quite easy. But for more creative pursuits this is much harder, and you have to encode some underlying set of rules or evaluation criteria (algorithmically determine good design from bad design). If your rules are too simplistic, you could end up with thousands of different solutions that all meet the criteria. If they are too strict, you could end up with no solutions, or all the generated solutions will look the same. That is what makes it so difficult to encode problems rooted in creativity/subjectivity/aesthetics. Interactive evolutionary algorithms usually rely on humans to assign a score ("fitness") to each candidate solution, but this is very labor intensive.
It is an interesting field that has been around for decades, even though it doesn't often get applied to interface design (for the aforementioned reasons).