The Stack Exchange communities menu is messy but likely not without a cause. You see, the actual example in the Stack Exchange sites for a better comparison would be the Tags search:
The reason is that in the Tags menu you already know what you want, so you'll just look for it and hopefully find it fast. For the communities menu you are meant to explore. You don't know what communities exist and as so you are provided a colorful, dynamic experience. I do think the communities menu is problematic, but not because is messy. Its because its hard to understand what the communities are about. If the name does not say everything you are bound to have to dig a bit (many times unnecessarily). Another problem is the almost absurd difference of box sizes (I think this bubble effect should be less exaggerated).
One point, however, is that when you give the user multiple unknown choices you usually provide some tools to make its life easier. Using the same example you have mentioned the communities menu has this top toolbar:
, where a classification appears. Other important feature is to change the messy list, for a very straightforward one:
But going back to your context. Personally I would start by profiling the expected user actions. When the User goes to your home page what will it want to do?
1) Search (Tags)
2) Explore (Communities)
3) Chose (Well known small list of options like your personal list of communities)
For 1: Let's say the user already known what it wants, and wants it fast. Every feature you implement must have as criteria the decrease of search time. As so autocomplete features are fundamental. Color coded tags depending on an intuitive classification (web languages, all-purpose, science, etc) is a strong possibility. Adding icons to your tags can also be positive (I'm a Python user and would easily recognize its logo everywhere).
For 2: The user is unsure of what it wants. It knows that is looking for some way of making plots in its webpage. How does it get a good list of possibilities (asking in stackoverflow is off-topic, although common ;p)? It would likely start by web, moving into visualization or plotting. StackExchange solves this by allowing any kind of sensible tag to exist. So the "Visualization" tag will lead you to questions about visualization. Crossing "Visualization" with "Python" would lead you to "Visualization" questions in the "Python" language (multi-tag search)
- there is an important point here that is the expectation that a user that is searching in this point is a registered one. As so it likely
has given its preferences in the registration process. This is an
important source of information. Consider if this is your case.
If this last alternative is your objective than I think your current homepage is close to a good solution.
- Can I search for "Python" and "Android Development" at the same time?
- Once I start writing will the list be updated automatically?
- Would it be a possibility to sort by most viewed (instead of alphabetically) when the search field is empty?
- Can the most famous titles have their respective icons?
I'm sure there are other things to take into consideration. I'll be sure to comment if any comes to mind.