I'm currently working in a project where we're building an application with a map as an important component. In the map there's a layer control with map layers grouped in to different categories. As the user expand one layer group she will see one to many layers. Now I've got a problem with more layers than there is room for in the control. The result right now is that the layers at the bottom is unaccessible but I need to find a solution. I would like to hear if there are any other intressant options beside from a traditional scrollbar and if some of you have seen some examples of this, especially in a map client (the layer control).
My first reaction is to caution you about changing the default scrolling UI. People know how a standard scrollbar works and they've developed habits for handling them. For example, I tend to use the mouse scrollwheel or the two-finger swipe on my trackpad to manipulate scrolling areas. But non-standard scrollbars don't always support those behaviors, making me click around on the thing to figure out how it works.
This reminds me of Photoshop's Layers window, which works just fine with grouped layers, all kinds of supporting buttons all over the place (plus on layers themselves), and a standard scroll bar on the right. I say scroll bar is perfectly fine, even in a small, constricted control like this.
You could also probably do an accordion, but that would not necessarily perform equally well in all resolutions/screen sizes.
You should check whether you can display less layers simultaneously through different kinds of grouping. Layer trees are common in GIS apps. Folders / accordions are also good options. There's also the possibility of a tabbed list with 2-3 groups of layers, but this only works in very specialized cases, when the layers are of very different types.
I agree with all the previous answers: a good options to the classic tree is accordion. Better grouping can help improving the list.
Decide how you provide feedback to the user that a group/theme is selected and how many layers within the group/theme are selected. Think about a well-balanced selection of layers that are turned on by default.