I'd recommend reading Communicating Design, which tells it about the specifics of how a UI is usually documented.
In general, there are layers of documentation:
- Who gets in contact with the system (context diagrams and/or personas)
- What the system is used for by them (use case listings, perhaps user stories)
- What is the flow of usage for each use case (flow diagrams)
- How the screens look like in each steps (wireframes)
- What are the visual and interaction patterns employed (pattern library)
- What is the actual visual language to be used (widget library, actual layouts)
I'd say, flow diagrams, wireframes and widgets are the most important ones perhaps.
These are built on top of each other: personas have goals, in order to reach these goals, they use the system for specific things (use cases), which in turn have a kind of flow or process (flow diagrams), the screen themselves have to look somehow (wireframes), there should be a kind of consistency between them (patterns), and these have to be implemented (widgets and layouts).
For a famous inside-organization Pattern library, look at the Yahoo Pattern Library
For wireframing, look at Balsamiq Mockups, Konigi Wireframes, Wireframe Showcase.