I'm currently working on the IA/UX for a mobile application and as I'm mapping it out it doesn't seem like a traditional site map makes sense nor does a flowchart. Can anyone point me to a good set of symbols that make sense for mapping out all the modal windows
In addition to the technique described by Rahul, I've also had success in applying Hagan Rivers' approach of "Application Maps" in certain contexts. Rather than an exact one-to-one representation of each screen, her technique tries to consider how a user "perceives" the flow of an application with regards to the menu system. She also uses the concepts of "hubs" and "spokes", which can handle modal dialog flows very well.
For a more detailed description of this technique, you might want to consider getting a copy of the 2010 UIE Web App Masters Tour proceedings (as I did) where she presented this technique in detail.
What we do is use a thick sharpie marker to sketch out the various screens in full, then we print/cut them out and tape them up to a wall (often a pane of glass in our office) with arrows pointing from one to the other. The benefits are that now you have the right fidelity to encourage discussion (it's not too detailed, but it's detailed enough to facilitate a conversation) and it's visible in an area where the entire team can see and comment on it. I prefer this method over putting a lot of time into dragging around widgets in a wireframing tool.
The biggest benefit this method has is that complexity becomes visible very early on as you try to place the different screens in logical areas on the wall. If it's too complex, you'll soon notice that you can't figure out where to put them all and still have it make sense. That's an early warning sign that you need to simplify. And simplicity should be a priority when designing mobile user interfaces.
Edit: Just came across UX Sticky Notes. You could try using these in combination with my suggestions above.
Give Axure a go. It's got a good balance of symbols for flow diagrams and wireframing to get you started. It should give you enough freedom to display the modal windows how you want. e.g. creating a hybrid flow/wireframe.
In terms of the actual design, I see what you mean. It may be tricky to keep it tidy without using number references, depending on how much you have to say. Good luck.
Before wireframing, write the title of each step or piece of functionality ("sign up", "log in", "Widget List", "Account Info") on a post it with a sharpie. Then slap those post it's on a whiteboard and start drawing boxes around items to group them and draw arrows to flow from page to page. Snap a digital photo in every configuration to document the process.