I am currently developing a new kind of UI for android device and I wanted to know how do I make prototype of it ? since none of the existing UI design tool has support for it. I wanted to include different gestures and movements and just about everything different.
As @Tony mentioned, pencil and paper is always the best place to start. It’s a great way to sketch ideas, edit and refine quickly and also generate feedback by literally walking around to other designers or stakeholders to get their feedback.
In my experience, a program that also lets you adjust the ‘sketchiness’ of the look and feel is always a big plus. This really helps when showing your user interface designs or interaction patterns to clients or stakeholders who would usually nitpick colours, type and image choice. My tip is to keep your prototype as ‘low-fidelity’ as possible until you move on to the visual design phase. I may be telling you something you already know and there are many applications that will let you adequately prototype your bespoke gestures and interactions, but the most important factor here is getting the workflow and design process right. Define, design, refine.
Honestly, pencil and paper. This is fast and cheap as well. Most solutions that will allow you to build interactive wire frames will have pre-built widgets. Thus, they won't fit your bespoke design. Pencil and paper is your best solution then moving onto HTML would be the next easiest. From there you can validate the concept and decide if you want to build a native app.