I did a search on Google Scholar and found a paper in which the authors develop a user-defined gesture set to use on mobile device. Resulting gesture set is motion based (OP was probably looking for touch based gestures) but it is more intuitive to the test users as they have defined it themselves.
Modern smartphones contain sophisticated sensors to monitor three-dimensional movement of the device. These sensors permit devices to recognize motion gestures—deliberate movements of the device by end-users to invoke commands. However, little is known about best-practices in motion gesture design for the mobile computing paradigm. To address this issue, we present the results of a guessability study that elicits end-user motion gestures to invoke commands on a smartphone device.
To our knowledge no research has been published describing the classification of motion gestures. As well, little research been done on end-user elicitation of motion gestures. Research on motion gestures has been focused on interaction techniques using motion input and tools to design motion gestures.
Lank, Li, Ruiz: User-Defined Motion Gestures for Mobile Interaction
I didn't found any papers on how would user-defined set of touch gestures look like or what would be the most intuitive touch gestures.
Though, come to think of it, user-defined set of touch gestures would probably look a lot like something we already have (as best practices, on design guides) because of all the exposure to touch screens.
Not surprisingly, there is at least one book on how touch interaction could be more natural.
Brave NUI World: Designing Natural User Interfaces for Touch and Gesture by Daniel Wigdor and Dennis Wixon
The second realization is that these input devices, while not themselves creating a better user experience, could be enablers for the creation of a UI that is more natural to use, and could fundamentally change the way we interact with technology. We dub this the natural user interface.
For a bonus here is Direct Manipulation Interfaces by Hutchins, Hollan and Norman.