One source of confusion could be that by dragging the scroll bar and dragging the page directly move the page in different directions. On touch devices you hardly ever use the scrollbar, so it doesn't matter on them. I'm not saying this is a major issue and that users wouldn't get used to it.
Another point against adapting universal dragging to desktop environments is the selection of text/elements, like already pointed out. On touch devices, selecting text needs some extra effort and affordances like drag handles appearing on long touches. I'd hate having to resort to such on desktop, too.
I really like the way Apple's MagicMouse kind of combines the actions of scrolling the document with a finger/mouse wheel and selections and other 'pointer operations'. MagicMouse lets the user to scroll the document by using fingers gestures on top of the mouse and move the pointer and make selections by moving the mouse itself.