I would consider two things here:
- Visual connection to action
- Common standard implementation
To the first point - visual connection: If you see an arrow that points up, you expect something to happen in that direction. You will automatically look up, not down. So every action that goes to a different direction will feel alien, detached. So this argument says: If arrow is pointing down, things should go down.
The second point - standard implementation: If you take a look at mobile standards, you will find, that arrows that point left and right control a back-and-forth progress: The "back" button has a left pointing arrow and the right will point to the right. The reason for this is most likely PAGES in a book, where the next page is "after the current one", which requires an action on the right.
The UP arrow usually represents a "return to top", or a collapse functionality, while the DOWN arrow - even by default html - implies actions that OPEN something that is hidden. What I try to say is: Making it differently will oppose common standards and rather confuse.
Thus, I would always let the arrow point in the direction that the action will happen: If it drops down, point to the bottom.