I noticed that on Etsy, when you mouse over products after conducting a search, two buttons display in the upper right hand corner to either "heart" the product or add the product to a list. How effective would that kind of interaction be with an add to cart button?
For a retail site, I would consider "Add To Cart" to be a piece of core functionality. Implementation of core functionality should not magically appear and disappear. While it seems relatively clear in your description, I think this can be confusing to users. If they have not hovered over an item yet, there will not have been an "Add to Cart" button/link available to them at all.
I suggest to put the category name above the filters and orders button. In this way is more clear that all content below the category belongs to the selected category (proximity). In addition in this way you can put a label in the order dropdown to make it more clear.