I'm having trouble trying to decide if it's better to click on a video and open it on a modal or in a new page? We want the user to focus on the video and see more videos, but do not leave the page where he is.
You should avoid putting content like this within a modal. A modal should be reserved for communicating critical information to a user that requires the user to take action. Modals are completely overused across the web today and it's easy to get confused with their intended/proper use.
With that said, you always have to take into account your business objectives with any interaction you introduce. It sounds like you want users to be able to preview the video before taking them out of the main page. If this is true, have you considered a popover (not a pop-up), a modeless panel, or a short looping preview on hover (I've seen this work on mobile as well)?
Viewing a video on page would be better than viewing it in a modal. If you don't want the users to move to other page, you can use a hybrid approach. You can use
inline modals - similar to google image search. When we click on an image, the enlarged view opens beneath and the view can be closed when not needed. The same approach can be used for videos.
I've often been inconvenienced by videos displaying in a modal dialog, for two related reasons:
It's too easy to lose my place, because dialogs are often (as implemented and as expected) easily dismissable by clicking outside them. But a video I chose to watch should keep playing until I tell it to stop and not lose my place with a misclick on a big target.
It unnecessarily removes other navigation/reading. For example, if your video has a slow part, or I'm on a slow network and waiting for it to buffer, I might want to read the associated article, or open some links in a new tab — or maybe the video mentioned something I want to read about or follow a link to before continuing.
Either displaying the video inline on the page, or on a new page, doesn't have these problems: if it's inline, I can scroll around and get to all the other page content while the video plays or is paused; if it's a link to a new page I can open a new tab, whereas dialogs as implemented usually refuse to open their content (or their page with the dialog displayed) in a new tab.
As you may notice, many of these issues can be avoided by careful programming, but displaying the video as "regular content on a page" is more robust with less effort.