Smallest screen size is a good place to start because of what you mentioned; it's easier to design from that state first and adapt upwards.
But, I think there's a more informed and human-centered approach to this. What do you know about how people will interact with your app? Specifically, what's the context and technology that they're using when they need your application?
Armed with that information, you will have a little more guidance on how the interface will create better outcomes based on context, and this is where "start small and scale up" can have some blind spots. The goals of someone interacting with your app on mobile could be vastly different than those of someone using it through a tablet or desktop, and simply scaling your interface might not be the best choice. More importantly, if it is the best choice, you have some information to validate that assumption.