You have to chose between two User Experience options: speed vs. feature. As it happens I'm in the current situation as well at work. We have a landing page with a lot of features, making it extremely heavy to load, and in production the infrastructure could be better.
So what do we do? Use cache as far as we can take it, reducing loading time - because the features are important (otherwise they wouldn't be there in the first place).
You can do the same. Use a map as light as possible without losing too many details. If your users start to interact with the map in scrolling or zooming you could load a little more details, just enough to make it readable and still keep reasonable loading time.
Don't take away good features because they are heavy - make the features lighter insted!