I'm working on a web project that displays a map to the user, using google map's JavaScript API. Inside the map, I will populate markers to display data to the user. Now, my problem is that my data is stored in a SODA database on another server, and there are more than 29,000 rows in that DB. On top of that, the database requires pagination every thousand rows, so my overhead for requesting and receiving the data is multiplied 30-fold as well. Needless to say, my load time for preparing that data for display to the user is much longer than I think this webpage should load. Currently, I was thinking of loading the page, showing a loading bar, and using AJAX to load the data while updating the loading bar as page after page of data comes through and is parsed/prepared for display. Personally, I would find this very annoying to have to do each time I open the website. I plan for all the functionality in the site to be handled on a single page, however; with AJAX grabbing additional data from the server when requested by the user. (This data size is negligible, not something I'm worrying about.) Is there some way I should be designing this to either: Significantly increase my load time. (By some combination of wizardry and sorcery I'm not aware of), or Make sure the user doesn't get frustrated during my load.
-
Have you thought about caching? I don't think webpages should need loading bars :/– FranchescaAug 30, 2014 at 6:18
-
@Franchesca Agreed. I was thinking about it, since the data isn't liable to change very often. However, I was planning on running the program inside programmr.com, and I'm not sure how/if-I-am-able-to save persistent files with PHP somewhere on their server. If I was running my own server, I would definitely end up chaching the data, and just updating it every once in a while.– Andrew GiesAug 30, 2014 at 6:28
-
You might get better help with this on another stack exchange site. Perhaps Super Users, Webmasters, or Stack Overflow?– FranchescaAug 31, 2014 at 7:37
3 Answers
How fast something feels depends on the absolute objective time the operation takes and a perception component, which is more subjective and can be slightly manipulated.
Some methods to consider:
- Can your data be displayed in layers (e.g. showing only hotels on the map, only restaurants, etc.)? If so, load one layer at a time and load the rest on demand or prefetching in the background.
- Any chance you can show a default view that can be precached (as a tutorial perhaps) and then allow the user to get to the "real" data?
- Try to be more descriptive and break down the process rather than just a global progress bar ("fetching initial results"; "calculating layout"; "validating locations"...)
- Provide useful animations (zoom in from the globe to the area in question?)
- If the wait time is horrible suggest the user leave the tab open and be notified via a sound, an email, etc. (this is if you cannot go below several minutes)
-
Interesting ideas. The data I'm presenting to the user is a google maps representation of all the parking meters in San Francisco, along with data such as operating costs, hours, number of spots the meter services, etc. All that data I can load and show on request easily. However on page load, I need at least the ID's of the meters (to grab the other data later) and the lat/long of each meter. THAT is just over 29,000 of them, which takes a while. Aug 30, 2014 at 6:34
-
I was also thinking of making the loading bar a little more comedic, for an entertainment factor. Ex, "Attempting to Parallel Park", "Checking Watch", "Moment of Panic", "Sprinting back to Car", "Feeding the Meter", etc. Those kind of things I think would make the app more interesting while it loads. Aug 30, 2014 at 6:36
Firstly - optimise the size of your data and your indexes. 29000 rows is a small dataset for a well indexed database to handle - suggest you read up on covering indexes: http://en.m.wikipedia.org/wiki/Database_index#Covering_index
The next stage is to consider that it makes zero sense to present 29000 icons to a user. Take a leaf out of Google's book and start by aggregating closely located data points on different scales. As the user zooms or drills in use Ajax to load another level of detail. This can be readily pre-computed and stored in the DB.
These two techniques should remove the need for the user waiting too long or any sort of loading progress bar.
In order to save on the load time on every page load you can use html5 to persistently store your json responses client side. You then check if the json you are about to request is already in the persistent store and only load it from the server if it isn't. You will need some way for the server to flush the cache on a future page load though for if the dataset has changed. There are more sophisticated cache schemes that could be used but I suggest you worry about the basics first. BUT DON'T DO ANY CACHING UNTIL EVERYTHING ELSE IS WORKING... Google "premature optimisation" to learn why.
-
Interesting. I'd quickly figured out that 29,000 icons would be feasible, but I hadn't been able to come up with a way to change it. I'd still like to combine this with caching somehow. Does anyone know about saving files programmatically on programmr? Aug 30, 2014 at 17:40
-
@AndrewG Added note recommending using html5 persistent storage as a data cache. Aug 30, 2014 at 18:38
-
Wouldn't the HTML5 persistent store still require me to attain all of my JSON from the DB for each client that opens the webpage? I figure if I can cache the data on the server, store a date/time of the last DB connection, and every time the data is requested, check if the date time is longer ago than 1 day (for examaple), return the cached data and begin the process of updating the cache from the DB in the background. Aug 30, 2014 at 19:10
-
A server cache would be fairly straightforward but unless your server is ridiculously under powered queries will execute very quickly. If you want a server cache whenever you respond to a json request store the parameters and response in memory for re-use similar to the client side - though there's really little value unless you have a huge user base in which case I'd run it in the cloud for scale anyway. I repeat your database should easily and efficiently deal with queries against a dataset this size if it's properly indexed and on any sensibly sized server. Only do it if it proves necessary Aug 30, 2014 at 19:27
-
The database isn't run my myself, I'm just making use of it. It supports queries very similar to (but not exactly like) MySQL. Aug 30, 2014 at 22:26
You may also try to:
Group your data by location for different zoom levels, i.e. show a single icon for several items at abstract zoom level 0 and draw items separately on a zoom level 1. Caching data for lower zoom levels will give you a nice boost.
Fetch data from the server by small chunks and render this chunks data to the user: it will let user to see some info while download is in progress (you may also show "total load progress" somewhere on the screen)
And, as others are mentioned already, optimize your data access by caching and on data write (i.e. add indices, group on data write and not read, etc)
-
Lots of people are saying, "Optimize your data by caching.". My problem isn't understanding I need to do that. My problem is, how am I supposed to? I'm going to eventually be running this program inside programmr's built-in web environment. As far as I know, each instance of the program will generate a new environment: I.E., I have no persistent file system available on the server. I'd be able to create and write to a cache, but if it disappears every time I reload the page, it does me no good. Does anyone know specifically how to save files persistently inside programmr's environment? Aug 30, 2014 at 22:32
-
@AndrewG add your own database at your own server and put user data in it (i.e. get the data from the remote server, analyze it and store in your own database for quick access) Aug 31, 2014 at 11:36