Search results - bring data by custom query or all data ? (Performance issues)

Question

We are developing a system which uses a web console. Once the user logs in, he can choose a date from a date slider and click the “Get” button to retrieve relevant data which can be more than 5000 records (The data that is retrieved is from the date that was chosen until “today”). For 5000+ records, it takes us ~5 minutes to bring the whole data so while the data is loading, the user can only read the already retrieved records without able to search \ edit \ filter.

Users can use the records to get an overall state about their system status or for investigating past events within a timeframe.

My question is: Should we allow the user to create a custom query before clicking the “Get” button and minimize the data\records retrieved from the DB (we lose the “live” search feature + show limited records based on the query) or leave it As-Is and just highlight the Loading and make the user wait until all the records are done retrieving (which will allow a search all over the data in “live”). What do you think users will prefer ?

Note: we do paging for the data\records (but still, only reading)

Thank you !

Answer 1

For the "All Data" query, you can do lazy loading. That is, load some of the queried records and then continue populating the records. You should always show a loading status though.

Moreover, you can give both the options to your users. Power users will really appreciate it. In your case, this performance issue is more of an engineering problem. Five minutes is too long a waiting time. I think your engineers can normalize the database in a better way. Let's look at an example from Stack Exchange Data Explorer.

Default Behaviour:

I picked an already define query (Top 500 users) and ran it against Stack Overflow Database. On the top-left corner, they show you the query execution time.

While the results are getting populated, a loading message is shown as follows:

The results are then shown in a simple table view.

Custom Behaviour

I changed the query to include Top 5000 users. They have a small link to "fork query" and customize it according to your needs. (Though I think that link can be highlighted in a better way.)

Step-wise execution time was shown again with the loading message. It took a little longer this time, but it was clear that query was being processed.

Hope this helps!

Answer 2

On the subject of moving through large datasets, these two tools may help you build a more efficient UX:

https://github.com/cmpolis/smart-table-scroll

http://nexts.github.io/Clusterize.js

Answer 3

You could use an infinite scrolling list, so you only load the first X amount of results (this should be fairly quick). If the user scrolls down towards the bottom, then you load in the next X amount ready for viewing.

Answer 4

That is a long wait. Isn't this contextual? Did you ask your users what they prefer? Are they happy to wait or will they prefer more control over the data? I am sure dev will tell more about ways of getting data but you may do something in the view. If you can at least make them filter with already downloaded data and still show that there is more in progress and may be a tiny progress bar or a counter showing data is loading, so that while they can filter the available data they know it may not be complete results.