We are developing a system which uses a web console. Once the user logs in, he can choose a date from a date slider and click the “Get” button to retrieve relevant data which can be more than 5000 records (The data that is retrieved is from the date that was chosen until “today”). For 5000+ records, it takes us ~5 minutes to bring the whole data so while the data is loading, the user can only read the already retrieved records without able to search \ edit \ filter.

Users can use the records to get an overall state about their system status or for investigating past events within a timeframe.

My question is: Should we allow the user to create a custom query before clicking the “Get” button and minimize the data\records retrieved from the DB (we lose the “live” search feature + show limited records based on the query) or leave it As-Is and just highlight the Loading and make the user wait until all the records are done retrieving (which will allow a search all over the data in “live”). What do you think users will prefer ?

Note: we do paging for the data\records (but still, only reading)

Thank you !

  • Is your question whether or not the gain of providing search tools which results in search results that the user actually is looking for is worth the loss of live update of a search? If so I guess it depends on the situation and target group, but generally I'd say providing tools which gives the user what they're actually looking for is worth a lot. Nov 17, 2014 at 10:30
  • 1
    I would only give them the first X. With a single count of the total. And give them an button to load all.
    – paparazzo
    Nov 17, 2014 at 15:24
  • Thank you both, I think I'll mix your answers into one solution by providing the first X and options of "Loading All" & "Custom Query"
    – Shushi
    Nov 18, 2014 at 16:58

4 Answers 4


For the "All Data" query, you can do lazy loading. That is, load some of the queried records and then continue populating the records. You should always show a loading status though.

Moreover, you can give both the options to your users. Power users will really appreciate it. In your case, this performance issue is more of an engineering problem. Five minutes is too long a waiting time. I think your engineers can normalize the database in a better way. Let's look at an example from Stack Exchange Data Explorer.

Default Behaviour:

I picked an already define query (Top 500 users) and ran it against Stack Overflow Database. On the top-left corner, they show you the query execution time.

enter image description here

While the results are getting populated, a loading message is shown as follows:

enter image description here

The results are then shown in a simple table view.

Custom Behaviour

I changed the query to include Top 5000 users. They have a small link to "fork query" and customize it according to your needs. (Though I think that link can be highlighted in a better way.)

Step-wise execution time was shown again with the loading message. It took a little longer this time, but it was clear that query was being processed.

enter image description here

Hope this helps!


On the subject of moving through large datasets, these two tools may help you build a more efficient UX:




You could use an infinite scrolling list, so you only load the first X amount of results (this should be fairly quick). If the user scrolls down towards the bottom, then you load in the next X amount ready for viewing.


That is a long wait. Isn't this contextual? Did you ask your users what they prefer? Are they happy to wait or will they prefer more control over the data? I am sure dev will tell more about ways of getting data but you may do something in the view. If you can at least make them filter with already downloaded data and still show that there is more in progress and may be a tiny progress bar or a counter showing data is loading, so that while they can filter the available data they know it may not be complete results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.