How to do filtering, searching on logs

Question

I'm working on a screen where system logs are rendering and default threshold per page is 10 and a load more button to fetch more data [No pagination].There are filtering, searching and sorting options available to help user to customize logs as per his needs.

But the question are :

1. Do I need to purpose threshold customize options [per page results : 10, 20, 30 ...]. Never saw this option with "load more type of pagination"

2. Should I filter(search, sort)upfront data (available on the screen) or complete data available on the server

Answer 1

Had a similar problem to point 2 recently with an incrementally loaded data grid.

When providing data to a user in this manner, they will instinctively expect search, filter, sort and group functionality to act on the whole data set, i.e. everything, not just what has been loaded client side.

The user doesn't care about your implementation. Why should they have to worry if data is on the client or the server? Why should they care if you need to do incremental loading or pagination due to volume of data? They expect a seamless experience because this is how the internet "works". You search everything with google, your search / filter / sort by category on Amazon includes all items, not just the one on the page.

Answer 2

If you're using a "Load More" dynamic for getting new log lines then you don't need pagination controls. You've already handled that user need. Having both would be confusing.

You should absolutely use upfront filtering, but give the user the option to switch to seeing the full raw data.

In most of the use-cases you're dealing with, the user will only be interested in some sub-set of the log entries e.g. looking for errors, looking for successful connections, looking for completed transactions.

However, in some smaller number of the use cases, the user is going to be browsing the logs but unsure of what they're looking for. This behaviour should still be possible in your system. So give them a control that lets them just look at the raw logs.