Design principles: Lean data vs. bloated data

Question

I've been tasked with designing tables to store civil infrastructure asset data. Users have asked for a multitude of fields in the tables. For instance, they've asked for 50+ fields in a roads table. The tables contain tens of thousands of records.

From my experience, 50+ fields in a table is way too many. Not from a system performance perspective, but from a data entry/data maintenance perspective. From past experience, I've found users can't keep up with that level of detail long-term. Maintaining 50+ fields for a specific asset is overwhelming, impractical and provides little value compared to the effort required.

I would like to set up a data that is lean, mean and reliable, rather than bloated, unreliable and probably full of nulls. It seems to me that principles like ‘less is more’, ‘keep it simple, stupid’, and ‘do a few things really well’ would be the key to success. But I struggle to communicate this to stakeholders when designing such systems.

So I'm wondering, are there any established design principles that talk about the approximate number of fields that can be reasonably/sustainably/accurately maintained by users? (In other words, a rationale for designing lean data.)

Answer 1

If there's some usage statistics available for your users, something like the 80/20 rule or Pareto Principle is a general indicator of where you should be focusing your efforts.

It is a more objective way to show the stakeholders that perhaps only 80% of the users (or whatever the figure might be) ever view or edit 20% of the fields, and therefore spending time and effort to maintain the rest of the information should not be as important and making sure that you spend time and effort to create the best experience for using that 20% of the information.

Answer 2

It is hard to answer your question without having more details.

However, when it comes to the approach to this task, I think that Maeda's SHE may be a good principle. As Maeda says, "the simplest way to achieve simplicity is through thoughtful reduction". SHE is one of the ways to get there (even though there may be various interpretations):

• Shrinking allows you to limit the overload by leaving just the elements giving User most value.

• Hiding will let you put the rest of them behind a curtain.

• Embodying requires from you to exaggerate the value of what is left.

This is a very general approach, though. In practice this could mean that you could:

• leave the most important fields in the main view,
• move the other ones to some "see more" section (expandable or available on click), even leaving them less functional deep there,
• focus on the ones that are left in the main view so that Users will see the value of simplicity and only go for the extended data/fields/controls when they really need them.

I understand it may be hard to implement, namely due to difficulties in deciding which ones are the most important and due to some dependencies between fields.

To tell more I would need to know more about the data fields and the topic, I am affraid.

Answer 3

I guess from DB perspective, normalization will come to your help, and you will certainly have guidelines which are focused on performance and other tangible factors.

I would instead draw your attention to the target users. When you mention that users are demanding for many fields vs stakeholders are demanding. You'd want to make an educated decision in favor of those who are going to use the system.

Under the hood, if your system is going to aid users to make some actionable decision based on the very nuances of the data about the roads then collecting lots of information makes sense. Oftentimes, the people collecting and storing the information will give you a direction of what sort of data can and should live together.

I suspect not many of your users are going to interact with tables directly. Those who are going to do can be educated upon the objective parameters like performance. Considering the User Centric Design approach, even if you collect 150+ fields for a road, you need to draw the line between technicalities of the implementation vs business objective.

More often than not, a clear focus on what they want to do with those fields and what sort of actionable inference the system is required to draw for them will give the direction and justification for the technical choices you will have to make.

Answer 4

To build on Michael Lai's answer, here's some additional info on the 80/20 rule from Universal Principles of Design:

80/20 Rule - A high percentage of effects in any large system are caused by a low percentage of variables.

The 80/20 rule asserts that approximately 80 percent of the effects generated by any large system are caused by 20 percent of the variables in that system. The specific percentages are not important, as measures of actual systems that indicate that the proportion of critical variables varies between 10 and 30 percent.

The 80/20 rule is useful for focusing resources and, in turn realizing greater efficiencies in design. For example, if the critical 20 percent of a product's features are used 80 percent of the time, design and testing resources should focus primarily on those features. The remaining 80 percent of the features should be re-evaluated to verify their value in the design. Similarly, when redesigning systems to make them more efficient, focusing on aspects of the system beyond the critical 20 percent quickly yields diminishing returns; improvements beyond the critical 20 percent will result in less substantial gains that are often offset by the introduction of errors or new problems into the system.

All elements in a design are not created equal. Use the 80/20 rule to assess the value of elements, target areas of redesign and optimization, and focus resources on an efficient manner. Noncritical functions that are part of the less-important 80 percent should be minimized or removed altogether from the design. When time and resources are limited, resist efforts to correct and optimize designs beyond the critical 20 percent, as such efforts yield diminishing returns. Generally, limit the application of the 80/20 rule to variables in a system that are influenced by many small and unrelated effects.