Showing duplicates in group or category

Question

I'm working on an application which handles billions of records, where lots of records could be duplicates. My task is to make it easy to find and correct the duplicate data, but I'm bit confused how to show duplicates records so that admin or any authorised user can take action on them

What i'm thinking :

I'll make pie chart with categories

Pie chat 1:

1. Name:

   1. First Name
   2. Middle Name
   3. Last Name
   4. Family name
   

Pie chart 2:

1. Address

   1. Official
   2. Residential
      1. Home number
      2. street Name
      3. Phone number
      4. Pin code

And there could be lots of fields(25+) per records like ID, bank a/c number, social security number and so on

eg: I have 5 records in a group where each records contain different mobile numbers but rest of the data is same. So i'll group them inside phone category

Question:

1. How do i tackle the situation (lots of groups) & lay them on the screen?
2. Making 25 + pie charts and then drill down to x level is cumbersome task. Any suggestions here?

Answer 1

To me this looks like reports of some kind, where admin should resolve issues with duplicates. The pie chart is more used to visualize data as parts of the whole information set. An example of the pie chart can be sales by region of total sales... like this:

In this case I think a simple list would do the job. Provide a list of possible duplicates where you can switch between showing the whole list and only list items with issues (where you notify admin).

^{download bmml source – Wireframes created with Balsamiq Mockups}

This list of issues is a drill down list where Name expands in the next screen to sub items of name (first, last, middle,...). This is a better way to display issues to be resolved by admins.

Answer 2

The most important question is "What do your administrators need to determine it's a duplicate?", and the second important question is (I guess) "What do the do about it?".

A pie chart (or any other graph or report) telling the administrator which kind of difference (mobile phone vs. street no vs. social security no) will not help at all in determining which records are duplicates. So I agree with @Benny that you should not use charts.

I think you should invest some algorithmic smartness into your program to make it help the admins: Which records are most likely duplicates? Which fields have the highest probability to vary, given it's the same person? For example, a typo on the mobile number might happen more often than a typo on the name. If social security numbers have a checksum integrated (I don't know if), a type on them is even less probable (assuming it was validated upon entry - if not, you can validate and flag incorrect ones). Also, if two fields vary, the probability the record identifies different people rises. Your program may even learn from the duplicate identification your administrators finally do, which of the fields are less relevant for unique identification.

Based on such indicators, your program should present a list of most likely duplicates. It should also indicate which of the duplicate is likely the correct one. For example, a corrupted social security number as the sole difference can be eliminated. Of, if mobile numbers differ, the newer record can be assumed to be correct since the person may have changed their provider.

Summary: I think your program should present a ranked list of likely duplicates, where the likely best correct candidate (which might be combined from several duplicates) is marked as a proposal. So the UI contains a ranked list of record groups, where in each group a selection of the correct one is possible, including a default selection. Your admins can fly through the list and select the correct candidate in each group, thereby reducing the list of duplicates.