An Intro to Resolving Duplicate Results in SQL Queries

by

I recently reviewed a SQL query that was returning duplicate results.  Since I expected, and needed, a single result from the query this was a problem.  Here’s an explanation of why this bug occurred, and several different ways it can be resolved.

My first attempt to remove the duplicates was to add the DISTINCT keyword to my query, but that didn’t fix the problem – I was still seeing duplicates.  As I was looking at the result set more closely I realized that the problem was in how the query was structured and the columns that it was returning in the results.

I needed to return a lot of data in this query, and it included joins across several tables.  Some of the tables that I was joining had one to many relationships with rows in other tables.  That meant that when my result set was created, each of those many results got returned as its own unique row.  For example, if I had one user joined to a table with four different category entries, the user would be returned in my result set four times, once for each category entry.  Since that wasn’t the result that I wanted, I needed to think more carefully about what data I was returning in my select.

Suppose you have a database that lists a number of users, and jobs that they have had. There is a users table, a jobs table, and a user_jobs table that joins the two.  Users can have had more than one job. Here is what the data looks like:

Let’s say you want to get a list of users, and also need some job data for those users.  You only need a single job, not all the jobs.   Here’s a sample query:

And here are the results:

As you can see, there is some duplication in the results. We wanted to return a single record for each user, but instead we get multiple records for some users in the results.  Looking across the result set, we can see that the “duplicate” results are not actually duplicates – the same user is being returned with multiple categories, making each row unique.  So if we only want to return one of each user, how do we do it?

There are a few potential solutions for this:

One would be to structure your data differently, for example so that there was a one to one relationship with each category.  This isn’t a good solution, as many cases, we’re dealing with data models that we didn’t create, that legacy code relies on, and that for whatever reason we can’t or don’t want to change.  So, how can we to get unique user results within this data model?

Another potential solution is to break this up into multiple queries.  Get the user list back first, then get a category associated with that user.

A third solution is to use a grouping within the initial query, to ensure that the data that is joined to the user data is also unique, thus a truly distinct data set is returned.

Now, we have a result set that only includes one of each user and some category data about that user.

As you can see, the data returned in a query set, and the joins you are making, needs to be carefully considered in order to return the correct results.

7 Comments

  1. Lloyd on said:

    I am having a problem regarding multiple join and duplicates
    I need to join 3 tables.
    I have
    one table with customer name, address and ID
    one table with vehicle number(reg) and customer ID
    one table with vehicle reg and entries for each time they have been captured on a camera

    I have to display the customer name, with count of cars they own, and count of times they were captured on the camera

    Can someone please help. It is a question on sqlzoo.net
    http://www.sqlzoo.net/wiki/Congestion_Medium
    (Number 5 – last question)

  2. Abraham Olatubosun on said:

    Msg 156, Level 15, State 1, Procedure ECEWS_VW_Adherence, Line 3
    Incorrect syntax near the keyword ‘declare’.

    code
    IF OBJECT_ID(‘ECEWS_VW_Adherence’,’V’) IS NULL
    BEGIN
    declare @sql1 varchar(4000)
    select @sql1 = ‘CREATE VIEW ECEWS_VW_Adherence
    AS
    declare @FacilityID nchar
    Select b.PatientEnrollmentID as PatientID
    ,c.RegimenAdhere as ARTAdherenceAssessmentPerformedDuringLast3Months
    ,c.UpdateDate as LastDateOfAssessment
    ,s.TestResults as HighestCD4SinceARTinitiation
    ,s.reportedbydate as DateHighestCD4Test
    ,@FacilityID as FacilityID
    From dbo.VW_PatientLaboratory s join dbo.VW_PatientDetail b on
    b.Ptn_Pk = s.Ptn_pk
    join dtl_Adherence_Reason c on s.Ptn_pk= c. Ptn_pk
    where s.TestName=”CD4”’
    exec (@sql1)
    END

  3. Arno on said:

    Many thanks for this example. You have solved my problem. Very helpful.

  4. Tracy on said:

    What if i want to create one unique row but create additional colunms for the job names like job name 1, job name 2 etc etc

  5. Eelco Hillenius on said:

    How about simply adding ‘group by users.id’ to your original query?

    • Ranjeet Kasture on said:

      This will also work.

    • Tito on said:

      Awesome. Thanks!!

Leave a Reply to Arno Cancel reply

Your email address will not be published. Required fields are marked