uniq Command in Linux: Filter Duplicate Lines

Quick take: The uniq command filters out adjacent duplicate lines, so you almost always run sort first. Use uniq -c to count occurrences, -d to show only duplicates, and -u for lines that appear once.

Introduction

The uniq command removes or reports repeated lines. Its one quirk defines how you use it: uniq only collapses adjacent duplicates, so the input must be sorted first. Paired with sort, it becomes a powerful tool for counting and de-duplicating data.

This guide covers removing duplicates, counting occurrences, and isolating lines that are repeated or unique.

Syntax

The basic syntax of the uniq command is:

uniq [OPTIONS] [INPUT]

Common Options and Parameters

The most useful options and parameters for the uniq command:

Practical Examples

Real uniq commands you can run today:

# Remove adjacent duplicates (sort first!)
sort access.log | uniq
# Count how many times each line appears
sort names.txt | uniq -c
# Show only the duplicated lines
sort list.txt | uniq -d
# Show only lines that appear once
sort list.txt | uniq -u
# Top 10 most frequent values
sort log | uniq -c | sort -nr | head
# Count unique visitors by IP
awk '{print $1}' access.log | sort | uniq -c | sort -nr

Tips and Best Practices

• Always sort before uniq — uniq only removes consecutive duplicates, so unsorted input gives wrong results.
• uniq -c followed by sort -nr is the standard way to build a frequency-ranked list.
• Use sort -u if you only need de-duplication without counts — it does both jobs in one command.

Final Thoughts

uniq is small but essential for analysing data — counting occurrences, finding duplicates, and isolating one-off lines. The golden rule is to sort first, because uniq only sees adjacent duplicates. Combined with sort and awk, it powers the log-analysis one-liners administrators rely on daily.