uniq — Linux basic command

Linux / Basic Linux commands / uniq.html

A command that removes or counts consecutive duplicate lines from sorted files.

Basic Information

What Is This Command?

uniq, short for unique, is a command that removes or counts consecutive duplicate lines from a sorted file or input. When used with sort, it can remove duplicates across the entire file.

Since it only removes consecutive duplicates, using it without sorting may not completely remove all duplicates.

What Problem Does It Solve?

• Removes duplicate entries from a list.
• Counts the occurrence of each item.
• Cleans up duplicate lines in logs.
• Extracts unique values in data analysis.

Main Features

Duplicate Removal

Removes consecutive duplicate lines.

• Use Case: When you need to eliminate duplicates from a sorted list.
• Result Obtained: Only unique lines remain.
• Limitation: If not sorted, non-consecutive duplicates will remain.

Count Duplicates (-c)

Displays the duplicate count for each line.

• Use Case: When checking which item appears most frequently.
• Result Obtained: The count is displayed before each line.
• Limitation: Must be used after sorting for accuracy.

Print Unique Items (-u)

Prints only lines that appear exactly once.

• Use Case: When finding items that occur only once.
• Result Obtained: Only solitary items are displayed.
• Limitation: Must be used after sorting for accuracy.

Based on Specific Fields (-f, -s)

Ignores characters at specific positions for comparison.

• Use Case: When grouping lines that are similar but differ slightly.
• Result Obtained: Duplicates matching the condition are removed.
• Limitation: Option combinations can be complex.

Installation and Execution

Verify Pre-installation

Included by default in all Linux distributions.

which uniq
uniq --version

Using with sort

Connect with sort via a pipe for full duplicate removal.

# Sort and then remove duplicates
cat file.txt | sort | uniq

# Count duplicates
cat file.txt | sort | uniq -c | sort -rn

Practical Usage Examples

Basic Duplicate Removal

Removes consecutive duplicates.

# Remove duplicates from a sorted file
touch colors.txt
echo -e "red\nblue\nblue\ngreen\ngreen\ngreen" > colors.txt
sort colors.txt | uniq

# Result:
# blue
# green
# red

# Compare with original
cat colors.txt

Count Duplicates

Counts how many times each item appears.

# Display duplicate count
cat colors.txt | sort | uniq -c

# Result:
#       1 blue
#       3 green
#       2 red

# Sort by frequency
cat colors.txt | sort | uniq -c | sort -rn
#       3 green
#       2 red
#       1 blue

Find Unique Items

Finds items that appear only once.

# Only lines that appear once
cat colors.txt | sort | uniq -u
# Result: blue (only once)

# Only duplicate lines
cat colors.txt | sort | uniq -d
# Result:
# green
# red

# Log analysis example
cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10