Uniq removes repeated lines.
uniq [OPTIONS...] [FILES]...
Uniq copies the input file, or the standard input, to the standard output, comparing adjacent lines. In the normal case, the second and succeeding copies of repeated lines are removed. Repeated lines must be adjacent in order to be found.
–u: Print unique lines. –d: Print (one copy of) duplicated lines. –c: Prefix a repetition count and a tab to each output line. Implies –u and –d. –f num: The first num fields together with any blanks before each are ignored. A field is defined as a string of non–space, non–tab characters separated by tabs and spaces from its neighbors. -cn num: The first num characters are ignored. Fields are skipped before characters.