How to completely erase the duplicated lines by linux tools?
<p>This question is not equal to <a href="https://stackoverflow.com/questions/23740545/how-to-print-only-the-unique-lines-in-bash">How to print only the unique lines in BASH?</a> because that ones suggests to remove all copies of the duplicated lines, while this one is about eliminating their duplicates only, i..e, change <code>1, 2, 3, 3</code> into <code>1, 2, 3</code> instead of just <code>1, 2</code>.</p>
<p>This question is really hard to write because I cannot see anything to give meaning to it. But the example is clearly straight. If I have a file like this:</p>
<pre><code>1
2
2
3
4
</code></pre>
<p>After to parse the file erasing the duplicated lines, becoming it like this:</p>
<pre><code>1
3
4
</code></pre>
<p>I know python or some of it, this is a python script I wrote to perform it. Create a file called <code>clean_duplicates.py</code> and run it as:</p>
<pre><code>import sys

#
# To run it use:
# python clean_duplicates.py &lt; input.txt &gt; clean.txt
#
def main():

    lines = sys.stdin.readlines()

    # print( lines )
    clean_duplicates( lines )

#
# It does only removes adjacent duplicated lines, so your need to sort them
# with sensitive case before run it.
# 
def clean_duplicates( lines ):

    lastLine    = lines[ 0 ]
    nextLine    = None
    currentLine = None
    linesCount  = len( lines )

    # If it is a one lined file, to print it and stop the algorithm
    if linesCount == 1:

        sys.stdout.write( lines[ linesCount - 1 ] )
        sys.exit()

    # To print the first line
    if linesCount &gt; 1 and lines[ 0 ] != lines[ 1 ]:

        sys.stdout.write( lines[ 0 ] )

    # To print the middle lines, range( 0, 2 ) create the list [0, 1]
    for index in range( 1, linesCount - 1 ):

        currentLine = lines[ index ]
        nextLine    = lines[ index + 1 ]

        if currentLine == lastLine:

            continue

        lastLine = lines[ index ]

        if currentLine == nextLine:

            continue

        sys.stdout.write( currentLine )

    # To print the last line
    if linesCount &gt; 2 and lines[ linesCount - 2 ] != lines[ linesCount - 1 ]:

        sys.stdout.write( lines[ linesCount - 1 ] )

if __name__ == &quot;__main__&quot;:

    main()
</code></pre>
<p>Although, while searching for duplicates lines remove seems to be easier to use tools as grep, sort, sed, uniq:</p>
<ol>
<li><a href="https://unix.stackexchange.com/questions/30173/how-to-remove-duplicate-lines-inside-a-text-file">How to remove duplicate lines inside a text file?</a></li>
<li><a href="https://stackoverflow.com/questions/21582610/removing-line-from-list-using-sort-grep-linux">removing line from list using sort, grep LINUX</a></li>
<li><a href="https://stackoverflow.com/questions/6712437/find-duplicate-lines-in-a-file-and-count-how-many-time-each-line-was-duplicated">Find duplicate lines in a file and count how many time each line was duplicated?</a></li>
<li><a href="https://stackoverflow.com/questions/9377040/remove-duplicate-entries-using-a-bash-script">Remove duplicate entries in a Bash script</a></li>
<li><a href="https://stackoverflow.com/questions/1444406/how-can-i-delete-duplicate-lines-in-a-file-in-unix">How to delete duplicate lines in a file without sorting it in Unix?</a></li>
<li><a href="https://stackoverflow.com/questions/19039036/how-to-delete-duplicate-lines-in-a-file-awk-sed-uniq-not-working-on-my-file">How to delete duplicate lines in a file...AWK, SED, UNIQ not working on my file</a></li>
</ol>