18
Oct
29

Bash one liner to randomize lines in file

Discovered that the bash shell has a variable called $RANDOM, which outputs a pseudo-random number every time you call it. Sweet! Allowed me to randomize the lines in a file for a process I needed to do, thusly:

for i in `cat unusual.txt`; do echo “$RANDOM $i”; done | sort | sed -r ’s/^[0-9]+ //’ > randorder.txt

In other words, put a random number on every line, sort the file, then take off the random numbers. Worked like a charm.

Enjoyed reading this post?
Subscribe to the RSS feed and have all new posts delivered straight to you.
29 Comments:
  1. Saint Aardvark 18 Oct, 2007

    Now that’s clever. I’ll have to remember that.

  2. red 12 Dec, 2007

    sed: illegal option — r

  3. mmichie 12 Dec, 2007

    Red: I used GNU sed on Linux, what Unix are you using?

  4. red 12 Dec, 2007

    I’m using bash on osx Leopard

  5. red 12 Dec, 2007

    my bad – i made a typo.

  6. mmichie 12 Dec, 2007

    Red: no prob :)

    It occurs to me that this thing has a problem whereby if the text file you were randomizing started with numbers and blank, the regular expression could be a bit too greedy. Just a quick hack that I came up with that could be improved.

  7. red 12 Dec, 2007

    looks like it’s sed -E on mac

  8. beh 29 Jan, 2008

    If you really want to be sure that contents of the file (eg. you have lot of lines that start with number) don’t have effect on sorting, you should do something like this:

    for i in `cat unusual.txt`; do echo “$RANDOM $i”; done; sed ’s/^/0000/’ | sed ’s/^0*\([0-9]\{5\}[ ].*$\)/\1/’ | sort | sed -r ’s/^[0-9]+ //’ > randorder.txt

    btw. this page if first Google hit when you search for .

  9. w23 6 Feb, 2008

    why don’t you just use shuf? like
    shuf unusual.txt > randorder.txt

  10. micdah 1 Mar, 2008

    Well I would say that using shuf would be quite an easier way to do it…
    But just as a comment on the original method, I had a bit of trouble getting it to work when the lines of my files contained spaces, causing each seperated word to get a line of its own..

    Anywho, don’t know if this is just on my setup / or that particular file, but I found a way that worked, very much inspired by your command – and it is as follows:
    while read -r line; do echo “$RANDOM $line”; done rand.txt

    Cheers… c”,)

  11. adrian 10 Mar, 2008

    Shuf doesn’t exist on the box i have. and i can’t seem to get:
    for i in `cat unusual.txt`; do echo “$RANDOM $i”; done | sort | sed -r ’s/^[0-9]+ //’ > randorder.txt
    keep getting the sed man page popping up.
    and i don’t really understand how:
    while read -r line; do echo “$RANDOM $line”; done rand.txt
    reads in a file in the first place? where is the input?

  12. adrian 10 Mar, 2008

    My bad, it wasn’t pasting right into the terminal, it works. (original solution)

  13. Shyam 15 May, 2008

    More efficient way (compared to the for loop)

    cat unusual.txt | while read line do….

  14. randomizer 17 May, 2008

    How about:

    sort -R unusual.txt > random.txt

    or printing just one random line:

    sort -R unusual.txt | tail -1

  15. random walker 29 Apr, 2009

    nice!

  16. random walker 29 Apr, 2009

    sort -R isn’t available in bash.

  17. Bill 30 Apr, 2009

    Dude, it’s like the Schwartzian transformation in bash!

  18. chad 30 Apr, 2009

    when I have a file with spaces where I want to iterator over each line I change the input field separator environment variable like:
    $ export IFS=’

    $ for line in `cat file`
    > {
    > echo “$RANDOM $line”
    > } | sort -n | sed -e ’s/^[0-9]+ //’ > random_file

  19. Dougal 3 May, 2009

    Just to add another tweak…

    while read i ;do echo “$RANDOM $i”; done randorder.txt

  20. Dougal 3 May, 2009

    My previous comment got screwed up by the formatting…

    First part: “while read i ; do echo “$RANDOM $i”; done randorder.txt”

  21. [...] té una funció $RANDOM que genera un nombre pseudo-aleatori en cada [...]

  22. bart 28 May, 2009

    How about:
    awk ‘BEGIN{srand()}{print rand(),$0}’ SOMEFILE | sort -n | cut -d ‘ ‘ -f2-

  23. Boris Abramov 25 Jun, 2009

    sort -R
    :)

  24. gabriellllo 6 Jul, 2009

    Great tip!

    Since I had numbers in my file, I used awk instead of sed, since I found it easier:
    for i in `cat list_to_randomize.txt`; do echo “$RANDOM $i”; done | sort | awk ‘{print $2}’ > randomized_list.txt

    Thanks a lot for this page!

    G.

  25. swordfish 2 Aug, 2009

    none of these worked for me. i found the solution on a far away google result : (be careful despite this is the first google result , the formatting sucks and copy-paste may give errors)

    awk ‘BEGIN {srand()} {print int(rand()*1000000) “\t” $0}’ file | sort -n | cut -f 2-

  26. Arthur 20 Sep, 2009

    sort -R will only work if the lines are unique. If there are duplicate lines, sort -R will put them next to each other.

    shuf on the other hand will sufficiently randomize a list, including NOT putting duplicate lines next to each other.

  27. manoflinux 23 Sep, 2009

    this actually works for me. its simpler and uses colrm which rocks.

    for i in `cat mapslist.txt`; do echo `printf “%08d” $RANDOM` $i; done | sort | colrm 1 10 >mapslist.txt.rand; mv mapslist.txt mapslist.old; mv mapslist.txt.rand mapslist.txt;

  28. Moses 10 Nov, 2009

    “why don’t you use shuf? Why not use sort -R?”
    If I had shuf, or if my local /bin/sort supported -R, I would use it. But I don’t, which is why I’m using something fork-y in bash.

    “why don’t you just get shuf?”
    Because I don’t have root and/or gcc on every machine I touch.

    HOWEVER, with what was mentioned above, I can make shuf:

    echo -e ‘#!/bin/bash’”\n”‘while read -r LINE; do echo “$RANDOM $LINE”; done |sort |sed -r ‘”’s/^[0-9]+ //’” > ~/bin/shuf
    chmod 0755 ~/bin/shuf

    Now I can put |shuf into my chains of pipes.

  29. Jadu Saikia 10 Jan, 2010
Post your comment



Celadon theme by the Themes Boutique