gzip files recursively

find . -type f -exec sh -c 'cat {} | gzip > {}.tmp; mv {}.tmp {}' \;

Linux paste and sed -- join and remove blanks

following commands 1) join the three files interleaving the lines of the files 2) remove any blank lines that may have been created if the files are not the same length

paste one.txt two.txt three.txt -d'\n'
sed '/^$/d' preQuery.txt > no_blanks

Linux command - remove lines

following command finds the lines in file1 that are missing from file2

fgrep -vxf file1_sorted file2_sorted > removed_ids

SSH without password

the following commands allow an ssh connection without a password

ssh-keygen -t rsa -P ""
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
ssh-copy-id hadoop@slave01