File: sort sort Sort The sort command allows us to sort the contents of a file either numerically or alphabetically. Sorting may vary by system but, there are a few rules all systems do adhere to. * Uppercase and lowercase are sorted seperately. Each are sorted alphabetically within their case. * Numbers are sorted by the first digit. The sorting order is 0,1,2,...9. * The remaining symbols such as ) % + ! are not grouped together when sorted. It separates the information in the file into columns using the blank (or tab) as the field seperator by default. By default, sort sorts the file from left to right. File: Using sort Using sort Using Sort General Syntax: sort filename This command will sort the filename on left most column. sort uses a space or a tab to separate the file into columns. Consider the contents of the file called sfile: sfile Original File Sorted File jjc azz ftv eab pll ecc tbb ftv eab gjb gjb jjc mwn joc mar mar ecc mwn azz pll jjc tbb To sort the file, we will use the following: sort sfile This will give the output, on standard output, as specified in the above table under the heading Sorted File. File: sort options sort options Sort Options Sorting Numeric Data By default, the sort command uses a "dictionary" ordering. If you want to sort numeric data, this will cause problems if the numbers have different numbers of digits. EX: if the file "sortfile" contains the following: 23 8 119 71 6 237 and you do: sort sortfile sort will put on standard output: 119 23 237 6 71 8 To sort numeric data in numeric order, use the -n option. sort -n sortfile Sorting in reverse order By default sort starts from the smallest (alphabetically first) and goes to the largest. If you want to reverse the order of the sort - start from the largest - you can use the -r option sort -r sortfile Ignoring case sort will treat all lower case letters as their upper case equivalents when you use the -f option. The "f" stands for "fold" as in "fold uppercase into lower". sort -f sortfile Try it out on a list of names with some first letters capitalized and some not. Sort on only alphanumeric characters sort will ignore any characters that are not digits, letters, or spaces when you use the -d option. The "d" could stand for "dictionary". sort -d sortfile This command will not consider characters such as "$" or "%" or "(" while sorting. Output only one copy of each line sort will output only one copy of any line that appears more than once in the input file when you use the -u option. The "u" stands for "unique". sort -u sortfile Because of the nature of sorting, lines that are identical will end up in the output as consecutive lines. Because of this, you can accomplish the same thing as sort -u sortfile by using the uniq command. sort sortfile | uniq The uniq command compares consecutive lines in the input file and filters them according to one of the following options: * uniq filename the default action is to remove all duplicates and print just one copy of each line * uniq -u filename print only lines that are unique (ie - have only one copy in the input file) * uniq -d filename print only lines that have duplicates (print just one copy of each such line) * uniq -c filename same as the default action (print one copy of each line) and include a count of the number of copies of the line Remember that uniq only compares consecutive lines, so it is usually used with data that has already been sorted. Saving sort Output to a file By default sort puts it's output on standard output. If you want to save the output to a file you can use the output re-direct: sort sortfile > sortfile.out Remember that you can not redirect the output to the same file, because the first thing the shell does when it encounters the re-direct character is delete any existing contents of the target file. Thus, when sort tries to read the file, it would be empty. Most Unix tools work one line at a time, but because sort cannot output anything until it has read all the lines in the file, you can tell sort to save the output back to the original file, instead of putting it on standard output, you can use the -o option: sort sortfile -o sortfile Notice that the file name for output must follow the -o, and remember that this will only work for sort, and this is not the same as the ">" character. File: sort by columns sort by columns Sort by Columns By default, the sort command will sort a file from left to right attempting to resolve any identical lines by searching across the line. If a file contains columns of information separated by white space, you can use the sort command to sort the file starting at a particular column. To do this, you can specify which column to begin the sort on using the -k option as follows: sort -k 2 filename The -k allows you to specify the number of the column to use as the sort key ("k" is for "key"). Now, I lied just a little. If there is more than one "white" char between the columns, then you need to use the -b option to tell sort to ignore extra white space. ("b" is for "blanks") So, the command would need to be: sort -b -k 2 filename Consider the following file called sfile2: sfile2 Original file Sorted File jjc 23245 joc 00232 ftv 03298 ftv 03298 pll 98987 azz 12345 tbb 87676 eab 20198 eab 20198 mwn 22281 gjb 65650 jjc 23245 mwn 22281 gjb 65650 mar 76776 mar 76776 ecc 99999 tbb 87676 joc 00232 pll 98987 azz 12345 ecc 99999 The output listed under Sorted File in the above table is a result of executing the following command. sort -b -k 2 sfile2 The file was sorted beginning with the second columns information. This is an alphabetic sort, sorting the values from 0..... to 9..... Please note that the numbers are sorted in "numeric" order only because they all have the same number of digits. See the next section about sorting numeric data. In the event of a tie, the sort command will check the rest of the columns until the end of the line and then return to the beginning of the line, trying to find a difference. To instruct sort where on the line to stop comparing columns, you can specify a stop column. To do this use the following syntax: sort -b -k 2,3 newfile This would tell sort to begin the sort at column 2 and stop the comparison on column 3. i.e. if there is a tie after all the columns up to the stop column have been checked, then sort will continue with the beginning of the line, instead of with the next column, as in the previous example. In all cases, sort will continue to check columns until it finds a difference, or the entire line has been checked. File: sort: Specifying a column separator sort: Specifying a column separator Sort - Specifying a Column Separator By default, the sort command will use white space to distinguish between columns (fields) in a file. To define what the field separator is going to be, use the -t option. sort -t! sortfile This will define the field separator to be a ! in the file called sortfile. Notice that I did not put a space between the -t and the separator character. A space is allowed there, but I like to put the character right next to the "t". To test out this command, the file /etc/passwd is a great template. This file uses the character : to separate columns (or "fields"), so you can try sorting by any column you like using the -t option. You will probably want to pipe the output to page or more, because the file is quite large. File: sort: Comparing multiple columns of different types sort: Comparing multiple columns of different types sort: Comparing multiple columns of different types Syntax: placement of option characters Consider the two commands: sort -n -k2 file and sort -k2n file Both commands will sort according to the 2nd column using numeric order. If there is a tie then sort goes back to the beginning of the line to try to resolve the tie. This is where the two commands differ. In the first case, the -n option is "global" so that sort will use it in all comparisons, including for the resolution of ties. In the second case, the -n option is restricted only to the 2nd column, so, sort will try to resolve the tie using the default dictionary order. Now consider the two commands: sort -k2n,3 file and sort -k2n -k3 file Here, in both casese we are sorting according to column 2 and resolving ties acording to column 3. This means that if there is a tie in column 2, then column 3 will be checked, instead of returning to the beginning of the line, as in the first example. Again, these two commands differ in the handling of the -n option. In the first case, the option holds for all the columns specified in the -k range. Therefore, if there is a comparison made on column 3, it will be a numeric comparison. This has unpredictable results if column 3 is not numeric. In the second case, the -n holds only for column 2 so sort will do comparisons on column 2 numerically and then, if there is a tie, sort will check column 3 according to the default dictionary order. Of course, everything we have mentioned here applies to any option on sort, not just the -n option. Also, please be careful when you are experimenting. It is often difficult to set up the exact scenario that you want to test without using a large amount of data. So, be sure to frequently review exactly what you are telling the sort command to do, and what features you are trying to test. Be careful that you do not get confused because you are expecting something other than what you are actually asking sort to do. Not that I have ever done that :) File: What is grep? What is grep? What is grep? What is grep? * grep allows us to search the contents of a file or files for a particular string * Example: You may have been developing a large C project and can not remember which module (file) contains a particular routine. Search for it by name using the grep command. File: How does grep work? How does grep work? How does grep work? grep performs a string search on the file(s) and returns a listing of all of the lines that contain the string-pattern (not word). It is importnat to remember that grep, by default, searches for the string that you specify, NOT the word. There is a difference. Syntax: grep string-pattern file(s) file(s) The files can be specified as multiple names on the line or can be specified using wildcards grep 'system' notes1 notes2 This command will grep for the string system in the files notes1 notes2 grep 'processor' *.c This command will grep for the string processor in the files *.c If there is no file specified, the search will default to the standard input. string-patterns String patterns can be regular alpha-numeric strings or string patterns that contain special characters to create regular expressions. It is always best to enclose the string-pattern in single quotes. There is a special set of metacharacters for use with grep. These are used when generating the regular expressions. The metacharacters are discussed in the section "grep - Using Regular Expressions". File: grep options grep options Some grep Options The grep command has a number of options that you can refrence via the man page. Let's look at just a few. Typical use of the options or flags is as follows: grep -option 'string' file(s) # w - treat the string pattern as a word By default, grep searches on a string pattern not a word. i.e. if you are searching for the string 'the' grep will return all occurrences of any string that contains 'the' such as 'the', 'then' and 'there'. When you specify the w option, grep searches on the word you specify and returns only the lines containing that word. A word is any string of alphanumeric characters (letters, numerals, and underscores) delimited by nonalphanumeric characters (punctuation or white space) or by the beginning or end of the line) Example: grep -w program *.c This would search for and locate any line that contains the word 'program' but, not 'programming'. # c - count number of LINES that contain the pattern. When this flag is used, only the number of lines that contain the string pattern is returned, NOT the number of occurrences of the pattern within the file. Example: grep -c system *.c # i - ignore case of string pattern The i option allows grep to ignore the case of the search string. Example: grep -i system *.c Any combination of lower and upper-case letters that form the string 'system' will be located. # v - "verbose" output The v option causes grep to match and output any line that does NOT contain the string. Example: grep -v '203' /etc/passwd Any line in /etc/passwd that does not contain the string '203' will be printed. This could be used to generate a list of all users NOT in user group 203. File: grep: Using regular expressions grep: Using regular expressions grep - Using Regular Expressions Using single quotes When using grep, it is always best to place single quotes around the string expression. The only time it is safe not to use quotes is when the search string is a simple alpha-numeric string that does not contain any white space or any of the following special characters. grep Metacharacters . Match any one character [ ] Match any one character listed within brackets [^] Match any one character except the ones listed within brackets ^ Match any line beginning with the pattern $ Match any line ending with the pattern Examples: * Using . grep 'lab.a' *.c This command will locate all lines that contain the string 'lab' and any one character and an 'a' within the files called *.c. Note: This will match 'lab.a' as well as 'lab1a', 'labKa', etc (see below: hiding special characters) * Using [ ] grep 'project[123]' *.c This command will locate all lines that contain the string 'project1', 'project2' or 'project3' within the files called *.c. Please note that the entire string '[123]' will match exactly one character. 'bob[smith]' will match 'bobs', 'bobm', etc... but not 'bobsmith'. * Using [^] grep 'test[^123]' program This command will locate all of the lines that contain the string 'test' followed by any character except a '1', '2', or '3' within the file called program. * Using ^ grep '^Hello' letter This command will match any line that begins with the string 'Hello' from the file called letter. Please note that if there are any blanks or tabs before the 'Hello' then that line will NOT be matched because 'Hello' is not at the begining of the line. * Using $ grep 'Good bye$' letter This command will match any line that ends with the string 'Good bye' from the file called letter. Please note again that white space (blanks and tabs) are significant here. If the line ends with 'Good bye ' (a blank after the 'e') then that line will NOT be matched. Please make sure that you practice using all of these special characters. Many unix tools, such as grep, sed, and awk, use regular expressions. Mastering the art of building regular expressions is an essential skill for a powerful unix user. Do not skimp on the practice time here. File: grep: Neutralizing the special characters grep: Neutralizing the special characters grep - Neutralizing the Special Characters To neutralize the special UNIX characters use ' ' and \ Examples: * grep 'file\.name' letter1 This will match any line that contains the string file.name * grep 'lab\[a1\]' letter1 This will match any line that contains the string lab[a1] * grep '\^bob' letter1 This will match any line that contains the string '^bob' anywhere on the line. Notice that you can search for the string 'bob^sue' without protecting the ^ because the ^ has special meaning only when it is at the beginning of the regular expression. * grep 'show me the $\$' letter1 This will match any line that contains the string 'show me the $$' Notice that you can search for the string '$500' without protecting the $ because the $ has special meaning only when it is at the end of the regular expression.