Unless otherwise noted, articles © 2005-2008 Doug Spencer, SecurityBulletins.com. Linking to articles is welcomed. Articles on this site are general information and are NOT GUARANTEED to work for your specific needs. I offer paid professional consulting services and will be happy to develop custom solutions for your specific needs. View the consulting page for more information.
Shell Scripting Tips
Shell Scripting Tips Written by Doug Spencer 12/05/2006
These are a few shell scripting tips to increase the ease and effectiveness of your scripts. This is by no means comprehensive, it is just an overview of some of the forgotten or little known commands that can be used to get things done. Most of these are BASH commands, but some work in Korn shell or Bourne shell as well.
First, one of my favorite UNIX commands: xargs. This command is great for performing repetitious tasks. Most looping stanzas can be replaced with a single line using xargs. The xargs command can be used in the following ways with the following input:
$ cat input.txt Item1 Item2 Item3
$ cat input.txt | xargs echo Item1 Item2 Item3
In the instance shown above, you see the input pipe to xargs puts the input on the command line as a space separated by default. The result of the command shown above is that the input is echoed to stdout.
$ cat input.txt | xargs -iZ echo This Z is on sale This Item1 is on sale This Item2 is on sale This Item3 is on sale
In the instance shown above, you see that the character "Z" is replaced by the input. A different character can be specified after the -i if you wan to use something else. This usage of xargs can be useful for a variety of instances.
Here is an example of how xargs can be used. You want to delete all files except those containing "mpeg" from the following files.
$ ls lambda last latex lav2wav lavinfo lavrec lame lastb lav2avi.sh lav2yuv lavpipe lavtrans lamed lastlog lav2mpeg lavaddwav lavplay lavvideo
You can do the following to verify you have selected the correct files to delete:
$ ls | grep -v mpeg lambda last lastb lastlog latex lav2avi.sh lav2wav lav2yuv lavaddwav lavinfo lavpipe lavplay lavrec lavtrans lavvideo
Then, when it is correct, run the following:
$ ls | grep -v mpeg | xargs rm $ ls lav2mpeg
You have now deleted all files in that directory except lav2mpeg, the only one that contained "mpeg" in the list. If you had additional files that should be skipped, simply grep them out in addition to the mpeg item.
The wait command will pause further execution of a script until processes running in the background finish. This is useful for running multiple tasks in parallel to facilitate quicker completion of a task. The wait command is a BASH builtin command and is also available in Korn shell. The following scipt will demonstrate using wait command.
#!/bin/bash echo This script demonstrates the wait command echo We will now start several long-running commands in the background sleep 10 & sleep 20 & sleep 15 & sleep 25 & echo Processes before wait completes: ps -fC sleep wait echo Processes after wait completes: ps -fC sleep
The output of running this script is:
debian:/tmp/test$ ./wc This script demonstrates the wait command We will now start several long-running commands in the background Processes before wait completes: UID PID PPID C STIME TTY TIME CMD user 30751 30750 0 20:46 pts/5 00:00:00 sleep 10 user 30752 30750 0 20:46 pts/5 00:00:00 sleep 20 user 30753 30750 0 20:46 pts/5 00:00:00 sleep 15 user 30754 30750 0 20:46 pts/5 00:00:00 sleep 25 Processes after wait completes: UID PID PPID C STIME TTY TIME CMD debian:/tmp/test$
As you can see, execution of the sleep commands were performed in parallel. Further execution of the script paused at the wait statement until all those background sleep commands completed, then execution resumed. This allows you to run many non-dependent processes in parallel, then wait until those are completed to run a command that depends on those processes. Especially on multiprocessor systems, this can greatly reduce the wall clock running time of many scripts.
Most branching statements can be replaced by logical equivalents. For instance, the following two statements produce the same result:
if [ "$RUNFILE" = "1" ]; then echo true else echo false fi
[ "$RUNFILE" = "1" ] && echo true || echo false
The use of logical equivalents is sometimes useful, but can obfuscate code in other instances.
Sometimes you need to run a program that may not return if it has difficulty or runs into an infinite loop. For those types of programs, you can use a timeout to monitor the process that may have problems and kill it if it does not complete in time. This can be accomplished using code similar to the following.
#!/bin/bash sleep 100 & # Long running command running in the background count=0 while pgrep -f "sleep 100" # while the command is running do sleep 10 count=$((count + 1)) [[ $count -ge 4 ]] && pkill -f "sleep 100" # Kill the process if it runs too long (4 times of checking in this case) done
Counting and Series
An easy way to use a series in a shell script is the seq command, which returns a sequence of numbers. If you want values from 10-20, the following command will generate those numbers:
$ seq 10 20 10 11 12 13 14 15 16 17 18 19 20
So, you can do something like:
seq 10 20 | xargs -iV ls -l fileV.txt
and get a list of files from 10-20 in the form file10.txt, file11.txt ...