Unless otherwise noted, articles © 2005-2008 Doug Spencer, SecurityBulletins.com. Linking to articles is welcomed. Articles on this site are general information and are NOT GUARANTEED to work for your specific needs. I offer paid professional consulting services and will be happy to develop custom solutions for your specific needs. View the consulting page for more information.

---

From SecurityBulletins.com

Shell Scripting Tips Written by Doug Spencer 12/05/2006

These are a few shell scripting tips to increase the ease and effectiveness of your scripts. This is by no means comprehensive, it is just an overview of some of the forgotten or little known commands that can be used to get things done. Most of these are BASH commands, but some work in Korn shell or Bourne shell as well.

Contents 1 xargs 2 wait 3 Logical Equivalency 4 Timeouts 5 Counting and Series

xargs

First, one of my favorite UNIX commands: xargs. This command is great for performing repetitious tasks. Most looping stanzas can be replaced with a single line using xargs. The xargs command can be used in the following ways with the following input:

$ cat input.txt 
Item1
Item2
Item3

$ cat input.txt | xargs echo 
Item1 Item2 Item3

In the instance shown above, you see the input pipe to xargs puts the input on the command line as a space separated by default. The result of the command shown above is that the input is echoed to stdout.

$ cat input.txt | xargs -iZ echo This Z is on sale
This Item1 is on sale
This Item2 is on sale
This Item3 is on sale

In the instance shown above, you see that the character "Z" is replaced by the input. A different character can be specified after the -i if you wan to use something else. This usage of xargs can be useful for a variety of instances.

Here is an example of how xargs can be used. You want to delete all files except those containing "mpeg" from the following files.

$ ls
lambda  last     latex       lav2wav    lavinfo  lavrec
lame    lastb    lav2avi.sh  lav2yuv    lavpipe  lavtrans
lamed   lastlog  lav2mpeg    lavaddwav  lavplay  lavvideo

You can do the following to verify you have selected the correct files to delete:

$ ls | grep -v mpeg
lambda
last
lastb
lastlog
latex
lav2avi.sh
lav2wav
lav2yuv
lavaddwav
lavinfo
lavpipe
lavplay
lavrec
lavtrans
lavvideo

Then, when it is correct, run the following:

$ ls | grep -v mpeg | xargs rm 
$ ls 
lav2mpeg

You have now deleted all files in that directory except lav2mpeg, the only one that contained "mpeg" in the list. If you had additional files that should be skipped, simply grep them out in addition to the mpeg item.

wait

The wait command will pause further execution of a script until processes running in the background finish. This is useful for running multiple tasks in parallel to facilitate quicker completion of a task. The wait command is a BASH builtin command and is also available in Korn shell. The following scipt will demonstrate using wait command.

#!/bin/bash
echo This script demonstrates the wait command
echo We will now start several long-running commands in the background

sleep 10 &
sleep 20 & 
sleep 15 & 
sleep 25 & 
echo Processes before wait completes:
ps -fC sleep

wait
echo Processes after wait completes:
ps -fC sleep

The output of running this script is:

debian:/tmp/test$ ./wc 
This script demonstrates the wait command
We will now start several long-running commands in the background
Processes before wait completes:
UID        PID  PPID  C STIME TTY          TIME CMD
user  30751 30750  0 20:46 pts/5    00:00:00 sleep 10
user  30752 30750  0 20:46 pts/5    00:00:00 sleep 20
user  30753 30750  0 20:46 pts/5    00:00:00 sleep 15
user  30754 30750  0 20:46 pts/5    00:00:00 sleep 25


Processes after wait completes:
UID        PID  PPID  C STIME TTY          TIME CMD
debian:/tmp/test$ 

As you can see, execution of the sleep commands were performed in parallel. Further execution of the script paused at the wait statement until all those background sleep commands completed, then execution resumed. This allows you to run many non-dependent processes in parallel, then wait until those are completed to run a command that depends on those processes. Especially on multiprocessor systems, this can greatly reduce the wall clock running time of many scripts.

Logical Equivalency

Most branching statements can be replaced by logical equivalents. For instance, the following two statements produce the same result:

   if [ "$RUNFILE" = "1" ]; then
       echo true
   else
       echo false
   fi

and

[ "$RUNFILE" = "1" ] && echo true || echo false

The use of logical equivalents is sometimes useful, but can obfuscate code in other instances.

Timeouts

Sometimes you need to run a program that may not return if it has difficulty or runs into an infinite loop. For those types of programs, you can use a timeout to monitor the process that may have problems and kill it if it does not complete in time. This can be accomplished using code similar to the following.

#!/bin/bash

sleep 100 & # Long running command running in the background
count=0
while pgrep -f "sleep 100" # while the command is running
do
  sleep 10
  count=$((count + 1))
  [[ $count -ge 4 ]] && pkill -f "sleep 100" # Kill the process if it runs too long (4 times of checking in this case)
done

Counting and Series

An easy way to use a series in a shell script is the seq command, which returns a sequence of numbers. If you want values from 10-20, the following command will generate those numbers:

$ seq 10 20
10
11
12
13
14
15
16
17
18
19
20

So, you can do something like:

seq 10 20 | xargs -iV ls -l fileV.txt

and get a list of files from 10-20 in the form file10.txt, file11.txt ...