Using "flock" to Prevent Multiple Instances of a Script from Running

In a previous post, I wrote about how you can use the lockfile command to ensure that only one instance of a script is running at a time. An alternative to lockfile is the flock command, which is used as follows:

flock /path/to/mylockfile cmd

By default, if the lock cannot be immediately acquired, flock will wait indefinitely until it becomes available. However, you can use the --nonblock (or -n) flag if you want flock to fail (with an exit code of 1) rather than wait if the lock cannot be immediately acquired. You can also specify how long flock should wait by passing in a --timeout in seconds.

A convenient form of flock often used within shell scripts is to use a file descriptor, as follows:

(
flock -n 9 || exit 1
# ... commands executed under lock ...
) 9>/path/to/mylockfile

If you want to prevent multiple instances of a shell script from running simultaneously, add the following boilerplate at the top of your script, which will cause the script to lock itself automatically on first run:

[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -en "$0" "$0" "$@" || :

Related posts:
Using "lockfile" to Prevent Multiple Instances of a Script from Running
Retrying Commands in Shell Scripts
Executing a Shell Command with a Timeout

Shell Scripting: Parsing options using getopt and getopts

This post shows how you can parse shell options using getopts and getopt.

Using getopts:

getopts is a bash built-in command. I find it a lot easier to use than getopt.

Here is an example of using getopts:

# options a and b are followed by a colon because they require arguments
while getopts "ha:b:" opt; do
  case "$opt" in
    h)
      echo "help"
     ;;
    a)
      option_a=$OPTARG
      ;;
    b)
      option_b=$OPTARG
      ;;
  esac
done
shift $((OPTIND-1))

echo "option_a: $option_a"
echo "option_b: $option_b"

# read positional parameters
echo "Param 1: $1"
echo "Param 2: $2"

Running it:

$ myscript.sh -a foo -b bar hello world
option_a: foo
option_b: bar
Param 1: hello
Param 2: world

Using getopt:

getopt supports long options and that's the only time I use it.

Here is an example of using getopt:

options=$(getopt -n "$0" -o ha:b: -l "help,alpha:,bravo:"  -- "$@")
(( $? != 0 )) && echo "Incorrect options provided" >&2 && exit 1
eval set -- "$options"

while true; do
  case "$1" in
    -h|--help)
        echo "help"
        ;;
    -a|--alpha)
        shift
        option_a="$1"
        ;;
    -b|--bravo)
        shift
        option_b="$1"
        ;;
    --)
        shift
        break
        ;;
  esac
  shift
done

echo "option_a: $option_a"
echo "option_b: $option_b"

# read positional parameters
echo "Param 1: $1"
echo "Param 2: $2"

Running it:

$ myscript.sh --alpha foo --bravo bar hello world
option_a: foo
option_b: bar
Param 1: hello
Param 2: world

Converting CSV to JSON in JavaScript

This post shows how you can convert a simple CSV file to JSON in JavaScript.

Consider the following sample CSV:

author,title,publishDate
Dan Simmons,Hyperion,1989
Douglas Adams,The Hitchhiker's Guide to the Galaxy,1979

The desired JSON output is:

[{"author":"Dan Simmons","title":"Hyperion","publishDate":"1989"},
{"author":"Douglas Adams","title":"The Hitchhiker's Guide to the Galaxy","publishDate":"1979"}]

The following JavaScript function transforms CSV into JSON. (Note that this implementation is quite naive and will not handle quoted fields containing commas!)

function toJson(csvData) {
  var lines = csvData.split("\n");
  var colNames = lines[0].split(",");
  var records=[];
  for(var i = 1; i < lines.length; i++) {
    var record = {};
    var bits = lines[i].split(",");
    for (var j = 0 ; j < bits.length ; j++) {
      record[colNames[j]] = bits[j];
    }
    records.push(record);
  }
  return records;
}

A simple test:

csv="author,title,publishDate\nDan Simmons,Hyperion,1989\nDouglas Adams,The Hitchhiker's Guide to the Galaxy,1979";
json = toJson(csv);
console.log(JSON.stringify(json));

To read a CSV file in JavaScript and convert it to JSON:

var rawFile = new XMLHttpRequest();
rawFile.open("GET", "books.csv", true);
rawFile.onreadystatechange = function () {
  if (rawFile.readyState === 4) {
    if (rawFile.status === 200 || rawFile.status == 0) {
      var allText = rawFile.responseText;
      var result = toJson(allText);
      console.log(JSON.stringify(result));
    }
  }
}
rawFile.send(null);

You might also like:
Converting XML to CSV using XSLT 1.0

Bash Redirection and Piping Shortcuts

Redirecting both stdout and stderr

In order to redirect both standard output and standard error to a file, you would traditionally do this:

my_command > file 2>&1

A shorter way to write the same thing is by using &> (or >&) as shown below:

my_command &> file

Similarly, to append both standard output and standard error to a file, use &>>:

my_command &>> file

Piping both stdout and stderr

To pipe both standard output and standard error, you would traditionally do this:

my_command 2>&1 | another_command

A shorter way is to use |& as shown below:

my_command |& another_command

Other posts you might like:
Shell Scripting - Best Practices
All posts with label: bash

Use sqsh, not isql!

Sqsh is a sql shell and a far superior alternative to the isql program supplied by Sybase. It's main advantage is that it allows you to combine sql and unix shell commands! Here are a few reason why I love it:

1. Pipe data to other programs
You can create a pipeline to pass SQL results to an external (or unix) program like less, grep, head etc. Here are a few examples:

# pipe to less to browse data
1> select * from data; | less

# a more complex pipeline which gzips data containing a specific word
2> select * from data; | grep -i foo | gzip -c > /tmp/foo.gz

# this example shows the use of command substitution
3> sp_who; | grep `hostname`

2. Redirect output to file
Just like in a standard unix shell, you can redirect output of a sql command to file:

# write the output to file
1> sp_helptext my_proc; > /tmp/my_proc.txt

3. Functions and aliases
You can define aliases and functions in your ~/.sqshrc file for code that you run frequently. Some of mine are shown below. (Visit my GitHub dotfiles repository to see my full .sqshrc.)

\alias h='\history'

# shortcut for select * from
\func -x sf
    \if [ $# -eq 0 ]
        \echo 'usage: sf "[table [where ...]]"'
        \return 1
    \fi
    select * from $*; | less -F
\done

# count rows in a table
\func -x count
    \if [ $# -eq 0 ]
        \echo 'usage: count "[table [where ...]]"'
        \return 1
    \fi
    select count(*) from $*;
\done

You can invoke them like this:

# select * from data table
1> sf "data where date='20140306'"

# count the rows in the employee table
2> count employee

# list aliases
3> \alias

4. History and reverse search

You can rerun a previous command by using the \history command or by invoking reverse search with Ctrl+r:

1> \history
(1) sp_who
(2) select count(*) from data
(3) select top 10 * from data

# invoke the second command from history
2> !2

# invoke the previous command
3> !!

# reverse search
4> <Ctrl+r>
(reverse-i-search)`sp': sp_who
4> sp_who

5. Customisable prompt
The default prompt is ${lineno}> , but it can be customised to include your username and database, and it even supports colours. It would be nice if there was a way to change the colour based on which database you were connected to (for example, red for a production database), but I haven't been able to figure out if this is possible yet. Here is my prompt, set in my ~/.sqshrc:

\set prompt_color='{1;33}' # yellow
\set text_color='{0;37}'   # white
\set prompt='${prompt_color}[$histnum][$username@$DSQUERY.$database] $lineno >$text_color '

6. Different result display styles
sqsh supports a number of different output styles which you can easily switch to. The ones I frequently use are csv, html and vert (vertical). Here is an example:

1> select * from employee; -m csv
123,"Joe","Bloggs"

2>select * from employee; -m vert
id:        123
firstName: Joe
lastName:  Bloggs

7. For-loops
A for-loop allows you to iterate over a range of values and execute some code. For example, if you want to delete data, in batches, over a range of dates, you can use a for-loop like this:

\for i in 1 2 3 4 5
    \loop -e "delete from data where date = '2014020$i';"
    \echo "Deleted 2014020$i"
\done

8. Backgrounding long-running commands
If you have a long-running command, you can run it in the background by putting an & at the end of the command. You can then continue running other commands, whilst this one runs in the background. You will see a message when the background command completes and you can use \show to see the results. Here is an example:

# run a command in the background
1> select * from data; &
Job #1 running [6266]

Job #1 complete (output pending)

# show the results of the backgrounded command
3> \show 1

Further information:
You can download sqsh here and then read the man page for more information.
You can take a look at my .sqshrc in my GitHub dotfiles repository.

Using "lockfile" to Prevent Multiple Instances of a Script from Running

This post describes how you can ensure that only one instance of a script is running at a time, which is useful if your script:

• uses significant CPU or IO and running multiple instances at the same time would risk overloading the system, or
• writes to a file or other shared resource and running multiple instances at the same time would risk corrupting the resource

In order to prevent multiple instances of a script from running, your script must first acquire a "lock" and hold on to that lock until the script completes. If the script cannot acquire the lock, it must wait until the lock becomes available. So, how do you acquire a lock? There are different ways, but the simplest is to use the lockfile command to create a "semaphore file". This is shown in the snippet below:

#!/bin/bash
set -e

# waits until a lock is acquired and
# deletes the lock on exit.
# prevents multiple instances of the script from running
acquire_lock() {
    lock_file=/var/tmp/foo.lock
    echo "Acquiring lock ${lock_file}..."
    lockfile "${lock_file}"
    trap "rm -f ${lock_file} && echo Released lock ${lock_file}" INT TERM EXIT
    echo "Acquired lock"
}

acquire_lock
# do stuff

The acquire_lock function first invokes the lockfile command in order to create a file. If lockfile cannot create the file, it will keep trying forever until it does. You can use the -r option if you only want to retry a certain number of times. Once the file has been created, we need to ensure that it is deleted once the script completes or is terminated. This is done using the trap command, which deletes the file when the script completes or when the shell receives an interrupt or terminate signal. I also like to use set -e in all my scripts, which makes the script exit if any command fails. In this case, if lockfile fails, the script will exit and the trap will not be set.

lockfile can be used in other ways as well. For example, instead of preventing multiple instances of the entire script from running, you may want to use a more granular approach and use locks only around those parts of your script which are not safe to run concurrently.

Note, that if you cannot use lockfile, there are other alternatives such as using mkdir or flock as described in BashFAQ/045.

Other posts you might like:
Shell Scripting - Best Practices
Retrying Commands in Shell Scripts
Executing a Shell Command with a Timeout

Retrying Commands in Shell Scripts

There are many cases in which you may wish to retry a failed command a certain number of times. Examples are database failures, network communication failures or file IO problems.

The snippet below shows a simple method of retrying commands in bash:

#!/bin/bash

MAX_ATTEMPTS=5
attempt_num=1
until command || (( attempt_num == MAX_ATTEMPTS ))
do
    echo "Attempt $attempt_num failed! Trying again in $attempt_num seconds..."
    sleep $(( attempt_num++ ))
done

In this example, the command is attempted a maximum of five times and the interval between attempts is increased incrementally whenever the command fails. The time between the first and second attempt is 1 second, that between the second and third is 2 seconds and so on. If you want, you can change this to a constant interval or random exponential backoff instead.

I have created a useful retry function (shown below) which allows me to retry commands from different places in my script without duplicating the retry logic. This function returns a non-zero exit code when all attempts have been exhausted.

#!/bin/bash

# Retries a command on failure.
# $1 - the max number of attempts
# $2... - the command to run
retry() {
    local -r -i max_attempts="$1"; shift
    local -r cmd="$@"
    local -i attempt_num=1

    until $cmd
    do
        if (( attempt_num == max_attempts ))
        then
            echo "Attempt $attempt_num failed and there are no more attempts left!"
            return 1
        else
            echo "Attempt $attempt_num failed! Trying again in $attempt_num seconds..."
            sleep $(( attempt_num++ ))
        fi
    done
}

# example usage:
retry 5 ls -ltr foo

Related Posts:
Executing a Shell Command with a Timeout
Retrying Operations in Java

Shell Scripting - Best Practices

Most programming languages have a set of "best practices" that should be followed when writing code in that language. However, I have not been able to find a comprehensive one for shell scripting so have decided to write my own based on my experience writing shell scripts over the years.

A note on portability: Since I mainly write shell scripts to run on systems which have Bash 4.2 installed, I don't need to worry about portability much, but you might need to! The list below is written with Bash 4.2 (and other modern shells) in mind. If you are writing a portable script, some points will not apply. Needless to say, you should perform sufficient testing after making any changes based on this list :-)

Here is my list of best practices for shell scripting (in no particular order):

1. Use functions
2. Document your functions
3. Use shift to read function arguments
4. Declare your variables
5. Quote all parameter expansions
6. Use arrays where appropriate
7. Use "$@" to refer to all arguments
8. Use uppercase variable names for environment variables only
9. Prefer shell builtins over external programs
10. Avoid unnecessary pipelines
11. Avoid parsing ls
12. Use globbing
13. Use null delimited output where possible
14. Don't use backticks
15. Use process substitution instead of creating temporary files
16. Use mktemp if you have to create temporary files
17. Use [[ and (( for test conditions
18. Use commands in test conditions instead of exit status
19. Use set -e
20. Write error messages to stderr

Each one of the points above is described in some detail below.

1. Use functions

   Unless you're writing a very small script, use functions to modularise your code and make it more readable, reusable and maintainable. The template I use for all my scripts is shown below. As you can see, all code is written inside functions. The script starts off with a call to the main function.

   #!/bin/bash
   set -e
   
   usage() {
   }
   
   my_function() {
   }
   
   main() {
   }
   
   main "$@"
   

2. Document your functions

   Add sufficient documentation to your functions to specify what they do and what arguments are required to invoke them. Here is an example:

   # Processes a file.
   # $1 - the name of the input file
   # $2 - the name of the output file
   process_file(){
   }
   

3. Use shift to read function arguments

   Instead of using $1, $2 etc to pick up function arguments, use shift as shown below. This makes it easier to reorder arguments, if you change your mind later.

   # Processes a file.
   # $1 - the name of the input file
   # $2 - the name of the output file
   process_file(){
       local -r input_file="$1";  shift
       local -r output_file="$1"; shift
   }
   

4. Declare your variables

   If your variable is an integer, declare it as such. Also, make all your variables readonly unless you intend to change their value later in your script. Use local for variables declared within functions. This helps convey your intent. If portability is a concern, use typeset instead of declare. Here are a few examples:

   declare -r -i port_number=8080
   declare -r -a my_array=( apple orange )
   
   my_function() {
       local -r name=apple
   }
   

5. Quote all parameter expansions

   To prevent word-splitting and file globbing you must quote all variable expansions. In particular, you must do this if you are dealing with filenames that may contain whitespace (or other special characters). Consider this example:

   # create a file containing a space in its name
   touch "foo bar"
   
   declare -r my_file="foo bar"
   
   # try rm-ing the file without quoting the variable
   rm  $my_file
   # it fails because rm sees two arguments: "foo" and "bar"
   # rm: cannot remove `foo': No such file or directory
   # rm: cannot remove `bar': No such file or directory
   
   # need to quote the variable
   rm "$my_file"
   
   # file globbing example:
   mesg="my pattern is *.txt"
   echo $mesg
   # this is not quoted so *.txt will undergo expansion
   # will print "my pattern is foo.txt bar.txt"
   
   # need to quote it for correct output
   echo "$msg"
   
   

   It's good practice to quote all your variables. If you do need word-splitting, consider using an array instead. See the next point.

6. Use arrays where appropriate

   Don't store a collection of elements in a string. Use an array instead. For example:

   # using a string to hold a collection
   declare -r hosts="host1 host2 host3"
   for host in $hosts  # not quoting $hosts here, since we want word splitting
   do
       echo "$host"
   done
   
   # use an array instead!
   declare -r -a host_array=( host1 host2 host3 )
   for host in "${host_array[@]}"
   do
       echo "$host"
   done
   

7. Use "$@" to refer to all arguments

   Don't use $*. Refer to my previous post: Difference between $*, $@, "$*" and "$@". Here is an example:

   main() {
       # print each argument
       for i in "$@"
       do
           echo "$i"
       done
   }
   # pass all arguments to main
   main "$@"
   

8. Use uppercase variable names for ENVIRONMENT variables only

   My personal preference is that all variables should be lowercase, except for environment variables. For example:

   declare -i port_number=8080
   
   # JAVA_HOME and CLASSPATH are environment variables
   "$JAVA_HOME"/bin/java -cp "$CLASSPATH" app.Main "$port_number"
   

9. Prefer shell builtins over external programs

   The shell has the ability to manipulate strings and perform simple arithmetic so you don't need to invoke programs like cut and sed. Here are a few examples:

   declare -r my_file="/var/tmp/blah"
   
   # instead of dirname, use:
   declare -r file_dir="{my_file%/*}"
   
   # instead of basename, use:
   declare -r file_base="{my_file##*/}"
   
   # instead of sed 's/blah/hello', use:
   declare -r new_file="${my_file/blah/hello}"
   
   # instead of bc <<< "2+2", use:
   echo $(( 2+2 ))
   
   # instead of grepping a pattern in a string, use:
   [[ $line =~ .*blah$ ]]
   
   # instead of cut -d:, use an array:
   IFS=: read -a arr <<< "one:two:three"
   

   Note that an external program will perform better when operating on large files/input.

10. Avoid unnecessary pipelines

    Pipelines add extra overhead to your script so try to keep your pipelines small. Common examples of useless pipelines are cat and echo, shown below:

    1. Avoid unnecessary cat

       If you are not familiar with the infamous Useless Use of Cat award, take a look here. The cat command should only be used for concatenating files, not for sending the output of a file to another command.

       # instead of
       cat file | command
       # use
       command < file
       

    2. Avoid unnecessary echo

       You should only use echo if you want to output some text to stdout, stderr, file etc. If you want to send text to another command, don't echo it through a pipe! Use a here-string instead. Note that here-strings are not portable (but most modern shells support them) so use a heredoc if you are writing a portable script. (See my earlier post: Useless Use of Echo.)

       # instead of
       echo text | command
       # use
       command <<< text
       
       # for portability, use a heredoc
       command << END
       text
       END
       

    3. Avoid unnecessary grep

       Piping from grep to awk or sed is unnecessary. Since both awk and sed can grep, you don't need the grep in your pipeline. (Check out my previous post: Useless Use of Grep.)

       # instead of
       grep pattern file | awk '{print $1}'
       # use
       awk '/pattern/{print $1}' file
       
       # instead of
       grep pattern file | sed 's/foo/bar/g'
       # use
       sed -n '/pattern/{s/foo/bar/p}' file
       

    4. Other unnecessary pipelines

       Here are a few other examples:

       # instead of
       command | sort | uniq
       # use
       command | sort -u
       
       # instead of
       command | grep pattern | wc -l
       # use
       command | grep -c pattern
       

11. Avoid parsing ls

    The problem is that ls outputs filenames separated by newlines, so if you have a filename containing a newline character you won't be able to parse it correctly. It would be nice if ls could output null delimited filenames but, unfortunately, it can't. Instead of ls, use file globbing or an alternative command which outputs null terminated filenames, such as find -print0.

12. Use globbing

    Globbing (or filename expansion) is the shell's way of generating a list of files matching a pattern. In bash, you can make globbing more powerful by enabling extended pattern matching operators using the extglob shell option. Also, enable nullglob so that you get an empty list if no matches are found. Globbing can be used instead of find in some cases and, once again, don't parse ls! Here are a couple of examples:

    
    shopt -s nullglob
    shopt -s extglob
    
    # get all files with a .yyyymmdd.txt suffix
    declare -a dated_files=( *.[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].txt )
    
    # get all non-zip files
    declare -a non_zip_files=( !(*.zip) )
    
    

13. Use null delimited output where possible

    In order to correctly handle filenames containing whitespace and newline characters, you should use null delimited output, which results in each line being terminated by a NUL (\000) character instead of a newline. Most programs support this. For example, find -print0 outputs filenames followed by a null character and xargs -0 reads arguments separated by null characters.

    # instead of
    find . -type f -mtime +5 | xargs rm -f
    # use
    find . -type f -mtime +5 -print0 | xargs -0 rm -f
    
    # looping over files
    find . -type f -print0 | while IFS= read -r -d $'\0' filename; do
        echo "$filename"
    done
    

14. Don't use backticks

    Use $(command) instead of `command` because it is easier to nest multiple commands and makes your code more readable. Here is a simple example:

    # ugly escaping required when using nested backticks
    a=`command1 \`command2\``
    
    # $(...) is cleaner
    b=$(command1 $(command2))
    

15. Use process substitution instead of creating temporary files

    In most cases, if a command takes a file as an input, the file can be replaced by the output of another command using process substitution: <(command). This saves you from having to write out a temp file, passing that temp file to the command and finally deleting the temp file. This is shown below:

    # using temp files
    command1 > file1
    command2 > file2
    diff file1 file2
    rm file1 file2
    
    # using process substitution
    diff <(command1) <(command2)
    

16. Use mktemp if you have to create temporary files

    Try to avoid creating temporary files. If you must, use mktemp to create a temporary directory and then write your files to it. Make sure you remove the directory after you are done.

    # set up a trap to delete the temp dir when the script exits
    unset temp_dir
    trap '[[ -d "$temp_dir" ]] && rm -rf "$temp_dir"' EXIT
    
    # create the temp dir
    declare -r temp_dir=$(mktemp -dt myapp.XXXXXX)
    
    # write to the temp dir
    command > "$temp_dir"/foo
    

17. Use [[ and (( for test conditions

    Prefer [[ ... ]] over [ ... ] because it is safer and provides a richer set of features. Use (( ... )) for arithmetic conditions because it allows you to perform comparisons using familiar mathematical operators such as < and > instead of -lt and -gt. Note that if you desire portability, you have to stick to the old-fashioned [ ... ]. Here are a few examples:

    [[ $foo == "foo" ]] && echo "match"  # don't need to quote variable inside [[
    [[ $foo == "a" && $bar == "a" ]] && echo "match"
    
    declare -i num=5
    (( num < 10 )) && echo "match"       # don't need the $ on $num in ((
    

18. Use commands in test conditions instead of exit status

    If you want to check whether a command succeeded before doing something, use the command directly in the condition of your if-statement instead of checking the command's exit status.

    
    # don't use exit status
    grep -q pattern file
    if (( $? == 0 ))
    then
        echo "pattern was found"
    fi
    
    # use the command as the condition
    if grep -q pattern file
    then
        echo "pattern was found"
    fi
    

19. Use set -e

    Put this at the top of your script. This tells the shell to exit the script as soon as any statement returns a non-zero exit code.

20. Write error messages to stderr

    Error messages belong on stderr not stdout.

    echo "An error message" >&2
    

If you have any other suggestions for my list, please share them in the comments section below!

Executing a Shell Command with a Timeout

Sometimes you may want to kill a command if it has been running for more than a specific time limit. For example, a shell script connecting to a network resource may hang for a long period of time if the resource is unavailable and it would be desirable to kill it and send out an alert.

This post describes different ways of running commands with time limits.

1) GNU coreutils timeout command
The easiest way to run a command with a time limit is by using the timeout command from GNU coreutils. For example, to run a command with a timeout of 2 minutes:

$ timeout 2m /path/to/command with args
$ echo $?
124

If the command has not completed within the specified time limit, the timeout utility will kill it (by sending it a TERM signal) and then exit with status 124.

2) The expect command
Another way to run a command with a timeout is by using expect as shown below:

$ expect -c "
    set echo '-noecho';
    set timeout 10;
    spawn -noecho /path/to/command with args;
    expect timeout { exit 124 } eof { exit 0 }"
$ echo $?
124

In the example above, the timeout is set to 10 seconds and expect will exit with a status of 124 when the command exceeds this time limit. Otherwise, it will exit with a status of 0. Unfortunately, you lose the exit code of the command you are running.

3) Using a custom timeout script
If you cannot use the two approaches above, you can write your own timeout script. Mine is shown below. It first starts a "watchdog" process which keeps checking to see if the command is running by executing kill -0 periodically. If it is still running after the time limit has been exceeded, the watchdog kills it.

#!/bin/bash
while getopts "t:" opt; do
  case "$opt" in
      t) timeout=$OPTARG ;;
  esac
done
shift $((OPTIND-1))

start_watchdog(){
  timeout="$1"
  (( i = timeout ))
  while (( i > 0 ))
  do
    kill -0 $$ || exit 0
    sleep 1
    (( i -= 1 ))
  done

  echo "killing process after timeout of $timeout seconds"
  kill $$
}

start_watchdog "$timeout" 2>/dev/null &
exec "$@"

Example:

$ timeout.sh -t 2 sleep 5
killing process after timeout of 2 seconds
Terminated

Useless Use of Grep

Most of us are familiar with the infamous Useless Use of Cat Award which is awarded for unnecessary use of the cat command. A while back, I also wrote about Useless Use of Echo in which I advised using here-strings and here-docs instead of the echo command. In a similar vein, this post is about the useless use of the grep command.

Useless use of grep | awk
awk can match patterns, so there is no need to pipe the output of grep to awk. For example, the following:

grep pattern file | awk '{commands}'

can be re-written as:

awk '/pattern/{commands}' file

Similarly:

grep -v pattern file | awk '{commands}'

can be re-written as:

awk '!/pattern/{commands}' file

Useless use of grep | sed
sed can match patterns, so you don't need to pipe the output of grep to sed. For example, the following:

grep pattern file | sed 's/foo/bar/g'

can be re-written as:

sed -n '/pattern/{s/foo/bar/p}' file

Similarly:

grep -v pattern file | sed 's/foo/bar/g'

can be re-written as:

sed -n '/pattern/!{s/foo/bar/p}' file

Useless use of grep in conditions
If you find yourself using grep in conditional statements to check if a string variable matches a certain pattern, consider using bash's in-built string matching instead. For example, the following:

if grep -q pattern <<< "$var"; then
    # do something
fi

can be re-written as:

if [[ $var == *pattern* ]]; then
    # do something
fi

or, if your pattern is a regex, rather than a fixed string, use:

if [[ $var =~ pattern ]]; then
    # do something
fi

Useless Use of Echo

Most of us are familiar with the Useless Use of Cat Award which is awarded for unnecessary use of the cat command. For example, in nearly all cases, cat file | command arg can be rewritten as <file command arg.

In a similar vein, this post is about the useless use of the echo command. In nearly all cases:

echo string | command arg

can be rewritten using a heredoc:

command arg << END
string
END

or, using a here-string:

command arg <<< string

Note: Here-strings are not portable (but most modern shells support them) so use the heredoc alternative shown above if you are writing a portable script!

Viewing CSV Files

I find CSV (or, to be more general, DSV) files difficult to read on Unix because you can't tell which column a value is in. So I always end up importing them into a spreadsheet which is a pain. Here is an example of a small pipe-delimited file containing book data:

Title|Author|CoAuthor|Year|ISBN
Carrie|Stephen King||1974|978-0385086950
The Human Web|William McNeill|Robert McNeill|2003|978-0393051797

It would be a lot easier to read, if I could convert the file into dictionaries of key:value pairs in order to see which columns the values were referring to, like this:

Title:Carrie
Author:Stephen King
CoAuthor:
Year:1974
ISBN:978-0385086950

Title:The Human Web
Author:William McNeill
CoAuthor:Robert McNeill
Year:2003
ISBN:978-0393051797

So, I wrote the following Bash script to convert a delimiter separated file into a collection of dictionaries. It uses awk to read the first row, which contains the column names, split it and store it in an array. It then prints out the remaining rows along with their column names which are looked up from the array.

#! /bin/bash
# CSV Viewer
# Usage: csv [-d delim] filename
# default delimiter is pipe.
#
# Input file:
# h1|h2|h3
# v1|v2|v3
# w1|w2|w3
#
# Output:
# h1: v1
# h2: v2
# h3: v3
#
# h1: w1
# h2: w2
# h3: w3

delim=|
while getopts "d:" OPTION
do
   case $OPTION in
     d) delim=$OPTARG; shift $((OPTIND-1)) ;;
   esac
done

if [ $# -eq 0 ]
then
    echo "Usage: csv [-d delim] filename" >&2
    exit 1
fi
awk -F "$delim" '{if(NR==1)split($0,arr);else for(i=1;i<=NF;i++)print arr[i]":"$i;print "";}' "$1"

Running the script:

sharfah@starship:~> csv.sh -d '|' file
Title:Carrie
Author:Stephen King
CoAuthor:
Year:1974
ISBN:978-0385086950

Title:The Human Web
Author:William McNeill
CoAuthor:Robert McNeill
Year:2003
ISBN:978-0393051797

Null Characters

On a few occasions (normally when I am running out of disk space), I have seen ^@ symbols appear in my log files. These are "file holes" and contain null characters. The null character (or NUL char) has an ASCII code of 0 and appears as ^@ when viewed in 'vi' or 'less'.

Create a dummy file containing null characters:
In order to create a file with null characters, simply print \000. For example:

sharfah@starship:~> perl -e \
'print "hello \000world\000\nfoo bar\n";' > file-with-nulls
sharfah@starship:~> less file-with-nulls
hello ^@world^@
foo bar

Find lines containing null characters:
Use the following command in order print out lines containing null characters:

sharfah@starship:~> perl -ne '/\000/ and print;' file-with-nulls \
| less
hello ^@world^@

You can also perform an octal dump of the file to check if it has null characters:

sharfah@starship:~> od -b file-with-nulls | grep ' 000'
0000000 150 145 154 154 157 040 000 167 157 162 154 144 000 012 146 157

Delete null characters:
There are various ways in which this can be done. In the following examples, I have used tr and sed to remove the unwanted characters.

sharfah@starship:~> tr -d '\000' < file-with-nulls | less
hello world
foo bar
sharfah@starship:~> sed 's/\x0//g' < file-with-nulls | less
hello world
foo bar

Do not use the strings command because it will create a newline when it encounters a null characeter. "The strings utility looks for ASCII strings in a binary file. A string is any sequence of 4 or more printing characters ending with a newline or a null character."

sharfah@starship:~> strings file-with-nulls | less
hello
world
foo bar

Further reading:
ASCII Character Set
Identifying and removing null characters in UNIX [stackoverflow]
File holes

Difference between $*, $@, "$*" and "$@"

They are all related to "all the arguments to the shell", but behave differently. The following script demonstrates each one of $*, $@, "$@" and "$*" in turn by echoing out the arguments passed in:

#! /bin/bash

echo +--- Using "\$*"

cnt=1
for i in $*
do
  echo arg$cnt: $i
  cnt=$(($cnt+1))
done

echo +--- Using \"\$*\"

cnt=1
for i in "$*"
do
  echo arg$cnt: $i
  cnt=$(($cnt+1))
done

echo +--- Using "\$@"

cnt=1
for i in $@
do
  echo arg$cnt: $i
  cnt=$(($cnt+1))
done

echo +--- Using \"\$@\"

cnt=1
for i in "$@"
do
  echo arg$cnt: $i
  cnt=$(($cnt+1))
done

Running the script, produces the following output:

sharfah@starship:~> echoArgs.sh  mars "kit kat" twix
+--- Using $*
arg1: mars
arg2: kit
arg3: kat
arg4: twix
+--- Using "$*"
arg1: mars kit kat twix
+--- Using $@
arg1: mars
arg2: kit
arg3: kat
arg4: twix
+--- Using "$@"
arg1: mars
arg2: kit kat
arg3: twix

When unquoted, $* and $@ do the same thing. They treat each "word" (sequence of non-whitespace characters) as a separate argument. This leads to the single argument "kit kat" being broken into two which may not be desired. When quoted, $* and $@ behave differently. "$*" treats the entire argument list as a single argument, but "$@" treats the arguments just as they were when passed in.

So, which should you use? In almost all cases you would use "$@" in order to preserve arguments with spaces (or quoted arguments) when being passed in on the command line or from one script to another. Never use the unquoted $* or $@ unless you are absolutely sure that you won't need to deal with arguments with spaces in them. If you need to combine all arguments into one, use "$*".

Unix Wrapper Script for Java Applications

All of my Java applications are deployed onto a Unix (Linux or Solaris) environment and so I need to have a unix wrapper shell script in order to start and stop them. I looked at Java Service Wrapper but I wanted something simpler. So I wrote my own little wrapper, based on JSW.

The wrapper allows you to invoke the following set of commands on your application:

• start: Starts up the application and writes its pid to a pid file. it also writes the hostname to the pid file. This is important, in case you try to run commands on the application on a different server. After starting, it waits a number of seconds and then checks to see if the application is still running.
• stop: Stops the application using the kill command. If it has not died within 10 seconds, it executes the harder kill -9 command.
• restart: Restarts the application by calling stop and then start.
• status:Reads the pid file and invokes kill -0 on the pid to see if the process is alive. If the process is not alive, an email alert is generated, if the $EMAIL variable has been set.
• dump: Generates a full Java thread dump by invoking kill -3 on the pid
• purge: Purges log files

Here is the wrapper script (bin/console.ksh):

#! /bin/ksh
# A generic wrapper script to stop and start components.
# Usage: console.ksh { start | stop | restart | status | purge | dump }
#

APP_NAME="APP"
APP_LONG_NAME="MyApplication"

#the location of the pid files.
PIDDIR="../pid"
PIDFILE="$PIDDIR/$APP_NAME.pid"
TIMESTAMP=$( date +%Y%m%d_%H%M%S )

#number of seconds to wait after starting up.
WAIT_AFTER_STARTUP=2

#the location of the log files
LOG_DIR="../log"
STDOUT="$LOG_DIR/${APP_NAME}.stdout.log"

PURGE_EXPRESSION="-mtime +2"

VMARGS="-Xms250M -Xmx3000M -verbose:gc"
MAINCLASS="com.blogspot.fahdshariff.MyApplication"
ARGS="arg1 arg2"

if [ -z ${JAVA_HOME} ]
then
 echo JAVA_HOME has not been set.
 exit 1
fi

START_COMMAND="${JAVA_HOME}/bin/java $VMARGS $MAINCLASS $ARGS"

#-----------------------------------------------------------------------------
# Do not modify anything beyond this point
#-----------------------------------------------------------------------------

getpid() {
  pid=""
  if [ -f "$PIDFILE" ]
  then
    if [ -r "$PIDFILE" ]
    then
      pid=`cat "$PIDFILE"|cut -d: -f1`
      host=`cat "$PIDFILE"|cut -d: -f2`
      currhost=`hostname`
      if [ "$host" != "$currhost" ]
      then
       echo "You are on the wrong host. $APP_LONG_NAME runs on $host."
       exit 1
      fi

      if [ "X$pid" != "X" ]
      then
       kill -0 $pid > /dev/null 2>&1
  if [ $? -ne 0 ]
  then
   # Process doesn't exist, so remove pidfile
   rm -f "$PIDFILE"
            echo "Removed stale pid file: $PIDFILE"
   pid=""
  fi
      fi
    else
      echo "Cannot read $PIDFILE."
      exit 1
    fi
  fi
}

#tests if the pid is alive
testpid() {
  kill -0 $pid > /dev/null 2>&1
  if [ $? -ne 0 ]
  then
   # Process is gone so remove the pid file.
 rm -f "$PIDFILE"
 pid=""
  fi
}

start() {
  echo "Starting $APP_LONG_NAME..."
  getpid
  if [ "X$pid" = "X" ]
  then
    if [ -s $STDOUT ]
    then
     #backup current file
     mv $STDOUT $STDOUT.sav.${TIMESTAMP}
    fi
    echo CLASSPATH is $CLASSPATH > $STDOUT
    echo ${START_COMMAND} >> $STDOUT
    nohup ${START_COMMAND} >> $STDOUT 2>&1 &
    echo $!:`hostname` > $PIDFILE
  else
    echo "$APP_LONG_NAME is already running."
    exit 1
  fi

  # Sleep for a few seconds to allow for intialization if required
  # then test to make sure we're still running.
  i=0
  while [ $i -lt $WAIT_AFTER_STARTUP ]
  do
    sleep 1
    i=`expr $i + 1`
  done
  if [ $WAIT_AFTER_STARTUP -gt 0 ]
  then
    getpid
    if [ "X$pid" = "X" ]
    then
      echo "WARNING: $APP_LONG_NAME may have failed to start."
      exit 1
    else
      echo "running ($pid)."
    fi
  else
    echo ""
  fi
}


stopit() {

  echo "Stopping $APP_LONG_NAME..."
  getpid
  if [ "X$pid" = "X" ]
  then
    echo "$APP_LONG_NAME was not running."
  else
    # Running so try to stop it.
    kill $pid
    if [ $? -ne 0 ]
    then
      echo "Unable to stop $APP_LONG_NAME."
      exit 1
    fi

    #  If it has not stopped in 10 tries, forcibly kill it
    savepid=$pid
    CNT=0
    TOTCNT=0
    while [ "X$pid" != "X" ]
    do
      # Show a waiting message every 5 seconds.
      if [ "$CNT" -lt "5" ]
      then
        CNT=`expr $CNT + 1`
      else
        echo "Waiting for $APP_LONG_NAME to exit..."
        CNT=0
      fi

      if [ $TOTCNT -gt 11 ]
      then
       echo "Killing by force (kill -9)"
       kill -9 $pid
      fi

      TOTCNT=`expr $TOTCNT + 1`
      sleep 1

      testpid
    done

    pid=$savepid
    testpid
    if [ "X$pid" != "X" ]
    then
      echo "Failed to stop $APP_LONG_NAME."
      exit 1
    else
      echo "Stopped $APP_LONG_NAME."
    fi
  fi
}

purge() {
  pcmd="find $LOG_DIR -follow -type f $PURGE_EXPRESSION -print -exec rm -f {} +"
  echo "Running purge: $pcmd"
  $pcmd
}

alert() {
  if [ "X$EMAIL" != "X" ]
  then
   echo "$APP_LONG_NAME is not running on `hostname`. Please check and restart." \
   | mailx -s "`hostname`: $APP_LONG_NAME is not running" "$EMAIL"
   echo "Sent alert to $EMAIL"
  fi
}

status() {
  getpid
  if [ "X$pid" = "X" ]
  then
    echo "$APP_LONG_NAME is not running."
    alert
    exit 1
  else
    echo "$APP_LONG_NAME is running (PID:$pid)."
    exit 0
  fi
}

dump() {
  echo "Dumping $APP_LONG_NAME..."
  getpid
  if [ "X$pid" = "X" ]
  then
    echo "$APP_LONG_NAME is not running."
  else
    kill -3 $pid

    if [ $? -ne 0 ]
    then
      echo "Failed to dump $APP_LONG_NAME."
      exit 1
    else
      echo "Dumped $APP_LONG_NAME."
    fi
  fi
}

####################

case "$1" in

  'start')
    start
    ;;

  'stop')
    stopit
    ;;

  'restart')
    stopit
    start
    ;;

  'status')
    status
    ;;

  'dump')
    dump
    ;;

  'purge')
    purge
    ;;

  *)
    echo "Usage: $0 { start | stop | restart | status | dump | purge }"
    exit 1
    ;;
esac

exit 0

Bash: Convert String to Array

This is how you can convert a string into an array:

s="124890"
for i in $(seq 0 $((${#s}-1)))
do
     arr[$i]=${s:$i:1}
done

${#s} refers to the length of the string s, which in this case is 6.
${s:$i:1} returns a substring of s, starting from $i, of length 1.

The seq command is a new one for me. Unfortunately, it doesn't seem to be available on Solaris, only Linux. From the man pages:

NAME
       seq - print a sequence of numbers

SYNOPSIS
       seq [OPTION]... LAST
       seq [OPTION]... FIRST LAST
       seq [OPTION]... FIRST INCREMENT LAST

To see how you can convert a string of digits to an array in other languages, read this stackoverflow question.

Fibonacci Shell Script

Here is a quick unix shell script which prints out the Fibonacci sequence:

0,1,1,2,3,5,8,13,21,34,55,89,144,...

The first two Fibonacci numbers are 0 and 1, and each remaining number is the sum of the previous two.

#!/bin/sh
prev=0
next=1

echo $prev

while(true)
do
 echo $next

 #add the two numbers
 sum=$(($prev+$next))

 #swap
 prev=$next
 next=$sum
 sleep 1
done

Creating a Report with SQLPlus

For those of you who have used SQL*Plus, you will know that it is a nightmare to get the output looking just the way you want it to. You have to battle with page sizes and column widths. (Why isn't there an option to set the column size automatically, I wonder?)

Here are a few things that I have learnt:

Silent Mode
Use the -s flag on your sqlplus command in order to inhibit output such as the SQL*Plus banner and prompt.

Spooling to a file
You need to spool in order to write the output of your sql commands to a file. Turn it off when you are done.

SQL> spool results.out
SQL> select 1 from dual;
SQL> spool off

Page Size
This refers to the number of rows on a single page. The default is 14 which means that after 14 lines, your table header will be repeated, which is ugly! In order to get around this, set your page size to the maximum of 50000. It would be nice if you could set it to unlimited.

SQL> show pagesize;
pagesize 24
SQL> set pagesize 50000

Line Size
This refers to how long your line can get before it wraps to the next line. If you are not sure how long your line can be, set the size to the maximum of 32767 and turn on trimspool in order to remove trailing blanks from your spooled file.

SQL> show linesize;
pagesize 80
SQL> set linesize 32767
SQL> set trimspool on

Column Size
You can specify the size of individual columns like this:

SQL> col employee_name format a40

Titles
Use TTITLE to display a heading before you run a query.

SQL> TTITLE LEFT 'My table heading'
SQL> select 1 from dual;

My table heading
         1
----------
         1

Use SKIP to skip lines e.g. SKIP 2 would be equivalent to pressing Return twice.

SQL> ttitle left 'My table heading' -
> SKIP 2 'Another heading' SKIP 2
SQL> select 1 from dual;

My table heading

Another heading

         1
----------
         1

Example script
The shell script below uses SQL*Plus to create a report.

#! /usr/bin/sh

#the file where sql output will go
OUT=/report/path/report.txt

#email this report?
EMAIL=Y

#oracle variables
ORACLE_HOME=/path/oracle/client
export ORACLE_HOME
SQLPLUS=$ORACLE_HOME/bin/sqlplus
export SQLPLUS
LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
TNS_ADMIN=/path/tnsnames
export TNS_ADMIN

#######################
#sqlplus - silent mode
#redirect /dev/null so that output is not shown on terminal
$SQLPLUS -s "user/pass@database" << END_SQL > /dev/null

SET ECHO OFF
SET TERMOUT OFF

SET PAGESIZE 50000

SET LINESIZE 32767
SET TRIMSPOOL ON

COL EMPLOYEE_NAME FORMAT A40

SPOOL $OUT

TTITLE LEFT 'EMPLOYEE REPORT' -
SKIP 2 LEFT 'Number of Employees:' SKIP 2

SELECT COUNT(*) AS total FROM employee
/

TTITLE LEFT 'Employee Names' SKIP 2

SELECT employee_name FROM employee
ORDER BY employee_name DESC
/

SPOOL OFF

END_SQL
#######################

#change tabs to spaces
expand $OUT > $OUT.new
mv $OUT.new $OUT

echo Finished writing report $OUT

if [ "$EMAIL" = "Y" ]
then
 to=someone@abc.com
 subject="Employee Report"
 mailx -s "$subject" $to < $OUT
 echo "Emailed report"
fi

Named Pipes with mkfifo [Unix]

Named pipes are useful for inter-process communication. Unlike anonymous pipes, any number of readers and writers can use a named pipe. They are very useful for letting other processes know that something has happened e.g. a file has been created etc.

Creating a named pipe
You can create a named pipe using the mkfifo command, which creates a special pipe file that remains in place until it is removed. Since it is a type of file, you can use the rm command to remove it when you are done.

sharfah@firefly:~> mkfifo mypipe
sharfah@firefly:~> ls
mypipe|

Writing to a named pipe
Since a named pipe is just a special type of file, you can write to it just as you would normally write to a file. However, if there are no processes reading from the pipe, the write call will block.

The following example script writes numbers into the pipe. If there are no readers, it will block on line 5.

COUNT=1
while (true)
do
 echo Writer$$: $COUNT
 echo $COUNT > mypipe
 COUNT=`expr $COUNT + 1`
 sleep 1
done

Reading from a named pipe
Reading from a named pipe is the same as reading from a normal file. You can cat a named pipe, tail it or read it as follows:

while (true)
do
   read line < mypipe
   echo Reader$$: $line
done

Multiple readers
If you have multiple readers reading from the same pipe, only one of the readers will receive the output. This is illustrated with the following example, in which I have launched one writer and two readers:

sharfah@firefly:~> writer.sh& reader.sh& reader.sh&
Writer10500: 1 
Reader10501: 1 
Writer10500: 2 
Reader10502: 
Reader10501: 2 
Writer10500: 3 
Reader10501: 3 
Reader10502: 
Writer10500: 4 
Reader10501: 
Reader10502: 4 
Writer10500: 5 
Reader10502: 5 
Reader10501: 
Writer10500: 6 
Reader10501: 6 
Reader10502: 
Writer10500: 7 
Reader10502: 7 
Reader10501: 
Writer10500: 8 
Reader10502: 8 
Reader10501: 
Writer10500: 9 
Reader10501: 9 
Reader10502: 
Writer10500: 10 
Reader10502: 10 

Get Yesterday's Date [Scripting]

Here is a useful snippet that you can include in your shell scripts if you need to find out what the previous day's date was.

Perl code:

$secsInADay=86400;
@yest=localtime(time-86400);
$year=$yest[5]+1900;
$month=$yest[4]+1;
$day=$yest[3];
print "$year$month$day";

One-liner for Shell Scripts

YEST=`perl -w -e '@yest=localtime(time-86400);printf "%d%.2d%.2d",$yest[5]+1900,$yest[4]+1,$yest[3];'`
echo $YEST

To get Tomorrow's Date

TOM=`perl -w -e '@tom=localtime(time+86400);printf "%d%.2d%.2d",$tom[5]+1900,$tom[4]+1,$tom[3];'`
echo $TOM