Sunday, October 20, 2013

Shell Scripting - Best Practices

Most programming languages have a set of "best practices" that should be followed when writing code in that language. However, I have not been able to find a comprehensive one for shell scripting so have decided to write my own based on my experience writing shell scripts over the years.

A note on portability: Since I mainly write shell scripts to run on systems which have Bash 4.2 installed, I don't need to worry about portability much, but you might need to! The list below is written with Bash 4.2 (and other modern shells) in mind. If you are writing a portable script, some points will not apply. Needless to say, you should perform sufficient testing after making any changes based on this list :-)

Here is my list of best practices for shell scripting (in no particular order):

  1. Use functions
  2. Document your functions
  3. Use shift to read function arguments
  4. Declare your variables
  5. Quote all parameter expansions
  6. Use arrays where appropriate
  7. Use "$@" to refer to all arguments
  8. Use uppercase variable names for environment variables only
  9. Prefer shell builtins over external programs
  10. Avoid unnecessary pipelines
  11. Avoid parsing ls
  12. Use globbing
  13. Use null delimited output where possible
  14. Don't use backticks
  15. Use process substitution instead of creating temporary files
  16. Use mktemp if you have to create temporary files
  17. Use [[ and (( for test conditions
  18. Use commands in test conditions instead of exit status
  19. Use set -e
  20. Write error messages to stderr

Each one of the points above is described in some detail below.

  1. Use functions

    Unless you're writing a very small script, use functions to modularise your code and make it more readable, reusable and maintainable. The template I use for all my scripts is shown below. As you can see, all code is written inside functions. The script starts off with a call to the main function.

    #!/bin/bash
    set -e
    
    usage() {
    }
    
    my_function() {
    }
    
    main() {
    }
    
    main "$@"
    
  2. Document your functions

    Add sufficient documentation to your functions to specify what they do and what arguments are required to invoke them. Here is an example:

    # Processes a file.
    # $1 - the name of the input file
    # $2 - the name of the output file
    process_file(){
    }
    
  3. Use shift to read function arguments

    Instead of using $1, $2 etc to pick up function arguments, use shift as shown below. This makes it easier to reorder arguments, if you change your mind later.

    # Processes a file.
    # $1 - the name of the input file
    # $2 - the name of the output file
    process_file(){
        local -r input_file="$1";  shift
        local -r output_file="$1"; shift
    }
    
  4. Declare your variables

    If your variable is an integer, declare it as such. Also, make all your variables readonly unless you intend to change their value later in your script. Use local for variables declared within functions. This helps convey your intent. If portability is a concern, use typeset instead of declare. Here are a few examples:

    declare -r -i port_number=8080
    declare -r -a my_array=( apple orange )
    
    my_function() {
        local -r name=apple
    }
    
  5. Quote all parameter expansions

    To prevent word-splitting and file globbing you must quote all variable expansions. In particular, you must do this if you are dealing with filenames that may contain whitespace (or other special characters). Consider this example:

    # create a file containing a space in its name
    touch "foo bar"
    
    declare -r my_file="foo bar"
    
    # try rm-ing the file without quoting the variable
    rm  $my_file
    # it fails because rm sees two arguments: "foo" and "bar"
    # rm: cannot remove `foo': No such file or directory
    # rm: cannot remove `bar': No such file or directory
    
    # need to quote the variable
    rm "$my_file"
    
    # file globbing example:
    mesg="my pattern is *.txt"
    echo $mesg
    # this is not quoted so *.txt will undergo expansion
    # will print "my pattern is foo.txt bar.txt"
    
    # need to quote it for correct output
    echo "$msg"
    
    

    It's good practice to quote all your variables. If you do need word-splitting, consider using an array instead. See the next point.

  6. Use arrays where appropriate

    Don't store a collection of elements in a string. Use an array instead. For example:

    # using a string to hold a collection
    declare -r hosts="host1 host2 host3"
    for host in $hosts  # not quoting $hosts here, since we want word splitting
    do
        echo "$host"
    done
    
    # use an array instead!
    declare -r -a host_array=( host1 host2 host3 )
    for host in "${host_array[@]}"
    do
        echo "$host"
    done
    
  7. Use "$@" to refer to all arguments

    Don't use $*. Refer to my previous post: Difference between $*, $@, "$*" and "$@". Here is an example:

    main() {
        # print each argument
        for i in "$@"
        do
            echo "$i"
        done
    }
    # pass all arguments to main
    main "$@"
    
  8. Use uppercase variable names for ENVIRONMENT variables only

    My personal preference is that all variables should be lowercase, except for environment variables. For example:

    declare -i port_number=8080
    
    # JAVA_HOME and CLASSPATH are environment variables
    "$JAVA_HOME"/bin/java -cp "$CLASSPATH" app.Main "$port_number"
    
  9. Prefer shell builtins over external programs

    The shell has the ability to manipulate strings and perform simple arithmetic so you don't need to invoke programs like cut and sed. Here are a few examples:

    declare -r my_file="/var/tmp/blah"
    
    # instead of dirname, use:
    declare -r file_dir="{my_file%/*}"
    
    # instead of basename, use:
    declare -r file_base="{my_file##*/}"
    
    # instead of sed 's/blah/hello', use:
    declare -r new_file="${my_file/blah/hello}"
    
    # instead of bc <<< "2+2", use:
    echo $(( 2+2 ))
    
    # instead of grepping a pattern in a string, use:
    [[ $line =~ .*blah$ ]]
    
    # instead of cut -d:, use an array:
    IFS=: read -a arr <<< "one:two:three"
    

    Note that an external program will perform better when operating on large files/input.

  10. Avoid unnecessary pipelines

    Pipelines add extra overhead to your script so try to keep your pipelines small. Common examples of useless pipelines are cat and echo, shown below:

    1. Avoid unnecessary cat

      If you are not familiar with the infamous Useless Use of Cat award, take a look here. The cat command should only be used for concatenating files, not for sending the output of a file to another command.

      # instead of
      cat file | command
      # use
      command < file
      
    2. Avoid unnecessary echo

      You should only use echo if you want to output some text to stdout, stderr, file etc. If you want to send text to another command, don't echo it through a pipe! Use a here-string instead. Note that here-strings are not portable (but most modern shells support them) so use a heredoc if you are writing a portable script. (See my earlier post: Useless Use of Echo.)

      # instead of
      echo text | command
      # use
      command <<< text
      
      # for portability, use a heredoc
      command << END
      text
      END
      
    3. Avoid unnecessary grep

      Piping from grep to awk or sed is unnecessary. Since both awk and sed can grep, you don't need the grep in your pipeline. (Check out my previous post: Useless Use of Grep.)

      # instead of
      grep pattern file | awk '{print $1}'
      # use
      awk '/pattern/{print $1}' file
      
      # instead of
      grep pattern file | sed 's/foo/bar/g'
      # use
      sed -n '/pattern/{s/foo/bar/p}' file
      
    4. Other unnecessary pipelines

      Here are a few other examples:

      # instead of
      command | sort | uniq
      # use
      command | sort -u
      
      # instead of
      command | grep pattern | wc -l
      # use
      command | grep -c pattern
      
  11. Avoid parsing ls

    The problem is that ls outputs filenames separated by newlines, so if you have a filename containing a newline character you won't be able to parse it correctly. It would be nice if ls could output null delimited filenames but, unfortunately, it can't. Instead of ls, use file globbing or an alternative command which outputs null terminated filenames, such as find -print0.

  12. Use globbing

    Globbing (or filename expansion) is the shell's way of generating a list of files matching a pattern. In bash, you can make globbing more powerful by enabling extended pattern matching operators using the extglob shell option. Also, enable nullglob so that you get an empty list if no matches are found. Globbing can be used instead of find in some cases and, once again, don't parse ls! Here are a couple of examples:

    
    shopt -s nullglob
    shopt -s extglob
    
    # get all files with a .yyyymmdd.txt suffix
    declare -a dated_files=( *.[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].txt )
    
    # get all non-zip files
    declare -a non_zip_files=( !(*.zip) )
    
    
  13. Use null delimited output where possible

    In order to correctly handle filenames containing whitespace and newline characters, you should use null delimited output, which results in each line being terminated by a NUL (\000) character instead of a newline. Most programs support this. For example, find -print0 outputs filenames followed by a null character and xargs -0 reads arguments separated by null characters.

    # instead of
    find . -type f -mtime +5 | xargs rm -f
    # use
    find . -type f -mtime +5 -print0 | xargs -0 rm -f
    
    # looping over files
    find . -type f -print0 | while IFS= read -r -d $'\0' filename; do
        echo "$filename"
    done
    
  14. Don't use backticks

    Use $(command) instead of `command` because it is easier to nest multiple commands and makes your code more readable. Here is a simple example:

    # ugly escaping required when using nested backticks
    a=`command1 \`command2\``
    
    # $(...) is cleaner
    b=$(command1 $(command2))
    
  15. Use process substitution instead of creating temporary files

    In most cases, if a command takes a file as an input, the file can be replaced by the output of another command using process substitution: <(command). This saves you from having to write out a temp file, passing that temp file to the command and finally deleting the temp file. This is shown below:

    # using temp files
    command1 > file1
    command2 > file2
    diff file1 file2
    rm file1 file2
    
    # using process substitution
    diff <(command1) <(command2)
    
  16. Use mktemp if you have to create temporary files

    Try to avoid creating temporary files. If you must, use mktemp to create a temporary directory and then write your files to it. Make sure you remove the directory after you are done.

    # set up a trap to delete the temp dir when the script exits
    unset temp_dir
    trap '[[ -d "$temp_dir" ]] && rm -rf "$temp_dir"' EXIT
    
    # create the temp dir
    declare -r temp_dir=$(mktemp -dt myapp.XXXXXX)
    
    # write to the temp dir
    command > "$temp_dir"/foo
    
  17. Use [[ and (( for test conditions

    Prefer [[ ... ]] over [ ... ] because it is safer and provides a richer set of features. Use (( ... )) for arithmetic conditions because it allows you to perform comparisons using familiar mathematical operators such as < and > instead of -lt and -gt. Note that if you desire portability, you have to stick to the old-fashioned [ ... ]. Here are a few examples:

    [[ $foo == "foo" ]] && echo "match"  # don't need to quote variable inside [[
    [[ $foo == "a" && $bar == "a" ]] && echo "match"
    
    declare -i num=5
    (( num < 10 )) && echo "match"       # don't need the $ on $num in ((
    
  18. Use commands in test conditions instead of exit status

    If you want to check whether a command succeeded before doing something, use the command directly in the condition of your if-statement instead of checking the command's exit status.

    
    # don't use exit status
    grep -q pattern file
    if (( $? == 0 ))
    then
        echo "pattern was found"
    fi
    
    # use the command as the condition
    if grep -q pattern file
    then
        echo "pattern was found"
    fi
    
  19. Use set -e

    Put this at the top of your script. This tells the shell to exit the script as soon as any statement returns a non-zero exit code.

  20. Write error messages to stderr

    Error messages belong on stderr not stdout.

    echo "An error message" >&2
    

If you have any other suggestions for my list, please share them in the comments section below!

23 comments:

  1. Your link to Difference between $*, $@, "$*" and "$@" points to the wrong place ;-)
    Next stop: a Sonar quality profile for shell scripts?

    ReplyDelete
  2. Well spotted, Gavin! I have fixed the link now. Thanks for reading :-)

    ReplyDelete
  3. Anonymous3:55 PM

    It's easier to port a shell, than a shell script, my shell-veteran friend likes to say. There's more to that then just a grain of truth. Other than that, I found myself quite in agreement!

    ReplyDelete
  4. You gathered a lot of useful recommendations that could be useful to *many* shell programmers! Maybe you could identify “bashisms” in your text, I noted 13 and 17. I am currently having a lot of trouble with bugs in Bash pledging its job management – Bash regularly core dumps!

    I just wrote a short text about a common error of shell beginners, consisting in implementing complex treatments in the shell, while these should be delegated to filters. I would love to read your comments on this text! http://unix-workstation.blogspot.de/2015/04/delegating-complex-treatments-to.html

    ReplyDelete
  5. Great post! A well-written resource to anyone looking to boost their Unix Shell Scripting.

    Thanks & Regards
    Elena Lauren

    ReplyDelete
  6. In our MacBook, technical support delegates at MacBook backing are very talented and experienced, who give users support at the earliest opportunity. You need to get a instant solution through MacBook technical support which is guaranteed you also. You don’t need to stress over it since we give you the fast technical support through Macbook Support Number :18003823046.

    ReplyDelete
  7. Our instantly QuickBooks Support team is perfect in taking down every QuickBooks error. We could assure you this with a guarantee. Call our QuickBooks Support contact number. Our QuickBooks Toll-free Support Number team will attend you.

    ReplyDelete
  8. Many QuickBooks Customer Service user always confused about one thing that which accounting software is the best and right option for their business. So here you are going. However, being an average or ordinary business person, taking care of acknowledged and professional accounting software, like QuickBooks, is not always easy.

    ReplyDelete
  9. We are widely recognized support providers for QuickBooks accounting solutions. Your QuickBooks software issues will start vanishing as soon as you receive linked to us at QuickBooks Toll-free Support Number.

    ReplyDelete
  10. Our QuickBooks Technical Support is obtainable for 24*7: Call @ QuickBooks Technical Support contact number any time.Take delight in with an array of outshined customer service services for QuickBooks via Intuit QuickBooks Phone Number at any time and from anywhere.It signifies that one can access our tech support for QuickBooks at any moment. Our backing team is dedicated enough to bestow you with end-to-end QuickBooks solutions when you desire to procure them for every single QuickBooks query

    ReplyDelete
  11. QuickBooks caused it to be simple for us to arrange and manage all our just about any transactions and analyze profits. And when you are facing issues into the functioning of QuickBooks then easily get instant support from our QuickBooks technical and highly qualified expert team. Get-in-touch with your QuickBooks Support Number team by dialing our toll-free number. Once you receive in touch with us, our best support technician shall show you in the Setup process in step-by-step.

    ReplyDelete
  12. QuickBooks users tend to be found in situations where they need to face most of the performance plus some other errors as a result of various causes inside their computer system. If you'd like any help for QuickBooks errors from customer care to obtain the treatment for these errors and problems, it is simple to connection with QuickBooks Support Number and obtain instant assistance with the guidance of our technical experts.

    ReplyDelete
  13. QuickBooks Support Phone Number supplies the Outmost Solution of one's Software Issues. Although, QuickBooks is a robust accounting platform that throws less errors when compared with others. It is usually been the absolute most challenging task to efficiently manage the business accounts in a geniune and proper way by just obtaining the best and proper solutions.

    ReplyDelete
  14. QuickBooks Customer Care Telephone Number: Readily Available For every QuickBooks Version.Consist of a beautiful bunch of accounting versions, viz.,QuickBooks Pro, QuickBooks Premier, QuickBooks Enterprise, QuickBooks POS, QuickBooks Mac, QuickBooks Windows, and QuickBooks Payroll, QuickBooks has grown to become a dependable accounting software that one may tailor depending on your industry prerequisite. As well as it, our QuickBooks Support Phone Number will bring in dedicated and diligent back-end helps for you for in case you find any inconveniences in operating any of these versions.

    ReplyDelete
  15. QuickBooks Helpline Phone Number +1855-236-7529. QuickBooks is a world-class Accounting and Financial Management software capable of smoothening the entire accounting process by automating the tasks. Bookkeepers and Accountants from all over the world blindly trust the software because of its high accuracy and reliability. Or contact at QuickBooks Web Connector Error
    Read more: https://tinyurl.com/y5k882kz

    ReplyDelete
  16. With the sudden increase in the pace of human lives, technology has taken an altogether a prominent space in it. Our team at QuickBooks Technical Support Phone Number +1 888-238-7409 provides assistance for one of the most helpful innovation of technology i.e. QuickBooks. Visit us:- https://www.enetquickbookenterprise.com/quickbooks-technical-support/
    QuickBooks Error Support Phone Number

    ReplyDelete
  17. Our QuickBooks Technical Support is obtainable for 24*7: Call @ QuickBooks Technical Support contact number any time.Take delight in with an array of outshined customer service services for QuickBooks via Intuit QuickBooks Support Number at any time and from anywhere.It signifies that one can access our tech support for QuickBooks at any moment. Our backing team is dedicated enough to bestow you with end-to-end QuickBooks solutions when you desire to procure them for every single QuickBooks query.

    ReplyDelete
  18. Well drafted and presented content! Your blog is quite engaging and a capturing one. I got access to any such type of content after long. So, thanks for it and keep writing. I further hope that your QuickBooks system might not be triggered by the QuickBooks Error 80070057. For resolving it, contact the QuickBooks customer care executives at QuickBooks Support Phone Number 1-833-441-8848.

    ReplyDelete
  19. Very well-structured and crafted writeup! Indeed, one of the best blogs I have ever been through. I just loved your content. Write more frequently and consistently. So, if you ever come across the QuickBooks Error 80070057, get it resolved through the assistance of QuickBooks experts via rapid concern calls. So, go for advanced QuickBooks assistance at QuickBooks Support Phone Number 1 833-441-8848.

    ReplyDelete
  20. Your blog is probably one of the best blogs, I’ve ever read. This was the best information for me. I now insist upon the frequent sharing of your writings. Well, if your working on QuickBooks gets corrupted via the occurrence if the QuickBooks Error 6000 83, avail the 24*7 consultation services with QuickBooks experts and instant resolution of problems at QuickBooks Support Phone Number 1-833-441-8848.

    ReplyDelete
  21. There is something different about your blog. All the facts and information seem so realistic. It is rare to find such a well-crafted post. However, if you ever come across technical doubts such as the QuickBooks Error -6150, -1006, report it and acquire the assistance of highly qualified experts at QuickBooks Support Phone Number 1-833-441-8848.

    ReplyDelete
  22. Lost connection to your Data file or Can’t able to locate the Data File? No matter, how complex the issues would be. Get instant & easy solutions for your queries by contacting us, on Quickbooks Helpline Number 800-986-4607. We provide the best support service to the Quickbooks user.

    ReplyDelete
  23. Facing error issue while tracking inventory? Get support from Quickbooks for Mac Support Phone Number +1-800-901-6679. By dialling to us, on our support Phone number you would get valuable aid to fix error issues in Quickbooks Mac. Our certified technicians are deployed for you, to offer reliable assistance. So, if having issues, make a call to us now. The support team deliver support anytime whenever come across trouble.

    ReplyDelete