Showing posts with label UNIX. Show all posts
Showing posts with label UNIX. Show all posts

Friday, July 10, 2020

Compute MD5 Checksum Hash on Windows and Linux

Use the following commands to print out the MD5 hash for a file.

On Windows:

> CertUtil -hashfile myfile.txt MD5
MD5 hash of file myfile.txt:
76383c2c0bfca944b57a63830c163ad2
CertUtil: -hashfile command completed successfully.

On Linux/Unix:

$ md5sum myfile.txt
76383c2c0bfca944b57a63830c163ad2 *myfile.txt

Saturday, February 28, 2015

Speeding up Grep on Large Files

Here are a few tips to improve the performance of grep on large files:

  1. Prefix your command with LC_ALL=C, in order to use the C locale with its smaller ASCII charset, instead of UTF-8
  2. Use grep -F to search for a fixed string (if possible), rather than a regex
  3. Remove the -i option, if you don't need it
LC_ALL=C grep -F searchString largeFile

Sunday, August 25, 2013

Executing a Shell Command with a Timeout

Sometimes you may want to kill a command if it has been running for more than a specific time limit. For example, a shell script connecting to a network resource may hang for a long period of time if the resource is unavailable and it would be desirable to kill it and send out an alert.

This post describes different ways of running commands with time limits.

1) GNU coreutils timeout command
The easiest way to run a command with a time limit is by using the timeout command from GNU coreutils. For example, to run a command with a timeout of 2 minutes:

$ timeout 2m /path/to/command with args
$ echo $?
124
If the command has not completed within the specified time limit, the timeout utility will kill it (by sending it a TERM signal) and then exit with status 124.

2) The expect command
Another way to run a command with a timeout is by using expect as shown below:

$ expect -c "
    set echo '-noecho';
    set timeout 10;
    spawn -noecho /path/to/command with args;
    expect timeout { exit 124 } eof { exit 0 }"
$ echo $?
124
In the example above, the timeout is set to 10 seconds and expect will exit with a status of 124 when the command exceeds this time limit. Otherwise, it will exit with a status of 0. Unfortunately, you lose the exit code of the command you are running.

3) Using a custom timeout script
If you cannot use the two approaches above, you can write your own timeout script. Mine is shown below. It first starts a "watchdog" process which keeps checking to see if the command is running by executing kill -0 periodically. If it is still running after the time limit has been exceeded, the watchdog kills it.

#!/bin/bash
while getopts "t:" opt; do
  case "$opt" in
      t) timeout=$OPTARG ;;
  esac
done
shift $((OPTIND-1))

start_watchdog(){
  timeout="$1"
  (( i = timeout ))
  while (( i > 0 ))
  do
    kill -0 $$ || exit 0
    sleep 1
    (( i -= 1 ))
  done

  echo "killing process after timeout of $timeout seconds"
  kill $$
}

start_watchdog "$timeout" 2>/dev/null &
exec "$@"
Example:
$ timeout.sh -t 2 sleep 5
killing process after timeout of 2 seconds
Terminated

Sunday, April 28, 2013

Useless Use of Grep

Most of us are familiar with the infamous Useless Use of Cat Award which is awarded for unnecessary use of the cat command. A while back, I also wrote about Useless Use of Echo in which I advised using here-strings and here-docs instead of the echo command. In a similar vein, this post is about the useless use of the grep command.

Useless use of grep | awk
awk can match patterns, so there is no need to pipe the output of grep to awk. For example, the following:

grep pattern file | awk '{commands}'
can be re-written as:
awk '/pattern/{commands}' file
Similarly:
grep -v pattern file | awk '{commands}'
can be re-written as:
awk '!/pattern/{commands}' file

Useless use of grep | sed
sed can match patterns, so you don't need to pipe the output of grep to sed. For example, the following:

grep pattern file | sed 's/foo/bar/g'
can be re-written as:
sed -n '/pattern/{s/foo/bar/p}' file
Similarly:
grep -v pattern file | sed 's/foo/bar/g'
can be re-written as:
sed -n '/pattern/!{s/foo/bar/p}' file

Useless use of grep in conditions
If you find yourself using grep in conditional statements to check if a string variable matches a certain pattern, consider using bash's in-built string matching instead. For example, the following:

if grep -q pattern <<< "$var"; then
    # do something
fi
can be re-written as:
if [[ $var == *pattern* ]]; then
    # do something
fi
or, if your pattern is a regex, rather than a fixed string, use:
if [[ $var =~ pattern ]]; then
    # do something
fi

Saturday, February 09, 2013

Selecting Specific Lines of a File Using Head, Tail and Sed

This post contains a few handy commands used to select specific lines from a file.

Print the first N lines

head -N file
Print the last N lines
tail -N file
Print all EXCEPT the first N lines
tail +$((N+1)) file
Print all EXCEPT the last N lines
head -n -N file
Print lines N to M (inclusive)
sed -n 'N,Mp' file
Print line N
sed 'Nq;d' file
Print all EXCEPT line N
sed 'Nd' file
Print multiple lines, I, J, K etc
Assuming I > J > K:
sed 'Ip;Jp;Kq;d' file
The last q tells sed to quit when it reaches the Kth line instead of looping over the remaining lines that we are not interested in.

Monday, December 31, 2012

Sed: Mutli-Line Replacement Between Two Patterns

This post has some useful sed commands which can be used to perform replacements and deletes between two patterns across multiple lines. For example, consider the following file:
$ cat file
line 1
line 2
foo
line 3
line 4
line 5
bar
line 6
line 7
1) Replace text on each line between two patterns (inclusive):
To perform a replacement on each line between foo and bar, including the lines containing foo and bar, use the following:
$ sed '/foo/,/bar/{s/./x/g}' file
line 1
line 2
xxx
xxxxxx
xxxxxx
xxxxxx
xxx
line 6
line 7
2) Replace text on each line between two patterns (exclusive):
To perform a replacement on each line between foo and bar, excluding the lines containing foo and bar, use the following:
$ sed '/foo/,/bar/{/foo/n;/bar/!{s/./x/g}}' file
line 1
line 2
foo
xxxxxx
xxxxxx
xxxxxx
bar
line 6
line 7
3) Delete lines between two patterns (inclusive):
To delete all lines between foo and bar, including the lines containing foo and bar, use the same replacement sed command as shown above, but simply change the replacement expression to a delete.

$ sed '/foo/,/bar/d' file
line 1
line 2
line 6
line 7
4) Delete lines between two patterns (exclusive):
To delete all lines between foo and bar, excluding the lines containing foo and bar, use the same replacement sed command as shown above, but simply change the replacement expression to a delete.
$ sed '/foo/,/bar/ {/foo/n;/bar/!d}' file
line 1
line 2
foo
bar
line 6
line 7
5) Replace all lines between two patterns (inclusive):
To perform a replacement on a block of lines between foo and bar, including the lines containing foo and bar, use:
$ sed -n '/foo/{:a;N;/bar/!ba;N;s/.*\n/REPLACEMENT\n/};p' file
line 1
line 2
REPLACEMENT
line 6
line 7
How it works:
/foo/{                   # when "foo" is found
  :a                     # create a label "a"
    N                    # store the next line
  /bar/!ba               # goto "a" and keep looping and storing lines until "bar" is found
  N                      # store the line containing "bar"
  s/.*\n/REPLACEMENT\n/  # delete the lines
}
p                        # print
6) Replace all lines between two patterns (exclusive):
To perform a replacement on a block of lines between foo and bar, excluding the lines containing foo and bar, use:
$ sed -n '/foo/{p;:a;N;/bar/!ba;s/.*\n/REPLACEMENT\n/};p' file
line 1
line 2
foo
REPLACEMENT
bar
line 6
line 7
References:
Sed - An Introduction and Tutorial by Bruce Barnett

Saturday, October 20, 2012

Joining Two Files with the Unix join Command

The join command is a useful tool for joining two files on a common field. It allows you to join two files, similar to the way you would join two tables in a SQL database.

The following example illustrates the power of the join command. You have two files, one containing a list of employees with their department ids and the other containing departments and their ids. You want to find out the names of the departments for each employee. You MUST first sort the files on the department id column (using the sort command) and then join them on that column.

$ cat employees.txt
Jones,33
Steinberg,33
Robinson,34
Smith,34
Rafferty,31
John,

$ cat departments.txt
31,Sales
33,Engineering
34,Clerical
35,Marketing

$ join -a 1 -t, -1 2 -2 1 -o 1.1 2.2 <(sort -t, -k2 employees.txt) <(sort -t, -k1 departments.txt)
John,
Rafferty,Sales
Jones,Engineering
Steinberg,Engineering
Robinson,Clerical
Smith,Clerical

Joining on multiple columns
The join command joins on a single field. What do you do if you want to join on multiple fields? You create a composite field by combining the multiple fields together! This can be done using awk. For example:
$ cat employees2.txt
Jones,33,50
Steinberg,33,51
Robinson,34,50
Smith,34,50
Rafferty,31,51

$ awk -F, '{print $2"_"$3","$0}' employees2.txt
33_50,Jones,33,50
33_51,Steinberg,33,51
34_50,Robinson,34,50
34_50,Smith,34,50
31_51,Rafferty,31,51

As you can see, an additional field has been created by concatenating the second and third fields of the file. Now you can join the files on the new composite field.

(File data courtesy of Wikipedia.)

Thursday, August 09, 2012

Running a command on multiple hosts

There are different ways you can run a command on multiple machines.

1. For loop
If you want to execute the same command on a few hosts, you can use a for loop as shown below:

for host in host1 host2 host3
do
    ssh $host "hostname; who -b"
done
The example above iterates over a list of hosts, and runs two commands on each one to print the name of the host and the time it was rebooted.

2. While loop
If your list of hosts is stored in a file, you can use a while loop as shown below:

while IFS= read -r host
do
    ssh -n $host "hostname; who -b"
done < /tmp/myhosts
You must provide the -n option to ssh, otherwise it will only run on the first host in your file and then the loop will terminate.

3. Parallel ssh
Parallel ssh (pssh) allows you to run a command on several hosts at the same time and is much faster than using a sequential loop if the number of hosts is large. You can specify how many parallel processes it uses to ssh to the various hosts (default is 32).

$ pssh
Usage: pssh [OPTIONS] -h hosts.txt prog [arg0] ..

  -h --hosts   hosts file (each line "host[:port] [user]")
  -l --user    username (OPTIONAL)
  -p --par     max number of parallel threads (OPTIONAL)
  -o --outdir  output directory for stdout files (OPTIONAL)
  -t --timeout timeout in seconds to do ssh to a host (OPTIONAL)
  -v --verbose turn on warning and diagnostic messages (OPTIONAL)
  -O --options SSH options (OPTIONAL)

$ pssh -h /tmp/myhosts -o /tmp/output "hostname; who -b"

Saturday, June 30, 2012

vim: Change statusline colour based on mode

Here is my vimrc statusline configuration:
" statusline
" format markers:
"   %t File name (tail) of file in the buffer
"   %m Modified flag, text is " [+]"; " [-]" if 'modifiable' is off.
"   %r Readonly flag, text is " [RO]".
"   %y Type of file in the buffer, e.g., " [vim]".
"   %= Separation point between left and right aligned items.
"   %l Line number.
"   %L Number of lines in buffer.
"   %c Column number.
"   %P percentage through buffer
set statusline=%t\ %m%r%y%=(ascii=\%03.3b,hex=\%02.2B)\ (%l/%L,%c)\ (%P)
set laststatus=2
" change highlighting based on mode
if version >= 700
  highlight statusLine cterm=bold ctermfg=black ctermbg=red
  au InsertLeave * highlight StatusLine cterm=bold ctermfg=black ctermbg=red
  au InsertEnter * highlight StatusLine cterm=bold ctermfg=black ctermbg=green
endif
It displays some useful information about the file and your position within it. It also automatically changes the colour of the statusline from red to green when you enter INSERT mode and back to red when you leave it.

This is what the status line looks like in INSERT mode:

Foo.java [RO][java] (ascii=097,hex=61) (158/667,23) (26%)

To see a description of all possible status line variables type :help statusline in vim.

To see my complete vimrc visit my GitHub dotfiles repository.

Saturday, May 26, 2012

MultiTail: Viewing Multiple Files with Custom Colorschemes

MultiTail is a program which allows you to tail multiple files in a single terminal. The feature I find most useful is its ability to highlight text in files using "colorschemes". There are a number of pre-defined colorschemes which can be found in the configuration file, multitail.conf.

Here is an example of using multitail. The command below tails two files: an apache access log and a tomcat catalina log using two different colorschemes.

$ multitail -cS apache /tmp/apache/access_log -cS log4j ${TOMCAT_HOME}/logs/catalina.out
You can also add additional colorschemes to your ~/.multitailrc. A colorscheme is simply a set of regular expressions to capture and highlight the text you are interested in. Here is my config file which contains my custom XML colour scheme.
check_mail:0

colorscheme:xml
# element text
cs_re_s:white:>([^<]*)<
# attribute key
cs_re_s:green: ([^ =]*)=
# attribute value
cs_re_s:red:=("[^"]*")
# element name
cs_re:blue,,bold:<[^>]*>
Used like this:
$  multitail -cS xml /var/log/config.xml
Related Post:
Highlighting Command Output with Generic Colouriser

Saturday, November 05, 2011

Regular Expressions in Bash

Traditionally, external tools such as grep, sed, awk and perl have been used to match a string against a regular expression, but the Bash shell has this functionality built into it as well!

In Bash, the =~ operator allows you to match a string on the left against an extended regular expression on the right and returns 0 if the string matches the pattern, and 1 otherwise. Capturing groups are saved in the array variable BASH_REMATCH with the first element, Group 0, representing the entire expression.

The following script matches a string against a regex and prints out the capturing groups:

#!/bin/bash

if [ $# -lt 2 ]
then
    echo "Usage: $0 regex string" >&2
    exit 1
fi

regex=$1
input=$2

if [[ $input =~ $regex ]]
then
    echo "$input matches regex: $regex"

    #print out capturing groups
    for (( i=0; i<${#BASH_REMATCH[@]}; i++))
    do
        echo -e "\tGroup[$i]: ${BASH_REMATCH[$i]}"
    done
else
    echo "$input does not match regex: $regex"
fi
Example usage:
sharfah@starship:~> matcher.sh '(.*)=(.*)' foo=bar
foo=bar matches regex (.*)=(.*)
    Group[0]: foo=bar
    Group[1]: foo
    Group[2]: bar

Saturday, October 08, 2011

Splitting a large file into smaller pieces

If you have a large file and want to break it into smaller pieces, you can use the Unix split command. You can tell it what the prefix of each split file should be and it will then append an alphabet (or number) to the end of each name.

In the example below, I split a file containing 100,000 lines. I instruct split to use numeric suffixes (-d), put 10,000 lines in each split file (-l 10000) and use suffixes of length 3 (-a 3). As a result, ten split files are created, each with 10,000 lines.

$ ls
hugefile

$ wc -l hugefile
100000 hugefile

$ split -d -l 10000 -a 3 hugefile hugefile.split.

$ ls
hugefile                hugefile.split.005
hugefile.split.000      hugefile.split.006
hugefile.split.001      hugefile.split.007  
hugefile.split.002      hugefile.split.008
hugefile.split.003      hugefile.split.009
hugefile.split.004

$ wc -l *split*
 10000 hugefile.split.000
 10000 hugefile.split.001
 10000 hugefile.split.002
 10000 hugefile.split.003
 10000 hugefile.split.004
 10000 hugefile.split.005
 10000 hugefile.split.006
 10000 hugefile.split.007
 10000 hugefile.split.008
 10000 hugefile.split.009
100000 total

Saturday, September 17, 2011

Faster SSH with Multiplexing

OpenSSH allows you to speed up multiple SSH connections to the same server using "multiplexing". The first connection acts as the "master" and any other connections reuse the master instance's network connection rather than initiating new ones.

In order to set this up add the following to your ~/.ssh/config file:

Host *
ControlMaster auto
ControlPath /tmp/%r@%h:%p
ControlMaster auto will use a master if one exists, or start a master otherwise. ControlPath is the path to the control socket used for connection sharing. %r, %h and %p are replaced with your username, host to which you are connecting and the port respectively.

In addition, you may want to add Ciphers arcfour in order to use the arcfour cipher which is faster than the default (aes128-cbc). The transfer rate of arcfour is about 90 MB/s, aes128-cbc is about 75 MB/s and the slowest is 3des-cbc, at 19 MB/s.

Saturday, August 20, 2011

LESSOPEN Powers Up Less

A really useful feature of the Unix less pager is LESSOPEN which is the "input preprocessor" for less. This is a script, defined in the LESSOPEN environment variable, which is invoked before the file is opened. It gives you the chance to modify the way the contents of the file are displayed. Why would you want to do this? The most common reason is to uncompress files before you view them, allowing you to less GZ files. But it also allows you to list the contents of zip files and other archives. I like to use it to format XML files and to view Java class files by invoking jad.

You can download a really useful LESSOPEN script from http://sourceforge.net/projects/lesspipe/ and then extend it if necessary.

To use it, simply add export LESSOPEN="|/path/to/bin/lesspipe.sh %s" to your bashrc.

You can then less:

  • directories
  • compressed files
  • archives, to list the files contained in them
  • files contained in archives e.g. less foo.zip:bar.txt
  • binary files

Sunday, July 24, 2011

Viewing CSV Files

I find CSV (or, to be more general, DSV) files difficult to read on Unix because you can't tell which column a value is in. So I always end up importing them into a spreadsheet which is a pain. Here is an example of a small pipe-delimited file containing book data:
Title|Author|CoAuthor|Year|ISBN
Carrie|Stephen King||1974|978-0385086950
The Human Web|William McNeill|Robert McNeill|2003|978-0393051797
It would be a lot easier to read, if I could convert the file into dictionaries of key:value pairs in order to see which columns the values were referring to, like this:
Title:Carrie
Author:Stephen King
CoAuthor:
Year:1974
ISBN:978-0385086950

Title:The Human Web
Author:William McNeill
CoAuthor:Robert McNeill
Year:2003
ISBN:978-0393051797
So, I wrote the following Bash script to convert a delimiter separated file into a collection of dictionaries. It uses awk to read the first row, which contains the column names, split it and store it in an array. It then prints out the remaining rows along with their column names which are looked up from the array.
#! /bin/bash
# CSV Viewer
# Usage: csv [-d delim] filename
# default delimiter is pipe.
#
# Input file:
# h1|h2|h3
# v1|v2|v3
# w1|w2|w3
#
# Output:
# h1: v1
# h2: v2
# h3: v3
#
# h1: w1
# h2: w2
# h3: w3

delim=|
while getopts "d:" OPTION
do
   case $OPTION in
     d) delim=$OPTARG; shift $((OPTIND-1)) ;;
   esac
done

if [ $# -eq 0 ]
then
    echo "Usage: csv [-d delim] filename" >&2
    exit 1
fi
awk -F "$delim" '{if(NR==1)split($0,arr);else for(i=1;i<=NF;i++)print arr[i]":"$i;print "";}' "$1"
Running the script:
sharfah@starship:~> csv.sh -d '|' file
Title:Carrie
Author:Stephen King
CoAuthor:
Year:1974
ISBN:978-0385086950

Title:The Human Web
Author:William McNeill
CoAuthor:Robert McNeill
Year:2003
ISBN:978-0393051797

Saturday, June 18, 2011

Efficiently Navigating Directories on UNIX

I find myself, like most developers, spending a lot of time navigating directories. Flipping back and forth between logs and application directories with long names can be quite tedious. So, with the help of a few new functions, aliases and config tweaks I've made the navigation process easier and more efficient. You no longer need to remember long paths because you can jump straight to them using their names. You can also choose to bookmark your favourite directories. Here is my setup:

1. Go up to a specific directory
I have a function called upto which allows you to jump up to any directory, on the current path, just by name. This is very useful if you are deep in a directory. I also have autocompletion for this function, so that it shows me valid directory names and completes them for me.

#
# Go up to the specified directory
#
upto(){
  if [ -z $1 ]; then
      echo "Usage: upto [directory]"
      return 1
  fi
  local upto=$1
  cd "${PWD/\/$upto\/*//$upto}"
}

#
# Completion function for upto
#
_upto(){
  local cur=${COMP_WORDS[COMP_CWORD]}
  d=${PWD//\//\ }
  COMPREPLY=( $( compgen -W "$d" -- $cur ) )
}
complete -F _upto upto
Example:
[/www/public_html/animals/hippopotamus/habitat/swamps/images] $ upto h[TAB][TAB]
habitat       hippopotamus
[/www/public_html/animals/hippopotamus/habitat/swamps/images] $ upto hippopotamus
[/www/public_html/animals/hippopotamus] $
2. Go up a specific number of directories
If you know how many levels you want to go up, you can use the up function e.g. up 5 will move you up 5 directories.
#
# Go up a specified number of directories
#
up(){
  if [ -z $1 ]
  then
    cd ..
    return
  fi
  local levels=$1
  local result="."
  while [ $levels -gt 0 ]
  do
    result=$result/..
    ((levels--))
  done
  cd $result
}
3. Go down to a specific directory
Sometimes you want to change to a directory but can't remember the path, or the path name is too long to type. I have a function called jd which allows you to jump down to a directory any level below the current one. It uses Bash's globstar feature so make sure you have it enabled (using shopt -s globstar). (Warning: this may be slow on large directory structures because of the searching involved.)
#
# Jumps to a directory at any level below.
# using globstar
#
jd(){
  if [ -z $1 ]; then
      echo "Usage: jd [directory]";
      return 1
  else
      cd **/$1
  fi
}
Example:
[/www/public_html/animals/hippopotamus/habitat/swamps/images] $ upto hippopotamus
[/www/public_html/animals/hippopotamus] $ jd images
[/www/public_html/animals/hippopotamus/habitat/swamps/images] $
4. CDPATH
The CDPATH variable is a colon-separated list of directories in which the shell looks for destination directories specified by the cd command. Mine is shown below. No matter what directory I am currently in, I can quickly jump to a project in my dev directory with cd <project> because it is on my path.
export CDPATH=".::..:../..:~:~/dev/"
5. Shell Options
I have set the following useful shell options in my .bashrc. The autocd option allows you to change to a directory without using the cd command and cdspell automatically corrects typos in directory names.
shopt -s cdspell     # correct dir spelling errors on cd
shopt -s autocd      # if a command is a dir name, cd to it
shopt -s cdable_vars # if cd arg is not a dir, assume it is a var
6. Quick Aliases
alias ..='cd ..'
alias ...='cd ../..'
alias ....='cd ../../..'
alias .....='cd ../../../..'
alias ......='cd ../../../../..'
7. Keeping a history of visited directories
I came across a useful post on Linux Gazette: History of visited directories in BASH. It contains a script which maintains a history of directories you have visited and then allows you to switch to them easily using a reference number. The command cd -- shows you your history and cd -2 would take you to the second item in your history list. For example:
[/www/public_html/animals] $ cd --
 1  /tmp
 2  /www/public_html/animals/hippopotamus/habitat/swamps/images
 3  /www/public_html/animals/lion
[/www/public_html/animals] $ cd -2
[/www/public_html/animals/hippopotamus/habitat/swamps/images] $
8. Bookmarks
I spend a lot of time moving between different directories especially between logs and application directories. I have implemented a bookmarking feature which allows you to bookmark your favourite directories and then change to them easily.
  • bm: bookmark the current directory
  • bcd: change to the specified bookmark
  • brm: remove a bookmark
  • bcl: clear all bookmarks
  • bll: list all bookmarks
#-------------------------------
# Directory Bookmark Functions
#-------------------------------

#
# Add a bookmark, if it doesn't exist
#
bm(){
  local val=$(pwd)
  for i in ${bookmarks[@]}
  do
    if [ "$i" == "$val" ]
    then
       return 1
    fi
  done
  num=${#bookmarks[@]}
  bookmarks[$num]=$val
}

#
# Goto specified bookmark
# or previous one by default
#
bcd(){
  index=$1
  if [ -z $index ]
  then
     index=$((${#bookmarks[@]}-1))
  fi
  local val=${bookmarks[$index]}
  if [ -z $val ]
  then
     echo "No such bookmark. Type bll to list bookmarks."
     return 1
  else
     cd "$val"
  fi
}

#
# Remove a bookmark
#
brm(){
  if [ $# -lt 1 ]
  then
     echo "Usage: brm [bookmark-index]"
     return 1
  fi
  if [ -z ${bookmarks[$1]} ]
  then
     echo "No such bookmark"
     return 1
  fi
  bookmarks=(${bookmarks[@]:0:$1} ${bookmarks[@]:$(($1 + 1))})
}

#
# Remove all bookmarks
#
bcl(){
    bookmarks=()
}

#
# List all bookmarks
#
bll(){
  if [ ${#bookmarks[@]} -ne 0 ]
  then
     local i=0
     while [ $i -lt ${#bookmarks[@]} ]
     do
       echo $i: ${bookmarks[$i]}
       ((i++))
     done
  fi
  return 0
}

If you have any useful directory-related functions, share them in the comments below!

Related Posts:

Monday, March 08, 2010

Null Characters

On a few occasions (normally when I am running out of disk space), I have seen ^@ symbols appear in my log files. These are "file holes" and contain null characters. The null character (or NUL char) has an ASCII code of 0 and appears as ^@ when viewed in 'vi' or 'less'.

Create a dummy file containing null characters:
In order to create a file with null characters, simply print \000. For example:

sharfah@starship:~> perl -e \
'print "hello \000world\000\nfoo bar\n";' > file-with-nulls
sharfah@starship:~> less file-with-nulls
hello ^@world^@
foo bar
Find lines containing null characters:
Use the following command in order print out lines containing null characters:
sharfah@starship:~> perl -ne '/\000/ and print;' file-with-nulls \
| less
hello ^@world^@
You can also perform an octal dump of the file to check if it has null characters:
sharfah@starship:~> od -b file-with-nulls | grep ' 000'
0000000 150 145 154 154 157 040 000 167 157 162 154 144 000 012 146 157
Delete null characters:
There are various ways in which this can be done. In the following examples, I have used tr and sed to remove the unwanted characters.
sharfah@starship:~> tr -d '\000' < file-with-nulls | less
hello world
foo bar
sharfah@starship:~> sed 's/\x0//g' < file-with-nulls | less
hello world
foo bar
Do not use the strings command because it will create a newline when it encounters a null characeter. "The strings utility looks for ASCII strings in a binary file. A string is any sequence of 4 or more printing characters ending with a newline or a null character."
sharfah@starship:~> strings file-with-nulls | less
hello
world
foo bar

Further reading:
ASCII Character Set
Identifying and removing null characters in UNIX [stackoverflow]
File holes

Tuesday, January 19, 2010

Difference between $*, $@, "$*" and "$@"

They are all related to "all the arguments to the shell", but behave differently. The following script demonstrates each one of $*, $@, "$@" and "$*" in turn by echoing out the arguments passed in:
#! /bin/bash

echo +--- Using "\$*"

cnt=1
for i in $*
do
  echo arg$cnt: $i
  cnt=$(($cnt+1))
done

echo +--- Using \"\$*\"

cnt=1
for i in "$*"
do
  echo arg$cnt: $i
  cnt=$(($cnt+1))
done

echo +--- Using "\$@"

cnt=1
for i in $@
do
  echo arg$cnt: $i
  cnt=$(($cnt+1))
done

echo +--- Using \"\$@\"

cnt=1
for i in "$@"
do
  echo arg$cnt: $i
  cnt=$(($cnt+1))
done
Running the script, produces the following output:
sharfah@starship:~> echoArgs.sh  mars "kit kat" twix
+--- Using $*
arg1: mars
arg2: kit
arg3: kat
arg4: twix
+--- Using "$*"
arg1: mars kit kat twix
+--- Using $@
arg1: mars
arg2: kit
arg3: kat
arg4: twix
+--- Using "$@"
arg1: mars
arg2: kit kat
arg3: twix
When unquoted, $* and $@ do the same thing. They treat each "word" (sequence of non-whitespace characters) as a separate argument. This leads to the single argument "kit kat" being broken into two which may not be desired. When quoted, $* and $@ behave differently. "$*" treats the entire argument list as a single argument, but "$@" treats the arguments just as they were when passed in.

So, which should you use? In almost all cases you would use "$@" in order to preserve arguments with spaces (or quoted arguments) when being passed in on the command line or from one script to another. Never use the unquoted $* or $@ unless you are absolutely sure that you won't need to deal with arguments with spaces in them. If you need to combine all arguments into one, use "$*".

Tuesday, January 12, 2010

Identify which process is using a port

Linux:
Use the lsof command and grep for the port number you are interested in:
sharfah@starship:~>  lsof -Pan -i tcp -i udp | grep :8343
java    27725 sharfah    6u  IPv6 20486040       TCP *:8343 (LISTEN)
Solaris:
If you have lsof installed on Solaris, then you can use the Linux method above. If you don't have or cannot install lsof, then use the pre-installed pfiles in a loop, as shown below:
sharfah@starship:~>  ps -ef | grep myuser | awk '{print $2}' | while read pid
>do
>echo $pid
>pfiles $pid| grep 12197
>done
19424
16132
16040
29373
15946
25178
 121: S_IFREG mode:0640 dev:289,6 ino:259883 uid:50006 gid:106 size:109318
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
15985
16052
18758

Wednesday, October 21, 2009

Bash Globbing

Globbing refers to the expansion of shell metacharacters to complete file names. For example, when you run ls *, the shell expands the wildcard * into a list of files and passes them as arguments to ls.
sharfah@starship:~> ls *
file1 File2
Dot Globbing
By default, Bash does not glob dot-files. This means that ls * will not pick up any files beginning with the dot (.) character. However, it is easy to change this by doing the following:
sharfah@starship:~> shopt -s dotglob
sharfah@starship:~> ls *
.dotFile1 file1 File2
Case-Insensitive Globbing
There is also an option to turn on case-insensitive globbing:
sharfah@starship:~> shopt -s nocaseglob
sharfah@starship:~> ls f*
file1 File2
To view a list of all your shell options, type shopt.