Showing posts with label commands. Show all posts
Showing posts with label commands. Show all posts

Saturday, February 28, 2015

Speeding up Grep on Large Files

Here are a few tips to improve the performance of grep on large files:

  1. Prefix your command with LC_ALL=C, in order to use the C locale with its smaller ASCII charset, instead of UTF-8
  2. Use grep -F to search for a fixed string (if possible), rather than a regex
  3. Remove the -i option, if you don't need it
LC_ALL=C grep -F searchString largeFile

Sunday, August 25, 2013

Executing a Shell Command with a Timeout

Sometimes you may want to kill a command if it has been running for more than a specific time limit. For example, a shell script connecting to a network resource may hang for a long period of time if the resource is unavailable and it would be desirable to kill it and send out an alert.

This post describes different ways of running commands with time limits.

1) GNU coreutils timeout command
The easiest way to run a command with a time limit is by using the timeout command from GNU coreutils. For example, to run a command with a timeout of 2 minutes:

$ timeout 2m /path/to/command with args
$ echo $?
124
If the command has not completed within the specified time limit, the timeout utility will kill it (by sending it a TERM signal) and then exit with status 124.

2) The expect command
Another way to run a command with a timeout is by using expect as shown below:

$ expect -c "
    set echo '-noecho';
    set timeout 10;
    spawn -noecho /path/to/command with args;
    expect timeout { exit 124 } eof { exit 0 }"
$ echo $?
124
In the example above, the timeout is set to 10 seconds and expect will exit with a status of 124 when the command exceeds this time limit. Otherwise, it will exit with a status of 0. Unfortunately, you lose the exit code of the command you are running.

3) Using a custom timeout script
If you cannot use the two approaches above, you can write your own timeout script. Mine is shown below. It first starts a "watchdog" process which keeps checking to see if the command is running by executing kill -0 periodically. If it is still running after the time limit has been exceeded, the watchdog kills it.

#!/bin/bash
while getopts "t:" opt; do
  case "$opt" in
      t) timeout=$OPTARG ;;
  esac
done
shift $((OPTIND-1))

start_watchdog(){
  timeout="$1"
  (( i = timeout ))
  while (( i > 0 ))
  do
    kill -0 $$ || exit 0
    sleep 1
    (( i -= 1 ))
  done

  echo "killing process after timeout of $timeout seconds"
  kill $$
}

start_watchdog "$timeout" 2>/dev/null &
exec "$@"
Example:
$ timeout.sh -t 2 sleep 5
killing process after timeout of 2 seconds
Terminated

Sunday, April 28, 2013

Useless Use of Grep

Most of us are familiar with the infamous Useless Use of Cat Award which is awarded for unnecessary use of the cat command. A while back, I also wrote about Useless Use of Echo in which I advised using here-strings and here-docs instead of the echo command. In a similar vein, this post is about the useless use of the grep command.

Useless use of grep | awk
awk can match patterns, so there is no need to pipe the output of grep to awk. For example, the following:

grep pattern file | awk '{commands}'
can be re-written as:
awk '/pattern/{commands}' file
Similarly:
grep -v pattern file | awk '{commands}'
can be re-written as:
awk '!/pattern/{commands}' file

Useless use of grep | sed
sed can match patterns, so you don't need to pipe the output of grep to sed. For example, the following:

grep pattern file | sed 's/foo/bar/g'
can be re-written as:
sed -n '/pattern/{s/foo/bar/p}' file
Similarly:
grep -v pattern file | sed 's/foo/bar/g'
can be re-written as:
sed -n '/pattern/!{s/foo/bar/p}' file

Useless use of grep in conditions
If you find yourself using grep in conditional statements to check if a string variable matches a certain pattern, consider using bash's in-built string matching instead. For example, the following:

if grep -q pattern <<< "$var"; then
    # do something
fi
can be re-written as:
if [[ $var == *pattern* ]]; then
    # do something
fi
or, if your pattern is a regex, rather than a fixed string, use:
if [[ $var =~ pattern ]]; then
    # do something
fi

Saturday, February 09, 2013

Selecting Specific Lines of a File Using Head, Tail and Sed

This post contains a few handy commands used to select specific lines from a file.

Print the first N lines

head -N file
Print the last N lines
tail -N file
Print all EXCEPT the first N lines
tail +$((N+1)) file
Print all EXCEPT the last N lines
head -n -N file
Print lines N to M (inclusive)
sed -n 'N,Mp' file
Print line N
sed 'Nq;d' file
Print all EXCEPT line N
sed 'Nd' file
Print multiple lines, I, J, K etc
Assuming I > J > K:
sed 'Ip;Jp;Kq;d' file
The last q tells sed to quit when it reaches the Kth line instead of looping over the remaining lines that we are not interested in.

Monday, December 31, 2012

Sed: Mutli-Line Replacement Between Two Patterns

This post has some useful sed commands which can be used to perform replacements and deletes between two patterns across multiple lines. For example, consider the following file:
$ cat file
line 1
line 2
foo
line 3
line 4
line 5
bar
line 6
line 7
1) Replace text on each line between two patterns (inclusive):
To perform a replacement on each line between foo and bar, including the lines containing foo and bar, use the following:
$ sed '/foo/,/bar/{s/./x/g}' file
line 1
line 2
xxx
xxxxxx
xxxxxx
xxxxxx
xxx
line 6
line 7
2) Replace text on each line between two patterns (exclusive):
To perform a replacement on each line between foo and bar, excluding the lines containing foo and bar, use the following:
$ sed '/foo/,/bar/{/foo/n;/bar/!{s/./x/g}}' file
line 1
line 2
foo
xxxxxx
xxxxxx
xxxxxx
bar
line 6
line 7
3) Delete lines between two patterns (inclusive):
To delete all lines between foo and bar, including the lines containing foo and bar, use the same replacement sed command as shown above, but simply change the replacement expression to a delete.

$ sed '/foo/,/bar/d' file
line 1
line 2
line 6
line 7
4) Delete lines between two patterns (exclusive):
To delete all lines between foo and bar, excluding the lines containing foo and bar, use the same replacement sed command as shown above, but simply change the replacement expression to a delete.
$ sed '/foo/,/bar/ {/foo/n;/bar/!d}' file
line 1
line 2
foo
bar
line 6
line 7
5) Replace all lines between two patterns (inclusive):
To perform a replacement on a block of lines between foo and bar, including the lines containing foo and bar, use:
$ sed -n '/foo/{:a;N;/bar/!ba;N;s/.*\n/REPLACEMENT\n/};p' file
line 1
line 2
REPLACEMENT
line 6
line 7
How it works:
/foo/{                   # when "foo" is found
  :a                     # create a label "a"
    N                    # store the next line
  /bar/!ba               # goto "a" and keep looping and storing lines until "bar" is found
  N                      # store the line containing "bar"
  s/.*\n/REPLACEMENT\n/  # delete the lines
}
p                        # print
6) Replace all lines between two patterns (exclusive):
To perform a replacement on a block of lines between foo and bar, excluding the lines containing foo and bar, use:
$ sed -n '/foo/{p;:a;N;/bar/!ba;s/.*\n/REPLACEMENT\n/};p' file
line 1
line 2
foo
REPLACEMENT
bar
line 6
line 7
References:
Sed - An Introduction and Tutorial by Bruce Barnett

Saturday, December 22, 2012

Useless Use of Echo

Most of us are familiar with the Useless Use of Cat Award which is awarded for unnecessary use of the cat command. For example, in nearly all cases, cat file | command arg can be rewritten as <file command arg.

In a similar vein, this post is about the useless use of the echo command. In nearly all cases:

echo string | command arg
can be rewritten using a heredoc:
command arg << END
string
END
or, using a here-string:
command arg <<< string
Note: Here-strings are not portable (but most modern shells support them) so use the heredoc alternative shown above if you are writing a portable script!

Saturday, October 20, 2012

Joining Two Files with the Unix join Command

The join command is a useful tool for joining two files on a common field. It allows you to join two files, similar to the way you would join two tables in a SQL database.

The following example illustrates the power of the join command. You have two files, one containing a list of employees with their department ids and the other containing departments and their ids. You want to find out the names of the departments for each employee. You MUST first sort the files on the department id column (using the sort command) and then join them on that column.

$ cat employees.txt
Jones,33
Steinberg,33
Robinson,34
Smith,34
Rafferty,31
John,

$ cat departments.txt
31,Sales
33,Engineering
34,Clerical
35,Marketing

$ join -a 1 -t, -1 2 -2 1 -o 1.1 2.2 <(sort -t, -k2 employees.txt) <(sort -t, -k1 departments.txt)
John,
Rafferty,Sales
Jones,Engineering
Steinberg,Engineering
Robinson,Clerical
Smith,Clerical

Joining on multiple columns
The join command joins on a single field. What do you do if you want to join on multiple fields? You create a composite field by combining the multiple fields together! This can be done using awk. For example:
$ cat employees2.txt
Jones,33,50
Steinberg,33,51
Robinson,34,50
Smith,34,50
Rafferty,31,51

$ awk -F, '{print $2"_"$3","$0}' employees2.txt
33_50,Jones,33,50
33_51,Steinberg,33,51
34_50,Robinson,34,50
34_50,Smith,34,50
31_51,Rafferty,31,51

As you can see, an additional field has been created by concatenating the second and third fields of the file. Now you can join the files on the new composite field.

(File data courtesy of Wikipedia.)

Thursday, August 09, 2012

Running a command on multiple hosts

There are different ways you can run a command on multiple machines.

1. For loop
If you want to execute the same command on a few hosts, you can use a for loop as shown below:

for host in host1 host2 host3
do
    ssh $host "hostname; who -b"
done
The example above iterates over a list of hosts, and runs two commands on each one to print the name of the host and the time it was rebooted.

2. While loop
If your list of hosts is stored in a file, you can use a while loop as shown below:

while IFS= read -r host
do
    ssh -n $host "hostname; who -b"
done < /tmp/myhosts
You must provide the -n option to ssh, otherwise it will only run on the first host in your file and then the loop will terminate.

3. Parallel ssh
Parallel ssh (pssh) allows you to run a command on several hosts at the same time and is much faster than using a sequential loop if the number of hosts is large. You can specify how many parallel processes it uses to ssh to the various hosts (default is 32).

$ pssh
Usage: pssh [OPTIONS] -h hosts.txt prog [arg0] ..

  -h --hosts   hosts file (each line "host[:port] [user]")
  -l --user    username (OPTIONAL)
  -p --par     max number of parallel threads (OPTIONAL)
  -o --outdir  output directory for stdout files (OPTIONAL)
  -t --timeout timeout in seconds to do ssh to a host (OPTIONAL)
  -v --verbose turn on warning and diagnostic messages (OPTIONAL)
  -O --options SSH options (OPTIONAL)

$ pssh -h /tmp/myhosts -o /tmp/output "hostname; who -b"

Saturday, May 26, 2012

MultiTail: Viewing Multiple Files with Custom Colorschemes

MultiTail is a program which allows you to tail multiple files in a single terminal. The feature I find most useful is its ability to highlight text in files using "colorschemes". There are a number of pre-defined colorschemes which can be found in the configuration file, multitail.conf.

Here is an example of using multitail. The command below tails two files: an apache access log and a tomcat catalina log using two different colorschemes.

$ multitail -cS apache /tmp/apache/access_log -cS log4j ${TOMCAT_HOME}/logs/catalina.out
You can also add additional colorschemes to your ~/.multitailrc. A colorscheme is simply a set of regular expressions to capture and highlight the text you are interested in. Here is my config file which contains my custom XML colour scheme.
check_mail:0

colorscheme:xml
# element text
cs_re_s:white:>([^<]*)<
# attribute key
cs_re_s:green: ([^ =]*)=
# attribute value
cs_re_s:red:=("[^"]*")
# element name
cs_re:blue,,bold:<[^>]*>
Used like this:
$  multitail -cS xml /var/log/config.xml
Related Post:
Highlighting Command Output with Generic Colouriser

Saturday, October 08, 2011

Splitting a large file into smaller pieces

If you have a large file and want to break it into smaller pieces, you can use the Unix split command. You can tell it what the prefix of each split file should be and it will then append an alphabet (or number) to the end of each name.

In the example below, I split a file containing 100,000 lines. I instruct split to use numeric suffixes (-d), put 10,000 lines in each split file (-l 10000) and use suffixes of length 3 (-a 3). As a result, ten split files are created, each with 10,000 lines.

$ ls
hugefile

$ wc -l hugefile
100000 hugefile

$ split -d -l 10000 -a 3 hugefile hugefile.split.

$ ls
hugefile                hugefile.split.005
hugefile.split.000      hugefile.split.006
hugefile.split.001      hugefile.split.007  
hugefile.split.002      hugefile.split.008
hugefile.split.003      hugefile.split.009
hugefile.split.004

$ wc -l *split*
 10000 hugefile.split.000
 10000 hugefile.split.001
 10000 hugefile.split.002
 10000 hugefile.split.003
 10000 hugefile.split.004
 10000 hugefile.split.005
 10000 hugefile.split.006
 10000 hugefile.split.007
 10000 hugefile.split.008
 10000 hugefile.split.009
100000 total

Sunday, April 17, 2011

Highlighting Command Output with Generic Colouriser

I recently started using Generic Colouriser, which is a tool that has the ability to colour the output from different programs based on regular expressions. It comes with config files for colouring output from commands such as diff, traceroute, ping, netstat and cvs, but it is also very easy to colour your own commands.

I wrote a config for ps which greys out the processes being run by root and highlights in green, the processes being run by me:

# configuration file for ps
# grey everything out
regexp=^[a-zA-Z]+ .*$
colours=bold black
======
# highlight my processes in green
regexp=^sharfah.*$
colours=green
You can then alias ps to use grc:
alias ps='grc ps'
I've also changed the log config, so that it greys out debug statements, colours warnings yellow and errors red. Most of my logs are produced by log4j. Here is my config:
# this configuration file is suitable for displaying log files
#errors
regexp=^.*(ERROR|Error|Exception).*$
colours=bold red
======
#stack trace
regexp=^\s+at [^:]*:\d*\)$
colours=red
======
regexp=^.*(FATAL|Fatal).*$
colours=on_red
======
regexp=^.*(WARNING|Warning).*$
colours=bold yellow
======
regexp=^.*(DEBUG|Debug).*$
colours=bold black
======
# this is a time
regexp=[^\d]*\d\d:\d\d:\d\d[\.,]{0,1}\d{0,1}\d{0,1}\d{0,1}
colours=bold blue
count=once
======
# ip number
regexp=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}
colours=bold magenta
count=more
I then created a bash function called cless which I use instead of less, to view my logs. I've also aliased tail.
cless ()
{
    if [ $# -eq 0 ]; then
        echo Usage: cless FILE;
        return 1;
    fi;
    grc cat $1 | less -iMF -R
}
alias tail='grc tail'
If you are getting bored of your bland terminal, try this out!

Sunday, February 20, 2011

XPaths with xmllint

xmllint is a command-line XML tool used to validate and pretty-print XML documents. More importantly, it offers an interactive shell mode which allows you to use xpaths to print out elements. For example, //body will print out the body element of an HTML document.

I wrote a useful bash function, which uses xmllint to evaluate xpaths really easily:

xpath()
{
    if [ $# -ne 2]; then
        echo "Usage: xpath xpath file"
        return 1
    fi
    xmllint --shell $2 <<< "cat $1" | sed '/^\/ >/d'
}
Example:
sharfah@starship:~> xpath "//body" index.html
<body>Hello World!</body>

Tuesday, January 12, 2010

Identify which process is using a port

Linux:
Use the lsof command and grep for the port number you are interested in:
sharfah@starship:~>  lsof -Pan -i tcp -i udp | grep :8343
java    27725 sharfah    6u  IPv6 20486040       TCP *:8343 (LISTEN)
Solaris:
If you have lsof installed on Solaris, then you can use the Linux method above. If you don't have or cannot install lsof, then use the pre-installed pfiles in a loop, as shown below:
sharfah@starship:~>  ps -ef | grep myuser | awk '{print $2}' | while read pid
>do
>echo $pid
>pfiles $pid| grep 12197
>done
19424
16132
16040
29373
15946
25178
 121: S_IFREG mode:0640 dev:289,6 ino:259883 uid:50006 gid:106 size:109318
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
        peername: AF_INET 10.232.160.164  port: 12197
15985
16052
18758

Thursday, May 21, 2009

find -exec vs xargs

If you want to execute a command on lots of files found by the find command, there are a few different ways this can be achieved (some more efficient than others):

-exec command {} \;
This is the traditional way. The end of the command must be punctuated by an escaped semicolon. The command argument {} is replaced by the current path name found by find. Here is a simple command which echoes file paths.

sharfah@starship:~> find . -type f -exec echo {} \;
.
./1.txt
./2.txt
This is very inefficient, because whenever find finds a file, it forks a process for your command, waits for this child process to complete and then searches for the next file. In this example, you will get the following child processes: echo .; echo ./1.txt; echo ./2.txt. So if there are 1000 files, there are 1000 child processes and find waits.

-exec command {} +
If you use a plus (+) instead of the escaped semicolon, the arguments will be grouped together before being passed to the command. The arguments must be at the end of the command.

sharfah@starship:~> find . -type f -exec echo {} +
. ./1.txt ./2.txt
In this case, only one child process is created: echo . ./1.txt ./2.txt, which is much more efficient, because it avoids a fork/exec for each single argument.

xargs
This is similar to the approach above, in that files found are bundled up (usually in batches of about 20-50 names) and sent to the command as few times as possible. find doesn't wait for your command to finish.

sharfah@starship:~> find . -type f | xargs echo
. ./1.txt ./2.txt
This approach is efficient and works well as long as you do not have funny characters (e.g. spaces) in your filenames as they won't be escaped.

Performance Testing
So which one of the above approaches is fastest? I ran a test across a directory with 10,000 files out of which 5,600 matched my find pattern. I ran the test 10 times, changing the order of the finds each time, but the results were always the same. xargs and + were very close, with \; always finishing last. Here is one result:

time find . -name "*20090430*" -exec touch {} +
real    0m31.98s
user    0m0.06s
sys     0m0.49s

time find . -name "*20090430*" | xargs touch
real    1m8.81s
user    0m0.13s
sys     0m1.07s

time find . -name "*20090430*" -exec touch {} \;
real    1m42.53s
user    0m0.17s
sys     0m2.42s
I'm going to be using the -exec command {} + method, because it is faster and can handle my funny filenames.

Friday, May 08, 2009

Solaris - CPU, Memory and Version

CPU Info:
In order to find information about processors on Solaris, use the psrinfo command:
sharfah@starship:~> psrinfo -v
Status of virtual processor 0 as of: 05/08/2009 09:53:17
  on-line since 05/03/2009 00:05:06.
  The i386 processor operates at 2612 MHz,
        and has an i387 compatible floating point processor.
Status of virtual processor 1 as of: 05/08/2009 09:53:17
  on-line since 05/03/2009 00:05:12.
  The i386 processor operates at 2612 MHz,
        and has an i387 compatible floating point processor.
Status of virtual processor 2 as of: 05/08/2009 09:53:17
  on-line since 05/03/2009 00:05:14.
  The i386 processor operates at 2612 MHz,
        and has an i387 compatible floating point processor.
Status of virtual processor 3 as of: 05/08/2009 09:53:17
  on-line since 05/03/2009 00:05:16.
  The i386 processor operates at 2612 MHz,
        and has an i387 compatible floating point processor.
Status of virtual processor 4 as of: 05/08/2009 09:53:17
  on-line since 05/03/2009 00:05:18.
  The i386 processor operates at 2612 MHz,
        and has an i387 compatible floating point processor.
Status of virtual processor 5 as of: 05/08/2009 09:53:17
  on-line since 05/03/2009 00:05:20.
  The i386 processor operates at 2612 MHz,
        and has an i387 compatible floating point processor.
Status of virtual processor 6 as of: 05/08/2009 09:53:17
  on-line since 05/03/2009 00:05:22.
  The i386 processor operates at 2612 MHz,
        and has an i387 compatible floating point processor.
Status of virtual processor 7 as of: 05/08/2009 09:53:17
  on-line since 05/03/2009 00:05:24.
  The i386 processor operates at 2612 MHz,
        and has an i387 compatible floating point processor.
Memory Info:
In order to find out how much physical memory is installed, use prtconf:
sharfah@starship:~> prtconf | grep Memory
Memory size: 65536 Megabytes
Version Info:
To show machine, software revision and patch revision information use the showrev command:
sharfah@starship:~> showrev
Hostname: starship
Hostid: 80f32709
Release: 5.10
Kernel architecture: i86pc
Application architecture: i386
Hardware provider:
Kernel version: SunOS 5.10 Generic_137112-06
sharfah@starship:~> uname -a
SunOS starship 5.10 Generic_137112-06 i86pc i386 i86pc
Processes:
In order to list the processes running, use prstat (equivalent to top).
sharfah@starship:~> prstat
  PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 4049 sharfah   1008K  840K sleep    0    0   0:03.17 0.3% find/1
14632 sharfah      114M   68M sleep   29   10   1:19.18 0.1% java/30
Related posts:
Linux - CPU, Memory and Version

Tuesday, March 03, 2009

Howto: Delete Empty Directories [Unix]

Consider the following directory structure:
/tmp
 |-->bar.txt
 |-->dir1/
 |-->dir2/
 |    |-->baz.txt
 |-->dir3/
 |-->foo.txt
There are two files called bar.txt and foo.txt, a non-empty directory called dir2 and two empty directories called dir1 and dir3.

This is how you can delete only the empty directories:

sharfah@starship:/tmp> unalias rmdir
sharfah@starship:/tmp> rmdir *
rmdir: directory "bar.txt": Path component not a directory
rmdir: directory "dir2": Directory not empty
rmdir: directory "foo.txt": Path component not a directory
You need to unalias rmdir just in case you have it aliased to "rm -rf"! You will notice that rmdir does not delete files or non-empty directories. Only dir1 and dir3 are deleted.

Another way to do it, using find:

sharfah@starship:/tmp> find . -type d -exec rmdir {} \;
rmdir: directory ".": Can't remove current directory or ..
rmdir: directory "./dir2": Directory not empty
Note that aliases aren't recognised by find, so even if you did have rmdir aliased, it would not use it.

Wednesday, February 25, 2009

Quick Maven Commands

Creating a Maven project
In order to create a new maven project called MyProject run the following command:
mvn archetype:create -DgroupId=fs.work -DartifactId=MyProject
This will create a new directory called MyProject with a pom.xml and the following tree structure:
MyProject
 |-->pom.xml
 |-->src
 |  |-->main
 |  |  |-->java
 |  |  |  |-->fs
 |  |  |  |  |-->work
 |  |  |  |  |  |-->App.java
 |  |-->test
 |  |  |-->java
 |  |  |  |-->fs
 |  |  |  |  |-->work
 |  |  |  |  |  |-->AppTest.java
The pom file looks like this:

  4.0.0
  fs.work
  MyProject
  jar
  1.0-SNAPSHOT
  MyProject
  http://maven.apache.org
  
    
      junit
      junit
      3.8.1
      test
    
  

Creating sub-modules
If you need to create sub-modules within your project, you need to change the packaging in the pom file (i.e. the "super" pom), to pom. Then, from within the MyProject directory issue the following commands to create sub-modules:
mvn archetype:create -DgroupId=fs.work -DartifactId=MyProjectWeb -Dpackaging=war
mvn archetype:create -DgroupId=fs.work -DartifactId=MyProjectModule1 -Dpackaging=jar
This creates the sub-modules and the directory tree now looks like this:
MyProject
 |-->pom.xml
 |-->src
 |  |-->main
 |  |  |-->java
 |  |  |  |-->fs
 |  |  |  |  |-->work
 |  |  |  |  |  |-->App.java
 |  |-->test
 |  |  |-->java
 |  |  |  |-->fs
 |  |  |  |  |-->work
 |  |  |  |  |  |-->AppTest.java
 |-->MyProjectModule1
 |  |-->pom.xml
 |  |-->src
 |  |  |-->main
 |  |  |  |-->java
 |  |  |  |  |-->fs
 |  |  |  |  |  |-->work
 |  |  |  |  |  |  |-->App.java
 |  |  |-->test
 |  |  |  |-->java
 |  |  |  |  |-->fs
 |  |  |  |  |  |-->work
 |  |  |  |  |  |  |-->AppTest.java
 |-->MyProjectWeb
 |  |-->pom.xml
 |  |-->src
 |  |  |-->main
 |  |  |  |-->java
 |  |  |  |  |-->fs
 |  |  |  |  |  |-->work
 |  |  |  |  |  |  |-->App.java
 |  |  |-->test
 |  |  |  |-->java
 |  |  |  |  |-->fs
 |  |  |  |  |  |-->work
 |  |  |  |  |  |  |-->AppTest.java
The pom file for MyProjectModule1 contains a reference to the parent and looks like this:

  
    MyProject
    fs.work
    1.0-SNAPSHOT
  
  4.0.0
  fs.work
  MyProjectModule1
  MyProjectModule1
  1.0-SNAPSHOT
  http://maven.apache.org
  
    
      junit
      junit
      3.8.1
      test
    
  

Deploying a jar to the repository
If you have a jar file called myarchive.jar which you want to upload to your maven repository, use the following command:
mvn deploy:deploy-file -Durl=scp://hostname/dir/to/maven -DrepositoryId=fs.repo -Dfile=myarchive.jar -DgroupId=fs.work -DartifactId=myarchive -Dversion=1.0 -Dpackaging=jar
This will create dir/to/maven/fs/work/myarchive/1.0/myarchive-1.0.jar in the maven repository.

Creating a dependency
To create a dependency on myarchive.jar, add the following dependency to your pom:


  fs.work
  myarchive
  1.0

Generating Eclipse .project and .classpath files
Use the following command:
mvn eclipse:eclipse
Skipping tests
To skip tests use the property maven.test.skip=true.
mvn -Dmaven.test.skip=true install
Release a project
Two commands must be invoked, in the following order:
mvn release:prepare
mvn release:perform
Other commands
mvn install
mvn clean
mvn compile
mvn jar:jar

Saturday, November 22, 2008

UNIX Fork Bomb :(){ :|:& };:

Whatever you do, please do NOT enter this code in your Unix terminal:
:(){ :|:& };:
This is one of the most elegant examples of a fork bomb, which works by creating a large number of processes very quickly in order to saturate the operating system's process table. Each process uses up CPU time and memory and so the system becomes unresponsive and is quickly brought to its knees! This is a form of denial-of-service attack.

So how does this command work? It might be easier to understand if I re-wrote it like this instead:

bomb()
{
   bomb | bomb &
}
bomb
You declare a function called bomb which calls itself recursively and pipes the output to another call of itself. The & puts the function call in the background so that the new child processes can never die. The semi-colon marks the end of the function and the final "bomb" launches the attack (by calling the function the first time).

You can only stop a fork bomb by destroying all instances of it. It is a difficult task to use another program to kill it because that would mean creating another process which the system may not have enough space for. The only guaranteed way of curing a fork bomb is to reboot.

Windows Fork bomb Example

:s
start %0
%0|%0
goto :s

Monday, October 06, 2008

Extract a tar.gz File in a Single Command [Unix]

The obvious way to extract a tar.gz archive is using the following two commands:
sharfah@starship:~> gunzip foo.tar.gz
sharfah@starship:~> tar xvf foo.tar
This is how you can achieve the same thing, but in a single command:
sharfah@starship:~> gunzip -c foo.tar.gz | tar xvf -
or alternatively:
sharfah@starship:~> gunzip < foo.tar.gz | tar xvf -
Both commands write the data produced from the gunzip command to standard out (-c in the first example and using < in the second). The output is then piped into the tar command. The "-" represents standard input.

Creating a tar.gz:
A tar.gz file is normally created using two commands as follows:

sharfah@starship:~> tar cvf foo.tar foodir
sharfah@starship:~> gzip foo.tar
Again, there is a single command to do the same thing:
sharfah@starship:~> tar cvf - foodir | gzip > foo.tar.gz

Friday, October 03, 2008

Checking CPU Utilisation on Linux

Here are a few commands which can be used to investigate CPU utilisation on Linux:

top
The top program provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel. The CPU usage shows the task's share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time.

top - 18:12:28 up 40 days, 18:35,  1 user,  load average: 0.13, 0.03, 0.01
Tasks:  98 total,   1 running,  96 sleeping,   0 stopped,   1 zombie
Cpu(s):  0.2% us,  0.0% sy,  0.0% ni, 99.4% id,  0.4% wa,  0.0% hi,  0.0% si
Mem:   8002512k total,  3845332k used,  4157180k free,    64624k buffers
Swap:  9437144k total,   771048k used,  8666096k free,  1831288k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    1 root      16   0   640   80   48 S  0.0  0.0  43:04.18 init
    2 root      RT   0     0    0    0 S  0.0  0.0   0:00.70 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.02 ksoftirqd/0
    4 root      RT   0     0    0    0 S  0.0  0.0   0:00.58 migration/1
    5 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/1
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.40 migration/2
    7 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/2
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.46 migration/3

sar
The sar command can be used to display a history of CPU usage:
10:00:01   CPU     %user     %nice   %system   %iowait   %idle
10:10:01   all     34.45      2.04     30.03      0.05   33.43
10:20:01   all     34.13      1.77     29.85      0.08   34.17

mpstat
The mpstat command can be used to show the percentage of CPU usage for each processor:
> mpstat -P ALL

18:02:42     CPU   %user   %nice %system %iowait    %irq
18:02:42     all   24.77   17.51   21.56    1.19    0.01
18:02:42       0   25.00   17.92   20.27    0.94    0.00
18:02:42       1   23.81   17.03   20.87    0.88    0.00
18:02:42       2   26.28   16.44   22.23    1.54    0.01
18:02:42       3   24.00   18.65   22.86    1.39    0.01
Order ps output
The following command displays the top 10 CPU users on your system. It involves listing the processes using ps and then sorting them by CPU usage:
> ps -eo pcpu,pid,user,cmd | sort -k 1 -r | head -10

%CPU   PID USER     CMD
 0.1    13 root     [events/3]
 0.1   103 root     [kswapd0]
 0.1   102 root     [kswapd1]
 0.0     9 root     [ksoftirqd/3]
 0.0     8 root     [migration/3]
 0.0  8982 root     [lockd]
 0.0  8981 root     [rpciod]
 0.0     7 root     [ksoftirqd/2]
 0.0    75 root     [kblockd/3]