Joining Two Files with the Unix join Command

The join command is a useful tool for joining two files on a common field. It allows you to join two files, similar to the way you would join two tables in a SQL database.

The following example illustrates the power of the join command. You have two files, one containing a list of employees with their department ids and the other containing departments and their ids. You want to find out the names of the departments for each employee. You MUST first sort the files on the department id column (using the sort command) and then join them on that column.

$ cat employees.txt
Jones,33
Steinberg,33
Robinson,34
Smith,34
Rafferty,31
John,

$ cat departments.txt
31,Sales
33,Engineering
34,Clerical
35,Marketing

$ join -a 1 -t, -1 2 -2 1 -o 1.1 2.2 <(sort -t, -k2 employees.txt) <(sort -t, -k1 departments.txt)
John,
Rafferty,Sales
Jones,Engineering
Steinberg,Engineering
Robinson,Clerical
Smith,Clerical

Joining on multiple columns
The join command joins on a single field. What do you do if you want to join on multiple fields? You create a composite field by combining the multiple fields together! This can be done using awk. For example:

$ cat employees2.txt
Jones,33,50
Steinberg,33,51
Robinson,34,50
Smith,34,50
Rafferty,31,51

$ awk -F, '{print $2"_"$3","$0}' employees2.txt
33_50,Jones,33,50
33_51,Steinberg,33,51
34_50,Robinson,34,50
34_50,Smith,34,50
31_51,Rafferty,31,51

As you can see, an additional field has been created by concatenating the second and third fields of the file. Now you can join the files on the new composite field.

(File data courtesy of Wikipedia.)

Java: Find an Available Port Number

In some cases, such as in unit tests, you might need to start up a server or an rmiregistry. What port number do you use? You cannot hardcode the port number because when your unit test runs on a continuous build server or on a colleague's machine, it might already be in use. Instead, you need a way to find an available port on the current machine.

According to IANA (Internet Assigned Numbers Authority), the ports that we are free to use lie in the range 1024-49151:

Port numbers are assigned in various ways, based on three ranges: System Ports (0-1023), User Ports (1024-49151), and the Dynamic and/or Private Ports (49152-65535)

The following utility class can help find an available port on your local machine:

import java.io.IOException;
import java.net.DatagramSocket;
import java.net.ServerSocket;
 
/**
 * Finds an available port on localhost.
 */
public class PortFinder {
 
  // the ports below 1024 are system ports
  private static final int MIN_PORT_NUMBER = 1024;
 
  // the ports above 49151 are dynamic and/or private
  private static final int MAX_PORT_NUMBER = 49151;
 
  /**
   * Finds a free port between 
   * {@link #MIN_PORT_NUMBER} and {@link #MAX_PORT_NUMBER}.
   *
   * @return a free port
   * @throw RuntimeException if a port could not be found
   */
  public static int findFreePort() {
    for (int i = MIN_PORT_NUMBER; i <= MAX_PORT_NUMBER; i++) {
      if (available(i)) {
        return i;
      }
    }
    throw new RuntimeException("Could not find an available port between " + 
                               MIN_PORT_NUMBER + " and " + MAX_PORT_NUMBER);
  }
 
  /**
   * Returns true if the specified port is available on this host.
   *
   * @param port the port to check
   * @return true if the port is available, false otherwise
   */
  private static boolean available(final int port) {
    ServerSocket serverSocket = null;
    DatagramSocket dataSocket = null;
    try {
      serverSocket = new ServerSocket(port);
      serverSocket.setReuseAddress(true);
      dataSocket = new DatagramSocket(port);
      dataSocket.setReuseAddress(true);
      return true;
    } catch (final IOException e) {
      return false;
    } finally {
      if (dataSocket != null) {
        dataSocket.close();
      }
      if (serverSocket != null) {
        try {
          serverSocket.close();
        } catch (final IOException e) {
          // can never happen
        }
      }
    }
  }
}

Testing Email with a Mock MailSender

If you have an application which sends out email, you don't want your unit tests doing that too, so you need to use a "mock mail sender". You can create one by extending JavaMailSenderImpl and overriding the send method so that it doesn't really send an email. Here is an example:

import java.util.Properties;

import javax.mail.internet.MimeMessage;

import org.springframework.mail.MailException;
import org.springframework.mail.MailPreparationException;
import org.springframework.mail.javamail.JavaMailSenderImpl;
import org.springframework.mail.javamail.MimeMessagePreparator;

public class MockMailSender extends JavaMailSenderImpl {

  @Override
  public void send(final MimeMessagePreparator mimeMessagePreparator) throws MailException {
    final MimeMessage mimeMessage = createMimeMessage();
    try {
      mimeMessagePreparator.prepare(mimeMessage);
      final String content = (String) mimeMessage.getContent();
      final Properties javaMailProperties = getJavaMailProperties();
      javaMailProperties.setProperty("mailContent", content);
    } catch (final Exception e) {
      throw new MailPreparationException(e);
    }
  }
}

The mock shown above stores the email body into the mail properties object. This is a quick-and-dirty way of getting access to the content of the email just in case you want to check it in your unit test.

Here is the associated Spring Java-based configuration:

@Configuration
public class MailConfig {

  @Bean
  public JavaMailSender mailSender() {
    final JavaMailSenderImpl sender = new JavaMailSenderImpl();
    sender.setHost("mail.host.com");
    return sender;
  }

  @Bean
  public Notifier notifier() {
    return new Notifier(mailSender());
  }
}

The unit-test configuration:

@Configuration
@Profile("unit-test")
public class UnitTestMailConfig extends MailConfig {
  @Override
  @Bean
  public JavaMailSender mailSender() {
   return new MockMailSender();
  }
}

For more information about sending emails with Spring 3, see the documentation here.

Related Posts:
Spring 3 - JavaConfig: Unit Testing

Installing Scala IDE for Eclipse Juno

Just spent ages trying to get the Scala IDE plugin working in Eclipse Juno (4.2).

Here are the links that finally worked for me:

• Scala IDE: http://download.scala-ide.org/nightly-update-juno-master-29x
• Scala Worksheet: http://download.scala-ide.org/nightly-build-worksheet-scalaide21-29/site

To install them go to Eclipse > Help > Install new software... and enter the URL in the Work with... textfield.

stackoverflow - 40k rep

Eight months after crossing the 30k milestone, I've now achieved a reputation of 40k on stackoverflow! The following table shows some interesting stats about my journey so far:

	0-10k	10-20k	20-30k	30-40k	Total
Date achieved	01/2011	05/2011	01/2012	09/2012
Questions answered	546	376	253	139	1314
Questions asked	46	1	6	0	53
Tags covered	609	202	83	10	904
Badges (gold, silver, bronze)	35 (2, 10, 23)	14 (0, 4, 10)	33 (2, 8, 23)	59 (3, 20, 36)	141 (7, 42, 92)

As I mentioned before, I have really enjoyed being a member of stackoverflow. For me, it has not simply been a quest for reputation, but more about learning new technologies and picking up advice from other experts on the site. I like to take on challenging questions, rather than the easy ones, because it pushes me to do research into areas I have never looked at before, and I learn so much during the process.

You can probably tell by the number of questions answered, that I haven't spent much time on stackoverflow recently. I've been busy at work and have also been participating in other stackexchange sites like superuser, serverfault and Unix and Linux.

50k, here I come!