Saturday, September 15, 2018

Java: Streaming a JDBC ResultSet as JSON

This post shows how you can convert a java.sql.ResultSet to JSON and stream it back to the caller. This is useful if you want to send a large dataset from a JDBC DataSource to a web application in JSON format. Streaming allows you to transfer the data, little by little, without having to load it all into the server's memory.

For example, consider the following ResultSet:

+---------+-----+
| name    | age |
+---------+-----+
| Alice   |  20 |
| Bob     |  35 |
| Charles |  50 |
+---------+-----+

The corresponding JSON is:

[
  { "name": "Alice",   "age": 20 },
  { "name": "Bob",     "age": 35 },
  { "name": "Charles", "age": 50 },
]

The following class (also available in my GitHub Repository) can be used to convert the ResultSet to JSON. Note that this class implements Spring's ResultSetExtractor, which can be used by a JdbcTemplate to extract results from a ResultSet.

/**
 * Streams a ResultSet as JSON.
 */
public class StreamingJsonResultSetExtractor implements ResultSetExtractor<Void> {

  private final OutputStream os;

  /**
   * @param os the OutputStream to stream the JSON to
   */
  public StreamingJsonResultSetExtractor(final OutputStream os) {
    this.os = os;
  }

  @Override
  public Void extractData(final ResultSet rs) {
    final var objectMapper = new ObjectMapper();
    try (var jg = objectMapper.getFactory().createGenerator(
                  os, JsonEncoding.UTF8)) {
      writeResultSetToJson(rs, jg);
      jg.flush();
    } catch (IOException | SQLException e) {
      throw new RuntimeException(e);
    }
    return null;
  }

  private static void writeResultSetToJson(final ResultSet rs,
                            final JsonGenerator jg)
                            throws SQLException, IOException {
    final var rsmd = rs.getMetaData();
    final var columnCount = rsmd.getColumnCount();
    jg.writeStartArray();
    while (rs.next()) {
      jg.writeStartObject();
      for (var i = 1; i <= columnCount; i++) {
        jg.writeObjectField(rsmd.getColumnName(i), rs.getObject(i));
      }
      jg.writeEndObject();
    }
    jg.writeEndArray();
  }
}

To use this in a web service with JAX-RS:

import javax.ws.rs.core.StreamingOutput;

@GET
@Path("runQuery")
public StreamingOutput runQuery() {
  return new StreamingOutput() {
    @Override
    public void write(final OutputStream os)
        throws IOException, WebApplicationException {
      jdbcTemplate.query("select name, age from person",
                   new StreamingJsonResultSetExtractor(os));
    }
  };
}

Saturday, September 08, 2018

Java: Mocking a ResultSet using Mockito

This post shows how you can mock a java.sql.ResultSet using Mockito. It can be used to help unit test code which performs operations on ResultSets (such as a ResultSetExtractor) without relying on an external datasource.

You can create a MockResultSet by providing a list of column names and a 2D array of data. For example:

var rs = MockResultSet.create(
           new String[] { "name", "age" }, //columns
           new Object[][] { // data
             { "Alice", 20 },
             { "Bob", 35 },
             { "Charles", 50 }
           });

The code for MockResultSet is shown below (also available in my GitHub Repository). Note that I have only mocked a few methods such as next, getString and getObject but it is quite easy to mock the rest by following the same pattern.

public class MockResultSet {

  private final Map<String, Integer> columnIndices;
  private final Object[][] data;
  private int rowIndex;

  private MockResultSet(final String[] columnNames,
                        final Object[][] data) {
    // create a map of column name to column index
    this.columnIndices = IntStream.range(0, columnNames.length)
        .boxed()
        .collect(Collectors.toMap(
            k -> columnNames[k],
            Function.identity(),
            (a, b) ->
              { throw new RuntimeException("Duplicate column " + a); },
            LinkedHashMap::new
            ));
    this.data = data;
    this.rowIndex = -1;
  }

  private ResultSet buildMock() throws SQLException {
    final var rs = mock(ResultSet.class);

    // mock rs.next()
    doAnswer(invocation -> {
      rowIndex++;
      return rowIndex < data.length;
    }).when(rs).next();

    // mock rs.getString(columnName)
    doAnswer(invocation -> {
      final var columnName = invocation.getArgumentAt(0, String.class);
      final var columnIndex = columnIndices.get(columnName);
      return (String) data[rowIndex][columnIndex];
    }).when(rs).getString(anyString());

    // mock rs.getObject(columnIndex)
    doAnswer(invocation -> {
      final var index = invocation.getArgumentAt(0, Integer.class);
      return data[rowIndex][index - 1];
    }).when(rs).getObject(anyInt());

    final var rsmd = mock(ResultSetMetaData.class);

    // mock rsmd.getColumnCount()
    doReturn(columnIndices.size()).when(rsmd).getColumnCount();

    // mock rs.getMetaData()
    doReturn(rsmd).when(rs).getMetaData();

    return rs;
  }

  /**
   * Creates the mock ResultSet.
   *
   * @param columnNames the names of the columns
   * @param data
   * @return a mocked ResultSet
   * @throws SQLException
   */
  public static ResultSet create(
                         final String[] columnNames,
                         final Object[][] data)
                         throws SQLException {
    return new MockResultSet(columnNames, data).buildMock();
  }
}

Saturday, June 02, 2018

Java 10: Collecting a Stream into an Unmodifiable Collection

Java 10 introduces several new methods to facilitate the creation of unmodifiable collections.

The List.copyOf, Set.copyOf, and Map.copyOf methods create new collection instances from existing instances. For example:

List<String> modifiable = Arrays.asList("foo", "bar");
List<String> unmodifiableCopy = List.copyOf(list);

// Note that since Java 9, you can also use "of" to create
// unmodifiable collections
List<String> unmodifiable = List.of("foo", "bar");

There are also new collector methods, toUnmodifiableList, toUnmodifiableSet, and toUnmodifiableMap, to allow the elements of a stream to be collected into an unmodifiable collection. For example:


// Java 10
Stream.of("foo", "bar").collect(toUnmodifiableList());

// before Java 10
Stream.of("foo", "bar").collect(
    collectingAndThen(toList(), Collections::unmodifiableList));

Monday, May 28, 2018

Java 10: The "var" keyword

Java 10 has introduced local variable type inference with the keyword var. This means that instead of writing:

Map<Department, List<Employee>> map = new HashMap<>();
// ...
for (Entry<Department, List<Employee>> dept : map.entrySet()) {
  List<Employee> employees = dept.getValue();
  // ...
}

you can use var to reduce boilerplate, as follows:

var map = new HashMap<Department, Employee>();
// ...
for (var dept : map.entrySet()) {
  var employees = dept.getValue();
  // ...
}

var removes the need to have explicit types written in your code. Not only does this reduce repetition but it also makes your code easier to maintain because if, for example, you decide to change the types of the objects stored in your map in the future, you would only need to alter one line of code.

Polymorphism:

Now let's take a look at how var behaves with polymorphic code. For example, if you have a class Shape with two subclasses, Square and Circle, what type will be inferred if you use var v = new Circle()? Let's try it out in JShell:

jshell> var v = new Circle();
v ==> Circle@4e9ba398

jshell> v.getClass();
$13 ==> class Circle

jshell> v = new Square();
|  Error:
|  incompatible types: Square cannot be converted to Circle
|  v = new Square();
|      ^----------^

As demonstrated above, v is of type Circle and if you try to reassign it to Square, the compiler will throw an error.

Anonymous classes:

One of the exciting things you can do with vars is create anonymous classes and refer to fields inside them! For example:

var person = new Object() {
  String name = "Joe";
  int age = 10;
};
System.out.println(person.name + ":" + person.age);

var infers the type of the anonymous class which means that you can use an anonymous class as a "holder" for intermediate values, without having to create and maintain a new class. Here is another example showing how you can create "temporary" person objects:

var persons = Stream.of("Alice", "Bob", "Charles")
    .map(s -> new Object() {
       String name = s;
       int age = 10;
    })
    .collect(toList());
persons.forEach(p -> System.out.println(p.name));
Other points to note:

You cannot use var without an explicit initialisation (assigning to null does not count) or within lambda expressions:

jshell> var v;
|  Error:
|  cannot infer type for local variable v
|    (cannot use 'var' on variable without initializer)
|  var v;
|  ^----^

jshell> var v = null;
|  Error:
|  cannot infer type for local variable v
|    (variable initializer is 'null')
|  var v = null;
|  ^-----------^

jshell> var v = () -> {}
|  Error:
|  cannot infer type for local variable v
|    (lambda expression needs an explicit target-type)
|  var v = () -> {};
|  ^---------------^

Saturday, May 26, 2018

HTML5 Date Input with jQuery Fallback

HTML5 introduced a new date input type which allows a user to enter a date using a date picker.

<input id="date" type="date" value="2018-05-26">

This is what it looks like in Chrome:

However, not all browsers support this input type. In unsupported browsers, such as Internet Explorer, you will simply see a text input field.

In this post, I will show how you can detect if a browser supports the date input type and how you can fall back to using jQuery's date picker if it doesn't.

Checking if the browser supports date input:

The following code creates an input element and sets its type to date. If the browser does not support date input, this operation will not work and the input type will degrade to text.

var input = document.createElement("input");
input.setAttribute("type", "date");
if (input.type !== "date") {
    console.log("browser does not support date input");
}
Alternatively, use the Modernizr library, which makes it easy to detect the features that a browser supports:
if (!Modernizr.inputtypes.date) {
    console.log("browser does not support date input");
}
Falling back to jQuery's date picker:

The JSFiddle below shows how you would use jQuery's date picker if the browser does not support date input.

Saturday, May 19, 2018

HTML: Disabling a Form on Submit

The following HTML snippet shows how you can disable the Submit button on a form to prevent multiple submissions. When the Submit button is clicked, the button is disabled and a progress bar is displayed.

Saturday, April 21, 2018

HTML5 - Styling a Progress Bar

The progress tag introduced in HTML5 can be used to represent the progress of a task.

For example, the following code:

<progress value="80" max="100"></progress>

displays:

Unsupported browser

Different browsers display the progress bar in different styles:

Changing the colour of the progress bar:
Now let's say that you wish to change the colour of the progress bar so that it is red if the value is less than the maximum and green when it equals the maximum. In order to do this, you can use the following CSS, which should work in Chrome, Firefox and IE:
[JSFiddle]

Creating a progress bar without HTML5:
If, like me, you would rather not have browser-specific CSS, then another approach is to create a progress bar without using the HTML5 progress tag. This is quite easy, as demonstrated here:
[JSFiddle]

Saturday, March 10, 2018

JUnit Hamcrest Matcher for JSON

This post shows how you can write JUnit tests to check if an object matches a JSON string. This is important if you are implementing REST services and want to test if your service produces the expected JSON response.

A useful library for comparing JSON objects is JSONassert. First, you have to convert your Java object into a JSON string (using Jackson, for example) and then compare it with your expected JSON string using JSONassert. (You could also convert your Java object into a JSONObject but I find that it's much easier to convert it into a string.)

The following snippet shows how you can compare an object (a List, in this case) against its JSON representation using JSONassert.

import org.skyscreamer.jsonassert.JSONAssert;
import com.fasterxml.jackson.databind.ObjectMapper;

List<String> fruits = Arrays.asList("apple", "banana");
String fruitsJSON = new ObjectMapper().writeValueAsString(fruits);
String expectedFruitsJSON = "[\"apple\", \"banana\"]";
JSONAssert.assertEquals(expectedFruitsJSON, fruitsJSON, true);

In order to make it simpler to write such unit tests, I have written a Hamcrest Matcher called IsEqualJSON for comparing JSON objects. It still uses JSONassert but allows you to express your tests in a more fluent way.

The following code shows how IsEqualJSON is used:

import static org.junit.Assert.*;
import static testutil.IsEqualJSON.*;

assertThat(Arrays.asList("apple", "banana"),
           equalToJSON("[\"apple\", \"banana\"]"));

// you can also have your expected JSON read from a file
assertThat(Arrays.asList("apple", "banana"),
           equalToJSONInFile("fruits.json"));

Here is the code for IsEqualJSON (also available in my GitHub Repository):

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.*;
import org.hamcrest.*;
import org.skyscreamer.jsonassert.*;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;

/**
 * A Matcher for comparing JSON.
 * Example usage:
 * <pre>
 * assertThat(new String[] {"foo", "bar"}, equalToJSON("[\"foo\", \"bar\"]"));
 * assertThat(new String[] {"foo", "bar"}, equalToJSONInFile("/tmp/foo.json"));
 * </pre>
 */
public class IsEqualJSON extends DiagnosingMatcher<Object> {

  private final String expectedJSON;
  private JSONCompareMode jsonCompareMode;

  public IsEqualJSON(final String expectedJSON) {
    this.expectedJSON = expectedJSON;
    this.jsonCompareMode = JSONCompareMode.STRICT;
  }

  @Override
  public void describeTo(final Description description) {
    description.appendText(expectedJSON);
  }

  @Override
  protected boolean matches(final Object actual,
                            final Description mismatchDescription) {
    final String actualJSON = toJSONString(actual);
    final JSONCompareResult result = JSONCompare.compareJSON(expectedJSON,
                                                             actualJSON,
                                                             jsonCompareMode);
    if (!result.passed()) {
      mismatchDescription.appendText(result.getMessage());
    }
    return result.passed();
  }

  private static String toJSONString(final Object o) {
    try {
      return o instanceof String ?
          (String) o : new ObjectMapper().writeValueAsString(o);
    } catch (final JsonProcessingException e) {
      throw new RuntimeException(e);
    }
  }

  private static String getFileContents(final Path path) {
    try {
      return new String(Files.readAllBytes(path), StandardCharsets.UTF_8);
    } catch (final IOException e) {
      throw new RuntimeException(e);
    }
  }

  @Factory
  public static IsEqualJSON equalToJSON(final String expectedJSON) {
    return new IsEqualJSON(expectedJSON);
  }

  @Factory
  public static IsEqualJSON equalToJSONInFile(final Path expectedPath) {
    return equalToJSON(getFileContents(expectedPath));
  }

  @Factory
  public static IsEqualJSON equalToJSONInFile(final String expectedFileName) {
    return equalToJSONInFile(Paths.get(expectedFileName));
  }
}

Sunday, February 25, 2018

Java 9: Enhancements to the Process API

Java 9 brings various improvements to the Process API, used for controlling and managing operating system processes.

Getting information about a process

There is a new ProcessHandle class which provides the process's pid, parent and descendants, as well as information about the start time and accumulated CPU time.

jshell> Process p = new ProcessBuilder("stress", "--cpu", "4", "--timeout", "5").start();
p ==> Process[pid=5572, exitValue="not exited"]

jshell> p.pid()
$2 ==> 5572

jshell> p.info().user()
$3 ==> Optional[fahd]

jshell> p.info().command()
$4 ==> Optional[/usr/bin/stress]

jshell> p.info().commandLine()
$5 ==> Optional[/usr/bin/stress --cpu 4 --timeout 120]

jshell> Arrays.toString(p.info().arguments().get())
$6 ==> "[--cpu, 4, --timeout, 120]"

jshell> p.info().startInstant()
$7 ==> Optional[2018-02-25T16:38:56.742Z]

jshell> p.info().totalCpuDuration().get().toMillis()
$8 ==> 0

It's strange that totalCpuDuration always returns 0 (a duration string of "PT0S"), no matter what command I run.

Note that I am invoking the Linux stress command in the example above. This is a useful tool for imposing a certain type of stress (e.g. creating cpu load) on your system.

Listing all running processes

The static ProcessHandle.allProcesses() method returns a stream of all processes visible to the current process.

ProcessHandle.allProcesses()
             .map(ProcessHandle::info)
             .map(ProcessHandle.Info::commandLine)
             .flatMap(Optional::stream)
             .forEach(System.out::println)

Triggering a function when a process exits

The Process.onExit method can be used to schedule a function when a process terminates. This method returns a CompletableFuture, which contains a variety of methods that can be called to schedule functions. Here is an example:

Process proc = new ProcessBuilder("sleep", "10").start();
proc.onExit()
    .thenAccept(p -> System.out.println("Process " + p.pid() + " exited with " + p.exitValue()));

Alternatively, to wait for a process to terminate, you can call Process.onExit().get().

Monday, January 01, 2018

fahd.blog in 2017

Happy 2018, everyone!

I'd like to wish everyone a great start to an even greater new year!

In keeping with tradition, here's one last look back at fahd.blog in 2017.

During 2017, I posted 17 new entries on fahd.blog. I am also thrilled that I have more readers from all over the world! Thanks for reading and especially for giving feedback.

Top 5 posts of 2017:

I'm going to be writing a lot more this year, so stay tuned for more great techie tips, tricks and hacks! :)

Related posts: