Kdb+/q - File Compression

Large tables can be compressed in Kdb+ by setting .z.zd. Compression of data can reduce disk cost and in some cases even improve performance for applications that have fast CPUs but slow disks.

.z.zd is a list of three integers consisting of logical block size, algorithm (0=none, 1=q, 2=gzip, 3=snappy, 4=lz4hc) and compression level.

Here is an example showing how to compress a table:

// Helper function that sets .z.zd and
// returns the previous value of .z.zd
.util.setZzd:{
  origZzd:$[count key `.z.zd;.z.zd;()];
  if[x~();
    system"x .z.zd";
    :origZzd;
  ];
  .z.zd:x;
  origZzd}

// create a table
td:([]a:1000000?10; b:1000000?10; c:1000000?10);

// save the table to disk without compression
`:uncompressed set td;

// save the table to disk using q IPC compression
origZzd:.util.setZzd[(17;1;0)];
`:compressed set td;
.util.setZzd[origZzd];

You can check compression stats by using the -21! function:

q)-21!`:compressed
compressedLength  | 5747890
uncompressedLength| 24000041
algorithm         | 1i
logicalBlockSize  | 17i
zipLevel          | 0i

The size of the file on disk is reduced from 22.8 MB to 5.5 MB after using q IPC compression.

Testing Expected Exceptions with JUnit 5

This post shows how to test for expected exceptions using JUnit 5. If you're still on JUnit 4, please check out my previous post.

Let's start with the following class that we wish to test:

public class Person {
  private final String name;
  private final int age;
    
  /**
   * Creates a person with the specified name and age.
   *
   * @param name the name
   * @param age the age
   * @throws IllegalArgumentException if the age is not greater than zero
   */
  public Person(String name, int age) {
    this.name = name;
    this.age = age;
    if (age <= 0) {
      throw new IllegalArgumentException("Invalid age:" + age);
    }
  }
}

To test that an IllegalArgumentException is thrown if the age of the person is less than zero, you should use JUnit 5's assertThrows as shown below:

import static org.hamcrest.CoreMatchers.*;
import static org.hamcrest.MatcherAssert.*;
import static org.junit.jupiter.api.Assertions.*;

import org.junit.jupiter.api.Test;

class PersonTest {

  @Test
  void testExpectedException() {
    assertThrows(IllegalArgumentException.class, () -> {
      new Person("Joe", -1);
    });
  }

  @Test
  void testExpectedExceptionMessage() {
    final Exception e = assertThrows(IllegalArgumentException.class, () -> {
      new Person("Joe", -1);
    });
    assertThat(e.getMessage(), containsString("Invalid age"));
  }
}

Related post: Testing Expected Exceptions with JUnit 4 Rules