In Java 16, Records and Pattern Matching have been made a final and permanent feature of the Java language!
I blogged about them when they were first released as preview language features back in Java 14 here:
Monday, April 05, 2021
Friday, April 02, 2021
kdb+/q - Display a Table as a Tree
This post shows how you can convert a keyed table to a hierarchical tree format in kdb+/q. This could be useful if you want to display data as a tree widget in a front-end.
Consider the following keyed table of world populations:
continent country city | population ---------------------------------------------| ---------- North America United States New York City | 8550405 North America United States Los Angeles | 3971883 North America Mexico Mexico City | 8918653 Europe United Kingdom London | 9126366 Europe Russia Moscow | 12195221 Europe Russia Saint Petersburg| 5383890 Africa Nigeria Lagos | 14862000 Africa Egypt Cairo | 9908788 Africa Egypt Giza | 8800000 Asia China Shanghai | 22315474 Asia India Mumbai | 12691836 Asia China Beijing | 11716620
We would like to display it as a tree of continent > country > city, as shown below (similar to a pivot table in Excel):
node | population
----------------------------| ----------
Total | 128441136
Asia | 46723930
China | 34032094
Shanghai | 22315474
Beijing | 11716620
India | 12691836
Mumbai | 12691836
Africa | 33570788
Egypt | 18708788
Cairo | 9908788
Giza | 8800000
Nigeria | 14862000
Lagos | 14862000
Europe | 26705477
Russia | 17579111
Moscow | 12195221
Saint Petersburg| 5383890
United Kingdom | 9126366
London | 9126366
North America | 21440941
United States | 12522288
New York City | 8550405
Los Angeles | 3971883
Mexico | 8918653
Mexico City | 8918653
In order to achieve this, we need to aggregate the data with different groupings, then combine the resultant tables and format it into a tree.
1. Grouping the data
First, we will add a dummy Total column to the table and then aggregate the table with the following groupings:
- Total
- Total, continent
- Total, continent, country
- Total, continent, country, city
The code for this is shown below:
// add Total column to the table. (td is a keyed table)
td:(`Total,keys[td]) xkey update Total:`Total from td;
keyCols:keys td;
// create a list of groupings
groupings:(1+til count keyCols) sublist\: keyCols;
// aggregate the table with each grouping
// this gives us a list of keyed tables (one per grouping)
tds:?[td;();;c!(sum;)each c:cols value td] each {x!x} each groupings;
// this step is optional but it's nice to sort each table on population
tds:`population xdesc'tds;
2. Joining the data
Next, we need to join the tables that were obtained as a result of the groupings. We do this by unkeying the tables and then using uj:
td:keyCols xkey (uj/) 0!'tds;
3. Formatting the data
Now let's add a Path column by concatenating the key columns:
td:![td;();0b;enlist[`Path]!enlist(`$sv';">";(string;(each;{x except `};(flip;enlist,keyCols))))];
td:(`Path,keyCols) xkey td;
This is what our tree looks like so far:
Path Total continent country city | population ---------------------------------------------------------------------------------------------------| ---------- Total Total | 128441136 Total>Asia Total Asia | 46723930 Total>Africa Total Africa | 33570788 Total>Europe Total Europe | 26705477 Total>North America Total North America | 21440941 Total>Asia>China Total Asia China | 34032094 Total>Africa>Egypt Total Africa Egypt | 18708788 Total>Europe>Russia Total Europe Russia | 17579111 Total>Africa>Nigeria Total Africa Nigeria | 14862000 Total>Asia>India Total Asia India | 12691836 Total>North America>United States Total North America United States | 12522288 Total>Europe>United Kingdom Total Europe United Kingdom | 9126366 Total>North America>Mexico Total North America Mexico | 8918653 Total>Asia>China>Shanghai Total Asia China Shanghai | 22315474 Total>Africa>Nigeria>Lagos Total Africa Nigeria Lagos | 14862000 Total>Asia>India>Mumbai Total Asia India Mumbai | 12691836 Total>Europe>Russia>Moscow Total Europe Russia Moscow | 12195221 Total>Asia>China>Beijing Total Asia China Beijing | 11716620 Total>Africa>Egypt>Cairo Total Africa Egypt Cairo | 9908788 Total>Europe>United Kingdom>London Total Europe United Kingdom London | 9126366 Total>North America>Mexico>Mexico City Total North America Mexico Mexico City | 8918653 Total>Africa>Egypt>Giza Total Africa Egypt Giza | 8800000 Total>North America>United States>New York City Total North America United States New York City | 8550405 Total>Europe>Russia>Saint Petersburg Total Europe Russia Saint Petersburg| 5383890 Total>North America>United States>Los Angeles Total North America United States Los Angeles | 3971883
4. Reordering the rows
The tree looks okay so far and you can stop there if you want but it would look better if child nodes were directly under their parents e.g. Shanghai should appear under China. In order to do this, we cannot simply use uj to combine our tables but we need to use the Over (/) accumulator to build the tree instead.
In order to get the row ordering correct, we add an id to each row, which will be a combination of the parent id and the row id. These id's look like this: 0, 0.0, 0.1, 0.1.1 etc. and will be used to sort the tree so that children appear under their parents.
Here is the final version of the code:
// Converts a table into a tree.
// @param td - a keyed table
// @param sortCol - the column to sort on
// @returns a table with a tree column
table2tree:{[td;sortCol]
// add Total column to the table
td:(`Total,keys[td]) xkey update Total:`Total from td;
keyCols:keys td;
// create a list of groupings
groupings:(1+til count keyCols) sublist\: keyCols;
// aggregate the table with each grouping
// this gives us a list of keyed tables (one per grouping)
tds:?[td;();;c!(sum;)each c:cols value td] each {x!x} each groupings;
// sort the tables
if[not null sortCol;tds:sortCol xdesc'tds];
// initial tree only has the Total row
tree:update id:"0",node:enlist "Total" from 0!first tds;
// build the tree using the over accumulator
tree:{[tree;td]
keyCols:keys td;
// join the parent id to the current table
td:td lj k xkey ?[tree;();0b;{x!x}(k:-1_keyCols),`id];
// update the id by concatenating the parent id to the row id
// we need to left-pad the row id so that sorting works correctly
// e.g. 1.3 should come before 1.10
td:update id:`$"."sv'flip(string id;(-1*count string count td)$string i) from td;
// add a node column which corresponds to the value of the last key column
td:![td;();0b;enlist[`node]!enlist last keyCols];
// add indentation to the node based on the depth (i.e. number of key columns)
indentation:(4*-1+count keyCols)#" ";
td:update node:(indentation,/:string node) from td;
// now add the table to tree
tree uj 0!td
}/[tree;1_tds];
// sort the tree on id
(`node,keyCols) xkey `id xasc tree}
This is what our final tree looks like:
q) data:3!("SSSI";enlist",") 0: `$"population.csv";
q) select node,population from table2tree[data;`population]
node population
-----------------------------------------
Total 128441136
Asia 46723930
China 34032094
Shanghai 22315474
Beijing 11716620
India 12691836
Mumbai 12691836
Africa 33570788
Egypt 18708788
Cairo 9908788
Giza 8800000
Nigeria 14862000
Lagos 14862000
Europe 26705477
Russia 17579111
Moscow 12195221
Saint Petersburg 5383890
United Kingdom 9126366
London 9126366
North America 21440941
United States 12522288
New York City 8550405
Los Angeles 3971883
Mexico 8918653
Mexico City 8918653
I also played around with adding lines to connect nodes of the tree but it got complicated very fast!
Can you think of a better way to do this? Let me know in the comments below!
Saturday, March 06, 2021
Creating Temporary Files with JUnit 5
This post shows you how to perform unit testing using temporary files with JUnit 5. If you're still on JUnit 4, please check out my previous post!
In JUnit 5, the @TempDir annotation is used to indicate that a field or method parameter of type Path or File is a temporary directory. Each test will use its own temporary directory and when the test method has finished executing, the directory and all its contents will be deleted. (If you want to share a temporary directory between tests, you need to make the field static.)
Here is an example:
import static org.hamcrest.CoreMatchers.*;
import static org.hamcrest.MatcherAssert.*;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
public class MyTest {
@TempDir
Path tempDir;
@Test
public void testWrite() throws IOException {
// Create a temporary file.
// This is guaranteed to be deleted after the test finishes.
final Path tempFile = Files.createFile(tempDir.resolve("myfile.txt"));
// Write something to it.
Files.writeString(tempFile, "Hello World");
// Read it.
final String s = Files.readString(tempFile);
// Check that what was written is correct.
assertThat("Hello World", is(s));
}
}
Related post: JUnit: Creating Temporary Files using the TemporaryFolder @Rule
Friday, January 01, 2021
fahd.blog in 2020
Happy 2021, everyone!
I'd like to wish everyone a great start to an even greater new year!
In keeping with tradition, here's one last look back at fahd.blog in 2020.
During 2020, I posted 10 new entries on fahd.blog. I am also thrilled that I have more readers from all over the world! Thanks for reading and especially for giving feedback.
Top 3 posts of 2020:I'm going to be writing a lot more this year, so stay tuned for more great techie tips, tricks and hacks! :)
Related posts:Tuesday, December 01, 2020
Parameterized Tests in JUnit 5
A parameterized test allows you to run a test against a varying set of data. If you find yourself calling the same test but with different inputs, over and over again, a parameterized test would help make your code cleaner. To create one in JUnit 5 you need to:
- Annotate the test method with
@ParameterizedTest - Annotate the test method with at lease one source e.g.
@ValueSource - Consume the arguments in the test method
The sections below describe some of the commonly used source annotations you can use to provide inputs to your test methods.
@ValueSource
This annotation lets you specify a single array of literal values that will be passed to your test method one by one, as shown in the example below:
@ParameterizedTest
@ValueSource(ints = {2, 4, 6})
void testIsEven(final int i) {
assertTrue(i % 2 == 0);
}
@CsvSource
This annotation allows you to specify an array of comma-separated values, which is useful if your test method takes multiple arguments. If you have a large number of arguments, you can use an ArgumentsAccessor to extract the arguments as opposed to creating a method with a long parameter list. For example:
@ParameterizedTest(name = "Person with name {0} and age {1}")
@CsvSource({ "Alice, 28",
"Bob, 30" })
void testPerson(final String name, final int age) {
final Person p = new Person(name, age);
assertThat(p.getName(), is(name));
assertThat(p.getAge(), is(age));
}
@ParameterizedTest(name = "Person with name {0} and age {1}")
@CsvSource({ "Alice, 28",
"Bob, 30" })
void testPersonWithArgumentAccessor(final ArgumentsAccessor arguments) {
final String name = arguments.getString(0);
final int age = arguments.getInteger(1);
final Person p = new Person(name, age);
assertThat(p.getName(), is(name));
assertThat(p.getAge(), is(age));
}
By the way, note how I have also customised the display name of the test using the {0} and {1} argument placeholders.
@CsvFileSource
This annotation is similar to CsvSource but allows you to load your test inputs from a CSV file on the classpath. For example:
@ParameterizedTest(name = "Person with name {0} and age {1}")
@CsvFileSource(resources = { "data.csv" })
void testPerson(final String name, final int age) {
final Person p = new Person(name, age);
assertThat(p.getName(), is(name));
assertThat(p.getAge(), is(age));
}
@MethodSource
This annotation allows you to specify a factory method which returns a stream of objects to be passed to your test method. If your test method has multiple arguments, your factory method should return a stream of Arguments instances as shown in the example below:
import static org.junit.jupiter.params.provider.Arguments.*;
@ParameterizedTest(name = "{0} is sorted to {1}")
@MethodSource("dataProvider")
void testSort(final int[] input, final int[] expected) {
Arrays.sort(input);
assertArrayEquals(expected, input);
}
static Stream<Arguments> dataProvider() {
return Stream.of(
arguments(new int[] { 1, 2, 3 }, new int[] { 1, 2, 3 }),
arguments(new int[] { 3, 2, 1 }, new int[] { 1, 2, 3 }),
arguments(new int[] { 5, 5, 5 }, new int[] { 5, 5, 5 }));
}
For more information, see the JUnit 5 User Guide on Parameterized Tests.
If you're still on JUnit 4 (why?!), check out my previous post on Parameterized Tests in JUnit 4.
Subscribe to:
Posts (Atom)