Sylvain Wallez

Getting meaningful stack traces from Rust tests returning a Result

Sylvain Wallez — Sun, 08 Jan 2023 16:34:57 +0100

I was recently writing a test for a side project (more on that in a future post), which looked like this:

#[test]
fn test_render_index_page() -> anyhow::Result<()> {

    let json = std::fs::read_to_string("tests/data/gdoc/index.json")?;
    let doc = serde_json::from_str(&json)?;
    let html = gdocs2hugo::gdoc_to_html::render(&doc)?;

    insta::assert_snapshot!("convert_test", html);

    Ok(())
}

The test returns an anyhow::Result that can capture the various error types returned by any of the first 3 lines.

Now what happens if one of those lines returns a failed result? For example when index.json doesn’t exist:

Error: No such file or directory (os error 2)
test test_render_index_page ... FAILED

failures:

---- test_render_index_page stdout ----
thread 'test_render_index_page' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/897e37553bba8b42751c67658967889d11ecd120/library/test/src/lib.rs:184:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

We know that some file wasn’t found, causing the test_render_index_page test to fail. In this simple example we only have one line that can cause this error, but in more complex tests we may open several files and the test result will not be helpful to determine which is the faulty one.

Also the panic message is about a failed assertion and not about our missing file. That error is output to stdout, which is output early in the test output and is easy to miss when you have lots of tests.

Side note: I wish Rust included the file name in this error, which would make it much more useful. I assume the reason is that it would require allocating a string to copy the path so that it’s owned by the Error object. We certainly don’t want to allocate memory every time we test if a file exists!

Let’s see if RUST_BACKTRACE=1 is more helpful:

Error: No such file or directory (os error 2)
test test_render_index_page ... FAILED

failures:

---- test_render_index_page stdout ----
thread 'test_render_index_page' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/897e37553bba8b42751c67658967889d11ecd120/library/test/src/lib.rs:184:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:142:14
   2: core::panicking::assert_failed_inner
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:218:23
   3: core::panicking::assert_failed
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:181:5
   4: test::assert_test_result
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/test/src/lib.rs:184:5
   5: publish_test::test_render_index_page::{{closure}}
             at ./tests/publish_test.rs:2:1
   6: core::ops::function::FnOnce::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ops/function.rs:248:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

The only line in our code is ./tests/publish_test.rs:2:1 which is the function declaration. But we knew that already…

We could replace every ? operator with unwrap() to panic early and have a meaningful location in the panic message, but that’s a lot of additional typing and it obscures the code. Also a good practice is to avoid unwrap() unless the result can never be an error to avoid panics, so we don’t want to introduce the cognitive dissonance of avoiding unwrap() in regular code while expecting its use in tests.

TestError: an error type that panics early

To make the stack trace more informative, we can use a little trick by panicking not in the test runner, but at the location where a ? causes the test to exit with an error. To achieve that we should remember that the ? operator is syntactic sugar for:

if let Error(err) = result { return Result::Err(From::from(err)) }

From above will resolve to the error variant of the test function return value. If the call to From::from panics, we will have an informative stacktrace.

So let’s replace anyhow::Result<()> with TestResult defined as follows:

#[derive(Debug)]
enum TestError {}

impl <Err: std::fmt::Display> From<Err> for TestError {
    #[track_caller]
    fn from(err: Err) -> Self {
        panic!("error: {}: {}", std::any::type_name::<Err>(), err);
    }
}

type TestResult = Result<(), TestError>;

If a ? operator is run with an error, TestError::from will panic. The track_caller attribute makes sure the panic will output the location of the caller (our test code) instead of the location of TestError::from which isn’t informative.

We can now use it as the result of our test function:

#[test]
fn test_render_index_page() -> TestResult {

    let json = std::fs::read_to_string("tests/data/gdoc/index.json")?;
    let doc = serde_json::from_str(&json)?;
    let html = gdocs2hugo::gdoc_to_html::render(&doc)?;

    return Result::Err(From::from(std::io::Error::last_os_error()));
    insta::assert_snapshot!("index_page", html);

    Ok(())
}

This results in the following stack trace:

test test_render_index_page ... FAILED

failures:

---- test_render_index_page stdout ----
thread 'test_render_index_page' panicked at 'error: std::io::error::Error - No such file or directory (os error 2)', tests/publish_test.rs:4:16
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Yay 🎉! That’s a lot better! The panic message gives us the error type, its message, and the exact location where it happened in the test code! And we no more have this split output where the real error message is away from the panic message.

Publishing TestError on crates.io

Although TestResult is very small, it’s pretty useful and deserves to be published on crates.io. We should prefer small crates, after all! Looking at how to name the crate, I discovered that… there was already a crate for that 😮! It even has the exact same TestResult type! I should have looked there first 😕

However my version is a bit more elaborate:

it accepts any kind of error, even those not implementing std::error::Error (anyhow::Error is one of them!)
the use of track_caller provides a more useful panic message
the panic message also contains the error type
TestError being an empty enum, it cannot be instantiated, so this prevents its use for anything else than implicit error conversion.

So I ended up packaging these changes as a PR for the existing crate. The maintainer liked it and promptly included it in a new release! So use the testresult crate to improve your test reports!

Happy testing!

'Go: the Good, the Bad and the Ugly'

Sylvain Wallez — Mon, 22 Jun 2020 15:10:11 +0200

This is an additional post in the “Go is not good” series. Go does have some nice features, hence the “The Good” part in this post, but overall I find it cumbersome and painful to use when we go beyond API or network servers (which is what it was designed for) and use it for business domain logic. But even for network programming, it has a lot of gotchas both in its design and implementation that make it dangerous under an apparent simplicity.

What motivated this post is that I recently came back to using Go for a side project. I used Go extensively in my previous job to write a network proxy (both http and raw tcp) for a SaaS service. The network part was rather pleasant (I was also discovering the language), but the accounting and billing part that came with it was painful. As my side project was a simple API I thought using Go would be the right tool to get the job done quickly, but as we know many projects grow beyond their initial scope, so I had to write some data processing to compute statistics and the pains of Go came back. So here's my take on Go woes.

Some background: I love statically typed languages. My first significant programs were written in Pascal. I then used Ada and C/C++ when I started working in the early 90's. I later moved to Java and finally Scala (with some Go in between) and recently started learning Rust. I've also written a substantial amount of JavaScript, because up to recently it was the only language available in web browsers. I feel insecure with dynamically typed languages and try to limit their use to simple scripting. I'm comfortable with imperative, functional and object oriented approaches.

This is a long post, so here's the menu to whet your appetite:

The Good
The Bad
The Ugly
Conclusion
A few days later: #3 on Hacker News!

The Good

Go is easy to learn

That's a fact: if you know any kind of programming language, you can learn most of Go's syntax in a couple of hours with the "Tour of Go", and write your first real program in a couple of days. Read and digest Effective Go, wander around in the standard library, play with a web toolkit like Gorilla or Go kit and you'll be a pretty decent Go developer.

This is because Go's overarching goal is simplicity. When I started learning Go it reminded me when I first discovered Java: a simple language and a rich but not bloated standard library. Learning Go was a refreshing experience coming from today's Java heavy environment. Because of Go's simplicity, Go programs are very readable, even if error handling adds quite some noise (more on this below).

But this may be false simplicity though. Quoting Rob Pike, simplicity is complicated, and we will see below that behind it there are a lot of gotchas waiting to bite us, and that simplicity and minimalism prevent writing DRY code.

Easy concurrent programming with goroutines and channels

Goroutines are probably the best feature of Go. They're lightweight computation threads, distinct from operating system threads.

When a Go program executes what looks like a blocking I/O operation, the Go runtime actually suspends the goroutine and resumes it when an event indicates that some result is available. In the meantime other goroutines have been scheduled for execution. We therefore have the scalability benefits of asynchronous programming with a synchronous programming model.

Goroutines are also lightweight: their stack grows and shrinks on demand, which means having 100s or even 1000s of goroutines is not a problem.

I once had a goroutine leak in an application: these goroutines were waiting for a channel to be closed before ending, and that channel was never closed (a common issue). The process was eating 90% of the CPU for no reason, and inspecting expvars showed 600k idle goroutines! I guess the CPU was used by the goroutine scheduler.

Sure, an actor system like Akka can handle millions of actors without a sweat, in part because actors don't have a stack, but they're far from being as easy to use as goroutines to write heavily concurrent request/response applications (i.e. http APIs).

Channels are how goroutines should communicate: they provide a convenient programming model to send and receive data between goroutines without having to rely on fragile low level synchronization primitives. Channels come with their own set of usage patterns.

Channels have to be thought out carefully though, as incorrectly sized channels (they're unbuffered by default) can lead to deadlocks. They also have a large number of gotchas and inconsistencies. We will also see below that using channels doesn't prevent race conditions because Go lacks immutability.

Great standard library

The Go standard library is really great, particularly for everything related to network protocols or API development: http client and server, crypto, archive formats, compressions, sending email, etc. There's even an html parser and a rather powerful templating engine to produce text & html with automatic escaping to avoid XSS (used for example by Hugo).

The various APIs are generally simple and easy to understand. They can sometimes look simplistic though: this is in part because the goroutine programming model means we just have to care about "seemingly synchronous" operations. This is also because a few versatile functions can also replace a lot of specialized ones as I found out recently for time calculations.

Go is performant

Go compiles to a native executable. Many users of Go come from Python, Ruby or Node.js. For them, this is a mind-blowing experience as they see a huge increase in the number concurrent requests a server can handle. This is actually pretty normal when you come from interpreted languages with either no concurrency (Node.js) or a global interpreter lock. Combined to the language simplicity, this explains part of the excitement for Go.

Compared to Java however, things are not so clear in raw performance benchmarks. Where Go beats Java though is on memory usage. Unless you're using Graal native-image which puts them in the same ballpark.

Go's garbage collector is designed to prioritize latency and avoid stop-the-world pauses, which is particularly important in servers. This may come with a higher CPU cost, but in a horizontally scalable architecture this is easily solved by adding more machines. Remember that Go was designed at Google, who are all but short on resources!

Compared to Java, the Go GC also has less work to do: a slice of structs is a contiguous array of structures, and not an array of pointers like in Java. Similarly Go maps use small arrays as buckets for the same purpose. This means less work for the GC, and also better CPU cache locality.

Go also beats Java for command-line utilities: being a native executable, a Go program has no startup cost contrarily to Java that first has to load and compile bytecode.

Language defined source code format

Some of the most heated debates in my career happened around the definition of a code format for the team. Go solves this by defining a canonical format for Go code. The gofmt tool reformats your code and has no options.

Like it or not, gofmt defines how Go code should be formatted and that problem is therefore solved once for all!

Standardized test framework

Go comes with a great test framework in its standard library. It has support for parallel testing, benchmarks, and contains a lot of utilities to easily test network clients and servers.

Go programs are great for operations

Compared to Python, Ruby or Node.js, having to install a single executable file is a dream for operations engineers. This is less and less an issue with the growing use of Docker, but standalone executables also means tiny Docker images.

Go also has some built-in observability features with the expvar package to publish internal statuses and metrics, and makes it easy to add new ones. Be careful though, as they are automatically exposed, unprotected, on the default http request handler. Java has JMX for a similar purposes, but it's much more complex.

Defer statement, to avoid forgetting to clean up

The defer statement serves a purpose similar to finally in Java: execute some clean up code at the end of the current function, no matter how this function is exited. The interesting thing with defer is that it's not linked to a block of code, and can appear at any time. This allows the clean up code to be written as close as possible to the code that creates what needs to be cleaned up:

file, err := os.Open(fileName)
if err != nil {
    return
}
defer file.Close()

// use file, we don't have to think about closing it anymore

Sure, Java's try-with-resource is less verbose and Rust automatically claims resources when their owner is dropped, but since Go requires you to be explicit about resource clean up, having it close to resource allocation is nice.

New types

I love types, and something that irritates/scares me is when for example we pass around persisted object identifiers as string or long everywhere. We usually encode the id's type in the parameter name, but this is a cause of subtle bugs when a function has several identifiers as parameters and some call mismatches parameter order.

Go has first-class support for new types, i.e. types that take an existing type and give it a separate identity, distinct from the original one. Contrarily to wrapping, new types have no runtime overhead. This allows the compiler to catch this kind of mistake:

type UserId string // <-- new type
type ProductId string

func AddProduct(userId UserId, productId ProductId) {}

func main() {
    userId := UserId("some-user-id")
    productId := ProductId("some-product-id")

    // Right order: all fine
    AddProduct(userId, productId)

    // Wrong order: would compile with raw strings
    AddProduct(productId, userId)
    // Compilation errors:
    // cannot use productId (type ProductId) as type UserId in argument to AddProduct
    // cannot use userId (type UserId) as type ProductId in argument to AddProduct
}

Unfortunately the lack of generics makes the use of new types cumbersome as writing reusable code for them requires to cast values to/from the original type.

The Bad

Go ignored advances in modern language design

In Less is exponentially more, Rob Pike explains that Go was meant to replace C and C++ at Google, and that its precursor was Newsqueak, a language he wrote in the 80's. Go also has a lot of references to Plan9, a distributed operating system the authors of Go developed in the 80's at Bell Labs.

There's even a Go assembly directly inspired from Plan9. Why not using LLVM that would have provided a wide range of target architectures out of the box? I may also be missing something here, but why is that needed? If you need to write assembly to get the most out of the CPU, shouldn't you use directly the target CPU assembly language?

Go creators deserve a lot of respect, but it looks like Go's design happened in a parallel universe (or their Plan9 lab?) where most of what happened in compilers and programming language design in the 90's and 2000's never happened. Or that Go was designed by system programmers who were also able to write a compiler.

Functional programming? No mention of it. Generics? You don't need them, look at the mess they produced in C++! Even if slice, map and channels are generic types as we'll see below.

Go's goal was to replace C and C++, and it's apparent that its creators didn't look much elsewhere. They missed their target though, as C and C++ developers at Google didn't adopt it. My guess is that the primary reason is the garbage collector. Low level C developers fiercely reject managed memory as they have no control on what happens and when. They like this control, even if it comes with additional complexity and opens the door to memory leaks and buffer overflows. Interestingly, Rust has taken a completely different approach with automatic memory management without a GC.

Go instead attracted users of scripting languages like Python and Ruby in the area of operation tools. They found in Go a way to have great performance and reduced memory/cpu/disk footprint. And more static typing too, which was new to them. The killer app for Go was Docker, that triggered its wide adoption in the devops world. The rise of Kubernetes strengthens this trend.

Interfaces are structural types

Go interfaces are like Java interfaces or Scala & Rust traits: they define behaviour that is later implemented by a type (I won't call it "class" here).

Unlike Java interfaces and Scala & Rust traits though, a type doesn't need to explicitly specify that it implements an interface: it just has to implement all functions defined in the interface. So Go interfaces are actually structural typing.

We may think that this is to allow interface implementations in other packages than the type they apply to, like class extensions that exist in Scala or Kotlin, or Rust traits, but this isn't the case: all methods related to a type must be defined in the type's package.

Go isn't the only language to use structural typing, but I find it has several drawbacks:

finding what types implement a given interface is hard as it relies on function definition matching. I often discover interesting implementations in Java or Scala by searching for classes that implement an interface.
when adding a method to an interface, you will find what types need to be updated only when they are used as values of this interface type. This can go unnoticed for quite some time. Go recommends to have tiny interfaces with very few methods, which is a way to prevent this.
a type may unknowingly implement an interface because it has the corresponding methods. But being accidental, the semantics of the implementation may be different from what is expected from the interface contract.

Update: for some ugliness with interfaces, see nil interface values below.

Interface methods don't support default implementations

Added after the Go 1.13 release. That may not seem like a big deal, but read on.

Go 1.13 introduced method chaining, adding a new Unwrap method to errors. Since Go interfaces don't support default implementations for their methods, adding a method to an existing interface would break a lot of existing code. So this new method is a "convention" rather than being part of the error interface. And because of that we can't just call err.Unwrap() to get the wrapped error. We have to use the separate function errors.Unwrap(err) which uses dynamic typing tests to check if Unwrap exists on its parameter.

Bye-bye compile-time checks, hello cumbersome syntax for what could have been a simple method call! Java faced a similar issue in JDK8 with the introduction of lambdas, and added default method implementation support to allow interfaces to evolve in a backwards-compatible way.

No enumerations

Go doesn't have enums, and in my opinion it's a missed opportunity.

There is iota to quickly generate auto-incrementing values, but it looks more like a hack than a feature. And a dangerous one, actually, since inserting a line in a series of iota-generated constants will change the value of the following ones. Since the generated value is the one that is used throughout the code, this can lead to interesting (not!) surprises.

This also means there is no way in Go to have the compiler check that a switch statement is exhaustive, and no way to describe the allowed values in a type.

The `:=` / `var` dilemma

Go provides two ways to declare a variable and assign it a value: var x = "foo" and x := "foo". Why is that?

The main differences are that var allows declaration without initialization (and you then have to declare the type), like in var x string, whereas := requires assignment and allows a mix of existing and new variables. My guess is that := was invented to make error handling a bit less painful:

With var:

var x, err1 = SomeFunction()
if (err1 != nil) {
  return nil
}

var y, err2 = SomeOtherFunction()
if (err2 != nil) {
  return nil
}

With :=:

x, err := SomeFunction()
if (err != nil) {
  return nil
}

y, err := SomeOtherFunction()
if (err != nil) {
  return nil
}

The := syntax also easily allows to accidentally shadow a variable. I was caught more than once by this, as := (declare and assign) is too close too = (assign), as shown below:

foo := "bar"
if someCondition {
  foo := "baz"
  doSomething(foo)
}
// foo == "bar" even if "someCondition" is true

Zero values that panic

Go doesn't have constructors. Because of that, it insists on the fact that the "zero value" should be readily usable. This is an interesting approach, but in my opinion the simplification it brings is mostly for the language implementors.

In practice, many types can't do useful things without proper initialization. Let's look a the io.File object that is taken as an example in Effective Go:

type File struct {
    *file // os specific
}

func (f *File) Name() string {
    return f.name
}

func (f *File) Read(b []byte) (n int, err error) {
    if err := f.checkValid("read"); err != nil {
        return 0, err
    }
    n, e := f.read(b)
    return n, f.wrapErr("read", e)
}

func (f *File) checkValid(op string) error {
    if f == nil {
        return ErrInvalid
    }
    return nil
}

What can we see here?

Calling Name() on a zero-value File will panic, because its file field is nil.
The Read function, and pretty much every other File method, starts by checking if the file was initialized.

So basically a zero-value File is not only useless, but can lead to panics. You have to use one of the constructor functions like Open or Create. And checking proper initialization is an overhead you have to pay at every function call.

There are countless types like this one in the standard library, and some that don't even try to do something useful with their zero value. Call any method on a zero-value html.Template: they all panic.

And there is also a serious gotcha with map's zero value: you can query it, but storing something in it will panic:

var m1 = map[string]string{} // empty map
var m0 map[string]string     // zero map (nil)

println(len(m1))   // outputs '0'
println(len(m0))   // outputs '0'
println(m1["foo"]) // outputs ''
println(m0["foo"]) // outputs ''
m1["foo"] = "bar"  // ok
m0["foo"] = "bar"  // panics!

This requires you to be careful when a structure has a map field, since it has to be initialized before adding entries to it.

So, as a developer, you have to constantly check if a structure you want to use requires to call a constructor function or if the zero value is useful. This is a lot of burden put on code writers for some simplifications in the language.

Go doesn't have exceptions. Oh wait... it does!

The blog post "Why Go gets exceptions right" explains in detail why exceptions are bad, and why the Go approach to require returning error is better. I can agree with that, and exceptions are hard to deal with when using asynchronous programming or a functional style like Java streams (let's put aside that the former isn't necessary in Go thanks to goroutines and the latter is simply not possible). The blog post mentions panic as "always fatal to your program, game over", which is fine.

Now "Defer, panic and recover" that predates it, explains how to recover from panics (by actually catching them), and says "For a real-world example of panic and recover, see the json package from the Go standard library".

And indeed, the json decoder has a common error handling function that just panics, the panic being recovered in the top-level unmarshal function that checks the panic type and returns it as an error if it's a "local panic" or re-panics the error otherwise (losing the original panic's stacktrace on the way).

To any Java developer this definitely looks like a try / catch (DecodingException ex). So Go does have exceptions, uses them internally but tells you not to.

Fun fact: a non-googler fixed the json decoder a couple of weeks ago to using regular errors bubbling up.

The Ugly

The dependency management nightmare

Update: since this post was written, Go 1.11 has introduced module support with resolution based on semantic versioning. This solves most of the issues explained below, although the choice of the minimum version for dependency resolution is debatable (see also this post on why Rust's Cargo chooses the maximum version).

Let's start by quoting Jaana Dogan (aka JBD), a well known gopher at Google, who recently vented her frustration on Twitter:

If dependency management is not resolved for another year, I am considering about quitting Go and never coming back. Dependency management issues routinely reverse all the joy I get from the language.
— JBD (@rakyll) March 21, 2018

Let's put it simply: there is no dependency management in Go. All current solutions are just hacks and workarounds.

This goes back its origins at Google, which famously uses a giant monolithic repository for all their source code. No need for module versioning, no need for 3rd party modules repositories, you build everything from your current branch. Unfortunately this doesn't work in the open Internet.

Adding a dependency in Go means cloning that dependency's source code repo in your GOPATH. What version? The current master branch at the time of cloning, whatever it contains. What if different projects need different versions of a dependency? They can't. The notion of "version" doesn't even exist.

Also, your own project has to live in GOPATH or the compiler won't find it. Want to have your projects cleanly organized in a separate directory? You have to hack per-project GOPATH or fiddle with symbolic links.

The community has developed workarounds with a large number of tools. Package management tools introduced vendoring and lock files holding the Git sha1 of whatever you cloned, to provide reproducible builds.

Finally in Go 1.6 the vendor directory was officially supported. But it's about vendoring what you cloned, and still not proper version management. No answer to conflicting imports from transitive dependencies that are usually solved with semantic versioning.

Things are getting better though: dep, the official dependency management tool was recently introduced to support vendoring. It supports versions (git tags) and has a version solver that follows semantic versioning conventions. It's not stable yet, but goes in the right direction. It still requires your project to live in GOPATH though.

But dep may not live long though as vgo, also from Google, wants to bring versioning in the language itself and has been making some waves lately.

So dependency management in Go is nightmarish. It's painful to setup, and you don't think about it while developing until it blows up when you add a new import or simply want to pull a branch of one of your team members in your GOPATH...

Let's go back to the code now.

Mutability is hardcoded in the language

There is no way to define immutable structures in Go: struct fields are mutable and the const keyword doesn't apply to them. Go makes it easy however to copy an entire struct with a simple assignment, so we may think that passing arguments by value is all that is needed to have immutability at the cost of copying.

However, and unsurprisingly, this does not copy values referenced by pointers. And the since built-in collections (map, slice and array) are references and are mutable, copying a struct that contains one of these just copies the pointer to the same underlying memory.

The example below illustrates this:

type S struct {
    A string
    B []string
}

func main() {
    x := S{"x-A", []string{"x-B"}}
    y := x // copy the struct
    y.A = "y-A"
    y.B[0] = "y-B"

    fmt.Println(x, y)
    // Outputs "{x-A [y-B]} {y-A [y-B]}" -- x was modified!
}

So you have to be extremely careful about this, and not assume immutability if you pass a parameter by value.

There are some deepcopy libraries that attempt to solve this using (slow) reflection, but they fall short since private fields can't be accessed with reflection. So defensive copying to avoid race conditions will be difficult, requiring lots of boilerplate code. Go doesn't even have a Clone interface that would standardize this.

Slice gotchas

Slices come with many gotchas: as explained in "Go slices: usage and internals", re-slicing a slice doesn't copy the underlying array for performance reasons. This is a laudable goal but means that sub-slices of a slice are just views that follow the mutations of the original slice. So don't forget to copy() a slice if you want to separate it from its origin.

Forgetting to copy() becomes more dangerous with the append function: appending values to a slice resizes the underlying array if it doesn't have enough capacity to hold the new values. This means that the result of append may or may not point to the original array depending on its initial capacity. This can cause hard to find non deterministic bugs.

In the code below we see that the effects of a function appending values to a sub-slice vary depending on the capacity of the original slice:

func doStuff(value []string) {
    fmt.Printf("value=%v\n", value)

    value2 := value[:]
    value2 = append(value2, "b")
    fmt.Printf("value=%v, value2=%v\n", value, value2)

    value2[0] = "z"
    fmt.Printf("value=%v, value2=%v\n", value, value2)
}

func main() {
    slice1 := []string{"a"} // length 1, capacity 1

    doStuff(slice1)
    // Output:
    // value=[a] -- ok
    // value=[a], value2=[a b] -- ok: value unchanged, value2 updated
    // value=[a], value2=[z b] -- ok: value unchanged, value2 updated

    slice10 := make([]string, 1, 10) // length 1, capacity 10
    slice10[0] = "a"

    doStuff(slice10)
    // Output:
    // value=[a] -- ok
    // value=[a], value2=[a b] -- ok: value unchanged, value2 updated
    // value=[z], value2=[z b] -- WTF?!? value changed???
}

Mutability and channels: race conditions made easy

Go concurrency is built on CSP using channels which makes coordinating goroutines much simpler and safer than synchronizing on shared data. The mantra here is "Do not communicate by sharing memory; instead, share memory by communicating". This is wishful thinking however and cannot be achieved safely in practice.

As we saw above there is no way in Go to have immutable data structures. This means that once we send a pointer on a channel, it's game over: we share mutable data between concurrent processes. Of course a channel of structures (and not pointers) copies the values sent on the channel, but as we saw above, this doesn't deep-copy references, including slices and maps, which are intrinsically mutable. Same goes with struct fields of an interface type: they are pointers, and any mutation method defined by the interface is an open door to race conditions.

So although channels apparently make concurrent programming easy, they don't prevent race conditions on shared data. And the intrinsic mutability of slices and maps makes them even more likely to happen.

Talking about race conditions, Go includes a race condition detection mode, which instruments the code to find unsynchronized shared access. It can only detect race problems when they happen though, so mostly during integration or load tests, hoping those will exercise the race condition. It cannot realistically be enabled in production because of its high runtime cost, except for temporary debug sessions.

Noisy error management

Something you will learn quickly in Go is the error handling pattern, repeated ad nauseam:

someData, err := SomeFunction()
if err != nil {
    return err;
}

Because Go claims to not support exceptions (although it does), every function that can end up with an error must have an error as its last result. This applies in particular to every function that performs some I/O, so this verbose pattern is extremely prevalent in network applications, which is Go's primary area.

Your eye will quickly develop a visual filter for this pattern and identify it as "yeah, error handling", but still it's a lot of noise and it's sometimes hard to find the actual code in the middle of error handling.

First of all, your functions should really return the error interface type and not a meaningful concrete type or the callers will encouter the dreaded "Why is my nil error value not equal to nil?" issue that has a dedicated FAQ entry (see also "nil interface values" below).

There are also a few gotchas, since an error result can actually be a nominal case, as for example when reading from the ubiquitous io.Reader:

len, err := reader.Read(bytes)
if err != nil {
    if err == io.EOF {
        // All good, end of file
    } else {
        return err
    }
}

In "Error has values" Rob Pike suggests some strategies to reduce error handling verbosity. I find them to be actually a dangerous band-aid:

type errWriter struct {
    w   io.Writer
    err error
}

func (ew *errWriter) write(buf []byte) {
    if ew.err != nil {
        return // Write nothing if we already errored-out
    }
    _, ew.err = ew.w.Write(buf)
}

func doIt(fd io.Writer) {
    ew := &errWriter{w: fd}
    ew.write(p0[a:b])
    ew.write(p1[c:d])
    ew.write(p2[e:f])
    // and so on
    if ew.err != nil {
        return ew.err
    }
}

Basically this recognizes that checking errors all the time is painful, and provides a pattern to just ignore errors in a write sequence until its end. So any operation performed to feed the writer once it has errored-out is executed even if we know it shouldn't. What if these are more expensive than just getting a slice? We've just wasted resources because Go's error handling is a pain. What if evaluating parameters has side effects? We've just introduced a serious bugs to simplify error handling...

Rust had a similar issue: by not having exceptions (really not, contrarily to Go), functions that can fail return Result<T, Error> and require some pattern matching on the result. So Rust 1.0 came with the try! macro and, recognizing the pervasiveness of this pattern, made it a first-class language feature. So you have the terseness of the above code while keeping a correct error-handling.

Transposing Rust's approach to Go is unfortunately not possible because Go doesn't have generics nor macros.

Nil interface values

This is an update after redditor jmickeyd shows a weird behaviour of nil and interfaces, that definitely qualifies as ugly. I expanded it a bit:

type Explodes interface {
    Bang()
    Boom()
}

// Type Bomb implements Explodes
type Bomb struct {}
func (*Bomb) Bang() {}
func (Bomb) Boom() {}

func main() {
    var bomb *Bomb = nil
    var explodes Explodes = bomb
    println(bomb, explodes) // '0x0 (0x10a7060,0x0)'
    if explodes != nil {
        println("Not nil!") // 'Not nil!' What are we doing here?!?!
        explodes.Bang()     // works fine
        explodes.Boom()     // panic: value method main.Bomb.Boom called using nil *Bomb pointer
    } else {
        println("nil!")     // why don't we end up here?
    }
}

The above code verifies that explodes is not nil and yet the code panics in Boom but not in Bang. Why is that? The explanation is in the println line: the bomb pointer is 0x0 which is effectively nil, but explodes is the non-nil (0x10a7060,0x0).

This is because interface values are fat pointers. The first element of this pair is the pointer to the method dispatch table for the implementation of the Bomb interface by the Explodes type, and the second element is the address of the actual Explodes object, which is nil.

The call to Bang succeeds because it applies to pointers to a Bomb: there is no need to dereference the pointer to call the method. The Boom method acts on a value and so a call causes pointers to be dereferenced, which causes a panic.

Note that if we had written var explodes Explodes = nil, then != nil would have not succeeded.

So how should we write the test in a safe way? We have to nil-check both the interface value and if non-nil, check the value pointed to by the interface object... using reflection!

if explodes != nil && !reflect.ValueOf(explodes).IsNil() {
    println("Not nil!") // we no more end up here
    explodes.Bang()
    explodes.Boom()
} else {
    println("nil!")     // 'nil' -- all good!
}

Bug or feature? The Tour of Go has a dedicated page to explain this behaviour and clearly says "Note that an interface value that holds a nil concrete value is itself non-nil".

Still, this is ugly and can cause very subtle bugs. It looks to me a like a big flaw in the language design to make its implementation easier.

Added in Dec '19: Other subtle bugs (or hard crashes, actually) can occur when a field of an interface type is updated concurrenly. Since assignment of a fat pointer isn't atomic, we may end up with a fat pointer pointing to a type and to a value of a different type. This breaks Go's memory safety and was demonstrated in an exploit during a security challenge. This problem can probably be found with Go's race detector if your tests exercise it.

Struct field tags: runtime DSL in a string

If you've used JSON in Go, you've certainly encountered something similar:

type User struct {
    Id string    `json:"id"`
    Email string `json:"email"`
    Name string  `json:"name,omitempty"`
}

These are struct tags which the language spec says are a string "made visible through a reflection interface and take part in type identity for structs but are otherwise ignored". So basically, put whatever you want in this string, and parse it at runtime using reflection. And panic at runtime if the syntax isn't right.

This string is actually field metadata, something that has existed for decades in many languages as "annotations" or "attributes". With language support, their syntax is formally defined and checked at compile time, while still being extensible.

Why did Go decide to use a raw string that any library can decide to use with whatever DSL it wants, parsed at run time?

Things can get awkward when you use multiple libraries: here's an example taken out of Protocol Buffer's Go documentation:

type Test struct {
    Label         *string             `protobuf:"bytes,1,req,name=label" json:"label,omitempty"`
    Type          *int32              `protobuf:"varint,2,opt,name=type,def=77" json:"type,omitempty"`
    Reps          []int64             `protobuf:"varint,3,rep,name=reps" json:"reps,omitempty"`
    Optionalgroup *Test_OptionalGroup `protobuf:"group,4,opt,name=OptionalGroup" json:"optionalgroup,omitempty"`
}

Side note: why are these tags so common when using JSON? Because in Go public fields must use UpperCamelCase, or at least start with an uppercase letter, whereas the common convention for naming fields in JSON is either lowerCamelCase or snake_case. Hence the need for tedious tagging.

The standard JSON encoder/decoder doesn't allow providing a naming strategy to automate the conversion, like Jackson does in Java. This probably explains why all fields in the Docker APIs are UpperCamelCase: that avoided the need for its developers to write these unwieldy tags for their large API.

Note however that the JSON parser is case insensitive, probably to make it easier to read camelCase JSON that is so common without requiring annotations. But this also means that writing the same data structure back will use the default UpperCamelCase naming and therefore produce a different JSON...

No generics... at least not for you

Update (2020-06): the Go team has come up with a rather nice proposal to add type parameters to Go, which will make this paragraph and the next one obsolete once it finally lands in the language. My only gripe is the builtin comparable constraint that could be defined as regular interface that any type (and not only builtin ones) can implement, and similarly the lack of widely used concepts like Ordered or Hashable. But that may come in due time.

It's hard to conceive a modern statically typed language without generics, but this is what you get with Go: it has no generics... or more precisely almost no generics, which as we'll see makes it worse than no generics at all.

The built-in slice, map, array and channel are generic. Declaring a map[string]MyStruct clearly shows the use of a generic type that has two parameters. Which is nice, as it allows type safe programming that catches all sorts of errors.

There are however no user-definable generic data structures. This means that you cannot define reusable abstractions that may work with any type, in a type-safe way. You have to use untyped interface{} and cast values to the proper type. Any mistake will only be caught at run time and will result in a panic. For a Java developer, it's like going back in the pre-Java 5 times, in 2004.

In "Less is exponentially more", Rob Pike surprisingly puts generics and inheritance in the same "typed programming" bag and says he favors composition over inheritance. Not liking inheritance is fine (I actually write a lot of Scala with little inheritance) but generics answer another concern: reusability while preserving type safety.

As we'll see below, the segregation between built-ins with generics and user-defined without generics has consequences on more than developer "comfort" and compile-time type safety: it impacts the whole Go ecosystem.

Go has few data structures beyond slice and map

The Go ecosystem doesn't have many data structures that provide added or different functionality from the built-in slice and map. Recent versions of Go added the containers package that provides a few of them. They all have the same caveat: they deal with interface{} values, meaning you lose all type safety.

Let's see an example with sync.Map which is a concurrent map with lower thread contention than guarding a regular map with a mutex:

type MetricValue struct {
    Value float64
    Time time.Time
}

func main() {
    metric := MetricValue{
        Value: 1.0,
        Time: time.Now(),
    }

    // Store a value

    m0 := map[string]MetricValue{}
    m0["foo"] = metric

    m1 := sync.Map{}
    m1.Store("foo", metric) // not type-checked

    // Load a value and print its square

    foo0 := m0["foo"].Value // rely on zero-value hack if not present
    fmt.Printf("Foo square = %f\n", math.Pow(foo0, 2))

    foo1 := 0.0
    if x, ok := m1.Load("foo"); ok { // have to make sure it's present (not bad, actually)
        foo1 = x.(MetricValue).Value // cast interface{} value
    }
    fmt.Printf("Foo square = %f\n", math.Pow(foo1, 2))

    // Sum all elements

    sum0 := 0.0
    for _, v := range m0 { // built-in range iteration on map
        sum0 += v.Value
    }
    fmt.Printf("Sum = %f\n", sum0)

    sum1 := 0.0
    m1.Range(func(key, value interface{}) bool { // no 'range' for you! Provide a function
        sum1 += value.(MetricValue).Value        // with untyped interface{} parameters
        return true // continue iteration
    })
    fmt.Printf("Sum = %f\n", sum1)
}

This is a great illustration of why there aren't many data structures in the Go ecosystem: they are a pain to use compared to the built-in slice and map. And for a simple reason: there are two categories of data structures in Go:

aristocracy, the built-in slice, map, array and channel: type safe and generic, convenient to use with range,
rest of the world written in Go code: can't provide type safety, clumsy to use because of required casts.

So library-defined data structures really have to provide solid benefits for us developers to be willing to pay the price of loosing type safety and the additional code verbosity.

The duality between built-in structures and Go code is painful in more subtle ways when we want to write reusable algorithms. This is an example from the standard library's sort package to sort a slice:

import "sort"

type Person struct {
    Name string
    Age  int
}

// ByAge implements sort.Interface for []Person based on the Age field.
type ByAge []Person

func (a ByAge) Len() int           { return len(a) }
func (a ByAge) Swap(i, j int)      { a[i], a[j] = a[j], a[i] }
func (a ByAge) Less(i, j int) bool { return a[i].Age < a[j].Age }

func SortPeople(people []Person) {
    sort.Sort(ByAge(people))
}

Wait... Seriously? We have to define a new type ByAge that has to implement 3 methods to bridge a generic (in the sense of "reusable") sort algorithm and the typed slice.

The only thing that should matter to us, developers, is the Less function that compares two objects and is domain-dependent. Everything else is noise and boilerplate required by the simple fact that Go has no generics. And we have to repeat it for each and every type that we want to sort. And every comparator too.

Update: Michael Stapelberg points me to sort.Slice that I missed. Looks better, although it uses reflection under the hood (eek!) and requires the comparator function to be a closure on the slice to sort, which is still ugly.

Every text explaining that Go doesn't need generics shows this as "the Go way" that allows having reusable algorithms while avoiding downcasting to interface{}...

Ok. Now to ease the pain, it would be nice if Go had macros that could generate this nonsensical boilerplate, right? Well, read on...

go generate: ok-ish, but...

Go 1.4 introduced the go generate command to trigger code generation from annotations in the source code. Well, "annotation" here actually means a magic //go:generate comment with strict rules: "the comment must start at the beginning of the line and have no spaces between the // and the go:generate". Get it wrong, add a space and no tool will warn you about it.

This covers several kinds of use cases:

Generating Go code from other sources: ProtoBuf / Thrift / Swagger schemas, language grammars, etc.
Generating Go code that complements existing code, such as stringer given as an example, that generates a String() method for a series of typed constants.
Add poor man's generics support. Since Go doesn't have generics, some clever band-aid solutions have emerged to generate them from template code.

First use case is ok, and the added value is that you don't have to fiddle with Makefiles and that generation instructions can be close to the generated code's usage.

For the second use case, many languages, such as Scala & Rust, have macros (which are mentioned in the design document) that have access to the source code's AST during compilation. Stringer actually imports the Go compiler's parser to traverse the AST. Java doesn't have macros but annotation processors play the same role.

Many languages also don't support macros so nothing fundamentally wrong here, except this fragile comment-driven syntax, which looks again like a quick hack that somehow does the job, and not something that was carefully thought out as coherent language design.

Oh, and did you know that the Go compiler actually has a number of annotations/pragmas and conditional compilation using this fragile comment syntax?

Conclusion

As you probably guessed, I have a love/hate relation with Go. Go is a bit like this friend that you like to hang out with because he's fun and great for small talk around beers, but that you find boring or painful when you want to have deeper conversations, and that you don't want to go on vacation with.

I like Go to quickly develop simple APIs (although JSON support is pretty basic compared to Java's Jackson or Rust's serde) or network stuff that goroutines make easy to reason about, I hate its limited expressiveness and half-baked type system when I have to implement business logic, and I hate all its quirks and gotchas waiting to hit you hard. Go requires a lot of attention and discipline to avoid them.

Up to recently there wasn't really an alternative in the space that Go occupies, which is developing efficient native executables without incurring the pain of C or C++. Rust is progressing quickly, and the more I play with it, the more I find it extremely interesting and superbly designed. I have the feeling that Rust is one of those friends that take some time to get along with, but that you'll finally want to engage with for a long term relationship.

Going back on more technical aspects, you'll find articles saying that Rust and Go don't play in the same park, that Rust is a systems language because it doesn't have a GC, etc. I think this is becoming less and less true. Rust is climbing higher in the stack with great web frameworks and nice ORMs. It also gives you that warm feeling of "if it compiles, errors will come from the logic I wrote, not language quirks I forgot to pay attention to".

We also see some interesting movements in the container/service mesh area with Buoyant (developers of Linkerd) developing their new Kubernetes service mesh Conduit as a combination of Go for the control plane (I guess because of the available Kubernetes libraries) and Rust for the data plane for its efficiency and robustness, and also Sozu proxy.

Swift is also part of this family or recent alternatives to C and C++. Its ecosystem is still too Apple-centric though, even if it's now available on Linux and has emerging server-side APIs and the Netty framework.

There is of course no silver bullet and no one-size-fits-all. But knowing the gotchas of the tools you use is important. I hope this blog post has taught you some things about Go that you weren't aware of, so that you avoid the traps rather than getting caught!

A few days later: #3 on Hacker News!

Update, 3 days after publishing: The reaction to this article has been amazing. It has made the front page of Hacker News (best rank I saw was #3) and /r/programming (best rank I saw was #5), and got some traction on Twitter.

The comments are generally positive (even on /r/golang/) or at least recognize that the article is balanced and tries to be fair. People on /r/rust of course liked my interest in Rust. Someone I never heard of even emailed me saying "I just wanted to let you know that I thought your write-up was the best ever. Thanks for all the work you put into it".

That was the hard part when writing it: try to be as factual and impartial as possible. This is of course not entirely possible as everyone has their own preferences, and why I focused on unexpected surprises and language ergonomics: how much the language helps you rather than getting in the way, or at least my way.

I also searched for code samples in the standard library or on golang.org and quoted people from the Go team, to base my analysis on authoritative material and avoid "meh, you quoted someone who got it wrong" reactions.

Writing this article used most of my evenings for two weeks, but it was lots of fun. And this is what you get when you do serious and honest work: lots of good vibes from teh Internets (if you ignore the few trolls and always grumpy people). Very motivating to write more in depth content!

Local date time calculations in Go

Sylvain Wallez — Thu, 22 Mar 2018 22:14:21 +0100

For a side project I'm writing in Go, I needed to send an email to each registered user during the night. Their night actually, according to each user's time zone. More precisely at a random time between 3am and 4am to spread email sending and avoid any rate limiting from SMTP providers or spam detection systems.

Coming from a Java/Scala background, I was looking for a large number of setters or offsetting methods on the time type. Here's what it would look like in Scala using classes in java.time:

val tz = user.timeZone // e.g. "Europe/Paris"

// the current wall clock time in the user's time zone
val userTime = ZonedDateTime.now(ZoneId.of(tz))

// tomorrow sometime between 3am and 4am
val emailSendTime = userTime
    .plusDays(1)
    .withHour(3)
    .plusSeconds(random.nextInt(3600))

// the instant (absolute timestamp) to be used by the task scheduler
val emailInstant = emailLocalTime.toInstant

Go puts simplicity first (too much IMHO but that's another discussion) and has only one type for time: time.Time. No wall clock time like Java's LocalDateTime or ZonedDateTime and no absolute Instant.

Similarly, time.Time has no setter method and only one offsetting method, AddDate(years, months, days). So with these in hand, how do we implement the above calculation?

Go has a couple of tricks here:

time.Time always has an associated time zone that by default is UTC.
time.Time has two destructuring functions, Time.Date() and Time.Clock that return the date and clock's constituents.
there is a Date(year, month, day, hour, min, sec, nsec, location) constructor function.

With that in hand, we can write the Go version:

location, _ := time.LoadLocation(user.TimeZone) // e.g. "Europe/Paris"

// the current wall clock time in the user's time zone
userTime := time.Now().In(location)

// destructure the time's date
year, month, day := userTime.Date()

emailSendTime := time.Date(
    year, month, day+1,        // date
    3, 0, rand.Intn(3600), 0,  // time (with nanosec)
    userTime.Location())

(yes, I skipped the dreaded if err != nil after LoadLocation)

An interesting thing to note regarding the Date function, is that we can pass out of range values for all parameters. For example, passing 18 for the month will set it to 6 and increment the year by one.

Nothing complicated here, but I had to "unlearn" what I knew from Java to find that a couple of simple functions were all that was needed to perform arbitrary time calculations. Now I need to look at how this works in Rust :-)

Micro benchmarks can make you short-sighted

Sylvain Wallez — Sat, 26 Nov 2016 11:02:10 +0100

In the last 24 hours, two articles related to code benchmarking showed up on my Twitter timeline, where the authors describe the optimization of a tiny bit of code via successive benchmarks. Both articles immediately triggered the same reaction in my head: "WTF, this is wrong! Why did they wrote this in the first place, and why do they focus on optimizing this bad solution?"

The second article even ends up with a solution that not only does not solve the original problem, but is really suboptimal. So in true xkcd#386 fashion I had to do something about it!

Both articles show that benchmark-driven optimization can lead people to focus on incrementally improving what is there rather than taking a step back and thinking about the original problem in a different way.

Sum of fizz and buzz

The first article, "What I learned from writing six functions that all did the same thing", describes a variant of Fizz buzz: given a number n, return the sum of all multiples of 3 & 5 that are between 0 and n. The initial solution is to build an array of multiples of 3 or 5, and then sum the members of this array. WTF? Two loops and an array? How can one even come up with this solution in the first place?

Reading the code, it seems trivial that the array not only useless but harmful by requiring memory allocation and two loops! After many incremental improvements to the sum of the array elements, the author finally takes a step back and replaces it by a single loop where the sum computed on the fly.

And he feels like a champion since it's 28 times faster... until someone comes up with the fact that it can be solved by adding/subtracting the sum of arithmetic progressions with common difference of 3, 5 & 15 (some math we learn in high school in France).

This article ends on a positive note though: even if there were 3 versions focusing on an incremental approach of the bad part of the solution (the array), the author finally takes a step back to consider things from a different angle, and ends up with a decent solution even if we could further reduce the number of useless iterations in the loop.

A loop that should have been a lookup table

The second article, "All The Small Things - JVM Profiling Lessons From The Trenches" starts by analyzing memory allocation in the CouchBase client library, wondering why there are so many allocations of an array of enums. The finding is that in Java, SomeEnumType.values() allocates an array at each call.

Allocating an array for a list of constants?

Let's study this first: the list of values for an enum is constant, so why do we need to allocate a brand new array at each call to values()? Simply because arrays in Java are mutable, and that the caller can thus modify the contents of the resulting array. Returning a singleton array would potentially cause weird bugs in subsequent callers of this method if someone modifies it.

It certainly almost never happens in real code, but the JDK had to make sure it is protected against it. A solution would have been for the method to return an unmodifiable List rather than a low-level mutable array.

Linear scan? Srsly?

Back to the article: they have an enum representing status codes and want to, given a numerical status code, find the corresponding enum value. The initial solution for this was to loop over all enum values until we find the one that has the correct code.

This is when the "WTF?!" kicks in: this code searches for a single element in a collection, given the value of a property of that element. Hmm... wait... doesn't it sound like a table lookup? It sure does! And the implementation is a very inefficient linear scan! Fortunately, looking at the enum definition, it looks like the most frequent statuses come first, alleviating a bit the cost of the scan. But still.

Then the article goes to great length about progressive improvements by adding fast path checks for common codes. This is indeed a good idea, but the codes in the fast path are mostly at the beginning of the list, so the performance gain shouldn't be high compared to a linear scan.

But more importantly, it does not solve the original problem, which was reducing memory allocation: for values out of the fast path, values() is still called, with the associated array allocation/copy.

Solving the original problem, even if badly

So before going to constant time lookup, let's remove array allocation, and keep a loop with no fast path. This is still a bad linear scan, but allows us some interesting comparison with the fast path solution (see the full code on GitHub):

// Original implementation, with no fast path
public static KeyValueStatus valueOfLoop(final short code) {
    for (KeyValueStatus value: values()) {
        if (value.code() == code) return value;
    }
    return UNKNOWN;
}

// Current code: fast path, but array allocation for other values
public static KeyValueStatus valueOf(final short code) {
    if (code == SUCCESS.code) {
        return SUCCESS;
    } else if (code == ERR_NOT_FOUND.code) {
        return ERR_NOT_FOUND;
    } else if (code == ERR_EXISTS.code) {
        return ERR_EXISTS;
    } else if (code == ERR_NOT_MY_VBUCKET.code) {
        return ERR_NOT_MY_VBUCKET;
    }

    for (KeyValueStatus keyValueStatus : values()) {
        if (keyValueStatus.code() == code) {
            return keyValueStatus;
        }
    }
    return UNKNOWN;
}

// No fast path, use a static array to avoid allocation when calling values()
private static final KeyValueStatus[] VALUES = values();

public static KeyValueStatus valueOfLoopOnConstantArray(final short code) {
    for (KeyValueStatus value: VALUES) {
        if (value.code() == code) return value;
    }
    return UNKNOWN;
}

Benchmark code:

@Param({
        "0", // 0x00, Success
        "1", // 0x01, Not Found
        "134", // 0x86 Temporary Failure
        "255", // undefined
        "1024" // undefined
})
public short code;

@Benchmark
public KeyValueStatus loopNoFastPath() {
    return KeyValueStatus.valueOfLoop(code);
}

@Benchmark
public KeyValueStatus loopFastPath() {
    return KeyValueStatus.valueOf(code);
}

@Benchmark
public KeyValueStatus loopOnConstantArray() {
    return KeyValueStatus.valueOfLoopOnConstantArray(code);
}

Benchmark results (benchmark name truncated to fit the page width):

Benchmark               (code)  Mode  Samples   Score  Score error  Units

loopNoFastPath               0  avgt       10  19.383        0.331  ns/op
loopNoFastPath               1  avgt       10  19.243        0.376  ns/op
loopNoFastPath             134  avgt       10  24.855        0.651  ns/op
loopNoFastPath             255  avgt       10  30.587        0.833  ns/op
loopNoFastPath            1024  avgt       10  30.619        1.209  ns/op

loopFastPath                 0  avgt       10   3.044        0.092  ns/op
loopFastPath                 1  avgt       10   3.040        0.051  ns/op
loopFastPath               134  avgt       10  25.070        0.637  ns/op
loopFastPath               255  avgt       10  31.215        1.089  ns/op
loopFastPath              1024  avgt       10  32.464        0.930  ns/op

loopOnConstantArray          0  avgt       10   2.975        0.086  ns/op
loopOnConstantArray          1  avgt       10   3.035        0.080  ns/op
loopOnConstantArray        134  avgt       10  10.215        0.269  ns/op
loopOnConstantArray        255  avgt       10  16.856        0.679  ns/op
loopOnConstantArray       1024  avgt       10  17.015        0.577  ns/op

We get some pretty interesting results here:

when looping on a constant array, avoiding memory allocation, the worst case scenario (255 & 1024, out of the value range) are still better than the best case scenario (0) with memory allocation.
for the values at the beginning of the list, iterating on a couple of values on the constant array isn't different that testing the fast path. It's easy to understand: the fast path is a partial loop unrolling.

So to solve the original memory allocation issue, rather than adding a fast path we could have just added a static copy of values() and be done with it!

Constant time lookup

But this approach is still bad, with a linear scan. Let's do a constant time lookup. For this, the classical Java approach is to use a HashMap, which we'll initialize once in a static block:

private static final HashMap<Short, KeyValueStatus> code2statusMap = new HashMap<>();

static {
    for (KeyValueStatus value: values()) {
        code2statusMap.put(value.code(), value);
    }
}

public static KeyValueStatus valueOfLookupMap(final short code) {
    return code2statusMap.getOrDefault(code, UNKNOWN);
}

And here's the benchmark:

Benchmark               (code)  Mode  Samples   Score  Score error  Units
lookupMap                    0  avgt       10   4.954        0.134  ns/op
lookupMap                    1  avgt       10   4.036        0.125  ns/op
lookupMap                  134  avgt       10   5.597        0.157  ns/op
lookupMap                  255  avgt       10   4.006        0.144  ns/op
lookupMap                 1024  avgt       10   6.752        0.228  ns/op

This is goes from 1/3 slower for the first values to more than 4 times faster for the last values. Time is no more increasing linearly, but we are slower on the most frequent values.

There is also a large variance, from 4.0 to 6.7 ns. Where does this come from? It's caused by hash collisions in the internal HashMap lookup table, causing lookups to degenerate into linear scan on the colliding key values.

So this looks like an acceptable solution, giving a more or less constant time lookup without large object allocation.

Don't forget boxing

There is still some allocation happening though: a Java HashMap expects objects for its keys, not primitive types. So we have an implicit boxing of code when looking up in the map. The JDK has an internal cache for small values to alleviate part of this problem. But for codes over 127 it still allocates a Short object.

The code of this allocation doesn't show up in the benchmarks results though, with 255 (not cached) being similar to 0 & 1 (cached). There's probably some HotSpot optimisation happening behind the scenes, furthermore considering that Short.valueOf() is an intrinsic function, i.e. calls to this method are hijacked by the JVM and implemented in native code (without going through JNI).

To completely remove this boxing, we could consider the excellent CarrotSearch's high performance primitive collections. But there's an even simpler solution.

A good old reverse lookup table

This is what immediately came to my mind when looking at KeyValueStatus source code. The code values range from 0 to 0xCC (204), which is small enough to allow keeping a simple lookup table in an array:

// Lookup table: code -> KeyValueStatus
private static final KeyValueStatus[] code2status = new KeyValueStatus[0x100];

static {
    Arrays.fill(code2status, UNKNOWN);
    for (KeyValueStatus keyValueStatus : values()) {
        if (keyValueStatus != UNKNOWN) {
            code2status[keyValueStatus.code()] = keyValueStatus;
        }
    }
}

public static KeyValueStatus valueOfLookupArray(final short code) {
    if (code >= 0 && code < code2status.length) {
        return code2status[code];
    } else {
        return UNKNOWN;
    }
}

Note that we pre-fill the array with UNKNOWN to avoid the cost of a if (result == null) at each call. Do a bit more upfront to do a lot less later.

The benchmark of array lookup, with the loop with fast path checks for comparison:

Benchmark               (code)  Mode  Samples   Score  Score error  Units
loopFastPath                 0  avgt       10   3.044        0.092  ns/op
loopFastPath                 1  avgt       10   3.040        0.051  ns/op
loopFastPath               134  avgt       10  25.070        0.637  ns/op
loopFastPath               255  avgt       10  31.215        1.089  ns/op
loopFastPath              1024  avgt       10  32.464        0.930  ns/op

lookupArray                  0  avgt       10   3.061        0.126  ns/op
lookupArray                  1  avgt       10   3.048        0.127  ns/op
lookupArray                134  avgt       10   3.070        0.084  ns/op
lookupArray                255  avgt       10   3.035        0.113  ns/op
lookupArray               1024  avgt       10   3.034        0.113  ns/op

There we are! Constant time lookup, no allocation, and as fast as the article's solution for values in the fast path.

Note: the code on GitHub has two more experiments (array lookup with a try/catch, and a large switch statement) but they don't give better results.

Conclusion

What can be learned from both articles is that benchmarks are awesome tools to actually measure improvements, but they can also be dangerous, by making people short-sighted and optimize only part of the problem, even on a 10-line code block.

Before going into nano-optimization driven by micro-benchmarks, take a step back and ask yourself "What is the problem I have to solve? How can I do it differently?"

And be curious, look at how things are implemented under the covers, and go down the rabbit hole once in a while to really understand how things work. You will learn a lot, and it will most certainly change the way you solve problems.

Ah, and don't forget at the end to check that you actually solved the original problem :-)

'Afternoon hack: a USB foot keyboard'

Sylvain Wallez — Fri, 29 Jul 2016 11:21:00 +0200

This post explains a quick afternoon hack to turn a foot pedal controller for music synthesizers into a programmable USB foot keyboard.

A couple of weeks ago, for a personal project, my wife needed to manually transcribe many hours of interview recordings. Text-to-speech software were failing miserably both because of the people's strong local accent and poor recording quality. To ease the task she used the free version of Express Scribe, a nice dedicated application with a powerful transport bar that works exactly like 20th century tapes: rewind/forward while playing back, slow down/up playback speed, etc.

How cool would it be to control this with the feet, so that she could have her hands free for typing? At least his was the dubious excuse I needed for a hack I had been thinking about for quite some time :-)

The hardware

The Korg EC5 is a sturdy 5 pedals foot controller aimed at Korg's synthesizers. It's a dedicated device that has no MIDI nor USB interface. Its 6 pin connector is just wires to the 5 pedal switches and a common ground. We'll turn it into a programmable 5 keys USB "keyboard". Here's the pin layout of the EC5 output connector:

I am a big fan of Teensy microcontrollers (which I buy from my friends at Snootlab): this cheap tiny thing has a lot of inputs and outputs and can emulate almost any USB device: mouse, joystick, keyboard, MIDI devices, and even flight simulator controls! We just need to cut the EC5 cable and solder the wires on the Teensy's ground and digital inputs 0 to 4. That's all there is for the hardware part!

The code

The software to turn pedal switches into USB key presses is simple, but requires to understand basic concepts of digital I/O. Discussing with my colleagues, I realized that most developers have learned digital electronic basics during their studies, but often quickly forgot about it. This is what actually motivated me to write this post.

Here's the source code. As for all programs written with the Arduino IDE, it's a single C++ file that instead of a standard main() has two methods: setup() to initialize the system, and loop() that is called over and over and where we read sensors, activate outputs, send data, etc.

/*
 * Korg EC5 mapping to control Express Scribe
 * You must select "Keyboard" from the "Tools > USB Type" menu in the Arduino IDE
 */

#include <Bounce.h>

// Create a Bounce object for each pedal. The Bounce object
// automatically deals with contact chatter or "bounce", and
// makes detecting changes very simple.
Bounce pedalA = Bounce(0, 10);
Bounce pedalB = Bounce(1, 10);
Bounce pedalC = Bounce(2, 10);
Bounce pedalD = Bounce(3, 10);
Bounce pedalE = Bounce(4, 10);

We start by creating 5 debouncers, one for each key, with a 10 milliseconds delay. Debouncing is mandatory in the hardware world: when a switch changes state, the measured value can be randomly 0 or 1 for a brief period of time, until the contact is stabilized. This debouncer is a great example of the amazing Arduino ecosystem, that provides a lot of components with a simple API, even if under the hood the implementation may be really tricky.

byte playMode = false;
void setup() {
  // Configure the pins for input mode with pullup resistors.
  pinMode(0, INPUT_PULLUP);
  pinMode(1, INPUT_PULLUP);
  pinMode(2, INPUT_PULLUP);
  pinMode(3, INPUT_PULLUP);
  pinMode(4, INPUT_PULLUP);
}

The calls to pinMode() configure the pins as digital inputs and activate the internal pull-up resistor. What is this, and why is it needed?

When a switch is closed, its pin is connected the ground, and we measure LOW (0 Volt). If the switch is open, its pin is "floating" and the measure is unpredictable. To have a stable value when the switch is open, we connect its output to Vcc (3.3 V for the Teensy) using a resistor. This has the effect of "pulling up" the voltage of the pin to a stable value that can be measured. The call to digitalRead() will then return HIGH if the measured voltage is higher than 2 Volt. Note that we cannot simply connect the input pin to Vcc, as it would create a short circuit when the switch is closed!

This value of 2 Volt comes from the fact that the pull-up resistor is combined with impedance of the microcontroller's analog to digital converter, which effectively creates a voltage divider.

The figure below outlines this:

If you want the nitty gritty details about calculating the value of pull-up resistors, read this Sparkfun tutorial or this reference documentation from Texas Instruments.

This pull-up resistor pattern is so common that microcontrollers now have embedded pull-up resistors that can be activated via software. This is what INPUT_PULLUP does: configure the pin as input, and activate the internal pull-up resistor so that we have a stable and measurable voltage when the switch is open, without requiring an external resistor. This makes less parts and also avoids mistakes while prototyping.

void loop() {
  // Let pedal switches debouncers update their state.
  pedalA.update();
  pedalB.update();
  pedalC.update();
  pedalD.update();
  pedalE.update();

This the beginning of the loop that will continuously track the change in pedal switches states and send USB key presses whenever changes happen.

  // Debouncers track "falling" and "rising" edges on the signal,
  // that are triggered by pedal press and release.
  if (pedalA.fallingEdge()) Keyboard.press(KEY_F3);
  if (pedalA.risingEdge()) Keyboard.release(KEY_F3);
  if (pedalB.fallingEdge()) Keyboard.press(KEY_F4);
  if (pedalB.risingEdge()) Keyboard.release(KEY_F4);

  if (pedalC.fallingEdge()) {
    // Flip play mode
    playMode = !playMode;
    Keyboard.press(playMode ? KEY_F5 : KEY_F6);
  }
  if (pedalC.risingEdge()) Keyboard.release(playMode ? KEY_F5 : KEY_F6);

  if (pedalD.fallingEdge()) Keyboard.press(KEY_F7);
  if (pedalD.risingEdge()) Keyboard.release(KEY_F7);
  if (pedalE.fallingEdge()) Keyboard.press(KEY_F8);
  if (pedalE.risingEdge()) Keyboard.release(KEY_F8);
}

Express Scribe allows defining keyboard shortcuts for its actions. The default configuration doesn't include mappings for everything we want to control with the feet, so we've configured it as follows:

F3 : reduce playback speed, mapped to pedal A
F4 : rewind, mapped to pedal B
F5 : play, mapped to pedal C
F6 : pause, mapped to pedal C
F7 : fast forward, mapped to pedal D
F8 : increase playback speed, mapped to pedal E

All pedal switches except pedal C have the same behavior: when the pedal is pressed (falling edge of the measured value) we send a key press event. Conversely when there the pedal is released (rising edge) we send a key release event.

Pedal C (the middle pedal) is a single command for both play and pause, which is much more comfortable than having to move the foot to play and pause. This is why pressing pedal C send either F5 or F6 depending on the value of "playMode" which is flipped at each press of this pedal.

Conclusion

Writing this article took a lot more time than actually doing this little hack! But I hope to have demystified the basics of digital input with microcontrollers. If you know how to program, you just need some basic soldering skills to start hacking. And this also is easy to learn.

Now how about the usefulness of this for my wife? Well... it turned out that coordinating feet and hand movement isn't something you master instantly, and in the end she ended up using mostly the function keys on the Mac's keyboard...

Oh well, that was a fun experiment nonetheless and I now have a new toy to control programs in unusual ways. I have some sort of track record here, having used a Wii remote to draw virtual grafittis on a wall, a Kinect to draw 3D virtual wireframes with an augmented reality headset, and other weird things!

Blog.reboot()

Sylvain Wallez — Sat, 18 Jun 2016 18:12:00 +0200

This blog is now a static website powered by the Pelican generator. It's the 3rd major change since I started blogging in 2002.

It began with Movable Type, a Perl blogging tool. There wasn't much choice at that time, and it did the job even if I did not like Perl (and still do not). In 2009 I switched to Drupal 6, which allowed me to have a nicer visual layout and have more control on the navigation, starting with clean URLs.

Now for a blog I don't need a full blown CMS for dynamic web pages, with the operational burden what comes with it. Also, the content writing UI in Drupal isn't that great (or wasn't, since I stayed with Drupal 6), and you ultimately end up writing HTML, even if hidden behind a fancy WYSIWYG editor. So much for semantic markup and separation of content and presentation.

Over time, the combination of Twitter's 140 characters instant gratification and a clunky writing UI led this blog to be really neglected. Now I have things to share that don't fit in a tweet!

Going static

So it was time for a change, and I was urged in that by my two sons Corentin and Théophile who started a "one blog post a month" challenge. Instead of just writing a blog post, I decided to reboot this blog so that the urge or writing would not be hindered by an unfriendly tool.

The 3 of us are now running our blogs with the Pelican static blog generator. I also did some quite extensive tests with Nikola, but in the end Pelican won because of its larger theme library that allowed me to experiment faster. Nikola's speed and live reloading while editing content are definitely better than Pelican's though, and now that I have settled on a theme I may come back to it.

Why not using the popular Jekyll or Hugo? The main reason is that I want to store my content in a format that is stable, well defined and with enough builtin features for documents that go beyond the simple set of headings, text and code snippets. I want to own my content, and I want it to last.

This rules out Markdown, since every tool comes with its own dialect and extensions, even if some are more popular than others, and even with the CommonMark initiative. Same for Textile who also seems to be falling into oblivion.

The remaining contenders were reStructuredText and AsciiDoc, who both have a formal specification. Their reference implementation is written in Python, so choosing a Python-based generator seemed a natural choice. Sure, other generators can use RestructuredText and AsciiDoc by forking external processes, but having the parser within the generators allows for tighter integration and greater generation speed. AsciiDoc also has the great AsciiDoctor Ruby implementation, but I'm no rubyist.

ReStructuredText has fewer syntactic artifacts than AsciiDoc, meaning less cognitive load for the same document complexity. So I'll use reStructuredText as the main writing format, knowing that the great Pandoc can do wonders converting from one syntax to another.

Setting up the new blog

The full configuration is published on GitHub. Actual content and deployment configurations are in a separate private repository.

The theme is based on StartBootstrap's Clean Blog and its Pelican implementation which I modified slightly. I still have to clean up the CSS and trim down Bootstrap to what's actually needed.

The banner images are randomly generated from a set of banner pictures (taken by me), unless I explicitly specify one in the post's metadata. I had this in the Drupal-based version, and love it since it gives a nice visual touch to pages without requiring scouting for an appropriate picture when none comes to mind quickly (choosing images is hard!)

The old content is still in HTML, and was extracted from Drupal with a tiny scraping script. It was actually easier than getting it out of the database, since over time I used various Drupal output formats. I'll progressively go through all past 369 (!) articles to convert them (again with Pandoc), clean up broken links and harmonize the layout.

The comments are now powered by Isso, a self-hosted Disqus clone, and I have imported the old comments from Drupal with a custom Python script. Again, I want to own my content, and comments are an integral part of it.

Publishing is not yet automated, and won't be until I a become a really proficient writer. Running a makefile once in while is good enough for now!

Hosting

And since this is a reboot, I also completely overhauled the infrastructure. This blog has been running for ages on a bare metal server, which I carefully nurtured and updated. Devops and immutable infrastructure are all the rage these days, so I used at home the same techniques I use at work: this blog is now running (warning: buzzwords ahead!) as a set of Docker containers managed with a simple docker compose file. The containers are running the Træfik reverse proxy in front of the Caddy file server and the Isso comment server.

If you don't know Træfik and Caddy, you should definitely have a look: they took a fresh start to web servers, taking into account the latest evolutions of the web: you get instant HTTP/2 and painless free SSL with Let's encrypt.

The server is a 3€/mo VPS which I can setup with a single command. Now I hear you say "if this is a static blog, why not using Github Pages"? Well, call me a masochist, but I actually enjoy tweaking my server. I've been doing it for more than a decade, and it's nice to have your own playground where you can experiment and break things!

Now enough with this meta-post. I've started collecting ideas and drafts for real content in this new shiny home!

Efficient storage of non-periodic time series with MongoDB

Sylvain Wallez — Wed, 21 Jan 2015 18:20:00 +0100

TL;DR: this post explains the MongoDB storage structure we use at Actoboard to efficiently store non periodic time series. Storing one Mongo document for each data point is woefully inefficient, so we store them in fixed-size segments, which speeds things up by more than an order of magnitude.

MongoDB makes the developer's life easier

When we started Actoboard, we decided to use MongoDB to store all our data. I already used it successfully on several projects, and it makes things a lot easier compared to an SQL database when you have a rich data model using a lot of polymorphic structures. You don't have this impedance mismatch between application objects and tables that has produced these object-relational monsters we've loved to hate.

MongoDB is nice for all our configuration data (data sources, widgets, etc): JSON documents are perfect for these fat polymorphic configuration objects, the volume of such data is limited, and the load is mostly read, avoiding the lock problems Mongo has in write-heavy scenarios.

Time series need special attention

Time series data is a different beast, with a write-mostly load of tiny objects consisting of dataset id, timestamp and value. The naive approach of storing each data point as a tiny document will yield poor write performance, as the database has to allocate space for every tiny document holding a data point and add it to the index, which will be almost as large as the actual data.

Read performance will be bad too as the data for a series will be spread in many places on disk, requiring lots of disk seeks when reading a time slice. Deletion of expired data will also create lots of holes on disk, increasing load on the database's space allocator.

We could have used a dedicated database to store time series, but when we started the project with a very small team (1.5 people) we tried to limit the number of moving parts in the architecture. We're ready however to switch to Cassandra when load becomes a problem: its P2P architecture and log-structured storage naturally avoids write contention and stores data on disk in chronological order. But we're not there yet.

The solution is therefore to store many data points in a single MongoDB document. This document has to be pre-allocated (filled with empty values) to avoid its size to change as new data is added, causing it to be relocated on disk. Storing a new data point then consists in doing in-place updates of an empty slot in the document.

The case of non periodic time series

There are many blog posts explaining how to store time series data, but they assume periodic sampling or predefined periodic aggregations. At Actoboard we have widely varying data collection rates ranging from once every minute to once a day. Users can also push values to their custom data sources whenever they want. So we needed something different.

Our time series data is stored in “value segments” containing a fixed number of data points. A segment has the following properties:

the time series identifier (we use 64 bits integers),
the end time of the previous segment (more on this later)
the start time of the next segment (ditto),
data point times, as an array of 64 bits timestamps,
data values, as an array of 64 bits floats.

The diagram below illustrates a series with 3 segments, the most recent one being partially filled.

The actual JSON object looks like this:

{
  "_id": NumberLong("133506131220758528"),
  "series": NumberLong("133814973280288768"),
  "owner" : NumberLong("112076101043355649"),
  "prevEnd" : NumberLong("1390410338000"),
  "nextStart" : NumberLong("1390485108000"),
  "time" : [ NumberLong("1390483902000"), NumberLong("1390483299000"), … ],
  "value" : [ 24.5, 26.32, … ],
  // other housekeeping properties
}

Because of this layout, we’ve put a strong constraint on our time series storage: data points can only be stored in chronological order. Out of order arrival is not allowed to avoid having to insert a datapoint into a filled interval, possibly leading to lots of segment manipulation. In practice this has never been a problem.

Initializing a new segment

When a new segment is created, its time and value arrays are pre-allocated with 160 zeroes. We calculated this length as being the maximum number of data points that can be stored in 4096 bytes, once encoded in BSON.

This 4 KiB size was chosen because MongoDB uses a power-of-two space allocation strategy, and this is also the standard Linux memory page size. Since MongoDB data files are memory-mapped, matching the page size uses the available RAM more efficiently.

On a new segment, the “prevEnd” value is set to the latest timestamp of the previously active segment (or the current timestamp if there’s none). Its “nextStart” is set to MAX_LONG (far far away in the future). More on this in the section on querying below.

Writing a new data point

When new data arrives, we need to know in which segment it should be written to. This is the role of a higher-level “series” object, which contains the active segment’s identifier and the next position to write to in this segment.

Time series object:

{
  "_id" : NumberLong("133814973280288768"),
  "lastDate" : NumberLong("1419347899000"),
  "lastValue" : 27.2,
  "segmentId" : NumberLong("254795601416290304"),
  "segmentPtr" : 121
}

In the nominal case, writing requires two database updates.

The first update decrements the segment pointer in the time series object and returns the new value. To ensure chronological order, we add the new data point’s timestamp as a query parameter, to prevent any update if it’s not posterior to the last one written.

var series = db.TimeSeries.findAndModify({
  query: { _id: seriesId, lastDate: { $lt: time }},
  update: { $set: { lastValue: value, lastDate: time }, $inc: { segmentPtr: -1 }},
  new: true // return the modified document
})

The result of this operation is an updated object, with the segment index position to write to. We can then perform our second update, storing the data at the correct place:

var fields = {};
fields["time." + series.segmentPtr] = timestamp;
fields["value." + series.segmentPtr] = value;

db.ValueSegments.update(
  { _id: series.segmentId },
  { $set: fields }
);

Now why decrementing the pointer, and not incrementing it? We initialize the pointer with the array’s length. So when the decrement operation returns a negative value, we know that the current segment is full and we must create a new one, without having to care about the actual array length. Fewer magic numbers in the code mean less bugs.
Creating a new segment is the non-nominal case that requires additional writes. It only happens once every 160 values.

Dealing with non ACIDity

MongoDB operations are atomic only at the document level. So what happens if the system crashes once we’ve updated the segment pointer but haven’t yet written the data? We end up with an empty data point in the value segment that has a zero timestamp. We deal with it at the application level by just ignoring such data points.

Also the application may crash after we’ve decremented the pointer and found that a new segment was created, but before it was actually written. In that case, decrementing again the pointer will yield numbers less than -1. Testing for negative values and not just -1 to create a new segment solves the problem.

Querying data

Querying time series data is always done with a time range, particularly with non-periodic time series, since we can’t know the exact time where a data point exists. However in order to draw the series in a given [tmin, tmax] range, we need to also get the data points that are immediately outside of this range, as depicted below:

Because of our layout, we fetch entire values segments (160 data points) from the database. Now to get the immediate outside data points, we can’t just query the segments whose data points are in the [tmin, tmax] range.

This is why we store the outer timestamps (previous segment end time and next segment start end time) in a value segment. This allows writing a query that returns segments containing data points right before or right after the wanted time range. Our query is then:

db.ValueSegments.find(
  { series: seriesId, prevEnd: { $lte: tmin}, nextStart: { $gte: tmax }},
  { sort: { prevEnd: 1 }}
)

We then get a set of sorted value segments, and we feed their data points into a pipeline that filters, transforms and aggregates according to the user’s graph configuration.

Performance

Our first tests using the naive approach of 1 document per data point yielded abysmal performance, in the 500 inserts/sec on our integration server.

With this new layout, things are much better, with about 10k inserts/sec. Not bad! Database size is also way smaller because the number of entries in the index has been divided by 160!

We only achieved this number after fixing a bug in Jongo, a thin layer on top of the Java MongoDB driver that allows using the mongo shell syntax to talk to the database. This bug was causing some integer values conversion that led to changing the BSON document layout, defeating the purpose of our value segments.

Conclusion

This may all sound quite complicated, but the code to implement this is rather small and straightforward. The hard part was finding the prevEnd/nextStart trick to fetch the outer data points with no additional requests.

Achieving 10k inserts/sec on a non-sharded setup is more than enough while our user base is still small, even more considering that we rate limit inserts to one per minute, and most of the metrics we collect are updated once every 10 minutes.

And on top of this time series storage, most metrics have an input filter that ensures we only write to the database when the metric value has actually changed, further reducing insertion rate and storage requirements. No need to write the same Twitter follower count every 10 minutes if it hasn’t changed!

GitHub hack: a common security flaw in webapp frameworks

Sylvain Wallez — Thu, 08 Mar 2012 12:55:00 +0100

GitHub has faced a spectacular hack: a disappointed developer has exploited a weakness in Ruby on Rails to gain commit access to the Rails master branch and create issues in the future. As a result, GitHub asks us to confirm our ssh keys, but this can have been used to change almost any data in the system.

How's it possible?

This weakness comes from the mass assignment feature in Rails. To ease development and increase productivity, Rails allows an object to be filled with any http request parameter whose name match an attribute of the object. All is well when people use the forms provided by your web pages, because they only send parameters for attributes that you (the developer) want to change. Now things are different when someone forges a request: your object is wide open, and they can change all of its attributes.

If you have a User class with properties id, name, password, role, then a user with role "visitor" can upgrade himself to "administrator" simply using the profile edition URL. Or change the password of another user to get access to his account by changing the id value, or... etc.

Most modern web frameworks have this mass assignment feature. They allow the developer to restrict the assignable attributes by means of whitelists or blacklists, but most if not all of them have it wide open by default. This is a comfortable developer feature, but how many of them will even think of protecting their model?

A couple of examples

Rails allows whilelisting and blacklisting in the model classes. It also has scoped protection, which is useful since the assignment constraints are different e.g. in a profile edition page and a user administration page.

In the Java world, Spring MVC is also wide open by default. It allows white/black listing, but this is defined at the controller level. It is rather cumbersome to use, since you have to make sure each and every controller has a @InitBinder method that correctly configures the binder using DataBinder.setAllowedFields().

Play! Framework is also wide open by default (told you, all of them!), but has an easy scoped blacklisting with the @NoBind annotation on model attributes. Whitelisting would be more secure though.

It is to be noted that component-oriented frameworks like Wicket should not suffer from this mass assignment issue, since they have a separate page model that isolates domain objects from the request. This comes however with additional complexity and bigger server side sessions.

And finally, this mass assignment issue is not limited to page controllers but also affects less visible elements, such as JSON encoder/decoders where mass assignment is the norm. Here again, JSON libraries like Jackson, GSon, Flexjson, etc make no restriction on the attribute names by default, and so the nice REST API that you have designed may well be a giant open door for attackers.

Conclusion

This hack on GitHub has given a lot of visibility to a problem that is largely present in many of today's web framework, which have traded comfort for security by default. We may see a lot of exploits of this flaw in the coming weeks on websites using a large variety of frameworks.

So make sure your webapp is not wide open, and let's hope framework developers will consider that security by default should have priority over comfort if the price to pay is to add a couple of annotations on the model classes.

My 2011: a year like no other

Sylvain Wallez — Sun, 01 Jan 2012 13:15:00 +0100

This blog has been mostly silent in 2011. Blame both Twitter that makes capturing quick thoughts so easy and a busy year as I started my freelance business. But a new year is starting, and along with the traditional wishes I owe an update to the people that still look around here. There have been many important events in the world this year, but I will talk about the selfish me, since this year has been like no other in my professional life.

I'm my own boss!

When I decided to go freelance, I thought it would be for a limited time while I worked on setting up a new startup project. It turned out this was a rather ambitious project, one of the kind I was a bit tired of, requiring a lot of work before reaching out to the world, and that the people I had planned to build it with were not the right ones for this kind of venture.

So I started to grow my freelance business, working 3 days a week architecting and developing the backend systems of a new M2M solution using J2EE (mostly Jetty and Spring), Hazelcast and MongoDB, and doing some short expert consulting gigs in various domains: data collection architecture, mobile geolocation app strategy, search engines, etc. I also did some fun side projects exploring augmented reality and Kinect hacks with my son and my friends at Tetalab, the Toulouse hackerspace.

I was also offered several interesting positions, but I didn't felt like being an employee again now, and most of them required relocating which isn't currently an option. So among others, I declined offers from Google, Facebook and even Apple who all seem to be starving for talents! Feels a bit weird to say "thanks, but no" to these big names...

I've also incorporated my freelance business for tax reasons, and the very difficult part was finding a name. After a long search and the help of startup name generators, we settled during a family gathering on Digispoon. No real website for now, as I don't really need it and don't have the time to set it up. And don't ask for a long explanation for this name: Digispoon just sounds nice, has this "digi" prefix that fits well with my various digital activities, and "spoon" has this double-O that so many web companies have, including my two former employers (Joost and Goojet, now rebooted as Scoop.it). A funny (or stupid) tagline could be "Add a spoonful of digital technology to your projects" :-)

If you have a nice tagline or logo idea, leave a comment!

New projects

In this period of change, two books really inspired me: The 4-hour work week, and Start small, Stay small, a developer's guide to launching a startup. In rather different ways, these books opened my eyes on the fact that there are other routes than either large companies or heavily funded startups. It is perfectly possible to build a one or two persons startup and grow a business that will bring enough money to live more than comfortably. And we live a golden age for this: hosting a web site has never been so cheap, and we have highly productive toolboxes available that make possible things we barely thought about a couple of years ago.

So this is where I am now: I've met people with business ideas and skills but little or no technical knowledge, and along with my part time freelance business I'm working on these projects. These are two different projects, which makes my work week a lot more than 4 hours, but these were interesting opportunities I did not wanted to miss. Nothing to show yet, but they'll both go live in a couple of weeks.

Technically, I'm using Play Framework, jQuery and its countless plugins and Less CSS, with MongoDB, MySQL and Lucene to store data and Hazelcast for cluster messaging and synchronization. A highly productive mix that allows me to go straight to the problem to solve rather than having to fight with boring infrastructure boilerplate.

2012

The year ahead will hopefully see these projets come to a fruitful life. Although I like it, my consulting business is more a means to eat and fund these projects than an end and who knows, maybe in 2013 I'll reach the mythical objective of working 4 hours a week and have plenty of time for all the things I've put aside because of lack of time. Not sure I'll ever reach that point though: I love technology too much!

And I promise I'll write more than a blog post a year :-)

Happy new year all!

How often do you redeploy your J2EE application?

Sylvain Wallez — Tue, 11 Jan 2011 11:37:00 +0100

The guys at ZeroTurnaround have published an interesting report on the development habits from over 1300 Java developers. One of the questions that really struck me was "how often do you redeploy?"

My answer to this question is "on my development machine, never". I use Jetty which is really easy to embed in a regular application. So all of the projets I work on have this tiny class in their src/run/java directory:

package net.bluxte.project;

import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.nio.SelectChannelConnector;
import org.eclipse.jetty.webapp.WebAppContext;

public class StartProject {
    
    public static void main(String[] args) throws Exception {
        Server server = new Server();
        SelectChannelConnector connector = new SelectChannelConnector();
        connector.setHost("127.0.0.1");
        connector.setPort(8888);
        connector.setForwarded(true);
        server.addConnector(connector);
        
        WebAppContext context = new WebAppContext("src/main/webapp", "/");
        context.setClassLoader(Thread.currentThread().getContextClassLoader());
        server.setHandler(context);
        
        server.start();
        server.join();
    }
}

This tiny class allows me to start the server with just a single click on Eclipse's "Run" or "Debug" icons in the toolbar, using the project's classpath where everything is already compiled by the IDE. No need to build a war file, no need to attach to a remote JVM for debugging, no need to wait for long package/deploy/restart operations. And of course code hotswap works just like it should and makes debugging a breeze.

Now the answer "on my development machine, never" is incomplete, and should be added "on the integration server, every time a source file changes" since Hudson picks up any changes, does a full build/test cycle and deploys the new version automatically on a test machine.

Complex "enterprisey" frameworks and specifications have made us forget that things can be simple if you think a bit, and that it also often pays to do a bit of "development design" to increase the developer's productivity without resorting to "helper tools" that only add some bloat to the process.

Sylvain Wallez

Getting meaningful stack traces from Rust tests returning a Result

TestError: an error type that panics early

Publishing TestError on crates.io

'Go: the Good, the Bad and the Ugly'

The Good

Go is easy to learn

Easy concurrent programming with goroutines and channels

Great standard library

Go is performant

Language defined source code format

Standardized test framework

Go programs are great for operations

Defer statement, to avoid forgetting to clean up

New types

The Bad

Go ignored advances in modern language design

Interfaces are structural types

Interface methods don't support default implementations

No enumerations

The := / var dilemma

Zero values that panic

Go doesn't have exceptions. Oh wait... it does!

The Ugly

The dependency management nightmare

Mutability is hardcoded in the language

Slice gotchas

Mutability and channels: race conditions made easy

Noisy error management

Nil interface values

Struct field tags: runtime DSL in a string

No generics... at least not for you

Go has few data structures beyond slice and map

go generate: ok-ish, but...

Conclusion

A few days later: #3 on Hacker News!

Local date time calculations in Go

Micro benchmarks can make you short-sighted

Sum of fizz and buzz

A loop that should have been a lookup table

Conclusion

'Afternoon hack: a USB foot keyboard'

Blog.reboot()

Efficient storage of non-periodic time series with MongoDB

GitHub hack: a common security flaw in webapp frameworks

My 2011: a year like no other

How often do you redeploy your J2EE application?

The `:=` / `var` dilemma