How and Why We Switched from Erlang to Python tells an intern’s tale, from whose perspective it runs like this: we used Erlang, “No one on our team is an Erlang expert” (despite “how crucial this service is to our product”!), and also would you please suspend brain activity while I make some performance claims.

Hold your horses. The good decision to rewrite critical services in a language they actually know aside, let’s look at their notes on perf:

Another thing to think about is the JSON library to use. Erlang is historically bad at string processing, and it turns out that string processing is very frequently the limiting factor in networked systems because you have to serialize data every time you want to transfer it. There’s not a lot of documentation online about mochijson’s performance, but switching to Python I knew that simplejson is written in C, and performs roughly 10x better than the default json library.

Let’s distill these claims:

• Erlang is historically bad at string processing
• string processing is very frequently the limiting factor in networked systems
• simplejson is written in C
• simplejson performs 10x better than the default json library

Further down:

I went into Mixpanel thinking Erlang was a really cool and fast language but after spending a significant amount of time … I understand how important code clarity and maintainability is.

Thus by implication?

• Erlang is not a really cool and fast language
• Erlang is somehow not conducive to code clarity and maintainability

This whole paragraph is just a mess and I can’t quote it without losing some of its slimy essence.

Again, in full:

I’ve learned a lot about how to scale a real service in the couple of weeks I’ve been here. I went into Mixpanel thinking Erlang was a really cool and fast language, but after spending a significant amount of time sorting through a real implementation, I understand how important code clarity and maintainability is. Scalability is as much about being able to think through your code as it is about systems-level optimizations.

By your own admission, no-one on your team is an Erlang expert; you “have trouble debugging downtime and performance problems”. Plenty of Erlang users don’t, so it suggests the problem with your team’s command of the enviroment is severe. Similarly, you earlier mention the “right way” to do something in Erlang, and immediately comment that your code didn’t do that at all – never mind that the “right way” mentioned was wrong.

Yikes.

So why does the word “Erlang” feature in the above-quoted paragraph at all?

There’s no reason to expect either code clarity or maintainability of a service developed over 2 years without an engineer skilled in the environment overseeing the architectecture.

I didn’t say Erlang in that sentence, and yet it has greater explanatory power than the intern’s claim for the same phenomenon.

I suspect their explanation is more controversial, however, and it’s easier to make these claims than arrive at the conclusion that the team’s own shortcomings were responsible for the technical debt accrued – and it makes for a better article. I choose my explanation:

• Erlang is somehow not conducive to code clarity and maintainability: there is not even anecdotal support in the article for this claim

That leaves 5 claims.

Let’s note an important confounding factor: the article is from August 2011. The state of Python and Erlang, and libraries for both have changed since.

As an aside: it’s easy to think that the performance claims they do indirectly make are incidental (and not essential) to the article.

But remove them, and note there’s not really an article any more; a prologue about mapping some concepts from an old codebase to new, and .. an epilogue espousing the virtues of code clarity and maintainability.

Ain’t nobody gonna argue with that, but, as noted above, just that alone does not a “How and Why We Switched from Erlang to Python” blog post make.

Let’s now dig into it – this won’t be much of an article without benchmarks either. Unlike their benchmarks, I’m actually comparing things in order to contrast; their decision to give benchmarks on the new system but not on the old is baffling at best.

I compared 4 Python JSON implementations and 3 Erlang ones:

• json (built-in in Python 2.7.4)
• simplejson 3.3.0 from PyPI, both with and without C extensions
• ujson 1.30 from PyPI
• mochijson and mochijson2 from mochiweb
• jiffy

simplejson is what the intern picked for their rewrite. mochijson is what they were using before.

All versions are current at time of writing.

Testing method:

• read 5.9MB of JSON from disk into memory
• start benchmark timer
• parse JSON from memory 10 times, each time doing some minimal verification that the parse was successful
• force a garbage collect
• stop our benchmark timer

The code is available on my GitHub account, and instructions to reproduce are found there.

Here are the results:

• ujson: 1,160ms
• jiffy: 1,271ms
• simplejson (with C): 1,561ms
• json: 2,378ms
• mochijson2: 8,692ms
• mochijson: 11,111ms
• simplejson (no C): 16,805ms

ujson wins! jiffy a close second! simplejson a close third! These results are the average of three runs each, but I did many more runs in testing the benchmark code and can say the variance was quite low.

So:

• simplejson performs 10x better than the default json library: this doesn't appear to be the case now. It may have been the case in 2011, depending on what the default json library was back then.
• Erlang is not a really cool and fast language: in this particular example the best Erlang library is on par with both of the best Python libraries – all three C-boosted, of course – and the best pure Erlang library runs in half the time as the apparently-best pure Python one. (json is C-boosted)

That leaves us with these claims unrefuted:

• Erlang is historically bad at string processing
• string processing is very frequently the limiting factor in networked systems
• simplejson is written in C

Erlang’s historical performance is somewhat irrelevant, but the claim stands nevertheless.

No evidence was advanced for the second claim: there was no way to determine whether faster string processing was responsible for any improvement in their benchmarks: we don’t even know if the benchmarks improved because we only were given one set (!). Of course, the changes being the entire system and not just string processing, before-and-afters would prove nothing, especially given the proficiency gap. Hence:

• string processing is very frequently the limiting factor in networked systems: maybe, maybe not, but picking the right library makes a big difference!

I mean, jeez; they could reduce their string processing (and thus the “limiting factor”?) by 33% if they switched from simplejson to ujson!

As for the third claim, if I don’t nitpick, it stands. Kinda.

Why did I feel the need to write this up?

I saw the article pop up on Hacker News today, 2 years after its publication. In fact, I’d seen the article not long after it was originally published, and I know it was on HN back then too. I don’t care about the fact that it was reposted; more that it was written at all.

It’s exactly the sort of useless bullshit that seems to fill a quarter of HN’s front page at any given stage: articles with titles like “Why I Don’t Use Vim”; “Why You Should Learn Vim”; “The Reason You Need To Understand System F If You Want To Ever Amount To Anything”; “Stop Thinking. Just Write.”; “How We Saved 95% Of Our Datacentre Costs By Rewriting Everything In Ada”; etc. etc. etc.

It’s edgy shit that grabs your attention and way oversells one point of view, at the expense of reason. This comment thread started out by acknowledging this trend, but was ruined by someone not catching the pattern being invoked. Usually there’s a nice novel point to be made somewhere if you can read the moderate subtext between the bold claims.

Unfortunately this article had no such point, and so turned out to be 100% garbage. But still, some people will read that article and go away thinking their prejudices about Erlang have been confirmed by someone who’s been in battle and seen it for themselves.

And really, this isn’t about Erlang. I don’t care what language or environment you use. Your coding philosophies are irrelevant, if you deliver – and Mixpanel are delivering, and clearly made the right choice to get rid of some technical debt there.

But don’t then try to shift any part of the responsibility of the decision to pay off that debt as being your tools’ faults, and especially not with such flawed logic.

I’ve started a project in Erlang recently, and I needed a big block of mutable memory – a massive byte array, buffer, whatever you want to call it. Erlang’s binary strings are immutable and so probably not suitable.

I figured the core distribution must’ve had something like this … nope. Spending 30 minutes Googling and checking docs twice and thrice over, but there’s clearly no mutable byte array in Erlang itself.

Is there a hack that approximates to this? Search … this StackOverflow almost seems hopeful at the end, referencing hipe_bifs:bytearray/2 and hipe_bifs:bytearray_update/3, but alas, they are so-named because they are part of the HiPE native compiler, and not Erlang itself. (I’m not currently using HiPE, and it would be nice to not be tied to it.)

Then, I thought, surely someone else has extended Erlang to do this. There are several modes of extending Erlang, but native implemented functions (NIFs) would be the obvious candidate for manipulating large chunks of memory.

Googling this yielded complete failure: people just don’t seem to be doing it. (If people are doing it and you know this, please, let me know.)

So I did it: k6_bytea.

The interface is simple and fairly pleasant:

1> Bytea = k6_bytea:from_binary(<<"Hello, world.">>).
<<>>
2> k6_bytea:set(Bytea, 7, <<"Thomas">>).
ok
3> k6_bytea:to_binary(Bytea).
<<"Hello, Thomas">>
4> Gigabyte = k6_bytea:new(1000000000).
<<>>
5> k6_bytea:delete(Gigabyte).
ok
6>

The gigabyte allocation caused a small notch on my memory performance graph:

perf

The obvious question remains: how does it perform, vis-à-vis binary strings?

Let’s make a contrived benchmark: initialise a large buffer, write all over it in a deterministic fashion, and copy data all over it. Let’s do it in parallel, too.

Here’s the benchmarking code for your perusal:

-module(k6_bytea_bench).
 
-export([run/0, binary_strings/1, k6_bytea/1]).
 
-define(COUNT, 1024).   % Parallel operations.
-define(SIZE, 102400).  % Size of buffer to work on.
 
run() ->
    measure(<<"Binary strings">>, ?MODULE, binary_strings),
    measure(<<"k6_bytea">>, ?MODULE, k6_bytea).
 
measure(Name, M, F) ->
    {SM, SS, SI} = erlang:now(),
    spawn_many(?COUNT, M, F, [self()]),
    receive_many(?COUNT, done),
    {EM, ES, EI} = erlang:now(),
    Elapsed = ((EM - SM) * 1000000 + (ES - SS)) * 1000 + ((EI - SI) div 1000),
    io:format("~s [~p ms]~n", [Name, Elapsed]),
    ok.
 
spawn_many(0, _, _, _) -> ok;
spawn_many(N, M, F, A) ->
    spawn(M, F, A),
    spawn_many(N - 1, M, F, A).
 
receive_many(0, _) -> ok;
receive_many(N, M) -> receive M -> receive_many(N - 1, M) end.
 
binary_strings(Done) ->
    B0 = <<0:(?SIZE*8)>>,
    B1 = binary_strings_set_bits(B0, lists:seq(0, ?SIZE - 1024, ?SIZE div 1024)),
    _ = binary_strings_copy_bits(B1, lists:seq(0, ?SIZE - 1024, ?SIZE div 1024)),
    Done ! done.
 
binary_strings_set_bits(B, []) -> B;
binary_strings_set_bits(B, [H|T]) ->
    <<LHS:H/binary, _:1/binary, RHS/binary>> = B,
    binary_strings_set_bits(<<LHS/binary, (H rem 255), RHS/binary>>, T).
 
binary_strings_copy_bits(B, []) -> B;
binary_strings_copy_bits(B, [H|T]) ->
    <<LHS:H/binary, Bit:1/binary, _:1/binary, RHS/binary>> = B,
    binary_strings_copy_bits(<<LHS/binary, Bit/binary, Bit/binary, RHS/binary>>, T).
 
k6_bytea(Done) ->
    B = k6_bytea:new(?SIZE),
    k6_bytea_set_bits(B, lists:seq(0, ?SIZE - 1024, ?SIZE div 1024)),
    k6_bytea_copy_bits(B, lists:seq(0, ?SIZE - 1024, ?SIZE div 1024)),
    k6_bytea:delete(B),
    Done ! done.
 
k6_bytea_set_bits(B, []) -> B;
k6_bytea_set_bits(B, [H|T]) ->
    k6_bytea:set(B, H, <<(H rem 255)>>),
    k6_bytea_set_bits(B, T).
 
k6_bytea_copy_bits(B, []) -> B;
k6_bytea_copy_bits(B, [H|T]) ->
    Bit = k6_bytea:get(B, H, 1),
    k6_bytea:set(B, H + 1, Bit),
    k6_bytea_copy_bits(B, T).

Over 3 runs, binary_strings averaged 24,015ms, and k6_bytea 198ms (0.83% time, or 121x speed).

There’s nothing very surprising about this; it’s large, unwieldy immutable data-structures vs. one mutable data-structure. It’s even the case that I have no idea if there are any better ways to simulate a byte array in Erlang, either with binary strings, or without!

The binary string-manipulating code above is ugly and error-prone, as it’s clearly not the purpose it was built for. If it should turn out that this really hasn’t been done better by someone else, then I encourage you to look to and improve k6_bytea for this purpose.

Lately I’ve been game programming. A few things have been coming to mind:

• The Sapir–Whorf hypothesis is totally true when it comes to programming languages.
• Dynamically typed languages encourage a whole lot of waste.
• There’s some sweet spot of expressivity, low level and non-shoot-yourself-in-the-footness in language that’s missing.

I will attempt to expound.

The languages I end up using on a day-to-day basis tend to be higher level. Non-scientific demonstration by way of my GitHub account’s contents at time of writing:

15 Ruby
9 OCaml
6 Go
6 Javascript
4 C
3 C++
2 Clojure
3 Perl
2 Haskell
...

In my professional career, I’ve concentrated on JavaScript, PHP, Erlang, Python and C#. The lowest level of these is, by far, Erlang. Perhaps it’s fairer to say that Erlang keeps me in check, perf-wise, more than any of the others.

So my mind has been a fairly high-level sort of mindset. I’ve made occasional trips to lower-level code, but there hasn’t been much of that lately, particularly as I’ve changed jobs and needed to concentrate solely on work stuff for a while.

Choosing C++ to write a game wasn’t too hard; it’s fairly standard in the industry, and I know it quite well. Bindings to libraries are plentiful, and getting the same codebase compiling on Windows, OS X and Linux is a well-understood problem that’s known to be solvable.

The thing is, C++ makes it abundantly clear when you’re doing something costly. This is something that occurs particularly to me now as I’ve not done this lower-level stuff in a while.

You wrote the copy constructor yourself, so you know exactly how expensive pushing a bunch of objects into a vector is. You chose a vector, and not a list, so you know exactly why you don’t want to call push_front so many times. You’re creating a ostringstream to turn all this junk into a string: it has a cost. Are we putting this on the stack or in the heap? Shall we use reference counting?

You make hundreds of tiny decisions all the time you’re using it; ones which are usually being abstracted away from you in higher level languages. It’s why they’re higher level.

And that’s basically all I have to say on that: the language makes you feel the cost of what you choose to do. Going to use a pointer? Better be sure the object doesn’t get moved around. Maybe I’ll just store an ID to that thing and store lookups in a map. How costly is the hashing function on the map key? You add such a lookup table in Ruby without a moment’s notice; here, you’re forced to consider your decision a moment. Every time you access the data, you’re reminded as well; it’s not something that ever goes out of mind.

Moreover, the ahead-of-time compilation means you can’t do costly runtime checks or casts unless you really want to (e.g. dynamic_cast), but again, the cost of doing so means you’ll never be caught unaware by slowing performance. In many (dynamic) higher level languages, basically every operation is laced with these.

So it’s suited for games programming, because performance is usually pretty important, and the language keeping performance on your mind means it’s not hard to consistently achieve high performance.

But C++’s deficiencies are also well-documented. It’s awful. It’s waiting to trip you up at every turn. After re-reading those talk slides, I figured I’d just port the code to C – until I remembered how much I used std::string, std::vector, std::list, and moreover, enjoyed the type- and memory-safety they all bring. I’m not particularly fond of giving that up and implementing a bunch of containers myself, or using generic containers and throwing away my type checks.

I think I’m after C with templates for structs (and associated functions), but I’m not sure yet. If you think I want C++, you probably need to re-read those notes.

The other solution is to only use as much of C++ as I like, and that’s basically what I do – but the language is still waiting to trip me up, no matter how much I try not to use it.

Time to think a bit about what the issues at hand really are.

I use Snapchat. It’s an app where you can take a photo or short (< 10 second) video and send it to your friends who use the service; they’ll then be able to see it, once, before it disappears forever.

Ostensibly, the app is for sexting, because there’s no fear that your photo will get spread around (no forwarding/etc.) or retained for longer than you’d like, but it seems like it’s not as much a sexter’s hangout as the media might want you to think.

My circle of friends use it basically as an extension of weird Twitter – most snaps I send and receive are strange angles of weird objects; the completely mundane but somehow therapeutic (7 seconds of the camera pointed outside the window of a tram, pointed at the ground moving below); or just closeups of Curtis Stone’s face, wherever we see him.

Of course, the promise that they won’t get retained is just that: a promise. Since your phone receives this image and shows it to you at some point, it must be downloaded by your phone. If it can be downladed by the phone, it can be downloaded by something else. We decided to find out how.

Here’s Contrigraph, a “data visualisation” (?) created by generating commits to match the Contribution Graph Shirt from GitHub.

It was pretty much a hack; first, I read the colours straight off the shirt, producing a big block of data like

0002342
2223322
2323241
2224333
3322122
2242231
...

Then we read that in one digit at a time, work out what day to start on so everything aligns correctly, and how many commits on a given day produce each gradient of colour. The result isn’t pretty:

start = Time.new 2012, 4, 23, 12, 0, 0, 0

tbl = {"0" => 0, "1" => 0, "2" => 1, "3" => 9, "4" => 14, "5" => 23}

3.times do 
  dbd.each do |n|
    tbl[n].times do
      `echo 1 >> graafik`
      `git commit -a -m 'contrigraph' --date='#{start.to_s[0..9]}T12:00:00'`
    end
    start += 86400
  end
end

Three times so this thing will scroll on for the next few years. Other values for tbl would work too; I just didn’t bother to do anything better. I’ve written clearer code, but that wasn’t really the point either.

I actually screwed this up twice: first I didn’t remember to treat the 0 entries correctly (i.e. I should have skipped those days, whereas I ignored them entirely); second, it seemed like I was getting hit by timezone issues where everything was shifted up a block.

In retrospect, I should have first produced a mini-contributions graph generator (i.e. one that takes a Git repository and produces what GitHub would do), validate that against an existing user/repo, then use that to ensure it’d work the first time. I did a similar thing to ensure I had the data correct, by producing a graph directly from the data:

This has come up a few times recently, so a little note:

/./ will match any character—except a newline.

/[\s\S]/, /[\w\W]/ and so forth are the only way to portably match every character including newlines.

what about /./m?

/./m does not match a newline (usually—see below).

The m modifier does not alter . in any way (usually—see below).

m changes what ^ and $ match—normally they’ll match only the start and end of the string, but m alters them to also match the start and end of any line, i.e. immediately after and prior a newline.

Mnemonic: m for “multiline”.

what about /./s?

Yes, that will do what you want—unless you’re using Ruby or JavaScript.

In Ruby, the s modifier is inexplicably unused—in 1.8, it stays on the Regexp object, but in no way affects matching, and in 1.9, it doesn’t even stay on the object, it just disappears. Note that these behaviours are different to using a complete nonsense modifier, like f, which causes a SyntaxError¹.

Even more inexplicable, this feature has been rolled into m instead! So in Ruby, you can use /./m to match a newline, and you can also use /^a/m to match an a at the beginning of the string, or after any newline in the string.

In JavaScript, the s modifier is absent entirely, and it’s not rolled into anything else. Use /[\s\S]/.

Mnemonic: s for “single line” (the string is treated as one line, in a twisted sense).

conclusion

Yay, regular expressions. Be sure what you mean.

To match any character including newlines:

• In JavaScript or any other language lacking the s modifier, use /[\s\S]/.
• In Ruby, use /./m, but be aware that this also modifies ^ and $. If unsure, use /[\s\S]/.
• If you have true PCREs, you may safely use /./s.

Note that this is often what you really want—rarely do you want to explicitly match every character except newlines. If you do that on purpose, at least leave a comment to that effect, otherwise your coworkers will just assume you didn’t know what you were doing.

Writers of all stripes enjoy engaging in the most cynical readings of human behavior because they think it makes them appear hyper-rational. But in fact here is a perfect example of how trying to achieve that makes you irrational. Human emotion is real. It is an observable phenomenon. It observably influences behavior. Therefore to fail to account for it when discussing coupling and relationships is the opposite of cold rationality; it is in fact a failure of empiricism.

L’Hote on Kate Bolick’s “All the Single Ladies”.

As programmers, we spend a lot of time just carting data from one place to another. Sometimes that’s the entire purpose of a program or library (data conversion whatevers), but more often it’s just something that needs to happen in the course of getting a certain task done. When we’re sending a request, using a library, executing templates or whatever, it’s important to be 100% clear on the format of the data, which is a fancy way of saying how the data is encoded.

Let’s do the tacky dictionary thing:

encoding (plural encodings)

1. (computing) The way in which symbols are mapped onto bytes, e.g. in the rendering of a particular font, or in the mapping from keyboard input into visual text.

2. A conversion of plain text into a code or cypher form (for decoding by the recipient).

I think these senses are a bit too specific—if your data is in a computer in any form, then it’s already encoded. The keyboard doesn’t even have to come into it.

coworker: If it were a critical task, how long to get that UI up and running?
me: Probably about the same amount of time if it were non-critical, all things considered
me: Maybe a little longer
me: Stress tends to make tasks drag out

Reference.

An employee of the company who mantains Mongo, who appears to be assigned Erlang driver maintenance lately, seems to deduce that the tests fail¹, and so changes an API which has been stable for 2 years in order to “make the tests pass”, despite also changing the tests.

WAT.