kivikakk.ee

Note: this is a pretty long article which does a deep dive into breaking some amateur crypto. I go on for quite a bit. Make a cup of tea before reading, and get ready to read some code!

introduction

Everyone knows it. Rolling your own cryptography is a terrible idea. Here’s Bruce Schneier writing about it in 1999. Here’s an excellent answer on the Infosec Stack Exchange about why you shouldn’t do it. Here’s another Scheiner post with an excellent opening sentence.

This, then, is a post about a broken homegrown cryptosystem; namely, that used in CodeIgniter, pre-2.2. This version was current until the release of CodeIgniter 2.2, on the 5th of June, 2014, and you can still find sites on it today.

The attack described in the post depends on a lot of things to go right (or wrong, if you will); it’s not just that they used a bad cipher, but also the fact that they rolled their own session storage, and implemented a fallback, and a dozen other things. This is probably typical for most bugs of this class; a bunch of bad decisions which aren’t thought through find their logical conclusion in complete insecurity.

Let’s get into it!

Read more

I made an extractor for Bemani IFS files. It’s been done a few times before, but this one’s reasonably usable compared to “drag and drop one file on this program, which spawns 50 files more, each of which you drag on this program to decompress, then each decompressed file to another program which spits up a window and has you drag a slider around until it looks right.”

Features:

  • You can open an IFS file and browse the data therein without saving them all out to a file.
  • You can export images as BMP, GIF, JPEG, PNG or TIFF.
  • The index used for guessing the aspect ratio of the image is saved and used automatically when browsing other images with the same pixel count.

It’s a .NET 2.0 application, and is in the public domain.

A screenshot!

a strange guide

This is a semi-comprehensive, non-prosaic (but hardly poetic) reference to vim. Approach with caution.

Modes:

  • normal: move the cursor, delete, enter insert mode, delete then insert, enter visual mode, yank and put (copy and paste)
  • insert: type stuff, move the cursor, scroll, put
  • visual: do things to the selected area, move your cursor about within it

And more:

  • registers: where your deleted and yanked text goes
  • text objects: even more ways to select text
  • what else?: what haven’t I covered?
  • CC0: this text is in the public domain

normal

move the cursor, delete, enter insert mode, delete then insert, enter visual mode, yank and put (copy and paste)

move the cursor

hjkl move the cursor left-down-up-right respectively. It’s good to remember these, but no big deal if you don’t; cursor keys usually work too. On Dvorak, you have jk on your left hand for vertical movement, and hl on your right for horizontal.

w moves a word forward. b moves a word backward. These stop at punctuation, too; if you want to skip a “word” where a word is just anything that’s not whitespace, W and B do the trick.

e moves forward until the end of a word, instead of the start of a word. E has the same rule as above.

These are a good start: using wbWB are pretty good for getting around quickly.

You can prefix any command with a number; 5w moves 5 words, 10B moves backwards 10, uh, ‘big’, words. These work on hjkl as well (and almost anything, honestly).

Go to the start and end of the line with ^$.

Go anywhere by searching with regular expressions; type /, and then complete the regular expression in the buffer beneath. n and N then find the next and previous match. These all function as movement commands in the below contexts, too. ? is the same as /, but backwards. Note that n and N are reversed when you’re in such a reverse search!

You can also go quickly to the following given character on the current line; f followed by the character instantly goes there. fa will put the cursor on the next a. Fa goes backwards. You can also go ‘til the next character, i.e. to the character immediately before, with t; ta puts the cursor just before the following a, and Ta just after the preceding one.

(You’ll note here that uppercase commands tend to vary the lowercase equivalents, by degree, inversion, or other shift in meaning. I do say “tend”, though, because there are plenty that don’t.)

<Enter> is another way of saying j, and <Space> another way of saying l.

) and ( move a sentence forward and backward respectively. } and { move by paragraphs instead.

]) and [( move to the next and previous unmatched ) and ( respectively, as do ]} and [{ for } and {. More understandably put: they move out to the next parenthesis or brace level in the given direction; they go “up” a level.

% moves to (and including) the matching token under the cursor; if your cursor is on a (, it’ll move to the matching ), etc.

G moves to the end of the document. If you prefix a number, it moves to that numbered line; 1G to the very top.

delete

d deletes stuff. You have to tell it what by following with a movement command: it deletes from the cursor position up to where you tell it. dw deletes the word you’re on, up to the start of the next word. dl deletes up to the following character; i.e. the single character your cursor is on.

Note that if you’ve hit d and change your mind, <Escape> will abort that command.

Operations spanning lines are a bit different. dd will delete the entirety of the current line. This repetition of a command to mean ‘this line’ is a pattern. You can use a number: 2dd will delete this line and the one after it. On that, 3dw deletes 3 words. (So does d3w.)

dj (delete down) will delete this and the next line. dk will delete this and the previous line.

D is the same as d$; it deletes to the end of the line.

d/blah<Return> will delete from the cursor until the next blah in the file. d?blah<Return> deletes back to the previous blah, including the entire matching word itself.

dta deletes from the cursor up to and not including the following a. dfa would include the a in its deletion. 2dta deletes up to and not including the second following a, etc.

x deletes a single character, as a handy shorthand. If you have the word programmer and the cursor positioned at the start, 10x will delete the whole word.

d) will delete to the end of the sentence; as will d} the paragraph. d% deletes the entirety of whatever the token under the cursor delimits; if your cursor is on a matched brace, the entire brace will be deleted, and so on.

enter insert mode

i will enter insert mode, leaving the cursor where it is.

a will enter insert mode, but places the cursor after the current character, i.e. appends.

I and A enter insert mode, but first move the cursor to the start or end of the line respectively. (These are handy.) They’re the same as ^i or $a respectively.

Note that numbers work with these too: 3i will enter insert mode, and whatever you type will be inserted thrice in total, once you leave the mode.

o and O enter insert mode, “opening” a line as they do. o opens a new line underneath (same as A<Return>), and O opens a line on the current line (same as I<Return><Escape>ki, or just ko if there’s a line above). You might need a mnemonic to remember which is which; O reaches higher up the screen than o, and so opens the line higher.

R enters replace mode; like insert, but overwriting text instead of inserting before it.

delete then insert

You can delete then insert in the same command; this is called “changing” text. Just use c instead of d, and it functions nearly identically as for deleting, and then enters insert mode on the spot. (cw is the same as dei; cc is the same as ^Di; the differences are that the whole object isn’t removed, that is to say, trailing spaces or newlines are kept. ce and cw do the same thing.)

C functions as Di. Change is often handy as c/, ct, cT, etc.

Note that numbers before change commands affect only the delete portion of the command; 2cw is literally like 2dei, not 2dei2i, in that it removes two words (excluding the trailing space), and then enters insert mode, but what you insert isn’t entered twice. This is almost always what you want.

s is like cl; it’ll change a character on your cursor. 10s erases the ten characters from under your cursor and enters insert. S is the same as cc (since otherwise what would it do different to C?).

r is like s but only for a single character; it waits for you to enter the character, which the one under the cursor will be replaced with. 10ra replaces the current and following 9 characters with the letter a.

enter visual mode

v enters visual mode. The cursor position at the time of entering visual mode becomes one end of the selection, and now you move the other.

Visual mode selects a range of text, traversing newlines as they occur sequentially in the text.

V enters visual line mode; this selects whole lines at a time, inclusive of both ends of the selection.

CTRL-V enters visual block mode; this selects a physical rectangular block of text. It can be a bit weird at first, especially with its behaviour crossing lines and empty lines, but quite useful, especially in combination with r, c, and s.

gv will enter visual mode with the last selection you had already made. (Note that your cursor won’t necessarily be anywhere near where it ends up.)

yank and put

y takes a movement like other commands; yy (copies) yanks the current line, yj the current and next, y2w (or 2yw) the two words from the cursor position onward, etc.

p puts (pastes) what was yanked after the cursor position. If you yanked linewise (like the first two examples above), the put is linewise: they are put as whole lines underneath the current one, as if you’d used o to open.

P yanks at the current cursor position, as if you used i to insert the text. If you yanked linewise, it puts as if you used O to open.

All delete commands, including change commands c, s and so on, actually have been the equivalent of “cut” operations. If you e.g. use cw to change a word, p will then put the deleted word.

See below about different registers in case you want to save some text without it getting clobbered by subsequent yanks or deletes.

insert

type stuff, move the cursor, scroll, put

type stuff

Type stuff! Go on!

move the cursor

If cursor keys are enabled, they probably will let you move the cursor in vim without exiting insert mode.

<Escape> will leave insert mode, by the way.

scroll

There’s a submode of insert mode that seems to be called expand mode; <CTRL-X> enables it. Any keypress not recognised by the mode will just be handled by regular insert mode and quit the expand mode; of note are <CTRL-E> and <CTRL-Y>, which scroll the window down and up respectively. Just start typing to exit the mode.

It’s worth noting these scroll in normal mode, too.

put

You can also put text while in insert mode. <CTRL-R>" will put whatever you last deleted or yanked; see below about registers for more on this; the " is specifying the register to put.

You can get help on insert-mode keys, by they way, by typing something like :help i_CTRL-R.

visual

do things to the selected area, move your cursor about within it

do things to the selected area

Any command, like c and d, will affect the selected mode en masse.

move your cursor about within it

Pressing o will change the end of the selection you’re moving to the other end; useful to adjust the selection.

The three visual modes, accessed with v, V and CTRL-V, are exited by pressing the appropriate key again. You can switch between visual modes by pressing the key for any other visual mode while already in one.

registers

When you delete or yank text, it’s put in a register; a clipboard, essentially. Each register is identified by a character, and they’re referred to by prefixing a ". "a is the a register, "F the F register, etc. There are special registers; the unnamed register, "" (i.e. the register name is double-quote), is what’s used by default by yank, delete and put.

To perform any register-affecting command with a given register, prefix the entire command with the register reference; "ay$ yanks the rest of the line into the "a register. "ap puts that. (Note that dcsxy always affect the "" register, as well as any specified one.)

"_ is the blackhole register; it causes the command to affect no registers. (This is the only exception to the parenthesis prior to this. Ahem.)

Per put in insert, <CTRL-R> waits for a register name; <CTRL-R>a in insert mode will put the contents of the "a register.

Registers "0 thru "9 have special meaning; see |quote_number| (type :help quote_number in normal mode) for more information.

If you specify a letter-named register with lowercase letters, you replace the contents; "ayw sets the contents of the "a register to the word from the cursor on. The same register reference but uppercase causes that register to be appended to.

More special registers exist; see |registers|.

text objects

Text objects are another way to select areas of text. They can’t be used as movement commands in normal mode in and of themselves, as they refer to regions; they may only be movement command arguments to commands that do something to regions, such as d and c. They are, however, very useful.

I lied, though: they can be used directly in visual mode, which selects the region they refer to. Experimenting with this is a good way to see how they work.

Text object commands start with either i or a; the former selects an object not including surrounding whitespace (“inner”), the latter includes surrounding whitespace. (Much like i leaves the cursor put when entering insert mode, but a moves it forward. It sort of makes sense.)

After that, you specify the kind of object; they’re based roughly on movement commands. iw, for instance, refers to the word your cursor is on top of. diw deletes the word your cursor is on. Note how this differs (usefully) from dw; dw deletes forwards from the cursor until the start of the next word, whereas diw deletes the entire word under the cursor, regardless of your cursor’s position.

Some common use cases: change the word under the cursor: ciw. Delete the word under the cursor entirely, closing any gaps: daw. Same, but take any punctuation touching the word with it, too: daW.

Sentences and paragraphs are accessed with s and p, again, following i or a. Give some of these a try in visual mode to see what I mean: enter visual mode, then try iw, then aw. Try ip, then ap. (Note that these will put you in visual line mode.)

Some of the most useful are the matching token operators: i[ (or i] ­ there’s no difference) will select the contents of the nearest matching [...], but not the [] themselves. a[ (or a]) include the tokens themselves.

Same works for {, (, <, ", ', `. Changing a string or HTML attribute quickly? ci". Want to quickly start a selection with a given block of code? va{.

Repetition works: 2di( deletes everything in the second-closest pair of parentheses, but not those parentheses themselves.

Editing HTML, t will work on HTML tags; it the contents of the nearest tag, at the contents and opening and closing tags themselves.

These are really lovely. See |text-objects| for more.

what else?

This is not even close to an exhaustive list of things in vim.

There are recordings (macros), which can be extraordinarily powerful; see |recording|. Note that they record and execute into and of the same registers that you paste in, meaning you can paste out a recording, modify in the text, then yank it back into the register for execution.

There are even more movement commands: * and # search for the word under the cursor, ; and , repeat your last tTfF command forward and backward (again with confusing reversing behaviour), ` and ' operate on marks set by m (see |mark|), and there are a million more commands prefixed by g (see |g|) that either modify the behaviour of the non g-prefixed command, or do something totally different. It’s worth a look.

Further, there’s the entire command-line mode, accessed by pressing : in normal mode. There are an enormous number of commands, many operating on ranges (pressing : in visual will start the command-line with a range matching the visually-selected lines); a useful one is s (see |:s|), for search-and-replace.

There’s ., which repeats your last command. Learn how it behaves in different circumstances and use it.

Keyword completion, triggered directly with CTRL-N and CTRL-P or through expand (^X) mode (see |i_CTRL-N|, |i_CTRL-X_CTRL-N|) just does a lookup based on tokens in open files. There’s omnicomplete (|compl-omni|, |i_CTRL-X_CTRL-O|), which can use plugins to intelligently autocomplete (like IntelliSense).

And a heck of a lot more, too.

Good luck!

thanks

Thanks to Nethanel Elzas for emailing in fixes and suggestions!

CC0

CC0
To the extent possible under law, kivikakk has waived all copyright and related or neighboring rights to “Being sorta useful in vim”. This work is published from: Australia.

It’s impressive how hard it is to use VSTO correctly.

When a COM handle crosses from unmanaged into managed code for the first time, a “runtime callable wrapper” is created for it. This wrapper maintains a reference count, incremented whenever the object is newly requested across the boundary.

The wrapper releases the COM object behind it when its finalizer is run, but this relies on all .NET references to the RCW itself becoming inaccessible to the GC and for the GC to pick it up. This may not reliably happen.

You can force the matter by decrementing the reference count on the RCW yourself: Marshal.ReleaseComObject does this, and should the reference count drop to zero, the COM object will get released immediately. This means taking a lot of care, though, since it’s not always obvious which VSTO calls result in the counter being incremented and which don’t. Something like the following can help work it out:

var refs = new object[3];
for (var i = 0; i < 3; ++i)
    refs[i] = my.VSTO.Call();
for (var i = 0; i < 3; ++i)
    Debug.WriteLine(Marshal.ReleaseComObject(refs[i]).ToString());

If each call to Marshal.ReleaseComObject returns the same number, it implies each call is incrementing the reference count; if the return values are decreasing, the call is not incrementing the count.

VSTO event callbacks are said to not need references released on passed-in COM objects, but an MSDN blog disagrees. There is no consensus here. For anything else at least: remember to release COM objects when you’re done with them, otherwise it can get painful.

Time zones in .NET

I’m a fairly new .NET developer. I worked with the framework a bit in 2005–6, but hadn’t really touched it since then. In the meantime, I’ve been sticking to the Linux ecosystem, and a little OS X, as mentioned in a previous article.

So, time zones. I know they’re a sore point for many environments, but most seem to dig themselves out of the hole and provide something that is, in the end, usable.

Ruby’s builtin Time is actually pretty darn good, and if you use Rails, ActiveSupport makes it even better. pytz seems .. alright. Databases generally have their heads screwed on straight. A lot of the time you can get away with just storing seconds since the epoch and call it a day, because there’s nothing more intrinsic built into the system.

Then I got my new job, and it was time get back into .NET. A lot has changed since 2006; it had only hit 2.0 then, mind.

So I felt confident I was using the latest, modern stuff. We target 4.0 and 4.5 across our projects, and there’s plenty nice about it.

Then I had to work with System.DateTime. Oh. Oh, gosh.

I quote the manual at you.

DateTime.Kind Property

Gets a value that indicates whether the time represented by this instance is based on local time, Coordinated Universal Time (UTC), or neither.

Kind is the lone field on a DateTime which has anything to do with time zones. It can take the value Local, Universal, or Unspecified. What does that even MEAN. Note that Kind is ignored in comparisons, too, which can only mean more fun for application developers.


It would be remiss of me to fail to note the paragraph in the docs which state:

An alternative to the DateTime structure for working with date and time values in particular time zones is the DateTimeOffset structure. The DateTimeOffset structure stores date and time information in a private DateTime field and the number of minutes by which that date and time differs from UTC in a private Int16 field. This makes it possible for a DateTimeOffset value to reflect the time in a particular time zone, whereas a DateTime value can unambiguously reflect only UTC and the local time zone’s time. For a discussion about when to use the DateTime structure or the DateTimeOffset structure when working with date and time values, see Choosing Between DateTime, DateTimeOffset, and TimeZoneInfo.

The linked page states that “although the DateTimeOffset type includes most of the functionality of the DateTime type, it is not intended to replace the DateTime type in application development.”

Intention or not, it should ALWAYS be used. It lists as a suitable application for the DateTimeOffset:

  • Uniquely and unambiguously identify a single point in time.

Because we don’t want that at any other time? When do you want a DateTime which non-specifically and ambiguously identifies several points in time?

On the other hand, listed as suitable for DateTime:

Retrieve date and time information from sources outside the .NET Framework, such as SQL databases. Typically, these sources store date and time information in a simple format that is compatible with the DateTime structure.

No fucking comment.

It continues:

Unless a particular DateTime value represents UTC, that date and time value is often ambiguous or limited in its portability. For example, if a DateTime value represents the local time, it is portable within that local time zone (that is, if the value is deserialized on another system in the same time zone, that value still unambiguously identifies a single point in time). Outside the local time zone, that DateTime value can have multiple interpretations. If the value’s Kind property is DateTimeKind.Unspecified, it is even less portable: it is now ambiguous within the same time zone and possibly even on the same system on which it was first serialized. Only if a DateTime value represents UTC does that value unambiguously identify a single point in time regardless of the system or time zone in which the value is used.

Completely useless. So, we’ll use DateTimeOffset in our application code, right?

Only the ecosystem hasn’t caught up.


Enter Npgsql, a Postgres driver for .NET with a frightening amount of code. It only works with DateTime objects when sending or receiving timestamps to or from Postgres.

Postgres has two column types: timestamp with time zone and timestamp without time zone (or timestamptz and timestamp, respectively). The former is about as good as a DateTime, but without trying to be more than it can: it doesn’t have Kind, which improves its usability by an order of magnitude. You can make a policy decision like “we’ll always store UTC timestamps”, and you’ve solved timezones in your application. They mark a specific point in time unambiguously.

Or you can just use timestamptz and they still unambiguously mark a specific point in time. It’s magic!

So how does Npgsql deal with this?

The genesis of this post was because we were noting strange behaviour: we had read a timestamptz out of the database, and then later SELECTed all rows where that column was strictly less than the value we read out. And yet that same row would be included in the result. It made no sense.

Turns out it really did make no sense.

The rest of this blog is a sequence of test cases which demonstrate just how bad the situation is.

[SetUp]
public void SetUp()
{
    _connection = new NpgsqlConnection(
        "Host=localhost; Port=5432; Database=time_zones_in_dot_net; " +
        "User ID=time_zones_in_dot_net");
    _connection.Open();
}

[TearDown]
public void TearDown()
{
    _connection.Close();
}

[Test]
public void TimeZonesSane()
{
    // introduction
    // ------------

    // This test assumes the *local* machine (running NUnit) has local time of +10.
    // It's agnostic to the time zone setting on the database server.
    // In other words, Postgres +1, .NET -100000000.

    // Render UTC (+0), Postgres (+3) and .NET (+10) distinguishable.
    _connection.Execute("SET TIME ZONE '+3'");

    // In the below tests we assert that the queries yield a .NET DateTime object
    // which, when .ToUniversal() is called on it, produces the given date in
    // "mm/dd/yyyy HH:MM:SS" format.

    // After that is the non-.ToUniversal() date in parenthesis.  This is *always* 10
    // hours ahead for a Local or Unspecified, and the same for Utc.  DateTime
    // objects have no knowledge of offset, only Kind.

    // There's also a character appended to represent what time zone "kind" it came
    // back with; "L" for Local, "?" for Unspecified, "U" for Universal.

    // As noted below, ToUniversal() on a Local or Unspecified returns a new DateTime
    // with Kind set to Universal, and unilaterally subtracts the time zone offset of
    // the machine the code is running on.


    // tests using string literals
    // ---------------------------

    // Not useful in themselves, because we'll never use string literals, but help to
    // demonstrate some initial weirdness.

    // string timestamp time zone unspecified: assumed to be in database local time.
    // Returns with Local Kind.
    QueryEqual("09/11/2013 03:47:03 (09/11/2013 13:47:03) L",
               "SELECT '2013-09-11 06:47:03'::timestamp with time zone");

    // string timestamp with time zone: should come back with the correct universal
    // value, with Local Kind.
    QueryEqual("09/11/2013 05:47:03 (09/11/2013 15:47:03) L",
               "SELECT '2013-09-11 06:47:03+1'::timestamp with time zone");


    // string timestamp without time zone: comes back 'unspecified' with the exact
    // datetime specified.  ToUniversal() assumes unspecified = local, so -10.
    // Returns with Unspecified Kind.
    QueryEqual("09/10/2013 20:47:03 (09/11/2013 06:47:03) ?",
               "SELECT '2013-09-11 06:47:03'::timestamp without time zone");

    // string timestamp with time zone, coerced to not have a time zone: as if the
    // time zone wasn't in the string.  Returns with Unspecified Kind.
    QueryEqual("09/10/2013 20:47:03 (09/11/2013 06:47:03) ?",
               "SELECT '2013-09-11 06:47:03+1'::timestamp without time zone");


    // tests using .NET values as parameters
    // -------------------------------------

    // These represent what we'll usually do.  They're also really messed up.

    // unadorned parameter: regardless of the DateTimeKind, the date is treated as
    // without time zone; the exact date given comes back, but with Unspecified Kind,
    // and so is as good as forced to local time.
    DateTimesEqual("09/10/2013 20:47:03 (09/11/2013 06:47:03) ?",
                   "SELECT @DateTime",
                   new DateTime(2013, 9, 11, 6, 47, 3, DateTimeKind.Local),
                   new DateTime(2013, 9, 11, 6, 47, 3, DateTimeKind.Unspecified),
                   new DateTime(2013, 9, 11, 6, 47, 3, DateTimeKind.Utc));

    // parameter specified as with time zone: regardless of the DateTimeKind, the
    // date is treated as in the database local time.  It comes back with Local Kind.
    DateTimesEqual("09/11/2013 03:47:03 (09/11/2013 13:47:03) L",
                   "SELECT (@DateTime)::timestamp with time zone",
                   new DateTime(2013, 9, 11, 6, 47, 3, DateTimeKind.Local),
                   new DateTime(2013, 9, 11, 6, 47, 3, DateTimeKind.Unspecified),
                   new DateTime(2013, 9, 11, 6, 47, 3, DateTimeKind.Utc));

    // parameter specified as without time zone: as for unadorned parameter.
    DateTimesEqual("09/10/2013 20:47:03 (09/11/2013 06:47:03) ?",
                   "SELECT (@DateTime)::timestamp without time zone",
                   new DateTime(2013, 9, 11, 6, 47, 3, DateTimeKind.Local),
                   new DateTime(2013, 9, 11, 6, 47, 3, DateTimeKind.Unspecified),
                   new DateTime(2013, 9, 11, 6, 47, 3, DateTimeKind.Utc));


    // discussion
    // -----------

    // DateTime parameters' kinds are ignored completely, as shown above, and are
    // rendered into SQL as a 'timestamp' (== 'timestamp without time zone').  When a
    // comparison between 'timestamp with time zone' and timestamp without time zone'
    // occurs, the one without is treated as being in database local time.

    // Accordingly, you should set your database time zone to UTC, to prevent
    // arbitrary adjustment of incoming DateTime objects.

    // The next thing to ensure is that all your DateTime objects should be in
    // Universal time when going to the database; their Kind will be ignored by
    // npgsql.  If you send a local time, the local time will be treated as the
    // universal one.

    // Note that, per the second group just above, 'timestamp with time zone' comes
    // back as a DateTime with Local Kind.  If you throw that right back into npgsql,
    // as above, the Kind will be summarily ignored and the *local* rendering of that
    // time treated as UTC.  Ouch.


    // conclusions
    // -----------

    // 'timestamp with time zone' is read as DateTime with Local Kind.  Note that the
    // actual value is correct, but it's invariably transposed to local time (i.e.
    // +10) with Local Kind, regardless of the stored time zone.  Calling
    // .ToUniversal() yields the correct DateTime in UTC.

    // DateTime's Kind property is ignored.  To work around, set database or session
    // time zone to UTC and always call ToUniversal() on DateTime parameters.

    // Don't use 'timestamp without time zone' in your schema.
}

private void DateTimesEqual(string expectedUtc, string query,
                            params DateTime[] dateTimes)
{
    foreach (var dateTime in dateTimes) {
        var cursor = _connection.Query<DateTime>(query, new {DateTime = dateTime});
        DatesEqual(expectedUtc, cursor.Single());
    }
}

private void QueryEqual(string expectedUtc, string query)
{
    DatesEqual(expectedUtc, _connection.Query<DateTime>(query).Single());
}

private static void DatesEqual(string expectedUtc, DateTime date)
{
    var code = "_";
    switch (date.Kind) {
        case DateTimeKind.Local:
            code = "L";
            break;
        case DateTimeKind.Unspecified:
            code = "?";
            break;
        case DateTimeKind.Utc:
            code = "U";
            break;
    }

    var uni = date.ToUniversalTime();
    Assert.AreEqual(expectedUtc,
                    string.Format("{0} ({1}) {2}",
                                  uni.ToString(CultureInfo.InvariantCulture),
                                  date.ToString(CultureInfo.InvariantCulture),
                                  code));
}

How and Why We Switched from Erlang to Python tells an intern’s tale, from whose perspective it runs like this: we used Erlang, “No one on our team is an Erlang expert” (despite “how crucial this service is to our product”!), and also would you please suspend brain activity while I make some performance claims.

Hold your horses. The good decision to rewrite critical services in a language they actually know aside, let’s look at their notes on perf:

Another thing to think about is the JSON library to use. Erlang is historically bad at string processing, and it turns out that string processing is very frequently the limiting factor in networked systems because you have to serialize data every time you want to transfer it. There’s not a lot of documentation online about mochijson’s performance, but switching to Python I knew that simplejson is written in C, and performs roughly 10x better than the default json library.


Let’s distill these claims:

  • Erlang is historically bad at string processing
  • string processing is very frequently the limiting factor in networked systems
  • simplejson is written in C
  • simplejson performs 10x better than the default json library

Further down:

I went into Mixpanel thinking Erlang was a really cool and fast language but after spending a significant amount of time … I understand how important code clarity and maintainability is.

Thus by implication?

  • Erlang is not a really cool and fast language
  • Erlang is somehow not conducive to code clarity and maintainability

This whole paragraph is just a mess and I can’t quote it without losing some of its slimy essence.

Again, in full:

I’ve learned a lot about how to scale a real service in the couple of weeks I’ve been here. I went into Mixpanel thinking Erlang was a really cool and fast language, but after spending a significant amount of time sorting through a real implementation, I understand how important code clarity and maintainability is. Scalability is as much about being able to think through your code as it is about systems-level optimizations.

By your own admission, no-one on your team is an Erlang expert; you “have trouble debugging downtime and performance problems”. Plenty of Erlang users don’t, so it suggests the problem with your team’s command of the enviroment is severe. Similarly, you earlier mention the “right way” to do something in Erlang, and immediately comment that your code didn’t do that at all – never mind that the “right way” mentioned was wrong.

Yikes.

So why does the word “Erlang” feature in the above-quoted paragraph at all?

There’s no reason to expect either code clarity or maintainability of a service developed over 2 years without an engineer skilled in the environment overseeing the architectecture.

I didn’t say Erlang in that sentence, and yet it has greater explanatory power than the intern’s claim for the same phenomenon.

I suspect their explanation is more controversial, however, and it’s easier to make these claims than arrive at the conclusion that the team’s own shortcomings were responsible for the technical debt accrued – and it makes for a better article. I choose my explanation:

  • Erlang is somehow not conducive to code clarity and maintainability: there is not even anecdotal support in the article for this claim

That leaves 5 claims.

Let’s note an important confounding factor: the article is from August 2011. The state of Python and Erlang, and libraries for both have changed since.


As an aside: it’s easy to think that the performance claims they do indirectly make are incidental (and not essential) to the article.

But remove them, and note there’s not really an article any more; a prologue about mapping some concepts from an old codebase to new, and .. an epilogue espousing the virtues of code clarity and maintainability.

Ain’t nobody gonna argue with that, but, as noted above, just that alone does not a “How and Why We Switched from Erlang to Python” blog post make.


Let’s now dig into it – this won’t be much of an article without benchmarks either. Unlike their benchmarks, I’m actually comparing things in order to contrast; their decision to give benchmarks on the new system but not on the old is baffling at best.

I compared 4 Python JSON implementations and 3 Erlang ones:

  • json (built-in in Python 2.7.4)
  • simplejson 3.3.0 from PyPI, both with and without C extensions
  • ujson 1.30 from PyPI
  • mochijson and mochijson2 from mochiweb
  • jiffy

simplejson is what the intern picked for their rewrite. mochijson is what they were using before.

All versions are current at time of writing.

Testing method:

  • read 5.9MB of JSON from disk into memory
  • start benchmark timer
  • parse JSON from memory 10 times, each time doing some minimal verification that the parse was successful
  • force a garbage collect
  • stop our benchmark timer

The code is available on my GitHub account, and instructions to reproduce are found there.

Here are the results:

  • ujson: 1,160ms
  • jiffy: 1,271ms
  • simplejson (with C): 1,561ms
  • json: 2,378ms
  • mochijson2: 8,692ms
  • mochijson: 11,111ms
  • simplejson (no C): 16,805ms

ujson wins! jiffy a close second! simplejson a close third! These results are the average of three runs each, but I did many more runs in testing the benchmark code and can say the variance was quite low.

So:

  • simplejson performs 10x better than the default json library: this doesn't appear to be the case now. It may have been the case in 2011, depending on what the default json library was back then.
  • Erlang is not a really cool and fast language: in this particular example the best Erlang library is on par with both of the best Python libraries – all three C-boosted, of course – and the best pure Erlang library runs in half the time as the apparently-best pure Python one. (json is C-boosted)

That leaves us with these claims unrefuted:

  • Erlang is historically bad at string processing
  • string processing is very frequently the limiting factor in networked systems
  • simplejson is written in C

Erlang’s historical performance is somewhat irrelevant, but the claim stands nevertheless.

No evidence was advanced for the second claim: there was no way to determine whether faster string processing was responsible for any improvement in their benchmarks: we don’t even know if the benchmarks improved because we only were given one set (!). Of course, the changes being the entire system and not just string processing, before-and-afters would prove nothing, especially given the proficiency gap. Hence:

  • string processing is very frequently the limiting factor in networked systems: maybe, maybe not, but picking the right library makes a big difference!

I mean, jeez; they could reduce their string processing (and thus the “limiting factor”?) by 33% if they switched from simplejson to ujson!

As for the third claim, if I don’t nitpick, it stands. Kinda.


Why did I feel the need to write this up?

I saw the article pop up on Hacker News today, 2 years after its publication. In fact, I’d seen the article not long after it was originally published, and I know it was on HN back then too. I don’t care about the fact that it was reposted; more that it was written at all.

It’s exactly the sort of useless bullshit that seems to fill a quarter of HN’s front page at any given stage: articles with titles like “Why I Don’t Use Vim”; “Why You Should Learn Vim”; “The Reason You Need To Understand System F If You Want To Ever Amount To Anything”; “Stop Thinking. Just Write.”; “How We Saved 95% Of Our Datacentre Costs By Rewriting Everything In Ada”; etc. etc. etc.

It’s edgy shit that grabs your attention and way oversells one point of view, at the expense of reason. This comment thread started out by acknowledging this trend, but was ruined by someone not catching the pattern being invoked. Usually there’s a nice novel point to be made somewhere if you can read the moderate subtext between the bold claims.

Unfortunately this article had no such point, and so turned out to be 100% garbage. But still, some people will read that article and go away thinking their prejudices about Erlang have been confirmed by someone who’s been in battle and seen it for themselves.

And really, this isn’t about Erlang. I don’t care what language or environment you use. Your coding philosophies are irrelevant, if you deliver – and Mixpanel are delivering, and clearly made the right choice to get rid of some technical debt there.

But don’t then try to shift any part of the responsibility of the decision to pay off that debt as being your tools’ faults, and especially not with such flawed logic.

k6_bytea

I’ve started a project in Erlang recently, and I needed a big block of mutable memory – a massive byte array, buffer, whatever you want to call it. Erlang’s binary strings are immutable and so probably not suitable.

I figured the core distribution must’ve had something like this … nope. Spending 30 minutes Googling and checking docs twice and thrice over, but there’s clearly no mutable byte array in Erlang itself.

Is there a hack that approximates to this? Search … this StackOverflow almost seems hopeful at the end, referencing hipe_bifs:bytearray/2 and hipe_bifs:bytearray_update/3, but alas, they are so-named because they are part of the HiPE native compiler, and not Erlang itself. (I’m not currently using HiPE, and it would be nice to not be tied to it.)

Then, I thought, surely someone else has extended Erlang to do this. There are several modes of extending Erlang, but native implemented functions (NIFs) would be the obvious candidate for manipulating large chunks of memory.

Googling this yielded complete failure: people just don’t seem to be doing it. (If people are doing it and you know this, please, let me know.)

So I did it: k6_bytea.

The interface is simple and fairly pleasant:

1> Bytea = k6_bytea:from_binary(<<"Hello, world.">>).
<<>>
2> k6_bytea:set(Bytea, 7, <<"Thomas">>).
ok
3> k6_bytea:to_binary(Bytea).
<<"Hello, Thomas">>
4> Gigabyte = k6_bytea:new(1000000000).
<<>>
5> k6_bytea:delete(Gigabyte).
ok
6>

The gigabyte allocation caused a small notch on my memory performance graph:

Screenshot of a GNOME desktop menubar, with a memory performance widget showing a very abrupt increase, then decrease, in memory allocated.

perf

The obvious question remains: how does it perform, vis-à-vis binary strings?

Let’s make a contrived benchmark: initialise a large buffer, write all over it in a deterministic fashion, and copy data all over it. Let’s do it in parallel, too.

Here’s the benchmarking code for your perusal:

-module(k6_bytea_bench).
 
-export([run/0, binary_strings/1, k6_bytea/1]).
 
-define(COUNT, 1024).   % Parallel operations.
-define(SIZE, 102400).  % Size of buffer to work on.
 
run() ->
    measure(<<"Binary strings">>, ?MODULE, binary_strings),
    measure(<<"k6_bytea">>, ?MODULE, k6_bytea).
 
measure(Name, M, F) ->
    {SM, SS, SI} = erlang:now(),
    spawn_many(?COUNT, M, F, [self()]),
    receive_many(?COUNT, done),
    {EM, ES, EI} = erlang:now(),
    Elapsed = ((EM - SM) * 1000000 + (ES - SS)) * 1000 + ((EI - SI) div 1000),
    io:format("~s [~p ms]~n", [Name, Elapsed]),
    ok.
 
spawn_many(0, _, _, _) -> ok;
spawn_many(N, M, F, A) ->
    spawn(M, F, A),
    spawn_many(N - 1, M, F, A).
 
receive_many(0, _) -> ok;
receive_many(N, M) -> receive M -> receive_many(N - 1, M) end.
 
binary_strings(Done) ->
    B0 = <<0:(?SIZE*8)>>,
    B1 = binary_strings_set_bits(B0, lists:seq(0, ?SIZE - 1024, ?SIZE div 1024)),
    _ = binary_strings_copy_bits(B1, lists:seq(0, ?SIZE - 1024, ?SIZE div 1024)),
    Done ! done.
 
binary_strings_set_bits(B, []) -> B;
binary_strings_set_bits(B, [H|T]) ->
    <<LHS:H/binary, _:1/binary, RHS/binary>> = B,
    binary_strings_set_bits(<<LHS/binary, (H rem 255), RHS/binary>>, T).
 
binary_strings_copy_bits(B, []) -> B;
binary_strings_copy_bits(B, [H|T]) ->
    <<LHS:H/binary, Bit:1/binary, _:1/binary, RHS/binary>> = B,
    binary_strings_copy_bits(<<LHS/binary, Bit/binary, Bit/binary, RHS/binary>>, T).
 
k6_bytea(Done) ->
    B = k6_bytea:new(?SIZE),
    k6_bytea_set_bits(B, lists:seq(0, ?SIZE - 1024, ?SIZE div 1024)),
    k6_bytea_copy_bits(B, lists:seq(0, ?SIZE - 1024, ?SIZE div 1024)),
    k6_bytea:delete(B),
    Done ! done.
 
k6_bytea_set_bits(B, []) -> B;
k6_bytea_set_bits(B, [H|T]) ->
    k6_bytea:set(B, H, <<(H rem 255)>>),
    k6_bytea_set_bits(B, T).
 
k6_bytea_copy_bits(B, []) -> B;
k6_bytea_copy_bits(B, [H|T]) ->
    Bit = k6_bytea:get(B, H, 1),
    k6_bytea:set(B, H + 1, Bit),
    k6_bytea_copy_bits(B, T).

Over 3 runs, binary_strings averaged 24,015ms, and k6_bytea 198ms (0.83% time, or 121x speed).

There’s nothing very surprising about this; it’s large, unwieldy immutable data-structures vs. one mutable data-structure. It’s even the case that I have no idea if there are any better ways to simulate a byte array in Erlang, either with binary strings, or without!

The binary string-manipulating code above is ugly and error-prone, as it’s clearly not the purpose it was built for. If it should turn out that this really hasn’t been done better by someone else, then I encourage you to look to and improve k6_bytea for this purpose.

Lately I’ve been game programming. A few things have been coming to mind:

  • The Sapir–Whorf hypothesis is totally true when it comes to programming languages.
  • Dynamically typed languages encourage a whole lot of waste.
  • There’s some sweet spot of expressivity, low level and non-shoot-yourself-in-the-footness in language that’s missing.

I will attempt to expound.

The languages I end up using on a day-to-day basis tend to be higher level. Non-scientific demonstration by way of my GitHub account’s contents at time of writing:

15 Ruby
9 OCaml
6 Go
6 Javascript
4 C
3 C++
2 Clojure
3 Perl
2 Haskell
...

In my professional career, I’ve concentrated on JavaScript, PHP, Erlang, Python and C#. The lowest level of these is, by far, Erlang. Perhaps it’s fairer to say that Erlang keeps me in check, perf-wise, more than any of the others.

So my mind has been a fairly high-level sort of mindset. I’ve made occasional trips to lower-level code, but there hasn’t been much of that lately, particularly as I’ve changed jobs and needed to concentrate solely on work stuff for a while.

Choosing C++ to write a game wasn’t too hard; it’s fairly standard in the industry, and I know it quite well. Bindings to libraries are plentiful, and getting the same codebase compiling on Windows, OS X and Linux is a well-understood problem that’s known to be solvable.

The thing is, C++ makes it abundantly clear when you’re doing something costly. This is something that occurs particularly to me now as I’ve not done this lower-level stuff in a while.

You wrote the copy constructor yourself, so you know exactly how expensive pushing a bunch of objects into a vector is. You chose a vector, and not a list, so you know exactly why you don’t want to call push_front so many times. You’re creating a ostringstream to turn all this junk into a string: it has a cost. Are we putting this on the stack or in the heap? Shall we use reference counting?

You make hundreds of tiny decisions all the time you’re using it; ones which are usually being abstracted away from you in higher level languages. It’s why they’re higher level.

And that’s basically all I have to say on that: the language makes you feel the cost of what you choose to do. Going to use a pointer? Better be sure the object doesn’t get moved around. Maybe I’ll just store an ID to that thing and store lookups in a map. How costly is the hashing function on the map key? You add such a lookup table in Ruby without a moment’s notice; here, you’re forced to consider your decision a moment. Every time you access the data, you’re reminded as well; it’s not something that ever goes out of mind.

Moreover, the ahead-of-time compilation means you can’t do costly runtime checks or casts unless you really want to (e.g. dynamic_cast), but again, the cost of doing so means you’ll never be caught unaware by slowing performance. In many (dynamic) higher level languages, basically every operation is laced with these.

So it’s suited for games programming, because performance is usually pretty important, and the language keeping performance on your mind means it’s not hard to consistently achieve high performance.

But C++’s deficiencies are also well-documented. It’s awful. It’s waiting to trip you up at every turn. After re-reading those talk slides, I figured I’d just port the code to C – until I remembered how much I used std::string, std::vector, std::list, and moreover, enjoyed the type- and memory-safety they all bring. I’m not particularly fond of giving that up and implementing a bunch of containers myself, or using generic containers and throwing away my type checks.

I think I’m after C with templates for structs (and associated functions), but I’m not sure yet. If you think I want C++, you probably need to re-read those notes.

The other solution is to only use as much of C++ as I like, and that’s basically what I do – but the language is still waiting to trip me up, no matter how much I try not to use it.

Time to think a bit about what the issues at hand really are.

I use Snapchat. It’s an app where you can take a photo or short (< 10 second) video and send it to your friends who use the service; they’ll then be able to see it, once, before it disappears forever.

Ostensibly, the app is for sexting, because there’s no fear that your photo will get spread around (no forwarding/etc.) or retained for longer than you’d like, but it seems like it’s not as much a sexter’s hangout as the media might want you to think.

My circle of friends use it basically as an extension of weird Twitter – most snaps I send and receive are strange angles of weird objects; the completely mundane but somehow therapeutic (7 seconds of the camera pointed outside the window of a tram, pointed at the ground moving below); or just closeups of Curtis Stone’s face, wherever we see him.

Of course, the promise that they won’t get retained is just that: a promise. Since your phone receives this image and shows it to you at some point, it must be downloaded by your phone. If it can be downladed by the phone, it can be downloaded by something else. We decided to find out how.

Read more

Here’s Contrigraph, a “data visualisation” (?) created by generating commits to match the Contribution Graph Shirt from GitHub.

It was pretty much a hack; first, I read the colours straight off the shirt, producing a big block of data like

0002342
2223322
2323241
2224333
3322122
2242231
...

Then we read that in one digit at a time, work out what day to start on so everything aligns correctly, and how many commits on a given day produce each gradient of colour. The result isn’t pretty:

start = Time.new 2012, 4, 23, 12, 0, 0, 0

tbl = {"0" => 0, "1" => 0, "2" => 1, "3" => 9, "4" => 14, "5" => 23}

3.times do 
  dbd.each do |n|
    tbl[n].times do
      `echo 1 >> graafik`
      `git commit -a -m 'contrigraph' --date='#{start.to_s[0..9]}T12:00:00'`
    end
    start += 86400
  end
end

Three times so this thing will scroll on for the next few years. Other values for tbl would work too; I just didn’t bother to do anything better. I’ve written clearer code, but that wasn’t really the point either.

I actually screwed this up twice: first I didn’t remember to treat the 0 entries correctly (i.e. I should have skipped those days, whereas I ignored them entirely); second, it seemed like I was getting hit by timezone issues where everything was shifted up a block.

In retrospect, I should have first produced a mini-contributions graph generator (i.e. one that takes a Git repository and produces what GitHub would do), validate that against an existing user/repo, then use that to ensure it’d work the first time. I did a similar thing to ensure I had the data correct, by producing a graph directly from the data: