Chapter 3: The fixtures that tested nothing

The memory test reported this:

file 4.6 MB, 100000 rows, peak heap 3.6 MB

Against a 50 MB budget. The shared string table design from chapter 2, with its contiguous blob and offset index and no pointers, appeared to be working extremely well.

The tell

3.6 MB was too good.

That fixture was supposed to contain 200,000 distinct strings. Even in the most compact representation imaginable, 200,000 strings of ~20 characters is several megabytes of text that has to exist somewhere. 3.6 MB was not consistent with having loaded a string table at all.

A good number and a number that means nothing look identical in test output. The only difference is whether it's consistent with the thing you think you measured.

One command:

$ unzip -l testdata/large.xlsx
  docProps/app.xml
  xl/theme/theme1.xml
  xl/worksheets/sheet1.xml     4769242 -> 26526406
  xl/styles.xml
  xl/workbook.xml
  [Content_Types].xml

No xl/sharedStrings.xml. Not in large.xlsx, not in simple.xlsx, not in any fixture.

Why

openpyxl, which generated every fixture, writes strings inline:

<c r="A1" t="inlineStr"><is><t>Name</t></is></c>

That is entirely legal. It is also a completely different code path from <c t="s"><v>0</v></c>, and it was the only one my tests had ever taken.

shared_strings.go is the file the memory design exists for, the one this port spends the most care on, and it had never executed. Every test passed. Coverage would have shown it, if I had looked at coverage instead of at a green tick and a good number.

The part that stings

I had already made the right decision for the right reason. The fixtures were generated by a third-party tool specifically so the tests would exercise the format rather than agree with a writer of my own.

That reasoning was sound and still insufficient. "Third-party" was the wrong requirement. The actual requirement is two writers that differ on the property you care about.

So the fixture set now uses three:

Tool	Writes	Covers
openpyxl	inline strings	<is><t> path
xlsxwriter	shared string table	<c t="s"> path
xlwt	BIFF8 .xls	the whole binary reader

Making it unable to happen again

Regenerating fixtures with different library versions could silently reintroduce this. So the property the other tests depend on is now itself a test:

// If a regenerated fixture quietly stopped using the shared string table, the
// string-table reader would go untested while every test still passed.
for _, f := range []string{"sst.xlsx", "large.xlsx"} {
        if !hasPart(f, "xl/sharedStrings.xml") {
                t.Errorf("%s has no xl/sharedStrings.xml; the shared string reader is untested", f)
        }
}
if present, known := hasPart("simple.xlsx", "xl/sharedStrings.xml"); known && present {
        t.Error("simple.xlsx now uses shared strings; the inline string reader is untested")
}

It asserts in both directions. One file must use the shared table; another must not. A change to either generator fails the build with a message saying which reader just became untested.

The honest number

With a real 400,000-entry table, where sharedStrings.xml is 15.4 MB uncompressed:

file 9.2 MB, 200000 rows, peak heap 26.9 MB

Still inside the 50 MB budget. Now it means something.

The generalisation

A passing test proves the assertion held. It does not prove the code ran.

A test whose fixture never reaches the code under test passes for free. It also passes faster, and in a memory test the absence shows up as a smaller number, which reads as success. The failure mode is disguised as the thing you wanted.

Suspiciously good results deserve the same scrutiny as bad ones. Nobody investigates a number that flatters them, which is exactly why it is worth building the habit.