go-excel-reader

I ported the core of a C# spreadsheet library to Go in a day. Every test passed. The memory benchmark reported 3.6 MB against a 50 MB budget.

The component those tests existed to exercise had never once executed.

What this is

ExcelDataReader is the C# library most .NET codebases reach for when they have to read a spreadsheet. It handles .xlsx, the legacy .xls binary format, and CSV, through a streaming cursor rather than by loading a workbook into memory.

go-excel-reader is a port of its core to Go. Pure Go, no CGo, one dependency. It reads all three formats through the same cursor model.

The finished thing works, and it is verified in a way I think is worth more than the code. It and the original C# library both dump a file to a canonical form, and the dumps are compared byte for byte. Not on files I wrote, but on ExcelDataReader's own 303-file test corpus, collected over a decade because each file broke something.

232 match exactly. 27 it refuses on purpose. 11 disagree and are written down. None are read here that the original refuses.

That last sentence is the one worth having. The rest of this is how those numbers moved.

I am not a spreadsheet-format expert. I had not written a line of BIFF or CFB parsing before this.

The plan was good. The tests were the problem.

My previous project was about a plan that was confidently wrong about the codebase it described. This time the plan was detailed and largely right. It named the record types, the magic bytes, the four RK encodings, and a gotchas table that was accurate.

So the failure moved somewhere else.

The shared string table is the reason an xlsx reader is hard to write efficiently. Almost every string in a spreadsheet lives in one shared table and each cell stores an index into it. It is the largest allocation a reader makes, and the whole memory design of this port is built around it.

I wrote it, benchmarked it, and got 3.6 MB on a 4.6 MB file. Comfortably inside budget.

The fixtures had no shared string table in them. openpyxl, which generated them, writes strings inline, as <c t="inlineStr">, rather than into xl/sharedStrings.xml. There was no such file in any fixture. The code had never run.

The tell was that the number was too good. A 200,000-entry table should cost several megabytes, and 3.6 MB was not consistent with having loaded one. Checking took one command.

With a real 400,000-entry table the honest figure is 26.9 MB. Still inside budget, but now it means something.

Then I did it again

The README said peak memory "tracks the widest row, not the file size."

That sentence describes what I designed, not what I measured. Rows do stream. But the string table is held fully in memory, and on a string-heavy workbook it is the memory profile, about 30 of the 39 MB. I shipped that claim in v0.1.0.

The fix was to stop asserting and measure. Two workbooks, identical row and cell counts, differing only in how many distinct strings they hold:

Workbook	Cells	Distinct strings	Peak heap
repeat.xlsx	1,000,000	100	3.5 MB
large.xlsx	1,000,000	400,000	26.9 MB

That is a better result than the claim I couldn't defend, and unlike the claim it can be falsified. A million cells in 3.5 MB shows rows are not accumulating. The gap between the two numbers is the string table and nothing else. Row count appears in neither.

Both are now tests. The baseline one matters more: if row streaming ever regressed into accumulation, the string-heavy fixture would still look plausible while the repeating one blew past its budget.

Then I stopped grading my own work

At this point everything passed and I had checked my twelve fixtures against the original C# library, byte for byte. That felt like strong evidence. It was evidence about twelve files I chose.

ExcelDataReader ships 379 test files of its own, real spreadsheets its maintainers collected over a decade because each one broke something. Names like DateFormatButNotDate.xls, ClipboardBiff8.xls, Issue467_EmptyContinueLeftoverbytes.xls.

So I pointed the same harness at all 303 of the .xls and .xlsx files:

Result	Before fixes	After
Byte-identical to ExcelDataReader	207	232
Both readers refuse (xlsb, encrypted, corrupt)	33	33
Both read it, output differs	37	11
Only ExcelDataReader reads it	26	27
Only mine reads it	0	0

That last row is the one I care about most. Nothing here parses a file the reference implementation refuses, which would mean inventing data.

It found three defects, each a different kind.

A silent one. Issue411.xls returned zero rows, no error, exit code 0. A caller gets an empty spreadsheet and no indication anything went wrong.

The cause is the best thing in this project. A compound file is a tree of storages, and a .xls containing an embedded object has nested storages with streams of their own. I was finding streams by scanning all directory entries linearly, so I resolved a Workbook belonging to the embedded object rather than the real one.

I had deleted the sibling and child pointers as dead fields, and written a comment explaining why that was fine:

"walking the tree would be asymptotically better on a directory far larger than any spreadsheet has"

Confidently wrong. Those pointers have nothing to do with performance. They are what gives an entry its scope. I removed them, rationalised it in a comment, and the rationalisation read as informed. Fixed, that file now says plainly that it is BIFF5 and unsupported. Silent data loss became an honest error.

A subtle one. ExcelDataReader distinguishes an empty inline string from an empty entry in the shared string table. The first is an absent value, the second is a value that happens to be empty. I treated both as text.

My first fix applied the rule everywhere. It repaired four files and broke eight. The corpus caught that immediately, which is the entire point: a one-line change that looks obviously right, measured against 303 files, is a question with an answer instead of an opinion.

A deliberate one. Error cells. EDR reports #DIV/0! as having no value; I returned the text. Mine is arguably more useful, but "arguably more useful" is not "identical," and my README said identical. It is now an option, off by default, and there is a table of known intentional differences.

The failure mode, again

Three times now, something reported success while measuring less than I thought. Same shape as the four that did it on the last project:

A passing test proves the assertion held. It does not prove the code ran.

That project's version was a check that could not distinguish "the thing is fine" from "I failed to look." This one could not distinguish "the code works" from "the code was never called." A test whose fixture doesn't reach the code under test passes for free, and it passes faster, and the speed reads as efficiency.

The third instance was the twelve fixtures themselves. They agreed with the reference implementation perfectly, and they were twelve files I chose, which is a sample that cannot surprise me. The corpus could, and did, eight times.

What helps is the same thing that helped last time: compare against something that can actually differ. So the fixtures are now written by three independent tools, two of which store strings differently on purpose, and a test asserts that property directly:

// If a regenerated fixture quietly stopped using the shared string table, the
// string-table reader would go untested while every test still passed.
for _, f := range []string{"sst.xlsx", "large.xlsx"} { ... }
if hasPart("simple.xlsx", "xl/sharedStrings.xml") { t.Error(...) }

The same reasoning produced a negative control in the BIFF string test. It parses the same bytes without the continuation boundaries and asserts the result comes out wrong. Without it, an assertion that "abcd" == "abcd" can pass for reasons having nothing to do with the code being right.

And then inputs nobody wrote

The corpus is 303 files that exist because a human hit a bug. Fuzzing is the same idea with the human removed, and it found a different kind of bug within twenty-five seconds:

runtime: out of memory: cannot allocate 17179869184-byte block
	xls.(*cfb).readDIFAT  cfb.go:166

A compound file header declares how many sectors its allocation table occupies. I used that number as an allocation size. A two-kilobyte file claiming 0xFFFFFFFF sectors asks for seventeen gigabytes and kills the process, and a Go fatal out-of-memory is not recoverable, so no caller can defend against it.

The two techniques found completely different things and neither would have found the other's. The corpus found semantic errors: a locale date format id I had never heard of, a serial out of range, records in an order I did not know was legal. Every one needed a real file produced by real software. The fuzzer found a safety error: a number used without asking whether it could be true. No human would ever write a file claiming four billion FAT sectors, so it will never appear in any corpus, and it is the one that gets your service killed.

2.1 million executions after the fix, no further failures.

The instrument was not in the box

Two days later I went back for the four disagreements I had left untriaged, and never got to them, because re-running the comparison from a clean checkout reported two files as newly broken that the committed baseline recorded as matching.

They were not broken. The C# side of the harness had been fixed during the triage. It was rendering two of ExcelDataReader's return types through a fallback that guaranteed a mismatch, and that fix was never committed. The repository held the version from before. So the committed harness could not reproduce the committed baseline, and had not been able to from the day both were written.

Nothing caught it, and the reason is worth stating plainly. The test that runs without .NET checks hashes of my own output, deliberately, so that the parity result survives without the other language's toolchain. That works. It also means the harness can rot completely while every test stays green, because nothing ever re-derives the verdicts. The number in the README was still true. The thing that proved it was gone.

This is the third variant of the same failure in one project. First, twelve fixtures I chose myself, which agreed with the reference and proved nothing. Then a memory benchmark reporting 3.6 MB because the code under test never ran. Now a verification harness that had stopped being able to verify. Each one passed. Each one measured nothing.

With the harness committed, HEAD reproduced the baseline file for file, and the four files turned out to be four bugs here in three classes: two CONTINUE records in the shared string table that my boundary handling could not express, a chart substream embedded inside a worksheet whose cached values I was reading as cells, and the one that stung: the same out-of-order-rows bug I had fixed in the .xls reader the day before, sitting unfixed in the .xlsx reader. I had written a page about why placing cells by absolute index mattered and had not gone to look at whether the other reader made the same assumption. It did. The two readers share no code, only an assumption about how a file numbers its rows, and the assumption was wrong in both.

233 of 303 then, up from 229, with nothing regressed and none of the remaining ten undiagnosed. It is 232 now, because fixing one of those ten cost a file of agreement on purpose: a producer writing an Excel serial out as ISO text carries the serial's float error into the string, and where the reference keeps 15:59:59.99999979 this rounds to the millisecond and says 16:00:00. That is the value Excel shows. Being right and being identical are not the same target, and the difference is now a row in a table rather than a surprise.

The fourth variant, and the one that nearly went to a stranger

Two days after that I sat down to write the findings up for the ExcelDataReader maintainers. Two of the remaining differences looked like bugs on their side, and sending them upstream seemed like the least I could do for a corpus that had found so many of mine.

Writing an issue means writing a reproduction, and a reproduction a maintainer can run cannot go through my harness. So I wrote twenty lines of C# that call their library directly and print the CLR type of every value it returns.

Both findings died.

The first was the timezone bug. Chapter 8 has a memorable little table showing one cell of strict/Open.xlsx decoding to 16:00, 11:00 and 02:00 under New York, UTC and Tokyo, and concludes that their output depends on the host. The clean program returns 11:00 under all three. The spread was mine. Their 3.9.0 release returns a DateTimeOffset carrying the host's offset for a file that has none, and my dumper renders a DateTimeOffset through .UtcDateTime. A fixed wall clock with a varying offset, converted to UTC, moves.

The second was whitespace, and it was backwards. I had written that they trim strings marked xml:space="preserve", and that I thought this was wrong of them. They preserve exactly those and trim the ones without the marker, which is what the XML specification permits and what Excel expects. My reader preserves unconditionally. On that file I am the one out of step, and it is now listed as a defect here.

So, the fourth variant. The first three were checks that measured nothing. This one measured something real and I described it as something else, which is worse, because there is a number and the number is correct. The dump is a lossy projection built to make two languages comparable. It collapses DateTimeOffset into UTC. It collapses int64 and float64 into the same bits. Both are deliberate, both are documented, and I read a conclusion about somebody else's library off it anyway.

A claim about my code is proved by the harness. A claim about their code has to be proved against their code. I own one of those and had been using it for both.

Giving it back

One finding survived, and it is the better one.

LocaleTime.xlsx and strict/LocaleTime.xlsx are the same workbook saved in the two OOXML flavours, and both sit in their repository. The first decodes to 1899-12-31 01:34:00, which is the value their own test already asserts. The second decodes to today's date with the same clock time, because a strict cell holding a time and no date is resolved against the current date. Read that file tomorrow and it gives a different answer.

That is filed as ExcelDataReader#759, with the reproduction, the line it comes from, and two candidate fixes. I offered to write whichever they prefer.

It is a small thing to hand back for a decade of test files that found bug after bug in my port in a single afternoon. It is also the only part of this project that improves anything for anybody who is not me.

What the port actually cost

The honest scorecard, because a post that only lists wins isn't worth reading:

Wrote it and it worked first run	numfmt, the xlsx reader, the CSV reader, the BIFF8 reader
Cost real time	Fixtures that tested nothing; a memory claim I couldn't defend
Found only by someone else's tests	A silent zero-row read; an empty-string rule; a parity claim that was narrower than it sounded
Deliberately not ported	xlsb, encryption, BIFF5 and earlier, AsDataSet()
Deliberately not optimised	unsafe.String on the string blob, since the budget is met without it

Four subsystems landed green on the first test run. That is not a boast; it is the setup for the point. The two things that went wrong were both in the layer that was supposed to tell me whether the code was right. Being good at the hard part bought me nothing, because the part that failed was the verification.

The two details that were genuinely hard

A date is not a type. A spreadsheet stores a date as a bare float64. Nothing in the cell says "date." The only signal is a number format reached through the style table, three hops away. Break any hop and a delivery date becomes the number 45217. Worse, scanning the format string for d/m/y is not enough: a format like "Paid on day "0.0 contains all of them inside a quoted literal. That one is a fixture now.

BIFF8's restated flags byte. A record payload can't exceed 8224 bytes, so a large string table arrives as one record plus a run of CONTINUE records. Concatenate them and almost everything works. But when a string is split across a boundary, the first byte of the continuation is not text. It restates the compression flag. Miss it and the table doesn't fail; it silently desynchronises, and every string after that point comes back garbled. The fixture generator writes random-length strings specifically to force the case.

Where it beats the original, and where the comparison is unfair

On the unfavourable input, 400,000 distinct strings, the worst case for this design:

Reader	Time	Peak RSS
go-excel-reader	3.2 s	40 MB
ExcelDataReader 3.9.0 (.NET 9)	3.3 s	95 MB
excelize v2.11.0	4.0 s	221 MB

The excelize row is not like for like on time: its cursor returns strings and does less type resolution than the other two. Memory is the meaningful column, and the README says so rather than letting the time column imply something it shouldn't.

The mechanism is one idea. The string table is stored as a single byte blob with an offset index instead of a []string. For 400,000 entries that is two allocations and zero pointers instead of 400,000 of each, and the garbage collector never traces into a []byte.

Deliberately not claimed

• 232 of 303 is not 303. 11 files are read by both and disagree. All eleven are diagnosed (elapsed-time cells, strict ISO time-only cells, whitespace, millisecond rounding, and one extra empty cell in Issue158.xls). Two of them are open defects here rather than choices: Issue158.xls, which has never been diagnosed, and the whitespace one, where this library preserves what the XML permits a reader to trim. The count is pinned by a test, but "96% agreement" is the honest headline, not "matches the original."
• The comparison pins the reference to TZ=UTC. ExcelDataReader resolves an OOXML strict time carrying no date against the current date, so those files decode differently depending on which day it is where you are. An earlier version of this page said its output depended on the timezone generally. That was wrong, and the correction is in the second afterword to chapter 9: the evidence for it came from our own dump rather than from the library.
• The dump is a lossy projection, on purpose. It renders int64 and float64 to the same IEEE bits, and collapses DateTimeOffset to UTC. That is what keeps float formatting out of the diff, and it means an IDENTICAL verdict says nothing about which Go numeric type a cell produced.
• The 27 files ExcelDataReader reads and this does not are mostly BIFF5-and-earlier and encrypted workbooks, both refused on purpose. A handful are not, and are open.
• The "streams a 500 MB workbook" claim is a design property, not a measurement. The largest file tested is 9.2 MB.
• golangci-lint was not run; it wasn't installed. go vet and staticcheck are clean.
• One machine, one architecture, one Go version.
• ExcelDataReader#759 is filed, not fixed. Nothing upstream has changed, and nothing here claims the C# library was improved. One issue with a reproduction and an offer to write the patch is the whole of the contribution so far.

Read the actual work

The chapters below are the engineering log in order, including both mistakes and the corrections. The decision log records what was ambiguous and what got traded away.