In Part 1, I built an LLM inference engine in pure SQL. It works. It's also painfully slow — around 6 minutes per token on a stock SQLite build.
But "slow" is a starting point, not a destination. If the computation is fundamentally relational, then the question becomes: how good can we make the relational engine itself?
So we start by patching the SQLite source.
⇒ The Bottleneck Every token generation runs through 30 transformer layers. Each layer does roughly 6 large matrix multiplications (Q, K, V projections, output projection, and two FFN projections). Each matmul is a SUM(a.val * b.val) GROUP BY dim over tens of millions of row pairs. That's several billion floating-point multiply-accumulate operations per token, executed one row at a time by a query engine built for transactional workloads, reading from a 21 GB file on disk. The constraints are real. But there is room in three places: how the binary is compiled, how the aggregation loop runs, and how the database pages are cached and read from disk.
⇒ Compile Smarter The stock build uses -O2. That's reasonable for a general-purpose tool. For a compute-bound inner loop doing floating-point accumulation, it leaves a lot on the table. The patched build uses: -O3 -march=native -funroll-loops -O3 enables auto-vectorization. -march=native tells the compiler it can use every instruction set available on this specific machine — AVX2, FMA, whatever is there. -funroll-loops unrolls inner loops that the B-tree cursor traversal and VDBE interpreter spend time in. The inference workload is single-threaded by nature — one sqlite3 process, one query at a time. So SQLITE_THREADSAFE=0 drops the mutex lock/unlock pairs that bracket every VDBE API call. They're not free. Smaller binary too: stripping out SQLITE_OMIT_LOAD_EXTENSION, SQLITE_OMIT_DEPRECATED, and SQLITE_OMIT_PROGRESS_CALLBACK reduces binary size, which means more of the instruction cache is available for the code that actually runs.
⇒ The Hot Loop in SUM()
This one surprised me. SQLite's SUM() aggregate uses the Kahan-Babushka-Neumaier algorithm to compensate for floating-point rounding error. It's a good algorithm. It also adds three floating-point operations, two volatile temporaries, and an fabs() branch to every single row of every aggregate.
For a matmul over a 576-dimensional weight matrix, SUM() is called roughly 330,000 times per output element. Per projection. Per layer. Per token.
KBN is the right choice for financial data or integer accumulation at risk of overflow. For transformer weights — finite REAL values where the argmax output only needs to be approximately correct — it is unnecessary overhead.
The patch adds a fast path in sumStep() in func.c:
/* Fast path for REAL columns: skip Kahan-Babushka-Neumaier compensation. */
p->rSum += sqlite3_value_double(argv[0]);
When the accumulator is already in floating-point mode (which it enters on the very first REAL it sees), subsequent REAL values bypass the compensation entirely. Integer overflow protection stays intact for the integer path.
⇒ Memory: Cache and mmap The weights table is 21 GB. The default SQLite page cache is 2 MB. That mismatch means most B-tree page reads are cache misses. The B-tree interior nodes for the weight indexes get evicted between queries and have to be re-read from disk on every layer. Raising the compiled-in default from 2 MB to 256 MB keeps those interior nodes hot. SQLite also supports memory-mapped I/O, where file reads become virtual memory accesses instead of read() syscalls — letting the OS page cache work rather than layering another copy on top. The default compiled-in mmap size is 0 (disabled). The SQL schema already sets PRAGMA mmap_size = 30000000000, but that doesn't help for the reads that happen before the PRAGMA fires. Setting SQLITE_DEFAULT_MMAP_SIZE to 20 GB compiled in: the database is memory-mapped from the first file open. The 00_schema.sql PRAGMA then raises it further for subsequent reads. Two final tuning details: WAL autocheckpoint raised from 1000 to 100000 frames (inference never writes, so checkpointing is pure overhead), and SQLITE_DEFAULT_SYNCHRONOUS left alone (the schema already handles this).
⇒ Results
| Build | Wall time | CPU time | CPU% |
| Stock (-O2) | 6m 05s | 309s | 88% |
| Patched (-O3 + all changes) | 4m 07s | 241s | 98% |
Next up: -ffast-math.