Emma's Blog

Decompression is up to 30% faster in CPython 3.15

Emma Smith — Tue, 11 Nov 2025 00:00:00 -0800

tl;dr
compression.zstd is the fastest Python Zstandard bindings with Python 3.15. Changes to code managing output buffers has led to a 25-30% performance uplift for Zstandard decompression and a 10-15% performance uplift for zlib for data at least 1 MiB in size. This has broad implications for e.g. faster wheel installations with pip and many other use cases.

Motivation

Since landing Zstandard support in CPython, I wanted to explore the performance of CPython's compression modules to ensure they were well-optimized. Furthermore, the maintainer of pyzstd and backports.zstd (a backport of compression.zstd to Python versions before 3.14) benchmarked the new compression.zstd module against 3rd party Zstandard Python bindings such as pyzstd, zstandard, and zstd, and found the standard library was slower than most other bindings!

Let's take a closer look at the benchmarks and how to read them:

Figures give timing comparison. For example, +42% means that the library needs 42% more time than stdlib/backports.zstd. The reference time column indicates an average time for a single run.

Emoji scale: ❤️‍🩹 -25% 🟥 -15% 🔴 -5% ⚪ +5% 🟢 +15% 🟩 +25% 💚

Okay, so hopefully we don't see a lot of red, meaning the reference standard library (stdlib) time is slower...

CPython 3.14.0rc3

Case stdlib pyzstd zstandard zstd

compress 1k level 3 <1ms ⚪ - 3.81% ⚪ - 1.17% 🟢 + 5.86%

compress 1k level 10 <1ms ⚪ + 1.91% 🟢 + 6.18% 🟢 + 9.83%

compress 1k level 17 <1ms 🟢 + 6.33% 🟢 + 7.67% 🟢 +12.92%

compress 1M level 3 7ms ⚪ + 0.60% 🔴 - 7.37% 🟢 +12.08%

compress 1M level 10 27ms 🟢 +10.39% ⚪ + 3.39% 🟢 +12.46%

compress 1M level 17 174ms ⚪ - 2.48% ⚪ - 3.91% ⚪ + 0.08%

compress 1G level 3 6.03s 🟩 +16.17% ⚪ - 2.94% ⚪ + 2.25%

decompress 1k level 3 <1ms 🟥 -15.14% 🔴 - 8.53% ⚪ - 2.37%

decompress 1k level 10 <1ms 🟥 -15.41% 🔴 - 9.22% ⚪ - 3.35%

decompress 1k level 17 <1ms 🔴 -11.16% 🔴 - 7.09% ⚪ + 2.07%

decompress 1M level 3 1ms 🔴 - 6.88% ⚪ - 4.03% 💚 +26.88%

decompress 1M level 10 1ms 🔴 - 6.69% ⚪ - 4.86% 💚 +25.63%

decompress 1M level 17 1ms 🔴 - 7.99% ⚪ - 4.96% 💚 +25.58%

decompress 1G level 3 1.49s 🟥 -19.41% 🟥 -17.58% 🟢 + 6.98%

decompress 1G level 10 1.62s ❤️‍🩹 -27.65% ❤️‍🩹 -26.48% 🔴 - 6.92%

decompress 1G level 17 1.67s 🟥 -24.01% 🟥 -23.04% ⚪ - 4.43%

Case	stdlib	pyzstd	zstandard	zstd
compress 1k level 3	<1ms	⚪ - 3.81%	⚪ - 1.17%	🟢 + 5.86%
compress 1k level 10	<1ms	⚪ + 1.91%	🟢 + 6.18%	🟢 + 9.83%
compress 1k level 17	<1ms	🟢 + 6.33%	🟢 + 7.67%	🟢 +12.92%
compress 1M level 3	7ms	⚪ + 0.60%	🔴 - 7.37%	🟢 +12.08%
compress 1M level 10	27ms	🟢 +10.39%	⚪ + 3.39%	🟢 +12.46%
compress 1M level 17	174ms	⚪ - 2.48%	⚪ - 3.91%	⚪ + 0.08%
compress 1G level 3	6.03s	🟩 +16.17%	⚪ - 2.94%	⚪ + 2.25%
decompress 1k level 3	<1ms	🟥 -15.14%	🔴 - 8.53%	⚪ - 2.37%
decompress 1k level 10	<1ms	🟥 -15.41%	🔴 - 9.22%	⚪ - 3.35%
decompress 1k level 17	<1ms	🔴 -11.16%	🔴 - 7.09%	⚪ + 2.07%
decompress 1M level 3	1ms	🔴 - 6.88%	⚪ - 4.03%	💚 +26.88%
decompress 1M level 10	1ms	🔴 - 6.69%	⚪ - 4.86%	💚 +25.63%
decompress 1M level 17	1ms	🔴 - 7.99%	⚪ - 4.96%	💚 +25.58%
decompress 1G level 3	1.49s	🟥 -19.41%	🟥 -17.58%	🟢 + 6.98%
decompress 1G level 10	1.62s	❤️‍🩹 -27.65%	❤️‍🩹 -26.48%	🔴 - 6.92%
decompress 1G level 17	1.67s	🟥 -24.01%	🟥 -23.04%	⚪ - 4.43%

Ouch. 10-25% slower is quite unfortunate! A silver lining is that most of the performance difference is in decompression, so that narrows the area that is in need of optimization.

After sitting down and thinking about it for a while, I came up with a few theories as to why compression.zstd would be slower compared to pyzstd and zstandard. My thinking was focused on noting differences in implementation I knew existed between the various bindings. First, both pyzstd and zstandard build against their own copies of libzstd (the C library implementing Zstandard compression and decompression). Meanwhile, CPython will build against the system- installed libzstd, which is older on my system. Maybe there is a performance improvement in the newer libzstd versions? Second, most of the performance difference is in decompression speed. Perhaps the implementation of compression.zstd.decompress() is inefficient? It uses multiple decompression instances to handle multi-frame input where pyzstd uses one, so perhaps that's the issue? Finally, maybe the handling of output buffers is slow? When decompressing data, CPython needs to provide an output buffer (location in memory to write to) to store the uncompressed data. If the creation/allocation of that output buffer is slow it could bottleneck the decompressor.

Premature Optimizations

These optimizations didn't work, so if you'd like to skip to the optimizations which worked, please move to the next section!

I decided to tackle these one at a time. First, I built pyzstd and zstandard against the system libzstd. Unfortunately, after re-running the benchmark, this yielded zero performance difference. Darn.

Next, I was pretty confident that compression.zstd.decompress() was at least partially the culprit of the worse performance. The current decompress() implementation is written in Python and creates multiple decompression contexts and joins the results together. Surely that had to lead to some performance degradation? I ended up re-implementing the decompress() function in C using a single decompression context to see if my theory was correct. To my chagrin, there was no performance uplift, and it may have even performed worse! For the curious, you can see my hacked together branch here. Goes to show that you can never be sure about performance bottlenecks based on code itself!

Properly Profiling CPython

With my first two attempts at optimizing Zstandard decompression in CPython unsuccessful, I realized that I should do what I probably should have done from the beginning: profile the code! I decided to use the standard library support for the perf profiler, as it would allow me to see both native/C frames such as inside libzstd or the bindings module _zstd, as well as Python frames.

So I went ahead and compiled CPython with some flags to improve perf data and ran a simple script which called compression.zstd.decompress() on a variety of data sizes. I highly recommend reading the Python documentation about perf support for more details but essentially what I ran was:

# in a cpython checkout
./configure --enable-optimizations --with-lto CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
make -j$(nproc)
cd ../compression-benchmarks
perf record -F 9999 -g -o perf.data ../cpython/python -X perf profile_zstd.py

After analyzing the profile with perf report --stdio -n -g, I noticed a significant bottleneck in the output buffer management code! Let's take a brief detour to discuss what the output buffer management code does and why it was the decompression bottleneck.

(Fast) Buffer Handling is Hard

When decompressing data, you feed the decompressor (libzstd in our case) a buffer (bytes in Python) that is then decompressed and needs to be written to a new buffer. Since this all happens in C, basically we need to allocate some memory for libzstd to write the decompressed data into. But how much memory? Well, in many cases, we don't know! So we need to dynamically resize the output buffer as it is filled up.

This is actually a pretty challenging problem because there are several constraints and considerations to be made. The buffer management needs to be fast for a variety of output buffer sizes. If you allocate too much memory up front, you'll waste time allocating unused memory and slow down decompressing small amounts of data. On the other hand, if you don't allocate enough, you'll have to make a lot of calls to the allocator, which will also slow things down as each allocation has overhead and leads to fragmenting the output data. The memory should not grow exponentially for large outputs, otherwise you could run out of memory for tasks that would normally fit into memory. Finally, each output from the decompressor can vary in size, given that it may need to buffer data internally.

Because of the complexity in managing an output buffer, there is code shared across compression modules in CPython to manage the buffer. This code lives in pycore_blocks_output_buffer.h. The code was modified four years ago to use an implementation which writes to a series of bytes objects stored in a list to hold the output of decompress calls. When finished, the bytes objects get concatenated together in _BlocksOutputBuffer_Finish, returning the final bytes object containing the decompressed data. When profiling Zstandard decompression, I found that greater than 50% (!) of decompression time was spent in _BlocksOutputBuffer_Finish! This seemed inordinately long, ideally this function should just be a few memcpys. So with this knowledge in hand, I tried to think of how best to optimize the output buffer code.

Sometimes Timing Works Out

Right around the time that I was working on this, PEP 782 was accepted. This PEP introduces a new PyBytesWriter API to CPython which makes it easier to incrementally build up bytes data in a safe and performant way at the Python C API level. It seemed like a natural fit for what the blocks output buffer code was doing, so I wanted to experiment with using it for the output buffer code. After modifying pycore_blocks_output_buffer.h to use PyBytesWriter, I re-ran the original benchmark to see if we had closed the performance gap:

Note: this benchmark was run on my local machine and the wall times are not comparable to the previous benchmark.

Case stdlib zstandard

compress 1k level 3 <1ms 💚 +61.02%

compress 1k level 10 <1ms 💚 +57.77%

compress 1k level 17 <1ms 💚 +364.86%

compress 1M level 3 5ms 💚 +40.02%

compress 1M level 10 32ms ⚪ - 0.99%

compress 1M level 17 126ms 🟩 +15.93%

compress 1G level 3 4.47s 💚 +48.69%

decompress 1k level 3 <1ms ⚪ + 4.67%

decompress 1k level 10 <1ms ⚪ + 4.79%

decompress 1k level 17 <1ms 🟢 + 5.38%

decompress 1M level 3 1ms 💚 +50.23%

decompress 1M level 10 1ms 💚 +41.94%

decompress 1M level 17 1ms 💚 +47.37%

decompress 1G level 3 1.80s 🟢 +12.87%

decompress 1G level 10 1.77s 🟢 +12.54%

decompress 1G level 17 1.80s 🟢 + 8.76%

Case	stdlib	zstandard
compress 1k level 3	<1ms	💚 +61.02%
compress 1k level 10	<1ms	💚 +57.77%
compress 1k level 17	<1ms	💚 +364.86%
compress 1M level 3	5ms	💚 +40.02%
compress 1M level 10	32ms	⚪ - 0.99%
compress 1M level 17	126ms	🟩 +15.93%
compress 1G level 3	4.47s	💚 +48.69%
decompress 1k level 3	<1ms	⚪ + 4.67%
decompress 1k level 10	<1ms	⚪ + 4.79%
decompress 1k level 17	<1ms	🟢 + 5.38%
decompress 1M level 3	1ms	💚 +50.23%
decompress 1M level 10	1ms	💚 +41.94%
decompress 1M level 17	1ms	💚 +47.37%
decompress 1G level 3	1.80s	🟢 +12.87%
decompress 1G level 10	1.77s	🟢 +12.54%
decompress 1G level 17	1.80s	🟢 + 8.76%

WOW! Not only have we closed the gap, compression.zstd is now faster than the popular zstandard 3rd-party module.

Validating Our Results

Wanting to validate the speedup, I decided to write up my own minimal benchmark suite at this point too, to compare between revisions of the standard library code and use pyperf, a benchmarking toolkit used in the venerable pyperformance benchmark suite.

So I went ahead and wrote up a benchmark for zstd which tests compression and decompression using default parameters for sizes 1 KiB, 1 MiB, and 1 GiB. I ran these benchmarks on main and my branch which uses PyBytesWriter.

zstd.compress(1K): Mean +- std dev: [main_zstd_3] 3.01 us +- 0.03 us -> [pybyteswriter_zstd_3] 3.00 us +- 0.03 us: 1.01x faster
zstd.compress(1M): Mean +- std dev: [main_zstd_3] 2.92 ms +- 0.02 ms -> [pybyteswriter_zstd_3] 2.89 ms +- 0.02 ms: 1.01x faster
zstd.compress(1G): Mean +- std dev: [main_zstd_3] 2.72 sec +- 0.01 sec -> [pybyteswriter_zstd_3] 2.67 sec +- 0.01 sec: 1.02x faster
zstd.decompress(1K): Mean +- std dev: [main_zstd_3] 1.40 us +- 0.01 us -> [pybyteswriter_zstd_3] 1.38 us +- 0.01 us: 1.01x faster
zstd.decompress(1M): Mean +- std dev: [main_zstd_3] 734 us +- 4 us -> [pybyteswriter_zstd_3] 546 us +- 3 us: 1.34x faster
zstd.decompress(1G): Mean +- std dev: [main_zstd_3] 790 ms +- 4 ms -> [pybyteswriter_zstd_3] 634 ms +- 3 ms: 1.25x faster

Geometric mean: 1.10x faster

For input sizes great than 1 MiB that's 25-30% faster decompression! In hindsight, this actually makes sense if you consider that libzstd's decompression implementation is exceptionally fast. lzbench, a popular compression library benchmark, found that libzstd can decompress data at greater than 1 GiB/s. This is much faster than bz2, lzma, or zlib, the other compression modules in the standard library. One of the motivations for adding Zstandard to CPython was it's performance. So it is not too surprising that the output buffer code would be a bottleneck, given that the existing compression libraries don't write as quickly to the output buffer. This also explains why compression isn't faster after changing the output buffer code. Compression is very CPU intensive so more time is spent in the compressor rather than writing to the output buffer. This also explains why the speedup is non-existent for decompressing 1 KiB of data - the first 32 KiB block that is allocated is plenty to store all of the output data, meaning all of the time is spent in the decompressor.

One final validation I wished to do was to check the performance of zlib, to ensure that the change did not regress performance for other standard library compression modules. I wrote a similar benchmark for zlib to the one I wrote for zstd, and found that there was also a performance increase with the output buffer change!

zlib.compress(1M): Mean +- std dev: [main] 13.5 ms +- 0.1 ms -> [pybyteswriter] 13.4 ms +- 0.0 ms: 1.00x faster
zlib.compress(1G): Mean +- std dev: [main] 11.4 sec +- 0.0 sec -> [pybyteswriter] 11.3 sec +- 0.0 sec: 1.00x faster
zlib.decompress(1K): Mean +- std dev: [main] 1.42 us +- 0.01 us -> [pybyteswriter] 1.39 us +- 0.01 us: 1.02x faster
zlib.decompress(1M): Mean +- std dev: [main] 1.29 ms +- 0.00 ms -> [pybyteswriter] 1.17 ms +- 0.00 ms: 1.10x faster
zlib.decompress(1G): Mean +- std dev: [main] 1.36 sec +- 0.00 sec -> [pybyteswriter] 1.17 sec +- 0.00 sec: 1.17x faster

Benchmark hidden because not significant (1): zlib.compress(1K)

Geometric mean: 1.05x faster

10-15% faster decompression on data of at least 1 MiB for zlib is pretty significant, especially when you consider that zlib is used by pip to unpack files in almost every wheel package Python users install.

Conclusion

With the improvements to output buffer handling, I was not only able to improve the performance of compression.zstd, but all of the compression module's decompression code. After stumbling over a few optimization ideas, I definitely learned my lesson to profile code before jumping to conclusions! You won't know what is a real bottleneck unless you can test it! Just having a benchmark is not enough!

The original issue I opened goes into a bit more detail about the process of benchmarking the compression modules, and the commit with the improvement has the diff of changes to adopt PyBytesWriter. One thing I'm proud of is that not only did the change improve performance, it also simplifies the implementation of the output buffer code and removed 60 lines of code in the process!

I did some more profiling of zlib to see if there were any more performance gains to be made, but the profile I gathered seems to indicate that 95+% of the time is spent in zlib's inflate implementation (with the rest in the CPython VM), so there is little if any room for further optimization in CPython's bindings for zlib. I think this is good, as it indicates Python users are getting the best performance they can in 3.15!

Going forward, I am planning on profiling compression code more, but the vast majority of the time spent there will probably be in the compressor since compression is so CPU intensive. Finally, I want to investigate optimizations related to providing more information about the final size of the output data. In some cases the output buffer is initialized to a small value and dynamically resized as output is produced, but ideally users would be able to provide more information about their workflow and see a performance improvement over it. I have a lot of other ideas related to compression I'd like to work on, check out my OSS TODO list for all of the random ideas I want to work on in the future!

Finding a miscompilation in Rust/LLVM

Emma Smith — Tue, 14 Oct 2025 00:00:00 -0700

Among my friends I have a reputation for ~~causing~~ stumbling across esoteric error messages. Whether that is SSL read: I/O error: Success (caused by a layered SSH connection hangup on Windows), or that time I tried installing NixOS on my laptop and os-prober failed to start (this was several years ago, so I am sure it is no longer an issue). I attribute these oddities to my curiosity, particularly around trying things that may or may not work and seeing if they do. Recently, I was trying to complete an item from my OSS TODO list when I came across a bug that stumped me for several days. Turns out sometimes even compilers have bugs...

My goal was to build CPython with Rust implementations of common compression libraries to see if the Rust libraries could be supported. CPython relies on C code to do many performance sensitive activities such as math and compression. I had recently read about the Trifecta Tech Foundation's initiative to re-write popular compression libraries in Rust. So far as of September 2025, they have pure-Rust re-implementations of zlib (the library used for zip and gzip files), and bzip2 that are available for use.

These Rust libraries not only bring increased memory safety, they're also as fast or faster than their C counter-parts. Additionally, zlib-rs is widely deployed in Firefox, to the point that it may have tripped over a CPU hardware bug(!). So I had confidence that at least zlib-rs would work out of the box.

To add support for these libraries to CPython, I made a branch with changes to the autoconf script to search for the Rust libraries through pkg-config. I built zlib-rs's C library with RUSTFLAGS="-Ctarget-cpu=native" for maximum speed, and then pointed CPython's build process to the built zlib_rs library. Everything built just fine. Next, I wanted to run the CPython zlib test suite to verify zlib-rs was working correctly. I mostly did this to make sure I had built things properly, I had no doubts the tests would pass.

And yet. I was shocked! zlib-rs is used in Firefox, cargo, and many other widely used tools and applications. Hard to believe it would have a glaring bug that would be surfaced by CPython's test suite. At first I assumed I had somehow made a mistake when building. I realized I had used my system zlib header when building, so maybe there was some weirdness with symbol compatibility?? No, re-building CPython pointing to the zlib-rs include directory didn't fix it. I tried running cargo test in the zlib-rs directory to make sure there wasn't something wrong I could catch there. No failures occurred.

At this point I was convinced it was probably a bug with how I was building things, or a bug in the cdylib (Rust lingo for "C library") wrapping zlib-rs since test Rust tests passed but the tests in CPython failed. To make my testing simpler, I captured the state of the test_zlib.test_combine_no_iv test using PDB and wrote a C program which does the same thing as the test, with deterministic inputs:

#include <stdio.h>
#include <string.h>
#include "zlib.h"

int main()
{
    unsigned char a[32] = {0x88, 0x64, 0x15, 0xce, 0x5e, 0x3b, 0x8d, 0x35,
                        0xdb, 0xd2, 0xb5, 0xfa, 0x8e, 0xa7, 0x73, 0x10,
                        0x66, 0x83, 0x1b, 0xd1, 0xde, 0x0f, 0x25, 0x86,
                        0xeb, 0xe5, 0x42, 0x44, 0xad, 0x62, 0xff, 0x11};
    uInt chk_a = crc32(0, a, 32);
    unsigned char b[64] = {0x31, 0xb8, 0xce, 0x94, 0x4d, 0x2b, 0xb9, 0x7e,
                        0xd5, 0x81, 0x7f, 0xc2, 0x40, 0xbf, 0x3d, 0xa5,
                        0x25, 0xa5, 0xf9, 0xdf, 0x53, 0x68, 0xc4, 0xf6,
                        0xbe, 0x06, 0x7d, 0xf3, 0xc7, 0xdc, 0x5b, 0x84,
                        0xce, 0xd2, 0xb2, 0xeb, 0x87, 0x62, 0x60, 0xe3,
                        0x10, 0x05, 0x64, 0x59, 0x15, 0xc4, 0x2d, 0x78,
                        0xc8, 0xf3, 0x14, 0x38, 0x87, 0x39, 0xb3, 0x58,
                        0xb5, 0x95, 0x07, 0x25, 0xd9, 0xc1, 0xac, 0x04};
    uInt chk_b = crc32(0, b, 64);
    unsigned char buff[96];
    memcpy(buff, a, 32);
    memcpy(buff + 32, b, 64);
    uInt chk = crc32(0, buff, 96);
    uInt chk_combine = crc32_combine(chk_a, chk_b, 64);
    printf("chk (%u) = chk_combine (%u)? %s\n", chk, chk_combine, chk == chk_combine ? "True" : "False");
    return (0);
}

This program also failed. Hm, okay, not an issue with CPython at least. I then translated the above test into Rust to add to the zlib-rs test suite, since the Rust tests passed. If it failed I could more easily debug the issue.

diff --git a/zlib-rs/src/crc32/combine.rs b/zlib-rs/src/crc32/combine.rs
index 40e3745..65c0143 100644
--- a/zlib-rs/src/crc32/combine.rs
+++ b/zlib-rs/src/crc32/combine.rs
@@ -66,6 +66,26 @@ mod test {

    use crate::crc32;

+    #[test]
+    fn test_crc32_combine_no_iv() {
+        for _ in 0..1000 {
+            let a: &[u8] = &[0x88, 0x64, 0x15, 0xce, 0x5e, 0x3b, 0x8d, 0x35, 0xdb, 0xd2, 0xb5, 0xfa, 0x8e, 0xa7, 0x73, 0x10, 0x66, 0x83, 0x1b, 0xd1, 0xde, 0x0f, 0x25, 0x86, 0xeb, 0xe5, 0x42, 0x44, 0xad, 0x62, 0xff, 0x11];
+            let b: &[u8] = &[0x31, 0xb8, 0xce, 0x94, 0x4d, 0x2b, 0xb9, 0x7e, 0xd5, 0x81, 0x7f, 0xc2, 0x40, 0xbf, 0x3d, 0xa5, 0x25, 0xa5, 0xf9, 0xdf, 0x53, 0x68, 0xc4, 0xf6, 0xbe, 0x06, 0x7d, 0xf3, 0xc7, 0xdc, 0x5b, 0x84, 0xce, 0xd2, 0xb2, 0xeb, 0x87, 0x62, 0x60, 0xe3, 0x10, 0x05, 0x64, 0x59, 0x15, 0xc4, 0x2d, 0x78, 0xc8, 0xf3, 0x14, 0x38, 0x87, 0x39, 0xb3, 0x58, 0xb5, 0x95, 0x07, 0x25, 0xd9, 0xc1, 0xac, 0x04];
+            let both: &[u8] = &[0x88, 0x64, 0x15, 0xce, 0x5e, 0x3b, 0x8d, 0x35, 0xdb, 0xd2, 0xb5, 0xfa, 0x8e, 0xa7, 0x73, 0x10, 0x66, 0x83, 0x1b, 0xd1, 0xde, 0x0f, 0x25, 0x86, 0xeb, 0xe5, 0x42, 0x44, 0xad, 0x62, 0xff, 0x11, 0x31, 0xb8, 0xce, 0x94, 0x4d, 0x2b, 0xb9, 0x7e, 0xd5, 0x81, 0x7f, 0xc2, 0x40, 0xbf, 0x3d, 0xa5, 0x25, 0xa5, 0xf9, 0xdf, 0x53, 0x68, 0xc4, 0xf6, 0xbe, 0x06, 0x7d, 0xf3, 0xc7, 0xdc, 0x5b, 0x84, 0xce, 0xd2, 0xb2, 0xeb, 0x87, 0x62, 0x60, 0xe3, 0x10, 0x05, 0x64, 0x59, 0x15, 0xc4, 0x2d, 0x78, 0xc8, 0xf3, 0x14, 0x38, 0x87, 0x39, 0xb3, 0x58, 0xb5, 0x95, 0x07, 0x25, 0xd9, 0xc1, 0xac, 0x04];
+
+            let chk_a = crc32(0, &a);
+            assert_eq!(chk_a, 101488544);
+            let chk_b = crc32(0, &b);
+            assert_eq!(chk_b, 2995985109);
+
+            let combined = crc32_combine(chk_a, chk_b, 64);
+            assert_eq!(combined, 2546675245);
+            let chk_both = crc32(0, &both);
+            assert_eq!(chk_both, 3010918023);
+            assert_eq!(combined, chk_both);
+        }
+    }
+
    #[test]
    fn test_crc32_combine() {
        ::quickcheck::quickcheck(test as fn(_) -> _);

Running cargo test passed! I was at my wits end! How could the C code fail but the Rust code succeed??

I felt like I had enough information that I reported the issue to zlib-rs. Let me interrupt this story to mention that I really want to thank Folkert de Vries (maintainer of zlib-rs) for help debugging this. They were extremely friendly and helpful in figuring out what was going wrong. Folkert responded to my issue that my C program sample works for them! Why would my machine be any different? I was running in the WSL at the time, maybe that could cause weirdness? I decided to write up a Containerfile to ensure I had a clean environment:

FROM ubuntu:24.04

RUN apt-get update && \
    apt-get install -y \
        build-essential \
        curl \
        git \
        pkg-config \
        libssl-dev

RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN curl -sSL https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
RUN echo "deb http://apt.llvm.org/noble/ llvm-toolchain-noble-20 main" > /etc/apt/sources.list.d/llvm.list
RUN apt-get update  && apt-get upgrade -y && apt-get install -y clang-20
RUN cargo install cargo-c
RUN mkdir /scratch
RUN git clone https://github.com/trifectatechfoundation/zlib-rs.git /scratch/zlib-rs
COPY ./test.c /scratch/zlib-rs/libz-rs-sys-cdylib/test.c
WORKDIR /scratch/zlib-rs/libz-rs-sys-cdylib
ENV RUSTFLAGS="-Ctarget-cpu=native" # comment this out to fix the bug
RUN cargo cbuild --release
RUN clang-20 -o test test.c -I ./include/ -static ./target/x86_64-unknown-linux-gnu/release/libz_rs.a
ENV LD_LIBRARY_PATH="target/x86_64-unknown-linux-gnu/release/"
ENTRYPOINT ["./test"]

While experimenting with setting up this container, I found a lead at last! If I compiled with RUSTFLAGS="-Ctarget-cpu=native", the program gave the wrong results. If I compiled without using native code generation, the program worked correctly. Bizarre!!

Backing up a bit, let me explain what RUSTFLAGS="-Ctarget-cpu=native" actually does (if you know already, please skip to the next paragraph). Compilers like rustc have feature flags for each target (aka OS + CPU architecture family) which allows them to optionally emit code that uses features of processors. For example, most x86 processors have sse2, and ARM64 processors have NEON or SVE. Newer processes usually come with newer features which provide optimized implementations of some useful thing, for example some x86 processors has optimized implementations of SHA hashing. Since not all computers have every feature, these need to be opted into at compile time. In the case of RUSTFLAGS="-Ctarget-cpu=native" I'm telling Rust "use all the features for my current processor." This is a way to eke out the most performance from a program. But in this case, it meant I had a bug on my hands! Folkert (maintainer of zlib-rs) suggested I try to narrow down exactly which instruction set extension was causing the issue. After a bit of binary searching, I found out it was avx512vl. AVX is an extension to provide SIMD and AVX512-VL is an extension which allows interoperability between 128/256-bit wide SIMD and faster 512-bit wide SIMD. This made a lot of sense in some ways, after all, I have an AMD R9 9950X, and one of it's features is AVX512 support! But how exactly did these AVX512 instructions get into the final binary?

NOTE:
As pointed out in a message on Mastodon, AVX512-VL is actually 11 years old! It was first introduced in Intel AVX512 implementations. However, AVX512 support in Rust is relatively new.

So enabling AVX512 was the culprit for the bug in crc32 calculations. Skimming over the zlib-rs code, I was a bit surprised to find that it does not explicitly use AVX-512 anywhere! In fact it uses the older SSE4.1 instruction set (presumably for maximum portability). So why was AVX512-VL causing these issues? Unfortunately, I don't know for sure. But I have a theory.

Rust uses LLVM as it's default backend (the bit of the compiler that emits instructions/binaries). LLVM probably realized it could use AVX512-VL instructions (available on my machine) to speed up the SSE4.1 code that zlib-rs is using. However, AVX512-VL is new enough that there was a bug in the compiler - a miscompilation - and the wrong code was emitted. I haven't found a smoking gun issue but it is probably one of these.

I am happy to report that this issue does not present itself with Rust 1.90+ or the latest release of zlib-rs. Many thanks again to Folkert for not only helping figure out the source of the issue, but also adding a mitigation to zlib-rs and cutting a new release to work around the miscompilation! Now the CPython test suite passes when linked against zlib-rs and I can continue my experiments...

Revamping my blog... again

Emma Smith — Sun, 07 Sep 2025 00:00:00 -0700

Background

Well, I've succumbed to the ever-present urge to completely change one's blog setup. It all started when I wanted to add my blog to the Awesome PyLadies' blogs repo. As part of the configuration you can add your blog's RSS feed (structured information about a blog's contents). But the configuration says:

if you wish to have your blog posts being promoted by the Mastodon bot; the RSS feed should be for Python-related posts

My previous blog generator was zola, which worked really well and was easy to set up! However, zola does not support per-tag (or "taxonomy" as zola calls them) feeds. I considered contributing support for this to zola, but I figured I'd look around at other static site generators and see what they support. My blog content is just a bunch of Markdown files after all, so it should be easy to move to another static site generator!

Yak shaving, for fun and profit

I came across Pelican, which was really appealing for a few reasons. First, it supported per-feed RSS feeds. But also, it is written in Python and I felt like it would be fitting since I am a Pythonista. So I decided I would try to port my blog to Pelican. As you may be able to tell by looking at the footer, I did so successfully :)

Setting up Pelican is actually super easy. I installed pelican with markdown support by running uv tool install pelican[markdown] and ran pelican-quickstart to set up a project. After answering a few prompts, I had a full project set up and could copy over the Markdown files used to write this blog. After changing the metadata from zola's format to Pelican's, I had a blog generated... with no theme.

Oh... I needed to see what themes were available. Fortunately Pelican makes this easy by going to the pelicanthemes.com website. That site has a number of community authored themes. Unfortunately, I didn't see any themes I loved.

Introducing pelican-theme-terminimal

So, I did the only natural thing to do and ported the zola theme I was using to Pelican. Fortunately, this wasn't actually too bad. Zola uses Tera for its templates, which is based on Jinja2, which is what Pelican uses. So for the most part I could minimally update the variables used and get the theme ported over easily. The layout between the two is slightly different so I had to restructure how things are designed, but overall it was pretty easy and enjoyable.

You can check out the theme's code here. I don't plan on working on the theme a ton more, mostly just to add features or customizations I want, but it is open source if anyone else wants to use it or submit patches.

The top priorities I have to work on are:

Links to RSS feeds
Mastodon verification

So yeah, my blog is now running on Pelican and Python 🎉

I have a few ideas to blog about over the next week or two so check back soon, or subscribe to my RSS feed.

Introducing the real me!

Emma Smith — Sat, 23 Nov 2024 00:00:00 -0800

I'm really excited for the opportunity to introduce my true self. A while back I began exploring my gender expression, and today marks an important step in my journey. So let me re-introduce myself, the real me this time.

Hi! 👋 I'm Emma, a trans woman. I use she/her pronouns.

I'm the same person, just more... me! So all the stuff in About Me is still true! Expect more posts about Python, packaging, and whatever other hobby projects I have going on.

Finally, I'd like to thank my wife and everyone who has supported me on this journey so far. I feel so lucky to be surrounded by so many supportive friends and family members. I absolutely could not have gotten this far without them.

New Rust crate: generational-arena-dom

Emma Smith — Sat, 08 Jul 2023 00:00:00 -0700

I've just published a new crate someone may find interesting. I recently had a take-home assessment where I used the Servo project's html5ever HTML parser crate. html5ever is the main crate that servo uses to parse HTML content on the web. This crate is very customizable, and you have to bring your own implementation of the DOM (which means you can handle memory management however you want!). They provide an example implementation of the DOM that uses Rc<RefCell<T>>s all over, which is a bit annoying to use with Rust's borrow checking model. This becomes particularly frustrating when dealing with frequent DOM mutations, as was the case in my project.

Fortunately, I recalled the benefits of using arenas for ergonomic memory management when working with trees. I also had recently read about Vale's generational arena usage, and I was inspired to build a DOM implementation based on a generational arena design. Generational arenas are nice because they don't suffer from the ABA problem, so it is thread-safe to add, update, and delete nodes in the DOM. I ended up coming across the generational-indextree crate, which uses tokens instead of references to refer to tree members, which made working with mutable elements of the DOM much easier!

Anyway, I encourage anyone who is interested to learn more to check out the project on my github or on the project page on crates.io

Using multiprocessing and sqlite3 together

Emma Smith — Fri, 19 May 2023 00:00:00 -0700

Note from the author: this is a pseudo TIL, but I hadn't seen it written down anywhere, hopefully someone finds it useful! Jump to "Solution" below if you don't care about the background.

Background

Generating Data

I recently started working on a reinforcement learning project, and I needed to generate a lot of training data. The project involves quantum compilers, so the data I generate is quantum circuits. For those unfamiliar, quantum circuits are just sequences of unitary matrices laid out in a particular order. I chose to store the circuit as a sequence of unitary gate names. The output of the data generation is the unitaries (numpy arrays) that are the result of multiplying the matrices in the circuit together.

I ended up wanting to generate somewhere in the region of a few hundred billion matrices, each of them very small. I knew off the bat that this would require a fair bit of time, and I wanted to take advantage of the 32 core server I own. Since I was using Python to generate this data, I used the multiprocessing module. Sadly I cannot yet take advantage of Python multithreading coming in 3.12.

Disk Space Woes

For saving the generated matrices, I started off by doing the simplest thing, just using plain-old np.savetxt to save the (pretty tiny) matrices to disk in each process after computing the product of the matrices in the quantum circuit. This... was problematic. I quickly ran into a disk out of space error. Normally this means I need to clear out space on whatever VM I am using, but there was one problem -- I still had hundreds of gigabytes of space left on disk!

I quickly deduced the error actually was caused by hitting the limit on entries in a directory, dang it CIFS! I briefly tried to make my own schema to split the unitaries into more directories to avoid this limit but I ended up hitting more file system limits. It was clear just writing files to disk wouldn't scale to the size of dataset I needed to generate.

Choosing a Database

Of course, dealing with so many small files, a database was the right solution to this problem. Why didn't I start with a database to begin with? Partially because I wanted to make it easy to load individual unitaries (np.loadtxt is an incredibly handy API). Also, I was just hacking this data generation script together.

I had one issue with switching to a database: I wanted something simple and lightweight, I didn't need anything fancy like postgres or the like. Sqlite is the obvious choice but sqlite does not by default support concurrent writes, which is exactly what I wanted to do!

Solution

So how can one achieve concurrent writes in Python using sqlite3? Sqlite by default uses a rollback log to maintain consistency. You can change the configuration so that sqlite uses a write-ahead log (WAL) mode as well, which allows for concurrent writes. You can enable WAL mode in Python by setting the journal mode:

# assume some Cursor object `cursor`
cursor.execute('PRAGMA journal_mode = WAL')

However, I started getting exceptions part way through. Some processes calculating the unitaries were being told the database was locked, even though it should not be (these processes should be writing to the WAL, which I want to always be available). Therefore I also set the sqlite pragma synchronous to OFF, which means that the WAL does not synchronize before checkpoints. Note this is dangerous because theoretically your database could become corrupted if the process crashes or the server shuts down. This is acceptable to me because I can always regenerate the database and either of these are very unlikely to occur while I run these data generation tasks. This can be done in Python like so:

# assume some Cursor object `cursor`
cursor.execute('PRAGMA synchronous = OFF')

In summary, by enabling the WAL and turning some sync'ing off, I was able to get multi-processed Python code to concurrently write to a sqlite database. This also gave a nice speed bump since sqlite is optimized for writing many small amounts of data to disk, a nice bonus!

Rust CLI tools apt repo

Emma Smith — Sat, 26 Nov 2022 00:00:00 -0800

tl;dr Go to http://apt.cli.rs and follow the instructions to add the apt repo

I guess its only fair to start my new blogging kick by catching up on a project I'd worked on several months ago which is pretty much "done." I really like several Rust tools, such as ripgrep, hyperfine, and fd. I wanted an easy way to install them on Debian-based systems that may not have them in the official repos, so I needed to create my own apt repo. I was originally considering making a ppa, or private package archive, since I mainly use Ubuntu, but I decided I wanted these tools available on Debian as well, so my only option was an apt repo.

It turns out that apt repos are really simple, you just need to serve a directory with e.g. nginx. It took me a while to find a tool I liked using for making the apt repo, however. I started with trying reprepro, but I found it was more annoying to use than I wanted, so I ended up using aptly, a newer apt repo management tool.

When setting up the repo, my criteria for inclusion were:

the tool must be able to build *.deb packages. This is the package format for Debian/Ubuntu so definitely required
the tool must build those packages. I wanted to use the official/same binaries as those in the Github release
the binary packages must be statically musl-linked. Since various versions of Debian/Ubuntu use different versions of glibc, you can get version compatibility errors if you don't statically link. Statically linking to musl is much easier than glibc, so this seemed to be the best route to avoid compatibility errors.

With these criteria in hand, I then set up the apt repo with the CLI tools that fit and wrote a script to try updating all of the packages, pulling from the official Github releases. I have subscribed to releases on Github from all the repos that are currently included in the apt repo so all I have to do is run the script when a new release comes out from one of the repos. I may end up setting up a cronjob that runs this update script, but I'm weighing whether it is worth any potential risks...

Anyway, if you are interested in using this apt repo, head on over to https://apt.cli.rs, I've written up the commands you need to get started using the apt repo.

If you have suggestions for Rust CLI tools that should be included, please open an issue on the github! If a package doesn't build musl-linked Debian packages, consider opening an issue on that project to add it. However, please do not spam maintainers asking for these packages to be built. I unfortunately do not have time to contribute to the packaging of every great Rust CLI tool, but I can add them to the apt repo if they are packaged already!

Anyway, hopefully someone else finds it useful!

Revamping my blog

Emma Smith — Fri, 25 Nov 2022 00:00:00 -0800

Well, I've decided I want to blog a bit more, in part thanks to Simon Willison's post suggesting keeping a blog is a good idea.

I've changed the UI a bit and I am now publishing via a Github action (I now just need to write a markdown file and push to a git repo!). I hope making it easier to blog will hopefully mean I blog a bit more.

One slight issue I ran into is I would push a commit, then my website would go down. The custom domain setting in Github pages would be reset to empty. At first, I thought this was a bug, how could this get reset so frequently! Turns out Github ties the custom domain to a file named CNAME in the root of the gh-pages branch (or whatever you configure to publish in the settings). I've now set that as a static asset to be included in the root so I won't run into this problem, but rather annoying behavior, my git repo shouldn't be used to store configuration!

PyBay 2019!

Emma Smith — Sun, 04 Aug 2019 00:00:00 -0700

I will be tabling at PyBay 2019 on August 18th! Hope to see you there :)

Stupid solutions for stupid problems

Emma Smith — Mon, 04 Jun 2018 00:00:00 -0700

A while back, a friend of mine was working on a coding challenge for some internship. The prompt said to create a dictionary with string keys and string values in your favorite programming language, without using the built in dictionary type.

This got me thinking of how I would solve this, and this terrible monster is what I came up with:

class DumbDict:
    __getitem__ = getattr
    __setitem__ = setattr

This is a very silly solution. I don't feel bad about it. Here it is in action:

>>>d = DumbDict()
>>>d['hi'] = 'test'
>>>d['hi']
'test'

How does it work? Well, in Python __getitem__ is the protocol for subscribtion access. When I write d['hi'] Python internally calls d.__getitem__('hi'). So whats the deal with the getattr call then?

In Python, everything is an object. By default, objects internally map attribute names to values. In my dictionary implementation, I take advantage of this to use Python's internals to create my dictionary.

The pros of this dictionary are that it is fast. It is close (within 10%) of the builtin dictionary type (there is a bit of overhead with function calls).

There are quite a few cons, the first of which several of the more knowledable among you have already realized. I'm totally cheating here. Well, maybe. Or I'm not. Technically, classes use something called types.MappingProxyType to manage attribute mappings. This is however an implementation detail, so its up to personal opinion whether I'm using a built in dictionary or not. Anyway, I think its pretty cool. And remember I did say it was stupid...

PyBay 2017

Emma Smith — Fri, 01 Sep 2017 00:00:00 -0700

The video of the panel at PyBay 2017 I was on was posted!

It was a pleasure to be a part of the panel, and a great experience to go to PyBay 2017. The SF Python community was kind and welcoming. I hope to go to their regular meetings if I can find time.

Typycal - Generate type stubs from runtime type information

Emma Smith — Wed, 26 Jul 2017 00:00:00 -0700

EDIT: This project stalled and is no longer being worked on. You can find the sources on my github. The original blog is below.

Note: This blog post assumes basic familiarity with typing

I am a collaborator on the mypy project, which implements a static type checker for Python. Static typing is useful for many reasons, such as making refactoring and other code maintenance easier. At PyCon this year, I heard from many developers using mypy who found it useful in understanding their code and finding type errors. Andreas Dewes gave a nice summary and analysis of typing in Python in a Europython talk. I highly recommend it if you are just starting typing or are not convinced it is worth it to read the slides from that talk.

At the moment, several companies and open source projects have moved to using static typing, such as Dropbox, Google, and Zulip, an open source chat application. One of the design goals of static typing in Python is to be optional and gradual. However, even with these goals, annotating code, especially for large code bases, can be a significant time drain. With this in mind, I was interested in trying to make it easier for people to adopt typing in their Python code. Some work has been put into making a more advanced type inference tool that would be able to generate useful type information such as Google's pytype project and mypy's stubgen script. Google and Facebook both have their own closed source runtime type inferencers. However, pytype can be inaccurate and crash on valid code (which is bad if you want to adopt static typing). In addition it only runs on Python 2 (though it can check Python 3 code). Stubgen is rather incomplete, and will likely never be able to infer the most complex cases. These are useful tools, but I believe that runtime introspection can do better.

These solutions seemed overly complex when all the type information needed is right in front of us: in the running code itself! At the time I was thinking about this problem, I coincidentally had been reading up on Pyjion and PEP 523: Adding a frame evaluation API to CPython when I realized I could use the new frame evaluation API to do runtime type introspection!

So what is this frame evaluation API?

The frame evaluation API was introduced mainly for JIT (just in time) compilers, and debuggers (PyCharm uses it). However, I wanted to use it to analyze the types in frames. You may be asking yourself: what is a frame? A frame is a data structure that Python uses to describe scopes and information about that scope. The simplest to understand frame is a function:

def test():
    ...

Modules, functions, generators, and comprehensions have their own scope so they get their own frame. So when I call os.path.abspath(path), I am creating a new frame. The frame data is represented by a C struct in CPython (if you aren't familiar with C, just think of it like a Python class with some attributes). It contains information about the function called, such as its name (abspath in our example), its file path, and most importantly its locals. Locals is a symbol table (a mapping of names to values) for the scope of the frame. For example, in abspath, the path argument is a local. Consider the following:

def hello(name):
    msg = "Hello %s!"
    print(msg % name)

Both name and msg are locals in the hello frame.

The frame evaluation API allows us to inspect the values of name and msg. The important part of locals is that function arguments are in the locals of a frame. We can get the type of these objects and log the argument types of the frame. Then we can execute the frame (run the Python code) and capture the return value (and type) as well. Thus the full signature of the function for each call is available. Here is basically how the code works (Python equivalent of the C code in typycal).

def typycal_evalframe(frame: frameobject, exc: int):
    """
    frame is the current frame to be executed
    exc indicates whether an exception has been thrown calling the frame.
    """
    if exc:
        return _PyEval_EvalFrameDefault(frame, exc)  # execute the frame as normal, this function is part of Python's private C API
    else:
        code = frame.f_code  # this is the bytecode of the frame (stuff to be executed), and some other useful info
        name = code.co_name
        file_name = code.co_filename
        if whitelisted(file_name, name):  # ignore stdlib and generators/comprehensions
            locals = frame.f_locals  # a dict of locals. names are keys, objects are values
            argc = code.co_argcount  # number of arguments passed to the function
            ret = _PyEval_EvalFrameDefault(frame, exc)  # run the frame, store the return value for analysis
            serialize_types(file_name, name, locals, argc, ret)
            return ret
        else:
            return _PyEval_EvalFrameDefault(frame, exc)  # don't want to analyze this frame, so execute it as normal

def hook():
    thread_state = thread_state_get() # Needed to tell Python to run our frame evaluation function, instead of the default
    thread_state.interp.eval_frame = typycal_evalframe  # assign our function to be called when a frame needs to be evaluated

def unhook():
    thread_state = thread_state_get()
    thread_state.interp.eval_frame = _PyEval_EvalFrameDefault(frame, exc)

The serialize_types function just takes the Python object and generates PEP 484 compliant type data that is written to a file. You can call the hook from your code:

import typycal
typycal.hook()
... # your code here

The hook will be put in place and typycal will be able to introspect frames, no decorators needed! We now have a means to serialize the type of the current frame! But how to get to *.pyi files?

Building *.pyi files

With this information logged, one can then run a Python program (part of typycal) to analyze the signatures and generate stub files. The hope is that companies and individuals will be able to run typycal on their source via unit tests or other uses, and come out with stubs that are a basis to typing their entire code. Currently typycal is implemented in C++, both to interop with CPython well and to be fast. It is rather unoptimized, and causes the execution of 12 million frames to slow by roughly 5x, which is obviously not ideal. Most of the slowness comes from checking if a frame should be analyzed. To not greatly burden I/O and keep the serialized data as minimal as possible, we want to exclude the standard library, which can add millions of frames to a medium sized code base. I have a few plans to reduce the time spent on I/O and other optimizations in mind.

Since getting type information for all types gets very complex, typycal for now handles int, str, tuple, list, callables, and None. dict should be relatively straightforward too. All other types will likely be Any'd since they may not be safe to put in a stub. This will likely change with time.

The future

I plan on spending the rest of the summer improving typycal to work decently well to the point that I can release it publicly. If you are are interested in getting a look at the library to play with, feel free to contact me, and I will consider sharing it. I also will be discussing this project and static typing in general at PyBay on August 11, and would be happy to answer questions in person.