Inside the ipscope MMDB Compiler: Packing 35.3M Prefixes into a Zero-Copy Radix Tree

#Why local binary radix trees beat network RPCs on high-throughput hot paths

When an edge proxy, Web Application Firewall (WAF), real-time ad-bidding engine, or API gateway processes 50,000 to 200,000 requests per second on a single multi-core machine, every microsecond on the request path counts. Even calling a local Redis or Memcached instance over a loopback socket introduces system call overhead, socket buffer copying, serialization framing, and kernel scheduler jitter that pushes p99 lookup latency to 250 to 800 microseconds.

By compiling the entire global IPv4 and IPv6 intelligence dataset into an immutable, memory-mapped (.mmdb) binary radix tree, your application resolves any IPv4 or IPv6 address in under 1 microsecond (380 to 710 nanoseconds at p50) using pure pointer arithmetic inside the operating system's mmap(2) virtual address space. There are no network sockets, no mutex locks, and zero heap allocations during the tree walk.

This deep dive explains how the ipscope MMDB compiler transforms 35.3 million raw CIDR prefixes and six intelligence families into a compact .mmdb file, how binary bit-tries and pointer deduplication work under the hood, and how to tune .mmdb lookups in Go and Rust for maximum CPU cache locality.

#Anatomy of the open MMDB v2.0 binary specification

Every .mmdb file produced by ipscope adheres strictly to the MaxMind DB Binary Format Specification v2.0. Because the wire format is an open standard, you never need a proprietary binary library to read ipscope databases. Standard open-source readers such as maxminddb-golang (Go), maxminddb (Rust), node-maxmind (Node.js), maxminddb (Python C-extension), and ngx_http_geoip2_module (NGINX C module) open ipscope .mmdb files directly without modifying a single line of reader code.

At the byte level, an .mmdb file is laid out as three contiguous sections:

+-------------------------------------------------------------------+
| 1. Binary Search Tree Section                                     |
|    Fixed-width node array (24-bit, 28-bit, or 32-bit records)     |
|    Implicitly indexed from Node 0 (Root) to Node N-1              |
+-------------------------------------------------------------------+
| 16-Byte Null Separator (0x00 * 16)                                |
+-------------------------------------------------------------------+
| 2. Data Section                                                   |
|    Deduplicated UTF-8 strings, IEEE-754 floats, unsigned ints,    |
|    nested maps, arrays, and 2/3/4-byte back-reference pointers    |
+-------------------------------------------------------------------+
| Metadata Start Marker: \xab\xcd\xefMaxMind.com                    |
+-------------------------------------------------------------------+
| 3. Metadata Map Section                                           |
|    node_count, record_size, ip_version (6), database_type,        |
|    build_epoch, languages, description                            |
+-------------------------------------------------------------------+

1. The Binary Search Tree Section

The search tree is a bitwise radix trie (specifically, a binary trie where each level inspects a single bit of the target IP address from most significant bit MSB 0 to least significant bit LSB 31 for IPv4 or LSB 127 for IPv6).

Rather than storing memory pointers or object headers, the tree is serialized as a flat array of fixed-size nodes. Each node consists of exactly two records:

• Left record (bit = 0)
• Right record (bit = 1)

The bit width of each record is defined by record_size in the metadata footer (24, 28, or 32 bits):

In a 28-bit tree, two 28-bit integers are packed into 7 bytes using bit-nibbles: the first 3 bytes hold the low 24 bits of the left record, the middle byte (byte[3]) splits its high 4 bits for the left record's upper nibble and its low 4 bits for the right record's upper nibble, and the final 3 bytes hold the low 24 bits of the right record. Packing nodes into 7 bytes instead of 8 bytes shrinks the search tree section by 12.5%, saving megabytes of L3 CPU cache.

How a reader interprets each record value (value vs node_count)

When a reader inspects bit i of the IP address and extracts the corresponding Left or Right integer R from the current node, three cases exist:

1. R < node_count: R is the index of the next internal node in the search tree. The reader advances to byte offset R * node_byte_size and inspects bit i + 1.
2. R == node_count: Explicit "not found" sentinel (for example, RFC 1918 private space 10.0.0.0/8 or unallocated bogon blocks). The walk terminates immediately in O(prefix_len) steps.
3. R > node_count: Leaf match! The prefix terminates here, and the byte offset of its intelligence payload inside the Data Section is computed directly as:

data_section_offset = (R - node_count) - 16

Because node_count is known when the file is opened, the reader never has to dereference a leaf node to check whether it is an internal node or a data pointer. A single integer comparison (R > node_count) ends the tree walk and yields the exact byte offset in the Data Section.

Unified IPv4 and IPv6 tree embedding

Every ipscope .mmdb file compiles with ip_version: 6 so a single database file resolves both IPv4 and IPv6 addresses without requiring two separate file descriptors.

To avoid walking 96 redundant zero bits on every IPv4 lookup, standard MMDB readers cache the tree node index reached after walking 96 zero bits (::/96 or ::ffff:0:0/96) at startup when Open() is called. When you pass an IPv4 address (32 bits) to Lookup(), the reader starts directly at the cached IPv4 root node and walks at most 24 to 32 steps (for /24 to /32 prefixes), completing the entire search tree traversal in fewer than 30 L1/L3 cache line reads.

#How our compiler shrinks 35.3M prefixes into 141 MB

If you naively serialized 35.3 million JSON-like maps into a binary file without structural optimization, the resulting database would exceed 8.4 gigabytes. Putting an 8.4 GB file on edge servers would blow past container memory limits and thrash the Linux page cache.

The ipscope MMDB compiler applies three aggressive optimization passes before emitting a single byte to disk:

Optimization 1: Sibling prefix coalescing (Bottom-Up Trie Pruning)

In raw BGP and RIR telemetry, adjacent /24 blocks often carry identical intelligence payloads. For example, if a cloud provider announces 198.51.100.0/24 and 198.51.101.0/24 from the exact same Frankfurt data center (AS24940), and neither subnet has distinct abuse velocity or scanner flags, keeping two separate /24 leaves wastes an internal tree node.

During compilation, our bottom-up trie coalescer computes a 128-bit cryptographic structural hash (XXH3_128) of the resolved payload at every leaf. In a post-order traversal:

• If node.left is a leaf and node.right is a leaf, and hash(node.left.payload) == hash(node.right.payload), the compiler collapses node itself into a single parent leaf (cidr_len - 1) and frees both child nodes.
• This recursive coalescing collapses millions of contiguous residential and cloud sub-allocations into their natural supernets without losing a single bit of intelligence fidelity.

Optimization 2: Canonical DAG deduplication in the Data Section

Even when two prefixes are geographically separated in the binary search tree (for example, AS13335 announcing 104.16.0.0/24 and 172.64.0.0/24), their network, connection, security, and geo records are frequently identical or share identical sub-maps.

Instead of emitting duplicate maps into the Data Section, our compiler builds a Directed Acyclic Graph (DAG) of every value in the database:

1. Whole-Record Sharing: If 12,400 distinct prefixes across the globe resolve to the exact same tuple of (geo, network, connection, security, security_extensions), the compiler writes that top-level map into the Data Section once. All 12,400 leaf records in the Binary Search Tree point to the exact same R = node_count + 16 + offset integer.
2. Sub-Map and String Interning: Even when two prefixes in Frankfurt differ in network.cidr or security_extensions.abuse_velocity_score, they still share the same geo map (Frankfurt am Main, Hesse, DE, Europe/Berlin) and the same network.organization string. The MMDB v2.0 specification supports a native Pointer control byte (type = 1) that encodes a 1-byte, 2-byte, 3-byte, or 4-byte offset back into the Data Section.
3. Frequency-Sorted Offset Placement: In MMDB v2.0, the smallest pointer representation (2 bytes total: 1 control byte + 1 offset byte) can address offsets 0 through 2,047 at the very beginning of the Data Section, and a 3-byte pointer addresses offsets 2,048 through 526,335. Our compiler sorts all interned strings and shared sub-maps by reference frequency in descending order before laying out the Data Section. The top 500 most referenced strings ("US", "DE", "residential", "hosting", "none", "America/New_York") and top ASN maps are guaranteed to sit in the first 2,048 bytes of the Data Section, so every reference to them across the entire file costs only 2 bytes.

#Decoding only what you need in Go (maxminddb-golang)

One of the biggest performance advantages of .mmdb over JSON or CSV is lazy partial decoding. When you pass a Go struct to db.Lookup(ip, &record), the reader walks the binary map keys and only decodes the fields present on your struct, skipping over nested maps you do not need.

package main

import (
	"fmt"
	"log"
	"net"
	"time"

	"github.com/oschwald/maxminddb-golang"
)

// EdgePolicyRecord decodes only the exact fields needed by our edge WAF,
// skipping city coordinates, postal codes, and extended metadata.
type EdgePolicyRecord struct {
	Geo struct {
		CountryCode         string  `maxminddb:"country_code"`
		DeclaredCountryCode string  `maxminddb:"declared_country_code"`
		PhysicalCountryCode string  `maxminddb:"physical_country_code"`
		IsVirtualLocation   bool    `maxminddb:"is_virtual_location"`
		Confidence          float64 `maxminddb:"confidence"`
	} `maxminddb:"geo"`
	Network struct {
		ASN          uint32 `maxminddb:"asn"`
		Organization string `maxminddb:"organization"`
	} `maxminddb:"network"`
	Connection struct {
		ConnectionType string `maxminddb:"connection_type"`
		IsMobile       bool   `maxminddb:"is_mobile"`
	} `maxminddb:"connection"`
	Security struct {
		IsVPN          bool     `maxminddb:"is_vpn"`
		IsTorExit      bool     `maxminddb:"is_tor_exit"`
		IsProxy        bool     `maxminddb:"is_proxy"`
		IsHosting      bool     `maxminddb:"is_hosting"`
		AnonymizerType string   `maxminddb:"anonymizer_type"`
		ReasonCodes    []string `maxminddb:"reason_codes"`
	} `maxminddb:"security"`
}

func main() {
	db, err := maxminddb.Open("/var/lib/ipscope/ipscope-complete.mmdb")
	if err != nil {
		log.Fatalf("failed to open mmdb: %v", err)
	}
	defer db.Close()

	ip := net.ParseIP("198.51.100.7")
	start := time.Now()

	var rec EdgePolicyRecord
	network, ok, err := db.LookupNetwork(ip, &rec)
	if err != nil {
		log.Fatalf("lookup failed: %v", err)
	}
	elapsed := time.Since(start)

	if ok {
		fmt.Printf(
			"Matched=%s Physical=%s Declared=%s ASN=AS%d (%s) VPN=%v Elapsed=%s\n",
			network.String(),
			rec.Geo.PhysicalCountryCode,
			rec.Geo.DeclaredCountryCode,
			rec.Network.ASN,
			rec.Network.Organization,
			rec.Security.IsVPN,
			elapsed,
		)
	}
}

Notice the use of db.LookupNetwork(ip, &rec): in addition to decoding the struct fields, LookupNetwork returns the exact *net.IPNet prefix leaf (198.51.100.0/24) where the radix tree walk terminated, using the depth counter i from the bit walk at zero extra cost.

#Zero-copy deserialization in Rust (maxminddb + serde)

In Rust, because the .mmdb file is mapped into memory for the lifetime of the Reader<Mmap>, you can borrow string slices (&'a str) directly from the underlying mmap pages without allocating String buffers on the heap:

use maxminddb::Reader;
use serde::Deserialize;
use std::net::IpAddr;

#[derive(Deserialize, Debug)]
pub struct GeoBlock<'a> {
    #[serde(borrow)]
    pub country_code: &'a str,
    #[serde(borrow)]
    pub declared_country_code: &'a str,
    #[serde(borrow)]
    pub physical_country_code: &'a str,
    pub is_virtual_location: bool,
    pub accuracy_radius_km: Option<u32>,
}

#[derive(Deserialize, Debug)]
pub struct SecurityBlock<'a> {
    pub is_vpn: bool,
    pub is_tor_exit: bool,
    pub is_proxy: bool,
    pub is_hosting: bool,
    #[serde(borrow)]
    pub anonymizer_type: &'a str,
}

#[derive(Deserialize, Debug)]
pub struct IpScopeLookup<'a> {
    #[serde(borrow)]
    pub geo: GeoBlock<'a>,
    #[serde(borrow)]
    pub security: SecurityBlock<'a>,
}

pub fn lookup_zero_copy<'a>(
    reader: &'a Reader<memmap2::Mmap>,
    ip: IpAddr,
) -> Result<Option<IpScopeLookup<'a>>, maxminddb::MaxMindDBError> {
    reader.lookup::<IpScopeLookup<'a>>(ip)
}

With #[serde(borrow)], country_code, declared_country_code, physical_country_code, and anonymizer_type are constructed as fat pointers (ptr, len) pointing directly into the read-only OS page cache of the .mmdb file. A single core on an AMD EPYC or Apple M-series processor executes over 1.8 million zero-copy lookups per second.

#Custom field-family pruning: Sizing your .mmdb for L3 CPU cache

Modern server CPUs (such as AMD EPYC Genoa/Turin or AWS Graviton4) feature 32 MB to 128 MB of L3 cache per CCD/socket. When the active working set of your .mmdb Binary Search Tree fits entirely inside L3 cache, pointer walks never stall on main DDR5 DRAM.

Not every workload needs all six intelligence families on every node. For example, an edge DDoS scrubber or CDN routing tier might only need geo_country, network, and anonymity (is_vpn, is_tor_exit, is_proxy), while leaving geo_city coordinates and security_extensions to backend analytics.

When you configure a custom database SKU in the ipscope Custom Database Builder, our compiler strips unselected field families before running Stage 1 (Bottom-Up Trie Coalescing). Because removing city-level subdivision splits allows thousands of adjacent /24 blocks within the same country and ASN to coalesce into /18 and /16 supernets, pruning unused families shrinks both the Data Section and the Binary Search Tree node_count:

#Frequently asked questions (FAQ)

Can I open ipscope .mmdb files with the standard maxminddb libraries in Python, Go, Rust, PHP, Java, and NGINX?

Yes. Every ipscope .mmdb artifact is 100% compliant with the open MMDB v2.0 binary specification. Any library or web server module that reads standard .mmdb files (maxminddb-golang, Rust maxminddb, Python maxminddb, Node maxmind, Java maxmind-db, and ngx_http_geoip2_module) can load ipscope .mmdb files directly. Only the decoded map keys follow the richer ipscope schema (geo, network, connection, security, security_extensions).

If I run 16 worker processes on the same Linux host, does opening a 141 MB .mmdb file use 16 * 141 MB = 2.25 GB of physical RAM?

No. When multiple processes open the same inode on disk using mmap(2) with MAP_SHARED or MAP_PRIVATE (read-only), the Linux kernel maps the exact same physical page-cache frames into each process's virtual page table. Across 16 NGINX, Go, or Python worker processes on one host, the physical RAM consumed by the .mmdb file is still only 141 MB total.

Should I call mlock(2) or pre-fault the .mmdb pages at startup?

On latency-critical edge proxies where the very first requests after process startup must not incur disk page faults, passing MAP_POPULATE (or sequentially reading the file into /dev/null during your container readiness probe before marking the pod healthy) pre-faults all .mmdb pages into the OS page cache so even cold-start lookups execute in under a microsecond.