# ipscope - Complete Technical & Product Reference Corpus (llms-full.txt)

Canonical Origin: https://ipscope.io
API Base URL: https://api.ipscope.io/v1
Documentation Corpus: https://ipscope.io/docs/llms-full.txt

> ipscope is an IPv4 and IPv6 network intelligence platform providing sub-millisecond REST API lookups and downloadable databases (`.mmdb`, `.csv.gz`, `.parquet`). Every IP lookup resolves physical vs. declared geolocation, BGP origin ASN, connection classification (residential, mobile/CGNAT, business, hosting), and active anonymizer/threat flags (VPN, proxy, Tor exit, datacenter, abuse velocity, internet scanner).

---

## 1. Complete Field Dictionary (JSON API & Offline Databases)

- **Country geolocation** (`group: country`, JSON block `geo`, backend family `geo_country`):
  - **Fields**: `country_code, declared_country_code, physical_country_code, is_virtual_location, location_type, confidence`
  - **Description**: Physical egress country versus declared registration country, with calibrated confidence and virtual-location detection when routing crosses borders.
- **City geolocation** (`group: city`, JSON block `geo`, backend family `geo_city`):
  - **Fields**: `city, subdivision, postal_code, latitude, longitude, accuracy_radius_km, timezone`
  - **Description**: Subdivision, city, postal code, coordinates, IANA time zone and accuracy radius in kilometers, separating verified metro POPs from wide-area pools.
- **Network & BGP ASN** (`group: asn`, JSON block `network`, backend family `network`):
  - **Fields**: `matched_network, asn, organization, bgp_prefix_length, allocation_status`
  - **Description**: Autonomous System Number, operator organization and the most-specific matched IPv4 or IPv6 CIDR prefix from the global BGP routing table.
- **Connection context** (`group: company`, JSON block `connection`, backend family `connection`):
  - **Fields**: `connection_type, is_residential, is_mobile, is_business, is_hosting`
  - **Description**: How the address connects to the internet: residential broadband, cellular mobile carrier, enterprise business, cloud hosting, education or government.
- **Anonymity & hosting** (`group: privacy`, JSON block `security`, backend family `anonymity`):
  - **Fields**: `is_vpn, is_proxy, is_tor_exit, is_hosting, anonymizer_type, confidence, reason_codes`
  - **Description**: Calibrated detection of VPN exits, open proxies, Tor exit nodes, private relays and cloud datacenters, backed by deterministic reason codes.
- **Threat & abuse velocity** (`group: abuse`, JSON block `security_extensions`, backend family `security_extensions`):
  - **Fields**: `abuse_velocity_score, is_known_scanner, observation_recency_hours`
  - **Description**: 30-day abuse velocity score, active internet scanner detection, and observation recency compiled from global threat and blocklist feeds.

---

## 2. API Plans & Rate Limits

Every plan includes a monthly lookup allowance. Keep your Prepaid Micro-USD Wallet at $0 and lookups past your quota fail closed with zero surprise bills, or top up your wallet to meter extra lookups automatically at $0.10 per 1,000 requests.
Overages rate: $0.10 / 1,000 lookups.

| Plan | Monthly Price | Monthly Quota | Sustained RPS | Burst | Intelligence Blocks | Batch Endpoint | SLA |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **Free** | Free ($0) | 30K | 5 req/s | 20 | country, asn | No | Standard |
| **Starter** | $12/mo | 250K | 10 req/s | 50 | country, city, asn, company | No | Standard |
| **Pro** | $39/mo | 2M | 30 req/s | 150 | country, city, asn, company, privacy, abuse | Yes (100 IPs/call) | 99.9% |
| **Business** | $99/mo | 10M | 100 req/s | 500 | country, city, asn, company, privacy, abuse | Yes (100 IPs/call) | 99.95% |
| **Enterprise** | $299/mo | 50M | 500 req/s | 2500 | country, city, asn, company, privacy, abuse | Yes (100 IPs/call) | 99.99% |

---

## 3. Downloadable Database Editions (`.mmdb`, `.csv.gz`, `.parquet`)

All database subscriptions include `.mmdb` (128-bit binary radix trie v2.0), `.csv.gz` (CIDR + start/end IPs), and `.parquet` (Zstd columnar blocks) compiled from the same 35.3M+ prefix state.

- **MMDB** (`mmdb`): Zero-copy binary tree for standard MMDB v2.0 readers in Rust, Go, Node, Python & NGINX
- **CSV** (`csv`): Canonical CIDR prefixes with start/end IPs, gzip-compressed (.csv.gz)
- **Parquet** (`parquet`): Zstd columnar blocks for SQL engines, OLAP data warehouses & SIEM pipelines

- **Monthly Cadence** (`monthly`): A new build every 30 days
- **Weekly Cadence** (`weekly`): A new build every 7 days
- **Daily Cadence** (`daily`): A new build every night by 06:00 UTC

### ipscope Lite (`IPScope-Lite-v1`)
- **Pricing**: Free (CC BY 4.0)
- **Included Families**: Country geolocation, Network & BGP ASN
- **Summary**: Country and BGP origin ASN for every routed IPv4 & IPv6 prefix. Free with attribution.

### IP to Country (`IPScope-Country-v1`)
- **Pricing**: $15/mo (monthly), $19/mo (weekly), $25/mo (daily)
- **Included Families**: Country geolocation
- **Summary**: Physical and declared country codes, location type, virtual-location flag and confidence.

### IP to ASN & Connection (`IPScope-ASN-Connection-v1`)
- **Pricing**: $22/mo (monthly), $29/mo (weekly), $45/mo (daily)
- **Included Families**: Network & BGP ASN, Connection context
- **Summary**: BGP origin ASN, organization, prefix length and residential/mobile/hosting connection context.

### IP to City Geolocation (`IPScope-City-v1`)
- **Pricing**: $45/mo (monthly), $65/mo (weekly), $85/mo (daily)
- **Included Families**: Country geolocation, City geolocation
- **Summary**: Country down to city: subdivision, postal code, coordinates, accuracy radius and IANA timezone.

### City + ASN + Connection (`IPScope-City-ASN-v1`)
- **Pricing**: $59/mo (monthly), $85/mo (weekly), $109/mo (daily)
- **Included Families**: Country geolocation, City geolocation, Network & BGP ASN, Connection context
- **Summary**: Full city geolocation, BGP origin ASN and connection type unified on the same 35.3M+ prefixes.

### Privacy, VPN & Hosting (`IPScope-Privacy-v1`)
- **Pricing**: $95/mo (weekly), $139/mo (daily)
- **Included Families**: Anonymity & hosting
- **Summary**: Verified VPN, proxy, Tor exit, relay and datacenter flags with confidence and deterministic reason codes.

### Complete Mega Database (`IPScope-Enterprise-Mega-v1`)
- **Pricing**: $189/mo (weekly), $279/mo (daily)
- **Included Families**: Country geolocation, City geolocation, Network & BGP ASN, Connection context, Anonymity & hosting, Threat & abuse velocity
- **Summary**: All 6 intelligence blocks: Country, City, ASN, Connection, Anonymity and Threat/Abuse velocity.

---

## 4. Product & Architecture Specifications

### Declared vs. physical IP geolocation built on routing reality (https://ipscope.io/products/geolocation)
Standard registration-only databases report the country where a block was registered years ago. ipscope separates declared registration country from physical BGP and RTT-triangulated location across 35.3M+ routed IPv4 and IPv6 prefixes.

**Key Metrics:**
- **Routed prefixes**: 35.3M+ (Unified IPv4 & IPv6 LPM tree)
- **Country precision**: 99.8% (Verified against active BGP & RTT)
- **Geofeed validation**: RFC 8805 (RPKI & origin-ASN corroborated)
- **Lookup latency**: < 1 µs (Local MMDB or sub-ms edge API)

#### Three country fields on every lookup: effective, declared, and physical [geo_country family]
Cloud operators and cross-border ISPs frequently lease IPv4/IPv6 ranges registered in one jurisdiction and announce them from a completely different continent. Every ipscope response exposes both the declared paperwork country and the physical routing country.

- country_code: Effective ISO 3166-1 alpha-2 country code for localization, tax, and compliance.
- declared_country_code: Country recorded in network allocation records or RFC 8805 geofeed.
- physical_country_code: Country where active BGP announcements and multi-vantage RTT probes place the router.
- is_virtual_location: Boolean flag set to true whenever declared and physical countries diverge.

#### City, subdivision, coordinates, timezone, and calibrated accuracy radius [geo_city family]
When geo_city is licensed on your API plan or custom database SKU, lookups include first-level administrative subdivisions, city names, postal codes, WGS84 coordinates, IANA timezones, and a realistic accuracy radius in kilometers.

- Honest accuracy_radius_km: 5 to 25 km for fixed FTTH/cable broadband, 50 to 250 km for cellular CGNAT pools.
- IANA timezone identifier (such as America/New_York or Europe/Berlin) for zero-friction scheduling.
- Reverse DNS IATA/UNLOCODE router PoP hints cross-checked against speed-of-light RTT constraints.

#### Geolocation field catalog
Fields returned in the geo object across the REST API and downloadable MMDB/CSV/Parquet databases.

| Field | Type | Family | Description |
| :--- | :--- | :--- | :--- |
| geo.country_code | string | null | geo_country | Effective ISO 3166-1 alpha-2 country code. |
| geo.declared_country_code | string | null | geo_country | Country registered in network allocation records or operator geofeed. |
| geo.physical_country_code | string | null | geo_country | Country verified by BGP routing and RTT triangulation. |
| geo.is_virtual_location | boolean | geo_country | True when declared_country_code differs from physical_country_code. |
| geo.location_type | string | geo_country | Resolution mode: physical, declared, virtual, anycast, or unknown. |
| geo.subdivision | string | null | geo_city | First-level state, province, or region name. |
| geo.city | string | null | geo_city | Resolved city name. |
| geo.accuracy_radius_km | integer | null | geo_city | Estimated radius in kilometers around the city centroid. |
| geo.timezone | string | null | geo_city | IANA time zone identifier. |

### BGP origin ASN, longest-matching CIDR prefix, and access network classification (https://ipscope.io/products/network)
Map any IPv4 or IPv6 address to its exact routed CIDR block, autonomous system number, canonical operator organization, and access network type (residential, mobile, business, or cloud hosting).

**Key Metrics:**
- **Active ASNs**: 118,000+ (Full DFZ IPv4 & IPv6 BGP table)
- **RIB ingestion**: Every 15m (Global route collector feeds)
- **Connection types**: 7 classes (Residential, mobile, hosting, business & more)
- **CIDR endpoint**: /v1/networks (Direct prefix & mask inspection)

#### Real-time BGP origin ASN and canonical operator naming [network family]
Instead of relying on static allocation netblocks that ignore sub-allocations and BGP more-specifics, ipscope builds its Radix/LPM tree directly from global BGP RIB dumps and RPKI Route Origin Authorizations.

- matched_network: The exact longest-matching routed CIDR prefix (/8 to /32 for IPv4, /16 to /128 for IPv6).
- network.asn: Origin Autonomous System Number normalized as an integer.
- network.organization: Canonical operator name cleaned of registry boilerplate.

#### Distinguish consumer broadband and cellular CGNAT from cloud datacenters [connection & hosting families]
Knowing the ASN is only half the story: large operators run both consumer FTTH broadband and enterprise cloud hosting under the same umbrella. Our prefix-level classifier labels each subnet by its actual traffic profile.

- connection_type: residential, mobile, business, hosting, education, government, or unknown.
- Boolean convenience flags: is_residential, is_mobile, is_business, and is_hosting.
- Identifies 4G/5G cellular carrier gateways so you never rate-limit thousands of mobile users sharing one CGNAT IP.

#### Network & connection schema
Fields included in the network and connection intelligence blocks.

| Field | Type | Family | Description |
| :--- | :--- | :--- | :--- |
| matched_network | string | null | network | Longest-matching routed CIDR prefix in the active snapshot. |
| network.asn | integer | null | network | BGP origin Autonomous System Number. |
| network.organization | string | null | network | Canonical autonomous system or network operator name. |
| connection.connection_type | string | connection | Primary access classification (residential, mobile, business, hosting, etc.). |
| connection.is_residential | boolean | connection | True for consumer FTTH, cable, and DSL broadband ISPs. |
| connection.is_mobile | boolean | connection | True for cellular carrier gateways (4G/5G CGNAT pools). |
| connection.is_business | boolean | connection | True for corporate leased lines and enterprise networks. |
| connection.is_hosting | boolean | hosting | True for cloud providers, colocation facilities, and VPS ranges. |

### Explainable VPN, proxy, Tor, and residential relay detection with reason codes (https://ipscope.io/products/privacy)
Detect commercial VPN exits, open SOCKS/HTTP proxies, Tor exit relays, datacenter hosting proxies, and high-velocity scanners with deterministic evidence codes you can log and audit.

**Key Metrics:**
- **Anonymizer classes**: 9 types (VPN, proxy, Tor, private relay & more)
- **Evidence codes**: Deterministic (reason_codes array on every match)
- **Abuse velocity**: 0 to 100 (Active honeypot & scanner scoring)
- **False-positive SLA**: < 24h review (Direct POST /v1/feedback pipeline)

#### Four boolean flags plus specific anonymizer_type and reason_codes [anonymity family]
Opaque risk scores make it impossible to explain why a legitimate customer was challenged. ipscope pairs calibrated confidence scores with explicit boolean flags and machine-readable reason_codes.

- is_vpn: Active commercial or self-hosted VPN exit nodes verified via WireGuard, OpenVPN, and IKEv2 handshakes.
- is_tor_exit: Synchronized hourly against the active Tor exit consensus.
- is_proxy & is_hosting: Open proxies, residential proxy relays, and cloud/VPS ranges.
- anonymizer_type: Distinguishes consumer privacy relays (private_relay) from high-risk anonymizers.

#### Honeypot abuse velocity and internet-wide scanner recency [security_extensions family]
For fraud engines, WAF rules, and SIEM enrichment pipelines that need active threat telemetry beyond static anonymizer lists, security_extensions tracks live hostile traffic observations.

- abuse_velocity_score: Normalized 0 to 100 score derived from credential-stuffing and darknet honeypot hits.
- is_known_scanner: Flags addresses actively sweeping TCP/UDP ports across our sensor network.
- observation_recency_hours: Hours elapsed since the most recent confirmed anonymizer or scanner packet.

#### Security & threat intelligence fields
Fields returned in security and security_extensions.

| Field | Type | Family | Description |
| :--- | :--- | :--- | :--- |
| security.is_vpn | boolean | anonymity | True when the IP is an active commercial or self-hosted VPN exit node. |
| security.is_tor_exit | boolean | anonymity | True when listed in the active Tor exit relay consensus. |
| security.is_proxy | boolean | anonymity | True for open HTTP/SOCKS proxies and residential proxy relays. |
| security.is_hosting | boolean | hosting | True when routed by a cloud, VPS, or colocation provider ASN. |
| security.anonymizer_type | string | anonymity | vpn, proxy, tor_exit, tor_relay, hosting_proxy, apple_private_relay, or none. |
| security.reason_codes | string[] | anonymity | Deterministic evidence codes (e.g. vpn_handshake_confirmed). |
| security_extensions.abuse_velocity_score | number | null | security_extensions | Normalized 0 to 100 abuse velocity score. |
| security_extensions.is_known_scanner | boolean | null | security_extensions | True when observed performing internet-wide port scanning. |
| security_extensions.observation_recency_hours | integer | null | security_extensions | Hours since the most recent active security observation. |

### Sub-millisecond IPv4 & IPv6 intelligence over HTTP/2 and HTTP/3 (https://ipscope.io/products/api)
Query a single IP, resolve your visitor’s caller IP without a key, or enrich 100 addresses per batch request pinned to an atomic Radix/LPM snapshot. Backed by 8 official zero-bloat server SDKs.

**Key Metrics:**
- **In-memory LPM p50**: 2.1 µs (Zero database I/O on the hot path)
- **Batch capacity**: 100 IPs / call (Pinned to a single snapshot version)
- **Official SDKs**: 8 languages (Built-in O(1) LRU cache & streaming)
- **Overflow metering**: $0.10 / 1K (Optional prepaid micro-USD wallet)

#### Single IP, caller IP, atomic 100-IP batch, and CIDR network inspection [LookupController]
Every API endpoint is served from an in-memory Radix/LPM routing snapshot with live BGP and geofeed enrichment on cache misses. Field projection happens automatically based on your API key’s licensedFamilies.

- GET /v1/ip/{ip}: Resolve any IPv4 or IPv6 address in sub-millisecond memory.
- GET /v1/ip/me: Keyless, proxy-aware caller IP lookup for CLI tools and edge diagnostics.
- POST /v1/ip/batch: Resolve up to 100 IPs in one round-trip with pre-debit validation.
- GET /v1/networks/{prefix}/{length}: Inspect canonical routing prefix records by CIDR.

#### REST API v1 endpoint summary
All endpoints are served over HTTPS from https://api.ipscope.io.

| Method | Endpoint | Auth Scope | Metering | Description |
| :--- | :--- | :--- | :--- | :--- |
| GET | /v1/ip/{ip} | lookup | 1 credit | Single IPv4 or IPv6 lookup with live BGP enrichment-on-miss. |
| GET | /v1/ip/me | Public (None) | Free (120/min) | Proxy-aware lookup of the connecting client IP. |
| POST | /v1/ip/batch | batch | 1 credit / IP | Batch lookup of 1 to 100 IPv4/IPv6 addresses. |
| GET | /v1/networks/{prefix}/{length} | lookup | Included | Canonical CIDR network prefix inspection. |
| GET | /v1/metadata/datasets | lookup | Included | Active and rollback snapshot versions and SHA-256 digests. |
| POST | /v1/feedback | lookup | Free | Submit geolocation or VPN false-positive corrections. |

### Zero-latency local IP intelligence in MMDB, CSV.gz, and Apache Parquet (https://ipscope.io/products/databases)
Run lookups inside your own application memory in under a microsecond with zero network hops and zero per-query fees. Choose from 7 ready-made editions or compile a custom field-family SKU.

**Key Metrics:**
- **Formats included**: 3 formats (MMDB, CSV.gz & Parquet on every build)
- **MMDB lookup time**: < 1 µs (Open MMDB v2.0 binary format)
- **Server licensing**: 1 server (Included (+$149/mo per extra server))
- **Rollback retention**: 30 builds (Immutable SHA-256 verified artifacts)

#### Seven preset editions or a custom build compiled to your exact schema [Preset & Custom SKUs]
Every database subscription compiles three synchronized files (.mmdb, .csv.gz, and .parquet) from the exact same verified snapshot that powers our real-time API. Pick a preset edition or use the Custom Database Builder to include only the field families your workload requires.

- Open MMDB v2.0 compatibility: open .mmdb files in standard C, Python, Go, Rust, Node.js, or Java readers.
- Vectorized analytics: query .parquet directly in columnar SQL engines and analytical data warehouses.
- Atomic updates: ETag (304 Not Modified), HTTP Range resume, and SHA256SUMS manifests.

#### Preset database editions
All editions include both IPv4 and IPv6 in a single unified file.

| Edition | Database Type | Included Families | Cadence | Starting Price |
| :--- | :--- | :--- | :--- | :--- |
| ipscope Lite | IPScope-Lite-v1 | Country + ASN | Monthly | Free (CC BY 4.0) |
| Country | IPScope-Country-v1 | geo_country | Monthly / Weekly / Daily | $15/mo |
| ASN & Network | IPScope-ASN-v1 | network | Monthly / Weekly / Daily | $22/mo |
| City Geolocation | IPScope-City-v1 | geo_country + geo_city | Monthly / Weekly / Daily | $45/mo |
| City + ASN | IPScope-City-ASN-v1 | geo_country + geo_city + network | Monthly / Weekly / Daily | $59/mo |
| Privacy, VPN & Hosting | IPScope-Privacy-v1 | geo_country + network + connection + hosting + anonymity | Weekly / Daily | $95/mo |
| Enterprise Mega | IPScope-Complete-v1 | All 6 intelligence families | Daily | $199/mo |

### Free Country + ASN database for IPv4 and IPv6, updated monthly (https://ipscope.io/products/lite)
ipscope Lite (IPScope-Lite-v1) packages our verified country geolocation and BGP origin ASN intelligence into a free monthly MMDB, CSV.gz, and Parquet release under Creative Commons Attribution 4.0.

**Key Metrics:**
- **License**: CC BY 4.0 (Free for commercial & open-source use)
- **IP families**: IPv4 + IPv6 (Unified dual-stack routing tree)
- **Formats**: MMDB · CSV · Parquet (All 3 formats included free)
- **Rebuild cadence**: Monthly (Published on the 1st of each month)

#### Production-grade Country and ASN resolution with zero cost [IPScope-Lite-v1]
Unlike crippled trial files that strip IPv6 or cap prefix depth, ipscope Lite ships the full routed IPv4 and IPv6 BGP table with declared vs. physical country codes and origin ASN metadata.

- Includes geo.country_code, geo.declared_country_code, geo.physical_country_code, and geo.is_virtual_location.
- Includes matched_network, network.asn, and network.organization.
- Simple attribution requirement: credit ipscope with a link to https://ipscope.io in your documentation or footer.

#### ipscope Lite vs. Paid Commercial Editions
Upgrade seamlessly without changing a single line of MMDB reader code.

| Capability | ipscope Lite (Free) | Commercial Editions |
| :--- | :--- | :--- |
| Country & Virtual Location (geo_country) | Included | Included |
| BGP Origin ASN & Organization (network) | Included | Included |
| City, Subdivision, Lat/Lon & Timezone (geo_city) | Not included | Included in City & Complete |
| Connection Type & Hosting (connection / hosting) | Not included | Included in Privacy & Complete |
| VPN, Tor, Proxy & Reason Codes (anonymity) | Not included | Included in Privacy & Complete |
| Rebuild Cadence | Monthly (every 30 days) | Daily, Weekly, or Monthly |
| Attribution Requirement | Required (CC BY 4.0 link) | Not required |

### Multi-source BGP, allocation, geofeed, and active RTT verification pipeline (https://ipscope.io/methodology)
Every ipscope snapshot is compiled from five independent measurement streams and must beat the active release on a disjoint ground-truth holdout set before promotion.

**Key Metrics:**
- **Signal sources**: 5 streams (BGP, Allocations, Geofeed, rDNS, Active RTT)
- **BGP collectors**: 40+ peers (Global route collector full-table feeds)
- **Holdout gate**: Automated (Blocks regressions before publication)
- **Snapshot swap**: Zero-downtime (Atomic pointer swap with SHA-256 check)

#### Corroborating registration declarations against physical network physics [5-Stage Compilation Pipeline]
No single registration table is authoritative for physical geography. Administrative records list corporate headquarters addresses, while operator geofeeds can contain typos or stale announcements. Our compiler cross-examines every claim against active routing and latency measurements.

- 1. BGP & RPKI Ingestion: Full DFZ IPv4 and IPv6 RIB dumps every 15 minutes establish canonical prefix boundaries and origin ASNs.
- 2. Allocation & Registration Normalization: Global internet delegation tables provide declared country and organization lineage.
- 3. RFC 8805 Geofeed Verification: Operator CSV geofeeds are ingested only when authenticated by netblock remarks and RPKI origin validation.
- 4. Active RTT & rDNS Triangulation: Multi-region probe nodes verify that claimed city locations satisfy speed-of-light latency bounds.
- 5. Holdout Quality Gate: Candidate snapshots are scored against verified ground-truth landmarks and VPN exits before atomic promotion.

#### Signal weighting & conflict resolution
How the compiler resolves disagreements between registry paperwork and active probes.

| Scenario | Declared Signal | Physical Signal | Compiler Resolution |
| :--- | :--- | :--- | :--- |
| Local ISP broadband | Declared country matches RTT probe country | RTT within 15 ms of regional PoP | location_type: "physical", is_virtual_location: false |
| Cross-border cloud subnet | Allocation registered in US | BGP + 2 ms RTT to Frankfurt PoP | declared: "US", physical: "DE", is_virtual_location: true |
| Global CDN / DNS anycast | Multiple PoPs announce same /24 | Low RTT from 3+ continents simultaneously | location_type: "anycast" |
| Unverified geofeed claim | CSV claims Tokyo | RTT from Tokyo > 180 ms, Amsterdam < 4 ms | Geofeed rejected; physical_country_code set to NL |

### Three synchronized artifact formats compiled from every snapshot (https://ipscope.io/databases/formats)
Every database subscription includes binary MMDB for sub-microsecond application lookups, Apache Parquet for vectorized warehouse joins, and gzipped CSV for relational bulk loading.

**Key Metrics:**
- **MMDB v2.0**: < 1 µs (mmap zero-copy binary radix tree)
- **Apache Parquet**: ZSTD columnar (Columnar SQL engines & data warehouses)
- **Gzipped CSV**: RFC 4180 (Relational COPY & bulk load)
- **Checksums**: SHA-256 (Signed SHA256SUMS on every release)

#### Compatible with standard MMDB readers in every language [1. Open MMDB Binary Format (.mmdb)]
Our compiler outputs strict MMDB binary format v2.0 files containing a unified IPv4 and IPv6 search tree and a deduplicated data section. Open the file with mmap and share it across worker processes with zero serialization overhead.

- Works out of the box with standard MMDB readers in C, NGINX, Envoy, Python, Node.js, Rust, Go, and Java.
- Nested dictionary structure matches our REST API JSON response (geo, network, connection, security, security_extensions).

#### Vectorized columnar scans in analytical SQL engines and relational databases [2. Apache Parquet (.parquet) & 3. Gzipped CSV (.csv.gz)]
For SIEM log backfills, data warehouse enrichment, and SQL analytics, nested fields are flattened into typed columns with cidr, ip_version, start_ip, and end_ip keys.

- Parquet files are ZSTD-compressed and row-group sorted by IP family and network start address for fast predicate pushdown.
- CSV.gz files follow strict UTF-8 RFC 4180 formatting for one-line relational COPY ingestion.

#### Artifact format comparison
All three formats are included at no extra cost on every database subscription.

| Format | File Extension | Compression | Primary Use Case | Typical Query Speed |
| :--- | :--- | :--- | :--- | :--- |
| Open MMDB v2.0 | .mmdb | Binary pointer deduplication | Real-time backend APIs, edge workers, reverse proxies | < 1 µs per IP via mmap |
| Apache Parquet | .parquet | ZSTD columnar | Columnar SQL engines & analytical data warehouses | Millions of rows/sec scan |
| Comma-Separated Values | .csv.gz | Gzip (RFC 1952) | Relational database COPY & custom ETL scripts | Sequential stream import |

### Nightly, weekly, or monthly builds with atomic SHA-256 swaps and 30-release rollback (https://ipscope.io/databases/updates)
Pull fresh database snapshots automatically from a stable authenticated URL. Conditional ETag headers, HTTP Range resume, and cryptographic checksums make cron and Kubernetes updates effortless.

**Key Metrics:**
- **Daily publish SLA**: 06:00 UTC (Fresh BGP, geofeed & VPN exit build)
- **Conditional GET**: 304 ETag (Zero bandwidth when file is unchanged)
- **Signed URLs**: GCS V4 HMAC (Resumable Range downloads)
- **Archive window**: 30 snapshots (Pin or roll back to any releaseId)

#### Match your rebuild cadence to how fast your signals change [Daily · Weekly · Monthly]
Country boundaries rarely change overnight, while VPN exit nodes and residential proxy pools rotate continuously. Choose Monthly, Weekly, or Daily rebuilds on any preset or custom database.

- Monthly (Included): Published on the 1st of every month at 06:00 UTC. Ideal for Country and ASN routing analytics.
- Weekly (+$25/mo): Published every Monday at 06:00 UTC. Ideal for City geolocation and ISP classification.
- Daily (+$65/mo): Published every night by 06:00 UTC. Recommended for VPN, proxy, Tor, and fraud prevention.

#### Rebuild cadence comparison
Switch cadences at any time from the dashboard or via PATCH /v1/mmdb/licenses/:id.

| Cadence | API Parameter | Publication Schedule | Monthly Add-On | Best For |
| :--- | :--- | :--- | :--- | :--- |
| Monthly | every_30_days | 1st of every month at 06:00 UTC | Included ($0) | Country localization, tax compliance, ASN reporting |
| Weekly | weekly | Every Monday at 06:00 UTC | +$25/mo | City geolocation, regional content personalization |
| Daily | daily | Every day by 06:00 UTC | +$65/mo | VPN/proxy blocking, fraud scoring, SIEM threat hunting |

### Clear per-server internal licensing with strict non-sharing protection (https://ipscope.io/databases/licensing)
Every commercial database subscription includes a license for 1 internal backend server by default (+$149/mo per additional server). Databases are strictly for internal organizational use and cannot be shared or sold.

**Key Metrics:**
- **Base license**: 1 server (Included in every database subscription)
- **Additional servers**: +$149 / mo (Per extra internal server (seats > 1))
- **Sharing & resale**: Prohibited (Cannot be shared, bundled, or sold)
- **ipscope Lite**: CC BY 4.0 (Free with attribution link)

#### 1 internal server included; scale server seats as your infrastructure grows [Commercial Subscriptions]
A commercial ipscope database subscription grants your organization a non-exclusive, non-transferable license to deploy the compiled .mmdb, .csv.gz, and .parquet artifacts on 1 internal server (seats: 1). If you deploy the database across multiple backend servers or edge instances, license the corresponding number of server seats.

- 1 internal server included: Run unlimited lookups on your licensed server with zero per-query fees.
- Additional internal servers (+$149/mo per server): Add or adjust server seats in the Custom Database Builder or via PATCH /v1/mmdb/licenses/:id.
- Strict non-sharing & non-resale rule: Databases and their raw records may never be shared with third parties, embedded into distributed software/appliances, sublicensed, or resold.

#### Database license rules at a glance
Applies to all preset and custom commercial database subscriptions.

| Rule / Tier | Permitted Scope | Monthly Rate |
| :--- | :--- | :--- |
| Base server license (seats: 1) | Deploy on 1 internal backend server inside your organization | Included |
| Additional servers (seats > 1) | Deploy on additional internal backend servers inside your organization | +$149/mo per extra server |
| Per-lookup / query fees on licensed server | Unlimited local lookups on licensed server seats | $0 (Unmetered locally) |
| External sharing, OEM bundling, or resale | Sharing, sublicensing, bundling into shipped software, or selling the database | Strictly prohibited |

---

## 5. Frequently Asked Questions (FAQ)

### How often is it updated?
The Lookup API always serves our latest promoted build automatically with zero downtime. Downloadable databases (MMDB, CSV.gz and Parquet) are available on Monthly, Weekly, or Daily rebuild schedules, and previous snapshots remain accessible if you ever need to roll back.

### Does it work with my existing MMDB reader code?
Yes. Every .mmdb file uses the standard MMDB binary format v2.0, so you can open it directly with standard open-source MMDB reader libraries in C, NGINX, Go, Node.js, Rust, Python, PHP and Java without changing your application code. Every build also ships as gzipped CSV (.csv.gz) and Apache Parquet (.parquet) for columnar SQL engines, OLAP data warehouses and SIEM pipelines.

### How do automated database updates work on my servers?
You can pull the latest MMDB, CSV.gz or Parquet artifact from a stable authenticated URL on a cron timer or Kubernetes CronJob. Download endpoints support ETag / If-None-Match (returning 304 Not Modified when your local file is already current), HTTP Range resume, and SHA256SUMS verification for atomic zero-downtime swaps.

### Can I look up multiple IPs or CIDR prefixes in one request?
Yes. Starting on the Pro plan, POST /v1/ip/batch resolves up to 100 IPv4 or IPv6 addresses in a single HTTPS call, and GET /v1/network/:cidr lets you inspect any routed CIDR prefix directly.

### What can I do with a database licence?
Every database subscription is licensed for 1 internal backend server inside your organization by default. If you need to deploy the database on more than 1 server, each additional server costs +$149/mo. Databases are strictly for internal server deployment and cannot be shared, bundled into third-party software, sublicensed, or sold. ipscope Lite is free under Creative Commons Attribution 4.0 (CC BY 4.0).

### What happens when I reach my monthly quota?
Every plan includes a monthly lookup allowance. Keep your Prepaid Micro-USD Wallet at $0 and lookups past your quota fail closed with zero surprise bills, or top up your wallet to meter extra lookups automatically at $0.10 per 1,000 requests.

### Is IPv6 covered?
Yes. All API endpoints and all downloadable database files cover both IPv4 and IPv6 in a single unified structure at the same price.

### Can I cancel at any time?
Yes, directly from the client dashboard with one click. Your API keys and database download access remain active through the end of your paid billing period. Annual subscriptions include 2 months free (17% off) and can be switched between monthly and yearly renewal at any time.

---

## 6. Engineering & Research Dispatches (`/blog`)

<!-- Article URL: https://ipscope.io/blog/cgnat-and-5g-mobile-carrier-geolocation-accuracy -->
---
slug: cgnat-and-5g-mobile-carrier-geolocation-accuracy
title: "Why Cellular 5G CGNAT Breaks Naive Rate Limiters and City Geolocation"
description: "How carrier-grade NAT pools share a single public IPv4 address across thousands of mobile subscribers, and how to use connection.is_mobile and accuracy_radius_km in production."
date: "2026-10-18"
readingTime: "14 min read"
category: "Geolocation"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "CGNAT, 5G, Mobile, Geolocation, Accuracy Radius, Rate Limiting"
featured: "false"
---

## Why cellular IP addresses do not behave like fixed home broadband

When backend engineers write IP-based rate limiters, fraud velocity rules, or city-level localization logic, they usually test from a laptop connected to fixed fiber-to-the-home (FTTH) or cable broadband. On fixed residential broadband, the ISP assigns a dedicated `/32` public IPv4 address (or a `/56` IPv6 prefix) to the household router, terminated at a local neighborhood Broadband Network Gateway (BNG). That IP address represents a single household and remains anchored to one city with a physical radius of `5 to 20 kilometers`.

Cellular 4G LTE and 5G Standalone (SA) networks operate on a completely different packet routing architecture. When a smartphone in a regional town connects to a local 5G radio cell tower (`gNodeB`), its packets **do not** egress onto the public internet at that cell tower.

Instead, the radio base station encapsulates the subscriber's IP packets inside a **GPRS Tunneling Protocol (`GTP-U`)** tunnel and backhauls them across the mobile operator's private MPLS/optical core network to a centralized **User Plane Function (`UPF` in 5G)** or **Packet Data Network Gateway (`PGW` in 4G)**. A single nationwide mobile carrier in Germany, the United States, or Japan might operate 40,000 radio cell towers, but only **6 to 15 regional UPF/PGW internet breakout centers** across the entire country!

Even more critically, because mobile carriers have hundreds of millions of active handsets and a limited pool of public IPv4 addresses, the regional UPF/PGW runs **Carrier-Grade NAT (`CGNAT` / `NAT44` / `NAT64`)**, multiplexing **2,000 to 8,000 simultaneous smartphones onto a single public IPv4 address**.

If your backend architecture ignores cellular CGNAT and regional UPF routing, you will inevitably ship three severe production bugs:

1. **Mass mobile customer lockouts during traffic spikes** (collateral damage from per-IP rate limiting).
2. **False "impossible travel" security alerts** when the 5G core re-anchors a subscriber's GTP tunnel between two regional UPFs.
3. **Over-confident city geolocation** (treating a regional mobile gateway in Munich as proof that the user is standing in downtown Munich rather than 120 km away in Nuremberg).

This engineering deep dive explains the packet flow of 4G/5G CGNAT, shows how ipscope models cellular subnets using `connection.is_mobile` and calibrated `geo.accuracy_radius_km`, and provides production Go and TypeScript patterns for CGNAT-safe rate limiting and session security.

## Inside the 4G/5G mobile packet core: Why your server only sees the UPF/PGW

Let us trace a single HTTPS request from a 5G smartphone located in **Augsburg, Germany** to your API server:

```txt
[ Smartphone in Augsburg ]
   |  Internal Carrier IP (100.64.18.42 / RFC 6598 Shared CGNAT Space)
   v
[ Local 5G Cell Tower (gNodeB in Augsburg) ]
   |  Encapsulates packet in GTP-U (UDP Port 2152) over Carrier Core MPLS
   v
[ Regional 5G User Plane Function (UPF) in Munich (80 km away) ]
   |  1. Decapsulates GTP-U header
   |  2. CGNAT translates 100.64.18.42:54112 -> Public IP 203.0.113.88:41029
   |     (Port-Block Allocation: 64 ports per subscriber, ~1,000-4,000 users/IP)
   v
[ Public Internet / DE-CIX Munich Peering ]
   |  Source IP seen by your server: 203.0.113.88
   v
[ Your API Gateway / Load Balancer ]
```

Two physical facts emerge from this packet diagram:

1. **The public internet only ever interacts with the UPF in Munich (`203.0.113.88`)**: Every traceroute, BGP route collector, and active RTT probe on the public internet terminates at the carrier's regional UPF firewall in Munich. The 80 km radio and GTP-U backhaul leg between Augsburg and Munich is completely invisible to Layer 3 internet routing.
2. **Port-Block Allocation (`PBA` / RFC 6888) packs thousands of users onto `203.0.113.88`**: By allocating a deterministic block of 16 to 64 ephemeral source ports (`1024..65535`) per active handset, a single `/32` public IPv4 address on a carrier CGNAT pool serves thousands of unrelated people riding the train, shopping, or browsing your app at the same second.

## How ipscope models cellular CGNAT pools accurately

When you query an IP address belonging to a 4G/5G carrier pool in ipscope, the response explicitly flags the cellular access topology (`connection.connection_type: "mobile"`, `connection.is_mobile: true`) and widens `geo.accuracy_radius_km` to match the true geographic service area of the regional UPF/PGW:

```json
{
  "ip": "203.0.113.88",
  "ip_version": 4,
  "matched_network": "203.0.113.0/24",
  "geo": {
    "country_code": "DE",
    "declared_country_code": "DE",
    "physical_country_code": "DE",
    "is_virtual_location": false,
    "location_type": "physical",
    "confidence": 0.91,
    "subdivision": "Bavaria",
    "city": "Munich",
    "latitude": 48.1351,
    "longitude": 11.582,
    "accuracy_radius_km": 120,
    "timezone": "Europe/Berlin"
  },
  "network": {
    "cidr": "203.0.113.0/24",
    "asn": 3320,
    "organization": "Deutsche Telekom AG",
    "rir": "RIPE"
  },
  "connection": {
    "connection_type": "mobile",
    "is_residential": false,
    "is_mobile": true,
    "is_business": false,
    "is_hosting": false
  },
  "security": {
    "is_vpn": false,
    "is_hosting": false,
    "is_tor_exit": false,
    "is_proxy": false,
    "anonymizer_type": "none",
    "confidence": 0.99,
    "reason_codes": []
  }
}
```

### How the ipscope compiler separates fixed broadband from mobile CGNAT inside the same ISP ASN

Many incumbent telecom operators (such as `Deutsche Telekom AS3320`, `Orange AS3215`, `AT&T`, `Vodafone`, or `NTT`) operate **both** fixed fiber/DSL residential broadband and nationwide 4G/5G cellular networks under the same parent ASN or sibling ASNs. An ASN-only lookup cannot tell whether `203.0.113.0/24` is a fixed FTTH neighborhood router block (`accuracy_radius_km: 12`) or a 5G mobile CGNAT gateway (`accuracy_radius_km: 120`).

Our compiler classifies `/24` IPv4 and `/48` IPv6 subnets within dual-play telecom ASNs using four subnet-level signals:

1. **Carrier rDNS & RPSL `netname` Pool Tokens**: Telecom engineers segregate BNG fixed pools (`dslb-`, `ftth-`, `bng-`, `static-`) from mobile packet gateway pools (`cgn-`, `pgw-`, `ggsn-`, `upf-`, `lte-`, `mobile-`, `gprs-`) in both `PTR` naming templates and RIR `netname`/`descr` attributes.
2. **TCP / IP Stack & Passive MTU/MSS Fingerprinting**: Fixed Ethernet/PPPoE broadband typically negotiates a `1492` or `1500` byte MTU (`TCP MSS 1452` or `1460`), whereas cellular GTP-U encapsulation overhead and mobile radio bearer framing consistently clamp TCP Maximum Segment Size (`MSS`) to cellular-specific values (`1360` to `1420` bytes).
3. **Ephemeral Port Allocation Entropy**: Fixed home routers allocate outbound source ports sequentially or across the full `32768..60999` Linux/macOS ephemeral range for a small number of devices, whereas carrier CGNAT appliances exhibit Port-Block Allocation (`PBA`) stepping across thousands of concurrent TLS sessions.
4. **RTT Variance Distribution**: Even when anchored to a Munich UPF, the tail distribution of radio access network (RAN) jitter across a `/24` cellular pool has a distinct signature compared to a passive optical network (GPON) fiber termination.

### Typical `accuracy_radius_km` across the four `connection_type` classes

| `connection.connection_type` | Boolean Flag | Typical `accuracy_radius_km` | Physical Network Termination Point |
| --- | --- | --- | --- |
| **`hosting`** | `is_hosting: true` | `5` to `10 km` | Exact cloud region or colocation data center facility |
| **`business`** | `is_business: true` | `5` to `15 km` | Dedicated enterprise DIA / leased fiber termination |
| **`residential`** | `is_residential: true` | `10` to `35 km` | Local neighborhood BNG / CMTS serving a city or suburb |
| **`mobile`** | `is_mobile: true` | `80` to `250 km` | Regional 4G/5G Packet Gateway (`PGW` / `UPF`) serving a state or multi-city region |

## Production Pattern 1: CGNAT-safe composite rate limiting in TypeScript

If your API gateway enforces a flat `60 requests / minute per IP` limit, a single busy train station or stadium where 150 users on the same mobile carrier open your app will exhaust the IP bucket in seconds and return `429 Too Many Requests` to every user on that carrier.

To fix this cleanly:

1. For **authenticated requests**, always key your primary rate limiter by `userId` or `sessionId` rather than raw IP.
2. For **unauthenticated endpoints** (catalog browsing, public search, OAuth login initiation), dynamically scale the IP bucket when `connection.is_mobile === true` and combine the IP with client-level entropy (such as an HTTP-only device cookie or TLS `JA4` fingerprint):

```ts
import { IpScopeClient } from '@ipscope/sdk-node';

const ipscope = new IpScopeClient({
  apiKey: process.env.IPSCOPE_KEY!,
  maxCacheEntries: 16_384,
  cacheTtlMs: 300_000,
});

export interface RateLimitConfig {
  bucketKey: string;
  maxRequestsPerMinute: number;
  windowSeconds: number;
}

export async function resolveRateLimitPolicy(
  clientIp: string,
  deviceCookieId: string | undefined,
  tlsJa4Fingerprint: string | undefined,
): Promise<RateLimitConfig> {
  const res = await ipscope.lookupIp(clientIp);
  const { connection, security, security_extensions } = res;
  const velocity = security_extensions?.abuse_velocity_score ?? 0;

  // 1. High-velocity scanner or active proxy: strict per-IP quarantine bucket
  if (velocity >= 75 || security.is_tor_exit || security.is_proxy) {
    return {
      bucketKey: `rl:strict:${clientIp}`,
      maxRequestsPerMinute: 10,
      windowSeconds: 60,
    };
  }

  // 2. Cellular 4G/5G CGNAT pool: partition bucket by (IP + device/TLS sub-key)
  //    and raise the aggregate IP ceiling so shared carrier NAT never starves legitimate users
  if (connection?.is_mobile && !security.is_vpn) {
    const subKey = deviceCookieId || tlsJa4Fingerprint || 'anon';
    return {
      bucketKey: `rl:mobile:${clientIp}:${subKey}`,
      maxRequestsPerMinute: 180,
      windowSeconds: 60,
    };
  }

  // 3. Standard residential or business broadband: per-IP bucket
  return {
    bucketKey: `rl:std:${clientIp}`,
    maxRequestsPerMinute: 120,
    windowSeconds: 60,
  };
}
```

## Production Pattern 2: Uncertainty-aware "Impossible Travel" detection in Go

Many enterprise IAM and zero-trust authentication systems run an **Impossible Travel** detector: if a user's session token is used from City A at `10:00:00` and from City B (`350 km` away) at `10:02:00`, the system computes an apparent travel speed of `10,500 km/h` (faster than a commercial jet) and revokes the user's session.

On 4G/5G mobile networks, this naive calculation causes constant false logouts. Why?

- A user is sitting on a high-speed train or driving between two cities, or the mobile carrier's core network performs a routine load-balancing handoff from the **Frankfurt UPF** (`accuracy_radius_km: 150`) to the **Cologne/Dusseldorf UPF** (`accuracy_radius_km: 140`).
- Even worse, if the user switches from Wi-Fi (`residential`, `accuracy_radius_km: 15`) to 5G cellular (`mobile`, `accuracy_radius_km: 180`) on the exact same phone, the IP centroid jumps `160 km` in `2 seconds`!

Instead of measuring the distance between the two point centroids `(lat_1, lon_1)` and `(lat_2, lon_2)`, you must compute the **minimum physically required distance between their uncertainty circles** by subtracting both `accuracy_radius_km` values from the Haversine great-circle distance:

```txt
D_effective = max(0, D_haversine(p1, p2) - R1 - R2)
```

If the two uncertainty circles overlap (`D_haversine <= R1 + R2`), then `D_effective = 0 km`, proving that the user could physically be in the exact same spot while switching between Wi-Fi and 5G or experiencing a carrier UPF handoff!

Here is the complete Go implementation:

```go
package sessionsec

import (
	"math"
	"time"
)

type GeoObservation struct {
	ObservedAt       time.Time
	CountryCode      string
	Latitude         float64
	Longitude        float64
	AccuracyRadiusKm float64
	IsMobile         bool
	IsVPN            bool
}

// EffectiveTravelDistanceKm computes the minimum distance between two IP geolocation
// uncertainty circles on the WGS84 sphere.
func EffectiveTravelDistanceKm(prev, curr GeoObservation) float64 {
	const earthRadiusKm = 6371.0

	dLat := (curr.Latitude - prev.Latitude) * math.Pi / 180.0
	dLon := (curr.Longitude - prev.Longitude) * math.Pi / 180.0
	lat1 := prev.Latitude * math.Pi / 180.0
	lat2 := curr.Latitude * math.Pi / 180.0

	a := math.Sin(dLat/2)*math.Sin(dLat/2) +
		math.Cos(lat1)*math.Cos(lat2)*math.Sin(dLon/2)*math.Sin(dLon/2)
	centroidDistanceKm := 2 * earthRadiusKm * math.Atan2(math.Sqrt(a), math.Sqrt(1-a))

	// Subtract both uncertainty radii (plus a 50km carrier UPF handoff buffer if either leg is mobile)
	mobileBufferKm := 0.0
	if prev.IsMobile || curr.IsMobile {
		mobileBufferKm = 50.0
	}

	effective := centroidDistanceKm - prev.AccuracyRadiusKm - curr.AccuracyRadiusKm - mobileBufferKm
	if effective < 0 {
		return 0
	}
	return effective
}

// IsImpossibleTravel returns true only when the effective minimum distance between
// uncertainty circles exceeds 900 km/h (commercial aircraft speed).
func IsImpossibleTravel(prev, curr GeoObservation) bool {
	if EffectiveTravelDistanceKm(prev, curr) == 0 {
		return false
	}
	hours := curr.ObservedAt.Sub(prev.ObservedAt).Hours()
	if hours <= 0.05 {
		// Enforce a minimum 3-minute floor to avoid division spikes
		hours = 0.05
	}
	requiredSpeedKmh := EffectiveTravelDistanceKm(prev, curr) / hours
	return requiredSpeedKmh > 900.0
}
```

By subtracting `prev.AccuracyRadiusKm` and `curr.AccuracyRadiusKm`, switching from home fiber in Augsburg (`R_1 = 15 km`) to a Munich 5G UPF (`R_2 = 120 km`, centroid distance `65 km`) yields `EffectiveTravelDistanceKm = 0 km`, eliminating 100% of false Wi-Fi-to-5G impossible-travel alerts while still catching real credential theft across continents (`D_effective > 4,000 km`).

## How IPv6 changes mobile rate limiting (but not UPF city radius)

Most major 4G/5G carriers now deploy dual-stack or IPv6-mostly (`RFC 8925` / `464XLAT`) cellular networks where each smartphone receives its own globally routable **`/64` IPv6 prefix**.

Understanding what IPv6 changes on mobile networks is critical for systems architects:

1. **IPv6 eliminates CGNAT port sharing across subscribers**: Two different smartphones on an IPv6 cellular network will never share the same `/64` IPv6 prefix. When your rate limiter keys IPv6 requests by the `/64` prefix (rather than `/128`, since iOS and Android rotate temporary RFC 8981 privacy interface identifiers within their assigned `/64`), every mobile subscriber gets an isolated rate-limit bucket!
2. **IPv6 does NOT change the physical UPF/PGW geolocation radius**: Even on IPv6, the subscriber's `/64` prefix is dynamically delegated out of a large `/36` or `/40` pool anchored at the regional 5G UPF in Munich, Frankfurt, or Dallas. Therefore, `connection.is_mobile` remains `true` and `geo.accuracy_radius_km` remains `80` to `250 km`.

## Frequently asked questions (FAQ)

### Why should I rate-limit IPv6 traffic by `/64` instead of `/128`?
Modern mobile and desktop operating systems (iOS, Android, macOS, Windows, Linux) implement **RFC 8981 Temporary Address Extensions**, which periodically generate new random 64-bit interface identifiers inside the subscriber's assigned `/64` subnet. Furthermore, anyone allocated a `/64` has `2^64` (`18.4 quintillion`) individual `/128` addresses at their disposal. Always normalize IPv6 addresses to their `/64` prefix when keying rate-limit counters.

### Can any IP geolocation provider resolve the exact cell tower a 5G phone is connected to?
No. Without on-device GPS/Wi-Fi permissions (`navigator.geolocation` in the browser or CoreLocation/FusedLocationProvider in a native mobile app), a remote web server only sees the source IP address of the carrier's internet breakout gateway (`UPF` / `PGW`). Any IP geolocation vendor claiming `5 km` street-level accuracy on cellular IP ranges is reporting false precision. That is why ipscope provides an honest, physics-backed `accuracy_radius_km` alongside `connection.is_mobile: true`.

---

<!-- Article URL: https://ipscope.io/blog/declared-vs-physical-geolocation -->
---
slug: declared-vs-physical-geolocation
title: "Why Registration Records Lie: Separating Declared and Physical IP Geolocation"
description: "How cross-border BGP announcements, IPv4 leasing marketplaces, and stale geofeeds break traditional GeoIP databases, and how active RTT triangulation resolves physical router location."
date: "2026-10-24"
readingTime: "14 min read"
category: "Geolocation"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "BGP, Geolocation, RTT, RFC 8805, IPv6, Compliance"
featured: "true"
---

## The structural flaw in registration-based IP geolocation

For the first twenty years of the commercial internet, IP geolocation databases relied on a straightforward assumption: the country listed on an IP block's Regional Internet Registry (RIR) allocation record was the country where packets addressed to that block actually terminated. When ARIN allocated a `/16` to a corporation headquartered in Wilmington, Delaware, legacy geolocation providers mapped all 65,536 addresses in that block to the United States.

In modern network engineering, legal registration geography and physical router topology diverge on millions of prefixes every single day. The exhaustion of the free IPv4 pools across all five RIRs (ARIN, RIPE NCC, APNIC, LACNIC, and AFRINIC) created a liquid global secondary market for IPv4 address space. Organizations routinely lease `/22` and `/24` blocks across continents without transferring the underlying RIR org record, announce those prefixes out of foreign Internet Exchange Points (IXPs), or re-home subnets across cloud availability zones in minutes using automated BGP orchestration.

If your application relies on a single `country_code` field derived purely from WHOIS, RDAP, or unverified self-published geofeeds, you are systematically misrouting edge traffic, miscalculating regional VAT or tax rules, and triggering false-positive compliance blocks.

This engineering deep dive examines the three root causes of registration-versus-physical divergence, explains how the ipscope compiler separates legal jurisdiction from physical router topology using speed-of-light latency triangulation, and provides concrete implementation patterns for routing, fraud scoring, and regulatory compliance.

## Three ways legal registration diverges from physical routing

### 1. Cross-border IPv4 leasing and Letter of Authorization (LOA) announcements

Since IPv4 runout, brokers and IP leasing platforms have enabled cloud providers, content delivery networks, and regional ISPs to lease address blocks on short-term contracts. Under standard routing practice, the original registrant (the lessor) retains administrative ownership of the `inetnum` or `NetRange` record in ARIN or APNIC, and issues a cryptographic ROA (Route Origin Authorization) via RPKI alongside a Letter of Authorization permitting the lessee's Autonomous System Number (ASN) to announce the `/24` over BGP.

Consider what happens when a bare-metal hosting provider in Frankfurt (`DE`) leases `198.51.100.0/24` from a legacy US telecom holder:

- The ARIN RDAP `NetRange` record still lists the registrant's corporate address in Virginia (`US`).
- Legacy geolocation vendors that refresh primarily from RIR WHOIS dumps continue to report `country_code: "US"`.
- In reality, every BGP update for `198.51.100.0/24` originates from an ASN peering at DE-CIX Frankfurt, and physical fiber latency from Frankfurt probes is under `1.4 ms`.

### 2. Multinational backbone sub-allocations without discrete inetnum objects

Global cloud providers, hyperscalers, and Tier-1 transit carriers register massive `/12` or `/14` IPv4 blocks and `/29` or `/32` IPv6 blocks under a single holding entity, typically in Seattle, Dublin, or Singapore. Internally, their traffic engineering controllers slice those parent blocks into thousands of `/24` IPv4 and `/48` IPv6 prefixes deployed across 35+ countries.

Many operators do not create separate RIR sub-allocation objects for every `/24` they move between Tokyo, Mumbai, and Sydney. Even when an operator publishes an RFC 8805 geofeed CSV, internal capacity migrations frequently happen days or weeks before the public CSV file is updated. During that window, registration-driven databases report the holding company's headquarters country rather than the active data center region.

### 3. Stale or unverified RFC 8805 geofeeds and geofeed spoofing

RFC 8805 (*A Format for Self-Published IP Geolocation Feeds*) was designed so network operators could publish a CSV mapping their CIDR blocks to ISO-3166 country and subdivision codes. While RFC 8805 solved the problem of slow WHOIS updates for honest operators, it introduced two new failure modes:

- **Operational drift**: A network engineering team re-announces a `/24` from Amsterdam to London during a peering migration and forgets to commit the updated row to their static `geofeed.csv` repository.
- **Intentional geo-arbitrage**: Commercial proxy networks and grey-market VPN providers intentionally publish RFC 8805 rows claiming that cheap datacenter subnets hosted in Frankfurt or Bucharest are located in high-value residential metros like New York, London, or Zurich in order to bypass streaming media geo-fences and ad-verification filters.

## Why collapsing geography into a single country field breaks production systems

When an IP intelligence provider collapses legal registration and physical location into a single `country_code` column, every downstream team in your organization is forced into a compromise:

| Engineering Team | What They Actually Need | What Goes Wrong with a Single Collapsed Field |
| --- | --- | --- |
| **Edge Routing & CDN** | **Physical router location** (`physical_country_code`) | Traffic from a Frankfurt server using a US-registered `/24` gets routed across the Atlantic to a US East PoP, adding `85 ms` of unnecessary RTT. |
| **Sanctions & Export Control** | **Both legal registrant and physical termination** (`declared_country_code` + `physical_country_code`) | Checking only physical routing misses an IP block legally owned by a sanctioned corporate entity; checking only registration blocks legitimate European infrastructure leasing legacy US space. |
| **Media Licensing & DRM** | **Physical termination + virtual location flag** (`physical_country_code` + `is_virtual_location`) | Proxy exits spoofing RFC 8805 geofeeds bypass regional licensing checks unless physical RTT triangulation overrides the self-declared CSV. |
| **Payments & Tax (VAT / GST)** | **Explicit reconciliation against billing country** | Checkout flows reject valid transactions when a multinational corporate VPN egresses in one country using an IP block registered to its parent company in another. |

## How ipscope separates declared and physical geography

Rather than hiding the disagreement between administrative records and active network topology inside a black-box heuristic, ipscope exposes three explicit country fields plus a boolean virtual-location indicator inside the `geo` block on every REST API response, MMDB record, and Parquet row:

```json
{
  "ip": "198.51.100.7",
  "ip_version": 4,
  "matched_network": "198.51.100.0/24",
  "geo": {
    "country_code": "DE",
    "declared_country_code": "US",
    "physical_country_code": "DE",
    "is_virtual_location": true,
    "location_type": "virtual",
    "confidence": 0.96,
    "subdivision": "Hesse",
    "city": "Frankfurt am Main",
    "latitude": 50.1109,
    "longitude": 8.6821,
    "accuracy_radius_km": 10,
    "timezone": "Europe/Berlin"
  },
  "network": {
    "cidr": "198.51.100.0/24",
    "asn": 24940,
    "organization": "Hetzner Online GmbH",
    "rir": "ARIN"
  }
}
```

### Field definitions and compiler semantics

| Field | Data Type | Compiler Source of Truth | Operational Meaning |
| --- | --- | --- | --- |
| `declared_country_code` | `string` (ISO-3166-1 alpha-2) | RIR RDAP/WHOIS allocation records (`inetnum`, `inet6num`, `NetRange`) and RPKI-authenticated RFC 8805 geofeeds | The legal or administrative jurisdiction where the IP block is registered or formally declared by its holder. |
| `physical_country_code` | `string` (ISO-3166-1 alpha-2) | Multi-vantage active RTT triangulation, BGP next-hop IXP locality, and rDNS router interface telemetry | The country where packets addressed to this prefix physically terminate on hardware. |
| `country_code` | `string` (ISO-3166-1 alpha-2) | Resolved effective country (`physical_country_code` when physical confidence exceeds threshold) | Primary country code for localization, default currency, and UI rendering. |
| `is_virtual_location` | `boolean` | `true` whenever `declared_country_code !== physical_country_code` | Indicates that the subnet is leased cross-border, announced remotely via BGP, or has a declared geofeed that conflicts with physical router physics. |
| `location_type` | `string` (`"physical"`, `"virtual"`, `"anycast"`) | Topology classifier | Distinguishes standard single-homed physical subnets from cross-border virtualized announcements and global BGP anycast prefixes. |

## How our compiler resolves physical location using speed-of-light physics

Network operators can put whatever text they want into a WHOIS remark or an RFC 8805 CSV file, but they cannot violate the speed of light in optical fiber.

In single-mode silica optical fiber, light propagates at approximately `200,000 km/s` (roughly two-thirds the speed of light in a vacuum due to a refractive index of `n ≈ 1.468`). Because an ICMP Echo, TCP SYN/ACK, or UDP probe must travel from the measurement vantage point to the target IP and back, the theoretical minimum round-trip propagation delay in straight-line fiber is `1 millisecond per 100 kilometers` of geographic distance. Once you account for non-linear fiber conduit routing, DWDM optical repeaters, and MPLS/BGP switching serialization, real-world terrestrial internet paths achieve roughly `65 to 85 kilometers of geographic radius per millisecond of minimum RTT`.

### Stage 1: BGP table ingestion and RIR bailiwick extraction

Every compilation cycle starts by ingesting full BGP routing tables (`RIB` dumps and real-time `UPDATE` streams) from route collectors peering at major IXPs worldwide, alongside daily RDAP and WHOIS snapshots from ARIN, RIPE NCC, APNIC, LACNIC, and AFRINIC.

For every routed IPv4 `/24` and IPv6 `/48`, the compiler extracts:

1. The originating ASN and its upstream transit or IXP peering adjacency graph.
2. The administrative registrant country from the most specific RIR allocation object.
3. Any RFC 8805 `#geofeed` URL attached to the allocation object via RFC 9092 remarks, validated against RPKI Route Origin Authorizations (ROAs).

### Stage 2: Reverse DNS (rDNS) router interface parsing

Before firing active latency probes, the compiler evaluates `PTR` records on gateway and router interfaces within the prefix and along the penultimate traceroute hops. Carrier backbone engineers name core and edge router interfaces using standardized IATA airport codes, UN/LOCODE city identifiers, and CLLI codes (for example, `ae4-200.cr1-fra1.ip4.gtt.net` or `et-0-0-2.br02.ams01.example.net`).

While rDNS on a leaf host can be set to anything by a customer, rDNS on the upstream provider's penultimate MPLS/BGP hop is controlled by the transit carrier. When multiple upstream peers converge on `fra` (Frankfurt) or `ams` (Amsterdam), the compiler records a strong topological prior for that metro area.

### Stage 3: Multi-vantage minimum-RTT trilateration

Whenever a prefix is newly announced, shifts its upstream BGP path, claims a new RFC 8805 city, or exhibits a disagreement between its RIR registration country and its upstream peering graph, our distributed measurement fleet executes lightweight TCP/ICMP minimum-RTT probes from vantage points across North America, Europe, Asia-Pacific, Latin America, and Africa.

We always take the **minimum observed RTT (`min_rtt`)** across multiple samples rather than the mean or median. Queuing delay on a congested router interface can inflate a single packet's RTT upward, but queuing delay can never make a packet arrive faster than the speed of light in fiber.

Consider our earlier example (`198.51.100.0/24`), where ARIN WHOIS and a stale geofeed claim the subnet is in Ashburn, Virginia (`US`):

| Measurement Vantage Point | Claimed Distance to Ashburn, VA | Measured `min_rtt` | Maximum Physical Radius (`~80 km/ms`) | Physically Compatible with Ashburn (`US`)? |
| --- | --- | --- | --- | --- |
| **Ashburn, US (`iad-01`)** | `12 km` | `84.2 ms` | `6,736 km` | Suspiciously high for local metro |
| **Newark, US (`ewr-01`)** | `380 km` | `81.6 ms` | `6,528 km` | Suspiciously high |
| **Amsterdam, NL (`ams-01`)** | `6,170 km` | `6.4 ms` | `512 km` | **Impossible** (`6,170 km > 512 km`) |
| **Frankfurt, DE (`fra-01`)** | `6,530 km` | `1.3 ms` | `104 km` | **Impossible** (`6,530 km > 104 km`) |
| **Zurich, CH (`zrh-01`)** | `6,640 km` | `5.8 ms` | `464 km` | **Impossible** (`6,640 km > 464 km`) |

A `1.3 ms` round-trip time from our Frankfurt vantage point proves with physical certainty that the target host lies within roughly `100 km` of Frankfurt am Main. It is physically impossible for a packet to leave Frankfurt, cross the Atlantic Ocean to Virginia, and return in `1.3 ms` (which would require travelling at over 10,000 times the speed of light in fiber).

When the trilateration solver sees this intersection of circles (`1.3 ms` from Frankfurt, `5.8 ms` from Zurich, `6.4 ms` from Amsterdam), it deterministically assigns:

- `declared_country_code: "US"` (preserving the ARIN administrative record for legal audit trails)
- `physical_country_code: "DE"` (reflecting the verified physical termination in Frankfurt)
- `country_code: "DE"`
- `is_virtual_location: true`
- `location_type: "virtual"`

## How BGP Anycast prefixes are handled (`location_type: "anycast"`)

A critical edge case in physical RTT triangulation is **BGP Anycast**. Content delivery networks, authoritative DNS providers, and DDoS scrubbing networks announce the exact same `/24` IPv4 or `/48` IPv6 prefix simultaneously from 50 to 300+ global Points of Presence.

When our measurement fleet probes an anycast prefix, the Frankfurt probe gets a `1.1 ms` reply (from the local Frankfurt anycast node), the Ashburn probe gets a `0.9 ms` reply (from the local Ashburn node), and the Tokyo probe gets a `1.4 ms` reply (from the local Tokyo node). If a naive trilateration solver tried to intersect three `100 km` circles centered in Frankfurt, Ashburn, and Tokyo, the intersection would be empty!

Our compiler detects anycast topology automatically when three or more geographically disjoint continents simultaneously observe sub-`10 ms` minimum RTTs to the same `/24` or `/48` prefix. When that condition holds, the compiler sets `location_type: "anycast"` and anchors the administrative metadata to the operator's primary registration while flagging that the prefix terminates at the nearest local PoP.

## Production implementation patterns

### Pattern 1: Dual-jurisdiction sanctions and export compliance in Python

If you operate a fintech, crypto exchange, or SaaS platform subject to OFAC, EU, or UK export controls, checking a single country code leaves a dangerous blind spot. A sanctioned corporate entity can lease infrastructure in a non-sanctioned country, or a non-sanctioned entity can route traffic into a restricted territory.

By inspecting both `declared_country_code` and `physical_country_code`, your compliance middleware catches both vectors cleanly:

```python
import asyncio
import os
from typing import TypedDict
from ipscope import AsyncIpScopeClient

RESTRICTED_JURISDICTIONS = {"CU", "IR", "KP", "SY"}

class ComplianceDecision(TypedDict):
    allowed: bool
    reason: str
    declared_country: str
    physical_country: str
    is_virtual: bool

async def evaluate_jurisdiction_policy(client_ip: str) -> ComplianceDecision:
    async with AsyncIpScopeClient(api_key=os.environ["IPSCOPE_KEY"]) as client:
        res = await client.lookup_ip(client_ip)
        geo = res["geo"]

        declared = geo["declared_country_code"]
        physical = geo["physical_country_code"]
        is_virtual = bool(geo["is_virtual_location"])

        if physical in RESTRICTED_JURISDICTIONS:
            return {
                "allowed": False,
                "reason": f"blocked_physical_termination:{physical}",
                "declared_country": declared,
                "physical_country": physical,
                "is_virtual": is_virtual,
            }

        if declared in RESTRICTED_JURISDICTIONS:
            return {
                "allowed": False,
                "reason": f"blocked_registered_allocation:{declared}",
                "declared_country": declared,
                "physical_country": physical,
                "is_virtual": is_virtual,
            }

        return {
            "allowed": True,
            "reason": "compliant_jurisdiction",
            "declared_country": declared,
            "physical_country": physical,
            "is_virtual": is_virtual,
        }

if __name__ == "__main__":
    decision = asyncio.run(evaluate_jurisdiction_policy("198.51.100.7"))
    print(decision)
```

### Pattern 2: Eliminating false checkout declines on cross-border cloud and corporate subnets

In e-commerce payment risk engines, a classic rule compares the customer's credit card billing country against the IP address country. When a multinational company routes employee laptop traffic through a regional cloud firewall whose IP block was leased from another RIR region, legacy databases report a country mismatch and trigger a false decline.

With ipscope, your risk engine can check whether the customer's billing country matches **either** the `physical_country_code` or, for verified business networks (`connection.is_business`), the `declared_country_code`, while stepping up authentication only when `security.is_vpn` or `security.is_proxy` is active:

```ts
import { IpScopeClient } from '@ipscope/sdk-node';

const client = new IpScopeClient({ apiKey: process.env.IPSCOPE_KEY });

export async function verifyBillingCountryMatch(
  clientIp: string,
  billingCountryIso2: string,
): Promise<{ status: 'match' | 'virtual_match' | 'mismatch'; details: string }> {
  const res = await client.lookupIp(clientIp);
  const { geo, connection, security } = res;

  if (geo.physical_country_code === billingCountryIso2) {
    return {
      status: 'match',
      details: `Physical termination matches billing country (${billingCountryIso2})`,
    };
  }

  if (
    geo.is_virtual_location &&
    geo.declared_country_code === billingCountryIso2 &&
    connection?.is_business &&
    !security.is_vpn &&
    !security.is_proxy
  ) {
    return {
      status: 'virtual_match',
      details: `Enterprise virtual subnet registered to ${geo.declared_country_code}, routed via ${geo.physical_country_code}`,
    };
  }

  return {
    status: 'mismatch',
    details: `Billing=${billingCountryIso2}, Physical=${geo.physical_country_code}, Declared=${geo.declared_country_code}`,
  };
}
```

## Frequently asked questions (FAQ)

### Which country field should I use for general website localization and CDN routing?
Use `geo.country_code` (or `geo.physical_country_code`). In standard unicast prefixes, `geo.country_code` resolves to the verified physical country where the user or server actually connects to the internet, ensuring low-latency PoP selection and accurate local language/currency defaults.

### Why does my IP show `declared_country_code: "US"` and `physical_country_code: "NL"`?
This means the IPv4 or IPv6 block is administratively registered in ARIN to a US legal entity (or declares `US` in its registration metadata), but the prefix is actively announced over BGP and physically terminated on routers located in the Netherlands (`NL`), as confirmed by sub-`5 ms` speed-of-light latency measurements from Dutch and German vantage points.

### Are `declared_country_code`, `physical_country_code`, and `is_virtual_location` included in downloadable `.mmdb` and `.parquet` files?
Yes. Every ipscope database SKU that includes the `geo_country` family (`IPScope-Country`, `IPScope-Privacy`, and `IPScope-Complete`) embeds `country_code`, `declared_country_code`, `physical_country_code`, `is_virtual_location`, `location_type`, and `confidence` directly inside the `.mmdb` binary tree, `.parquet` columns, and `.csv.gz` exports.

---

<!-- Article URL: https://ipscope.io/blog/detecting-vpn-exit-nodes-and-residential-proxies -->
---
slug: detecting-vpn-exit-nodes-and-residential-proxies
title: "Deterministic VPN & Residential Proxy Detection with Explainable Reason Codes"
description: "Why black-box fraud scores fail in production, and how ipscope combines WireGuard/OpenVPN protocol probes, MTU/MSS telemetry, and ASN classification to emit auditable reason codes."
date: "2026-10-15"
readingTime: "14 min read"
category: "Security"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "VPN, Proxy, Tor, Security, Fraud Prevention, Reason Codes"
featured: "false"
---

## Why opaque IP risk scores fail in production engineering

If you have ever maintained a login, signup, or payment checkout pipeline, you have likely experienced the failure mode of black-box IP reputation APIs. A third-party vendor returns a single opaque number like `"fraud_score": 88`, your middleware blocks the request, and twenty minutes later a high-value enterprise customer or mobile subscriber opens an urgent support ticket because they were locked out of their account.

When your security engineers inspect the logs, `"fraud_score": 88` tells them nothing actionable:

- Was the IP address an active Tor exit node or a high-velocity credential-stuffing botnet?
- Was it a Fortune 500 employee routing traffic through a corporate cloud security gateway (`Zscaler`, `Cloudflare Gateway`, `Netskope`) hosted on a datacenter ASN?
- Was it an iPhone user in Safari with Apple iCloud Private Relay enabled?
- Or was it simply a shared 5G cellular CGNAT gateway where one bad actor out of 4,000 subscribers triggered a static blocklist three days ago?

When an IP intelligence provider conflates **network access topology**, **anonymizer protocol state**, and **recent abuse velocity** into a single undocumented integer, you cannot write surgical security policies. You are forced to choose between high false-positive rates that hurt conversion and lax thresholds that let automated abuse through.

In ipscope, we designed our security architecture around a strict separation of concerns:

1. **Access network topology (`connection`)**: Classifies whether the IP belongs to a `residential`, `mobile`, `business`, or `hosting` network.
2. **Structural anonymizer state (`security`)**: Identifies active VPNs, residential/datacenter proxies, Tor exits, and consumer privacy relays, backed by deterministic `reason_codes`.
3. **Time-decayed hostile activity (`security_extensions`)**: Tracks active scanning and brute-force cadence via `abuse_velocity_score` (`0` to `100`) and `observation_recency_hours`.

## Anatomy of the ipscope `security` and `connection` blocks

Every lookup licensed for the `anonymity` and `connection` families returns explicit boolean flags, an enumerated `anonymizer_type`, a calibrated `confidence` score (`0.0` to `1.0`), and an array of machine-parsable `reason_codes`:

```json
{
  "ip": "198.51.100.42",
  "ip_version": 4,
  "matched_network": "198.51.100.0/24",
  "connection": {
    "connection_type": "hosting",
    "is_residential": false,
    "is_mobile": false,
    "is_business": false,
    "is_hosting": true
  },
  "security": {
    "is_vpn": true,
    "is_hosting": true,
    "is_tor_exit": false,
    "is_proxy": false,
    "anonymizer_type": "vpn",
    "confidence": 0.98,
    "reason_codes": [
      "vpn_handshake_confirmed",
      "commercial_vpn_exit_enumerated",
      "datacenter_asn_range"
    ]
  },
  "security_extensions": {
    "abuse_velocity_score": 14,
    "is_known_scanner": false,
    "observation_recency_hours": 6
  }
}
```

### Understanding `anonymizer_type` values

| `anonymizer_type` | Description | Typical Risk Profile |
| --- | --- | --- |
| `"none"` | Direct residential, mobile, business, or un-tunneled cloud connection | Baseline traffic (inspect `is_hosting` and `abuse_velocity_score` for bot activity) |
| `"vpn"` | Commercial VPN service or self-hosted WireGuard / OpenVPN / IPsec exit node | Medium risk: common among privacy-conscious users and geo-hoppers; step up to MFA/Passkey on sensitive mutations |
| `"proxy"` | Open HTTP/SOCKS4/SOCKS5 proxy or residential proxy exit node | High risk: heavily used by credential-stuffing frameworks, sneaker bots, and ad-fraud farms |
| `"tor"` | Active Tor network exit relay verified against hourly consensus | High risk for account creation and payment checkout; legitimate for anonymous reading |
| `"private_relay"` | Consumer OS/browser privacy relay (such as Apple iCloud Private Relay) | Low fraud risk: hardware-attested consumer devices masking client IP within the user's real country/region |

## How ipscope detects VPNs, proxies, and relays across four signal layers

### Layer 1: Active cryptographic protocol handshakes (`vpn_handshake_confirmed`)

Many commercial VPN providers lease ordinary datacenter `/24` blocks from cloud and bare-metal hosts without registering the blocks under the VPN brand's name. Passive WHOIS analysis only sees the underlying hosting company (for example, `M247`, `Datacamp`, `Choopa`, or `Hetzner`).

To identify active VPN concentrators deterministically, our distributed scanner fleet performs non-intrusive, protocol-specific handshake verification across the IPv4 and active IPv6 address space:

- **OpenVPN (UDP/TCP)**: Sends a valid OpenVPN `P_CONTROL_HARD_RESET_CLIENT_V2` control packet and verifies whether the listener responds with a `P_CONTROL_HARD_RESET_SERVER_V2` session state machine reply or requires `tls-auth` / `tls-crypt` HMAC framing characteristic of OpenVPN daemons.
- **IKEv2 / IPsec (`UDP 500` / `UDP 4500`)**: Sends a standard `IKE_SA_INIT` proposal and inspects the responder's cryptographic transform selection, vendor ID payloads (`VID`), and NAT-Traversal encapsulation headers.
- **WireGuard and Shadowsocks / V2Ray / Trojan**: Combines TLS ALPN fingerprint telemetry, port allocation entropy, and continuous exit-node enumeration across 95+ commercial VPN providers. When a user connects through a multi-hop VPN where the ingress IP differs from the egress IP, our continuous egress enumeration probes record the exact exit `/32` and attach `commercial_vpn_exit_enumerated`.

### Layer 2: Residential proxy exit detection (`residential_proxy_exit`)

Detecting datacenter VPNs is only half the battle. Modern scraping and credential-stuffing operations route requests through **residential proxy networks** (often powered by SDKs embedded in free mobile/desktop apps, residential bandwidth-sharing containers, or compromised home routers). Because a residential proxy exit resides on a legitimate consumer ISP (`connection.is_residential: true`, `connection.is_hosting: false`), ASN-only checks completely miss it.

Our compiler identifies residential proxy exits using three orthogonal signals:

1. **Reverse-Tunnel Beacon & Relay Node Correlation**: Residential proxy networks require persistent control connections from the home device back to central proxy aggregator gateways. By tracking the aggregator infrastructure and enumerating proxy exit pools, we identify active residential proxy IPs while they are participating in the relay mesh.
2. **Multi-Tenant Session & TLS/TCP Fingerprint Divergence**: A genuine household router typically exhibits a small, stable set of OS TCP stack parameters and TLS client fingerprints (`JA4`) over a 1-hour window. A residential proxy exit multiplexing thousands of automated bot requests exhibits rapid, simultaneous rotation of incompatible OS/TLS signatures and anomalous destination ASN fan-out within minutes.
3. **Ephemeral Port Exhaustion & Darknet Hit Correlation**: When residential IPs suddenly participate in coordinated distributed scans or credential-stuffing waves observed across our global darknet sensor network, `residential_proxy_exit` and `abuse_velocity_score` elevate in tandem.

### Layer 3: Hourly Tor directory consensus and exit list synchronization (`tor_consensus_exit`)

The Tor network consists of guard, middle, and exit relays. Only **exit relays** (`Exit` flag in the directory consensus, plus actual outbound exit addresses discovered via `TorDNSEL` / `exit-addresses` descriptors) originate traffic toward public web servers.

Our ingestion pipeline syncs directly with the Tor directory authorities every 60 minutes. We distinguish between middle/guard relays (which do not egress user HTTP traffic) and active exit IPs, setting `is_tor_exit: true`, `anonymizer_type: "tor"`, and `reason_codes: ["tor_consensus_exit"]`.

### Layer 4: Consumer privacy relay separation (`private_relay`)

One of the most common sources of false positives in legacy GeoIP databases is treating **Apple iCloud Private Relay** (and similar hardware-attested consumer oblivious HTTP relays) as hostile anonymous proxies.

Unlike commercial VPNs that allow users to spoof arbitrary foreign countries, consumer privacy relays enforce strict regional locality (preserving the user's true country and typically their metro/time zone) and require hardware-backed device attestation to enter the relay ingress. Moreover, several major privacy relay operators publish official RFC 8805 egress feeds so web services can localize content properly.

By classifying these egress blocks with `anonymizer_type: "private_relay"` and `reason_codes: ["consumer_privacy_relay"]`, ipscope lets your security team allow legitimate iOS and macOS users through checkout without friction while still enforcing strict controls on commercial VPNs and SOCKS proxies.

## Reference dictionary of `security.reason_codes`

Every flag in the `security` block is backed by one or more deterministic `reason_codes` so your engineering and support teams can audit every decision in your logs:

| Reason Code | Trigger Condition | Primary Associated Flag |
| --- | --- | --- |
| `vpn_handshake_confirmed` | Active OpenVPN, IKEv2/IPsec, or WireGuard protocol listener verified on host | `is_vpn: true` |
| `commercial_vpn_exit_enumerated` | Verified outbound exit IP of a commercial VPN provider | `is_vpn: true` |
| `datacenter_asn_range` | Prefix belongs to a dedicated cloud, VPS, or colocation hosting ASN | `is_hosting: true` |
| `tor_consensus_exit` | Active Tor exit relay in the current hourly directory consensus | `is_tor_exit: true` |
| `open_proxy_listener` | Active HTTP CONNECT, SOCKS4, or SOCKS5 proxy verified on host | `is_proxy: true` |
| `residential_proxy_exit` | Residential or mobile IP actively routing traffic for a proxy relay pool | `is_proxy: true` |
| `consumer_privacy_relay` | Verified egress IP of an attested consumer privacy relay (e.g. iCloud Private Relay) | `anonymizer_type: "private_relay"` |
| `rdns_vpn_token` | Reverse DNS (`PTR`) or operator WHOIS explicitly designates VPN exit infrastructure | `is_vpn: true` |

## Building an explainable authentication & checkout policy in TypeScript

Here is a complete, production-ready policy engine using `@ipscope/sdk-node` that handles consumer privacy relays, shared mobile CGNAT gateways, commercial VPNs, residential proxies, and Tor exits with full audit logging:

```ts
import { IpScopeClient, type IpLookupResponse } from '@ipscope/sdk-node';

const client = new IpScopeClient({
  apiKey: process.env.IPSCOPE_KEY!,
  maxCacheEntries: 8192,
  cacheTtlMs: 60_000,
});

export type PolicyAction = 'allow' | 'step_up_mfa' | 'challenge_captcha' | 'block';

export interface SecurityPolicyDecision {
  action: PolicyAction;
  ruleId: string;
  reasonCodes: string[];
  anonymizerType: string;
  abuseVelocity: number;
}

export async function evaluateCheckoutRequest(
  clientIp: string,
  userCountryBillingIso2: string,
): Promise<SecurityPolicyDecision> {
  const res: IpLookupResponse = await client.lookupIp(clientIp);
  const { geo, connection, security, security_extensions } = res;
  const velocity = security_extensions?.abuse_velocity_score ?? 0;

  // Rule 1: Block active Tor exits and high-velocity scanners on payment checkout
  if (security.is_tor_exit || velocity >= 80 || security_extensions?.is_known_scanner) {
    return {
      action: 'block',
      ruleId: 'HIGH_RISK_TOR_OR_SCANNER',
      reasonCodes: security.reason_codes,
      anonymizerType: security.anonymizer_type,
      abuseVelocity: velocity,
    };
  }

  // Rule 2: Allow attested consumer privacy relays without friction if country matches
  if (security.anonymizer_type === 'private_relay') {
    const countryMatches = geo.physical_country_code === userCountryBillingIso2;
    return {
      action: countryMatches ? 'allow' : 'step_up_mfa',
      ruleId: countryMatches ? 'ALLOW_PRIVATE_RELAY_LOCAL' : 'STEP_UP_PRIVATE_RELAY_FOREIGN',
      reasonCodes: security.reason_codes,
      anonymizerType: security.anonymizer_type,
      abuseVelocity: velocity,
    };
  }

  // Rule 3: Challenge residential or open proxies (high botnet / credential stuffing correlation)
  if (security.is_proxy) {
    return {
      action: velocity >= 40 ? 'block' : 'challenge_captcha',
      ruleId: 'PROXY_DETECTED',
      reasonCodes: security.reason_codes,
      anonymizerType: security.anonymizer_type,
      abuseVelocity: velocity,
    };
  }

  // Rule 4: Step up to Passkey / WebAuthn / 2FA for commercial VPNs or direct datacenter egress
  if (security.is_vpn || (security.is_hosting && !connection?.is_business)) {
    return {
      action: 'step_up_mfa',
      ruleId: 'VPN_OR_HOSTING_EGRESS',
      reasonCodes: security.reason_codes,
      anonymizerType: security.anonymizer_type,
      abuseVelocity: velocity,
    };
  }

  // Rule 5: Clean residential, mobile, or business network
  return {
    action: 'allow',
    ruleId: 'CLEAN_ACCESS_NETWORK',
    reasonCodes: security.reason_codes,
    anonymizerType: security.anonymizer_type,
    abuseVelocity: velocity,
  };
}
```

## Handling residential proxy churn and dynamic ISP reassignment

Unlike datacenter `/24` blocks where VPN servers stay online for months, residential IPv4 addresses are reassigned via DHCP when home routers reboot or ISP PPPoE sessions cycle. If a residential IP was part of a proxy pool on Tuesday, and the ISP reassigns that IP to a different household on Thursday, permanently flagging the IP as `is_proxy: true` would penalize an innocent subscriber.

To prevent stale residential proxy flags, our compiler enforces a strict **observation TTL and confidence decay curve** on all `residential` and `mobile` access networks (`connection.is_residential || connection.is_mobile`):

- Datacenter VPN and hosting classifications (`connection.is_hosting: true`) are anchored to stable subnet and protocol listener state verified across consecutive sweeps.
- Residential and mobile proxy classifications (`residential_proxy_exit`) require recurring telemetry confirmation within a rolling **24-to-72-hour window**. Once proxy relay activity ceases on a residential `/32`, its proxy confidence decays automatically and the IP returns to a clean `residential` state on the next daily compilation cycle.

## Frequently asked questions (FAQ)

### Why are both `is_vpn: true` and `is_hosting: true` set on most commercial VPN IPs?
Almost all commercial VPN providers (`Mullvad`, `NordVPN`, `ExpressVPN`, `ProtonVPN`, `IVPN`) rent bare-metal or virtual servers inside commercial data centers (`connection_type: "hosting"`). Therefore, the IP address is simultaneously located inside a hosting provider's ASN (`is_hosting: true`) and operating as a verified VPN exit node (`is_vpn: true`). Keeping both booleans independent allows you to distinguish a raw cloud server (`is_hosting: true, is_vpn: false`) from a commercial VPN exit (`is_hosting: true, is_vpn: true`) and a residential VPN/proxy (`is_hosting: false, is_vpn: true`).

### Should I block all `is_vpn: true` traffic at the firewall level?
Rarely. Approximately 15% to 25% of technical users, remote software engineers, and privacy-conscious consumers keep a commercial VPN or corporate split-tunnel active during normal browsing. Blocking all VPNs at Layer 3/4 turns away legitimate users. Instead, allow read-only browsing and normal authenticated sessions on VPNs, and use `security.is_vpn` to trigger step-up authentication (such as Passkeys, 2FA, or email verification) on high-risk actions like account registration, password resets, API key creation, and payment checkout.

### Are `reason_codes` and `anonymizer_type` available in the offline `.mmdb` and `.parquet` files?
Yes. Every database SKU that includes the `anonymity` family (`IPScope-Privacy` and `IPScope-Complete`) contains the full `security` map, including `is_vpn`, `is_hosting`, `is_tor_exit`, `is_proxy`, `anonymizer_type`, `confidence`, and the `reason_codes` list.

---

<!-- Article URL: https://ipscope.io/blog/honeypot-abuse-velocity-and-internet-scanner-scoring -->
---
slug: honeypot-abuse-velocity-and-internet-scanner-scoring
title: "Scoring Real-Time Abuse Velocity: How security_extensions Detects Scanners and Credential Stuffing"
description: "How ipscope aggregates distributed darknet sensor hits, SSH/TLS brute-force telemetry, and port-sweep cadence into a calibrated 0-to-100 abuse_velocity_score with exponential time decay."
date: "2026-10-22"
readingTime: "14 min read"
category: "Security"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "Security, Abuse Velocity, Scanners, Botnets, WAF, Fraud Prevention"
featured: "false"
---

## Why static IP blocklists lose to ephemeral botnets and residential DHCP churn

Traditional IP threat intelligence relies on static, flat blocklists: CSV or text files of "bad IPv4 addresses" distributed once per day. In modern production security engineering, static blocklists fail in two opposite directions simultaneously:

1. **Too slow for cloud-native and proxy-rotated attacks**: Automated credential-stuffing frameworks, CVE exploitation bots, and LLM scrapers spin up ephemeral cloud instances or cycle through residential proxy pools in minutes. By the time a flat daily blocklist is compiled and pushed to your firewall, the attacker has already rotated to a fresh `/24` or a new set of residential proxy exits.
2. **Too sticky for dynamic residential and cellular ISPs**: When a compromised home IoT device or malware-infected laptop sits behind a consumer ISP router, its outbound attack traffic originates from a dynamically leased residential IPv4 address. When that home router reboots or the ISP rotates PPPoE/DHCP leases 24 hours later, that IPv4 address is assigned to a completely innocent household. Static blocklists that lack continuous time decay keep blocking that residential IP for weeks or months, generating endless customer lockouts.

To solve both problems, ipscope cleanly decouples **structural network anonymity** (`security`) from **time-decayed hostile traffic velocity** (`security_extensions`).

This engineering post explains how our global darknet and honeypot sensor mesh collects hostile packet telemetry, how we compute the calibrated `0` to `100` `abuse_velocity_score` using multi-ASN target entropy and exponential half-life decay, and how to wire `security_extensions` into your API gateway, WAF, and login endpoints.

## Anatomy of the `security_extensions` block

When your API plan or custom database SKU includes the `security_extensions` intelligence family, every IP lookup returns three real-time activity metrics alongside the core `security` block:

```json
{
  "ip": "198.51.100.214",
  "ip_version": 4,
  "matched_network": "198.51.100.0/24",
  "connection": {
    "connection_type": "hosting",
    "is_residential": false,
    "is_mobile": false,
    "is_business": false,
    "is_hosting": true
  },
  "security": {
    "is_vpn": false,
    "is_hosting": true,
    "is_tor_exit": false,
    "is_proxy": false,
    "anonymizer_type": "none",
    "confidence": 0.99,
    "reason_codes": [
      "datacenter_asn_range"
    ]
  },
  "security_extensions": {
    "abuse_velocity_score": 92,
    "is_known_scanner": true,
    "observation_recency_hours": 1
  }
}
```

### Field definitions in `security_extensions`

| Field | Data Type | Range / Values | Operational Definition |
| --- | --- | --- | --- |
| `abuse_velocity_score` | `uint8` | `0` to `100` | Calibrated, time-decayed severity and breadth score of hostile scanning, brute-forcing, and exploit attempts observed from this IP. `0` indicates zero observed hostile telemetry. |
| `is_known_scanner` | `boolean` | `true` or `false` | `true` when the IP exhibits systematic horizontal port sweeps (`SYN` / UDP scanning) or automated vulnerability enumeration across multiple independent `/16` networks. |
| `observation_recency_hours` | `uint32` (`nullable`) | `0` to `720` (`null` if `0` score) | Elapsed hours since our sensor fleet last observed hostile or scanning packets from this IP address. |

## How our sensor mesh captures hostile telemetry

Our telemetry ingestion pipeline aggregates packet and application-layer events from three distinct sensor tiers deployed across 45+ autonomous systems and all five RIR regions:

### 1. Passive darknet (network telescope) radiation sensors

We monitor unallocated and unrouted-to-application `/24` and `/23` IPv4 blocks and `/48` IPv6 honeypot subnets that host zero legitimate public services (no DNS `A`/`AAAA` records, no web links, no TLS certificates in CT logs). Because no legitimate human browser or misconfigured client has any reason to send TCP `SYN` packets to unused darknet address space, every unsolicited packet arriving at a darknet sensor is, by definition, either:

- Horizontal internet-wide port scanning (`ZMap`, `Masscan`, `Nmap`),
- Automated worm / botnet propagation probing (`TCP/22`, `TCP/23`, `TCP/445`, `TCP/3389`, `TCP/6379`, `TCP/27017`), or
- Spoofed UDP reflection backscatter (which we filter out by requiring a completed 3-way TCP handshake or stateful protocol payload before attributing application-layer abuse).

### 2. High-interaction protocol honeypots (SSH, HTTP/TLS, SMTP, Redis, Database)

Completing a TCP 3-way handshake proves that the source IP is **not spoofed** (the sender had to receive our `SYN-ACK` sequence number and return a valid `ACK`). Behind our handshake listeners, lightweight protocol state machines classify the exact intent of the connection:

- **Credential Brute-Forcing**: Repeated SSH password/key authentication attempts, SMTP `AUTH LOGIN` dictionary attacks, PostgreSQL/MySQL/Redis authentication sweeps, and HTTP `POST /wp-login.php`, `/api/v1/login`, or OAuth token endpoint stuffing.
- **Vulnerability & Path Exploitation**: HTTP requests probing for exposed `.env` files, `.git/config`, cloud metadata SSRF headers, Log4Shell/Spring4Shell payloads, PHPUnit RCE paths, and unauthenticated Kubernetes/Docker API endpoints.
- **TLS / JA4 Fingerprint Clustering**: Captures the exact TLS ClientHello structure (`JA4`) and HTTP/2 settings frame to correlate distributed botnet nodes executing the same automation binary across thousands of source IPs.

## The mathematics of `abuse_velocity_score`: Breadth, Intent, and Exponential Decay

A common mistake in naive threat-scoring systems is scoring an IP based on **raw packet count**. If you score by raw packet count, a single developer who accidentally runs an aggressive `nmap -p 1-65535` scan against one test server looks worse than a stealthy botnet node sending 1 login attempt per minute against 500 different banks.

To prevent single-target noise from skewing the score, the ipscope compiler calculates `abuse_velocity_score` across three normalized dimensions:

```txt
Score(ip, t) = round( 100 * (0.45 * Breadth + 0.35 * Intent + 0.20 * Cadence) * exp(-lambda * delta_t_hours) )
```

### Dimension 1: Target Breadth & ASN Entropy (`Breadth`, `45%` weight)

Instead of counting total packets sent to a single sensor, `Breadth` measures the **Shannon entropy and count of distinct destination `/16` networks and distinct destination ASNs** touched by the source IP:

- A source IP that sends 10,000 packets to a single destination IP receives a low breadth score because the activity is localized to one host.
- A source IP that hits 40 distinct `/16` sensors across 25 different autonomous systems in North America, Europe, and Asia within 15 minutes approaches `Breadth = 1.0` and triggers `is_known_scanner = true`.

### Dimension 2: Protocol Intent Severity (`Intent`, `35%` weight)

Not all internet scanners have the same intent. Academic and commercial internet topology mapping projects (`Censys`, `Shodan`, `Shadowserver`) perform regular port sweeps and publish clear `PTR` reverse DNS records, WHOIS abuse contacts, and HTTP opt-out headers. Conversely, exploit botnets (`Mirai`, `Mozi`, credential-stuffing clusters) send active RCE payloads and dictionary attacks after completing the TCP handshake.

| Observed Behavior Tier | Intent Multiplier (`Intent`) | `is_known_scanner` | Typical Score Range (When Active) |
| --- | --- | --- | --- |
| **Zero observed darknet/honeypot hits** | `0.00` | `false` | `0` |
| **Low-rate single-port probing / background noise** | `0.25` | `false` | `10` to `29` |
| **Identified benign research / topology scanner** | `0.45` | `true` | `35` to `55` |
| **Unattributed horizontal port / vulnerability scanner** | `0.75` | `true` | `60` to `84` |
| **Active credential brute-forcing or RCE exploit payloads** | `1.00` | `true` | `85` to `100` |

### Dimension 3: Cadence Burstiness (`Cadence`, `20%` weight) and Exponential Time Decay (`exp(-lambda * delta_t)`)

Every observation is timestamped to the second. Let `delta_t_hours` equal `observation_recency_hours` (the hours elapsed since the most recent confirmed TCP/application-layer hit from `ip`).

We apply an exponential decay factor `exp(-lambda * delta_t_hours)` calibrated to the underlying `connection.connection_type`:

- **Residential and Mobile CGNAT (`is_residential` or `is_mobile`)**: Uses a fast decay half-life (`half_life = 12 hours`). If a compromised home device stops transmitting because the ISP rotated its DHCP lease, an `88` score drops to `44` after 12 hours, `22` after 24 hours, and `< 6` after 48 hours.
- **Dedicated Cloud / Datacenter Hosting (`is_hosting`)**: Uses a standard decay half-life (`half_life = 24 hours`), because static cloud IPs rarely rotate to consumer households.

## Practical score bands for WAF and API Gateway policies

When integrating `abuse_velocity_score` into your edge middleware, we recommend dividing the `0..100` scale into four operational action bands:

| `abuse_velocity_score` Band | Classification | Recommended Edge / Gateway Action |
| --- | --- | --- |
| **`0` to `19`** | **Clean / Baseline** | Normal traffic handling and standard rate limits. |
| **`20` to `49`** | **Elevated Background Noise** | Typical of shared NAT gateways or historical low-volume probing; log for telemetry, apply standard authenticated rate limits. |
| **`50` to `74`** | **Active Reconnaissance / Suspected Automation** | Require Proof-of-Work / Turnstile / CAPTCHA on unauthenticated endpoints; tighten per-IP rate limits by `4x`. |
| **`75` to `100`** | **High-Velocity Attacker / Exploit Botnet** | Block unauthenticated mutations (`/login`, `/signup`, `/password-reset`, `/checkout`); drop scanner probes at the edge WAF before hitting origin databases. |

## Production implementation: Adaptive rate limiting and scanner blocking in Node.js

Here is how to combine `connection`, `security`, and `security_extensions` in an Express, Fastify, or Next.js edge middleware to block active scanners and throttle brute-force bots without ever locking out shared 5G cellular CGNAT users:

```ts
import { IpScopeClient } from '@ipscope/sdk-node';

const ipscope = new IpScopeClient({
  apiKey: process.env.IPSCOPE_KEY!,
  maxCacheEntries: 16_384,
  cacheTtlMs: 60_000,
});

export interface EdgeInspectionResult {
  action: 'allow' | 'throttle' | 'challenge' | 'block';
  rateLimitPerMinute: number;
  reason: string;
}

export async function inspectIncomingRequest(
  clientIp: string,
  requestPath: string,
  isAuthenticatedSession: boolean,
): Promise<EdgeInspectionResult> {
  const res = await ipscope.lookupIp(clientIp);
  const { connection, security, security_extensions } = res;

  const velocity = security_extensions?.abuse_velocity_score ?? 0;
  const isScanner = security_extensions?.is_known_scanner ?? false;
  const recencyHours = security_extensions?.observation_recency_hours ?? 999;

  const isSensitiveAuthRoute =
    requestPath.startsWith('/api/auth/') ||
    requestPath.startsWith('/api/checkout');

  // 1. Drop active high-velocity scanners and exploit bots on unauthenticated requests
  if (!isAuthenticatedSession && (velocity >= 80 || (isScanner && recencyHours <= 6))) {
    return {
      action: 'block',
      rateLimitPerMinute: 0,
      reason: `high_abuse_velocity:${velocity}:recency_${recencyHours}h`,
    };
  }

  // 2. Challenge medium-high velocity IPs or open proxies hitting auth/checkout routes
  if (isSensitiveAuthRoute && (velocity >= 50 || security.is_proxy || security.is_tor_exit)) {
    return {
      action: 'challenge',
      rateLimitPerMinute: 10,
      reason: `sensitive_route_challenge:velocity_${velocity}:type_${security.anonymizer_type}`,
    };
  }

  // 3. Shared 4G/5G Mobile CGNAT gateways: allocate high aggregate concurrency bucket
  //    so 4,000 mobile users sharing one carrier IP do not starve each other
  if (connection?.is_mobile && !security.is_vpn && !security.is_proxy) {
    return {
      action: 'allow',
      rateLimitPerMinute: 1200,
      reason: 'mobile_cgnat_shared_pool',
    };
  }

  // 4. Commercial VPN or raw datacenter egress: tighter bucket on unauthenticated traffic
  if (security.is_vpn || security.is_hosting) {
    return {
      action: velocity >= 35 ? 'throttle' : 'allow',
      rateLimitPerMinute: isAuthenticatedSession ? 120 : 30,
      reason: `hosting_or_vpn_egress:${security.anonymizer_type}`,
    };
  }

  // 5. Clean residential or enterprise business broadband
  return {
    action: 'allow',
    rateLimitPerMinute: 300,
    reason: 'clean_access_network',
  };
}
```

## Enforcing fail2ban-style SSH and admin protection in Go

For infrastructure teams protecting SSH bastions, Kubernetes API servers, or VPN concentrators, you can query `security_extensions` directly in Go (`github.com/badele-llc/ipscope-go`) or via a local `.mmdb` database to reject known internet scanners before spending CPU cycles on cryptographic key exchange:

```go
package main

import (
	"context"
	"fmt"
	"os"
	"time"

	ipscope "github.com/badele-llc/ipscope-go"
)

func ShouldAllowAdminIngress(ctx context.Context, client *ipscope.Client, remoteIP string) (bool, string) {
	res, err := client.LookupIP(ctx, remoteIP)
	if err != nil {
		// Fail-open to local auth if network lookup times out
		return true, "lookup_fallback"
	}

	if res.SecurityExtensions != nil {
		if res.SecurityExtensions.IsKnownScanner && res.SecurityExtensions.AbuseVelocityScore >= 60 {
			return false, fmt.Sprintf(
				"blocked_scanner(score=%d,recency=%dh)",
				res.SecurityExtensions.AbuseVelocityScore,
				res.SecurityExtensions.ObservationRecencyHours,
			)
		}
	}

	if res.Security.IsTorExit || res.Security.IsProxy {
		return false, fmt.Sprintf("blocked_anonymizer(%s)", res.Security.AnonymizerType)
	}

	return true, "allowed"
}

func main() {
	client := ipscope.NewClient(os.Getenv("IPSCOPE_KEY"), ipscope.WithTimeout(2*time.Second))
	allowed, reason := ShouldAllowAdminIngress(context.Background(), client, "198.51.100.214")
	fmt.Printf("allowed=%v reason=%s\n", allowed, reason)
}
```

## Frequently asked questions (FAQ)

### What is the difference between `security.confidence` and `security_extensions.abuse_velocity_score`?
They measure two completely orthogonal properties:

- **`security.confidence` (`0.0` to `1.0`)** measures our structural certainty that an IP address belongs to an anonymizing service (such as a commercial VPN, SOCKS proxy, or Tor exit). A clean, privacy-focused commercial VPN server may have `security.is_vpn: true, security.confidence: 0.99` while having an `abuse_velocity_score` of `0` if no hostile scanning or brute-force traffic has originated from it.
- **`security_extensions.abuse_velocity_score` (`0` to `100`)** measures the intensity, target breadth, and recency of hostile scanning or exploit traffic observed from that IP address, regardless of whether the IP is a VPN, a compromised cloud VM, or an infected home router.

### How does `observation_recency_hours` help me avoid false positives on offline `.mmdb` databases?
If your infrastructure downloads the `.mmdb` database on a **Weekly** or **Monthly** schedule rather than **Daily**, you can inspect `observation_recency_hours` alongside the database's `build_epoch` to see how fresh the abuse observation was at compile time. For customers using `security_extensions` in offline `.mmdb` or `.parquet` deployments, we strongly recommend configuring your database subscription to the **Daily (`06:00 UTC`)** update cadence so exponential score decay and newly detected scanner IPs are refreshed every 24 hours.

---

<!-- Article URL: https://ipscope.io/blog/ipscope-vs-maxmind-ipinfo-iplocate-ipgeolocation-ipscanner -->
---
slug: ipscope-vs-maxmind-ipinfo-iplocate-ipgeolocation-ipscanner
title: "ipscope vs. MaxMind, IPinfo, IPLocate, IPGeolocation, and IPScanner: The 2026 Engineering Comparison"
description: "An architectural and pricing comparison of IP intelligence providers: evaluating declared vs. physical geolocation, VPN and residential proxy detection, MMDB and Parquet database parity, and total cost of ownership."
date: "2026-10-27"
readingTime: "12 min read"
category: "Engineering"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "Comparison, MaxMind, IPinfo, IPLocate, IPGeolocation, IPScanner, MMDB, Geolocation"
featured: "true"
---

## Executive summary: Why engineering teams switch to ipscope

Choosing an IP intelligence provider in 2026 affects three critical layers of production infrastructure: **routing and localization accuracy**, **fraud and anonymizer detection**, and **unit economics across API and self-hosted database deployments**.

Legacy providers built their data pipelines when IPv4 blocks rarely crossed national borders and administrative allocation records matched physical router geography. Today, cloud operators lease `/22` and `/24` blocks across continents, commercial VPNs rotate residential proxy exits hourly, and data engineering teams require identical schemas across real-time edge APIs, local `.mmdb` binary trees, and columnar `.parquet` data lake tables.

**ipscope** was architected from the BGP table up to solve the structural blind spots of legacy providers:

1. **Separated physical vs. declared geolocation**: Unlike MaxMind, IPLocate.io, ipgeolocation.io, and ipscanner.io (which return a single country field that frequently reflects stale registration paperwork or unverified geofeeds), ipscope returns `country_code`, `declared_country_code`, `physical_country_code`, and `is_virtual_location` on every lookup, backed by multi-vantage active RTT latency triangulation.
2. **One canonical schema across API, MMDB, CSV, and Parquet**: Every ipscope database build ships in `.mmdb`, `.csv.gz`, and `.parquet` with the exact same 6 field families (`geo_country`, `geo_city`, `network`, `connection`, `anonymity`, `security_extensions`) returned by `GET /v1/ip/{ip}`.
3. **Self-serve modular database licensing without enterprise gatekeeping**: While IPinfo.io gates full downloadable databases behind custom enterprise contracts starting around `$1,000+/mo` and MaxMind splits City, ISP, and Anonymous IP into separate expensive `.mmdb` files, ipscope offers transparent modular database pricing starting at `$29/mo` (or `$279/mo` for the Complete 6-block database).
4. **No credit multipliers for security and VPN fields**: Unlike credit-based APIs that charge 2x to 4x credits when enabling threat or proxy flags, 1 IP lookup equals 1 request on ipscope regardless of how many licensed field families you query.

## Head-to-head feature and architecture matrix

The table below compares **ipscope** against **MaxMind (GeoIP2)**, **IPinfo.io**, **IPLocate.io**, **ipgeolocation.io**, and **ipscanner.io** across the capabilities backend and security engineers evaluate in production:

| Capability | ipscope | MaxMind (GeoIP2) | IPinfo.io | IPLocate.io | ipgeolocation.io | ipscanner.io |
| --- | --- | --- | --- | --- | --- | --- |
| **Physical vs. declared country split** | **Yes** (`physical_country_code` + `declared_country_code` + `is_virtual_location`) | No (single country field) | No (single country field) | No (single country field) | No (single country field) | No (single country field) |
| **Active RTT triangulation** | **Yes** (multi-vantage speed-of-light verification) | Partial (proprietary model, no split field) | Yes (active probing, single country output) | No (registration + geofeed aggregation) | No (registration + ISP feeds) | No (edge threat focus) |
| **Downloadable formats included** | **MMDB, CSV.gz, Parquet** (all 3 included) | MMDB, CSV (no native Parquet) | MMDB, CSV, Parquet (Enterprise tiers) | MMDB, CSV | CSV, MMDB (selected tiers) | None (API & edge worker focus) |
| **Single unified MMDB file for all blocks** | **Yes** (1 `.mmdb` or custom SKU mix) | No (separate City, ISP, Anonymous IP `.mmdb` files) | Paid Enterprise bundle | Yes (tier-dependent) | Separate Geo and Security DBs | N/A |
| **VPN, Proxy, Tor & Hosting detection** | **Yes** (`is_vpn`, `is_proxy`, `is_tor_exit`, `is_hosting`, `anonymizer_type`) | Requires separate GeoIP2 Anonymous IP SKU | Yes (Core/Plus tiers and above) | Yes (included across tiers) | Yes (Security plan / extra credits) | Yes (primary product focus) |
| **Abuse velocity & scanner telemetry** | **Yes** (`abuse_velocity_score`, `known_attacker`, `is_scanner`, `is_bot`) | Excluded from standard GeoIP2 DBs | Separate Abuse / Enterprise dataset | Basic threat flag | Threat score on higher tiers | Bot & signup gate rules |
| **Batch API throughput** | **100 IPs / request** (`POST /v1/ip/batch`) + `O(1)` streaming SDKs | No native batch endpoint in standard GeoIP2 WS | Batch endpoint supported | Bulk lookup supported | Bulk lookup up to 50 IPs | Single IP / edge middleware |
| **Official typed SDKs** | **8 languages** (Python, Node, Go, Rust, Java, .NET, Ruby, PHP) | 7 languages (MMDB readers + Web Service clients) | Official SDKs available | Community & core wrappers | Official wrappers | JS & edge middleware |
| **Self-serve database starting price** | **$29/mo** (modular) or **$279/mo** (Complete 6-block) | ~$100/mo to $1,100+/mo across split SKUs | ~$1,000+/mo (Enterprise DB contracts) | ~$100/mo+ | ~$150/mo+ per dataset | No offline database catalog |

## Deep dive 1: ipscope vs. MaxMind (GeoIP2 and GeoLite2)

MaxMind created the `.mmdb` binary format and remains the default baseline in legacy stacks. However, engineering teams migrating off MaxMind GeoIP2 and GeoLite2 consistently cite three operational pain points:

### 1. Split `.mmdb` files and fragmented schemas
In MaxMind's catalog, city geolocation (`GeoIP2-City.mmdb`), ASN/ISP metadata (`GeoIP2-ISP.mmdb`), connection classification (`GeoIP2-Connection-Type.mmdb`), and VPN/proxy flags (`GeoIP2-Anonymous-IP.mmdb`) are distributed as separate binary files. To enrich a single incoming request with location, ASN, and VPN detection at the edge, your service must open, memory-map, and traverse **three to four separate radix trees** per lookup.

In **ipscope**, a single `.mmdb` file (or your custom-configured SKU) stores all 6 intelligence blocks in one unified binary radix tree. One `O(prefix_length)` pointer walk resolves geolocation, BGP origin ASN, connection type, and anonymizer telemetry in under `400 nanoseconds`. Better yet, ipscope `.mmdb` artifacts adhere strictly to the open MMDB binary specification v2.0, meaning you can load an ipscope `.mmdb` file into your existing reader library with a one-line file path change.

```python
import maxminddb

# Drop-in replacement: open a single unified ipscope MMDB file
with maxminddb.open_database("/var/lib/ipscope/ipscope-complete.mmdb") as reader:
    record = reader.get("198.51.100.42")
    # Single lookup resolves physical geo, BGP ASN, and VPN telemetry together
    print(record["geo"]["physical_country_code"])  # "DE"
    print(record["geo"]["declared_country_code"])  # "US"
    print(record["network"]["asn"])                # 24940
    print(record["security"]["is_vpn"])            # True
```

### 2. Registration bias in GeoLite2 and slow geofeed correction
MaxMind GeoLite2 relies heavily on administrative allocation records and unverified operator geofeeds, resulting in widespread misclassifications whenever a US-registered cloud or CDN announces an IPv4 prefix out of a European or Asian datacenter. Even in paid GeoIP2, MaxMind outputs only one `country.iso_code` field, forcing your application to guess whether that country represents the legal registrant or the physical router.

**ipscope** exposes both dimensions explicitly (`physical_country_code` and `declared_country_code`) alongside `is_virtual_location`, allowing payment compliance systems and low-latency edge routers to make deterministic decisions.

### 3. Native Apache Parquet included at no extra cost
MaxMind distributes databases only in `.mmdb` and `.csv`. Data engineering teams running columnar data warehouses and analytical query engines must build and maintain custom ETL jobs to convert multi-gigabyte CSV archives into typed columnar tables. Every ipscope database subscription includes pre-compiled Zstandard-compressed `.parquet` files with integer `ip_start_int` / `ip_end_int` bounds for instant `O(log N)` range joins.

## Deep dive 2: ipscope vs. IPinfo.io

IPinfo.io is a strong competitor known for its active measurement network and developer ergonomics. When comparing **ipscope vs. IPinfo.io**, the decisive differences center on **data transparency**, **API plan granularity**, and **database accessibility**:

1. **Database pricing accessibility (`$279/mo` vs. `$1,000+/mo`)**: IPinfo.io positions its full downloadable databases (especially bundles including Privacy/VPN detection and Company data) as high-ticket enterprise contracts frequently starting at `$1,000+/mo` with annual commitments. **ipscope** makes every database SKU self-serve from day one: you can license the standalone Privacy & VPN database for `$99/mo` or the Complete 6-block database (Geolocation + ASN + Company + Privacy + Abuse Velocity) for `$279/mo` on a monthly build schedule (`$419/mo` for nightly daily builds).
2. **Physical vs. declared country separation**: While IPinfo uses active probing to select a single country code, it overwrites the declared registration country in the response. Security and compliance teams frequently need to know *when* a discrepancy exists (for example, an IP legally registered to an offshore entity but physically routed through a rack in Amsterdam). ipscope's `declared_country_code !== physical_country_code` check surfaces this signal natively.
3. **Predictable API quotas and wallet overages**: IPinfo's mid-market plans step abruptly from Core (`~$49/mo` for 150k requests) to Plus/Max (`~$163+/mo` for 250k requests). **ipscope** offers a `$49/mo` Starter plan with **250,000 requests/mo**, a `$149/mo` Growth plan with **1,000,000 requests/mo** (including full Privacy & VPN detection), and a `$399/mo` Business plan with **5,000,000 requests/mo** and all 6 intelligence blocks, backed by a flat `$0.10 per 1,000` prepaid wallet overage rate so production traffic never hits a hard 429 wall.

## Deep dive 3: ipscope vs. IPLocate.io

IPLocate.io offers a clean developer API with a generous daily free tier (`1,000 req/day`). However, as engineering teams scale into production fraud prevention, SIEM enrichment, and high-QPS edge routing, they encounter key differences between **ipscope and IPLocate.io**:

- **Active RTT verification vs. passive feed aggregation**: IPLocate.io primarily aggregates administrative allocation records, BGP routing tables, and published RFC 8805 geofeeds. When an operator publishes a spoofed or stale RFC 8805 geofeed (a common technique used by proxy providers to make datacenter IPs appear in residential metros), passive aggregators ingest the spoofed location directly. **ipscope** runs every geofeed claim through an automated multi-vantage ICMP/TCP RTT triangulation gate: if the speed-of-light latency floor from our nearest vantage probes contradicts the claimed city, ipscope overrides `physical_country_code` with the empirically measured router location and flags `is_virtual_location: true`.
- **Abuse velocity and attacker telemetry**: Beyond static boolean flags (`is_vpn`, `is_proxy`), ipscope's `security_extensions` block tracks rolling 24-hour scanner and brute-force velocity (`abuse_velocity_score` from `0` to `100`, `known_attacker`, `is_scanner`, `is_bot`, and `last_seen_hours_ago`).
- **Columnar Parquet & Custom SKU Compilation**: ipscope compiles custom database SKUs containing only the exact field families your service licenses, minimizing memory footprint on edge nodes while shipping `.mmdb`, `.csv.gz`, and `.parquet` simultaneously.

## Deep dive 4: ipscope vs. ipgeolocation.io

ipgeolocation.io provides geolocation, astronomy, currency, and security endpoints. When backend architects evaluate **ipscope vs. ipgeolocation.io**, three structural differences stand out:

1. **No credit multipliers on security lookups**: On ipgeolocation.io, querying standard geolocation costs 1 credit, but enabling security/threat detection or bulk enrichment can consume multiple credits per IP, multiplying your effective monthly cost. On **ipscope**, 1 IP address always equals 1 lookup unit (`$0.10 / 1K` overage), whether you request `geo_country` alone or all 6 field families including `anonymity` and `security_extensions`.
2. **BGP prefix and network hierarchy depth**: ipscope treats BGP routing as a first-class primitive, returning the exact longest-prefix match (`matched_network`), `prefix_length`, origin `asn`, `organization`, `allocation_status`, and dedicated `/v1/asn/{asn}` and `/v1/network/{cidr}` endpoints.
3. **Strict latency SLOs and constant-memory streaming SDKs**: Instead of untyped REST wrappers, ipscope maintains 8 idiomatic, zero-dependency SDKs with bounded LRU caching, automatic jittered backoff, and `lookupStream()` iterators capable of enriching 100,000+ IPs in constant `O(1)` memory.

## Deep dive 5: ipscope vs. ipscanner.io

ipscanner.io is designed primarily as a website traffic filter, CMS plugin, and edge middleware gate for blocking bots and disposable signups. While useful for no-code site protection, **ipscope** serves a fundamentally broader infrastructure and data engineering scope:

- **Full-stack data ownership (API + MMDB + CSV + Parquet)**: ipscanner.io is an API and edge-rule service without downloadable `.mmdb` binary databases or `.parquet` warehouse datasets. If your architecture requires zero-latency (`< 1 microsecond`) in-process lookups inside Go, Rust, or Java microservices, or offline SQL enrichment in your data warehouse, ipscope provides the complete offline database artifact pipeline alongside its global Anycast API.
- **Granular schema for custom risk engines**: Rather than coupling IP intelligence to opinionated website-blocking rules, ipscope gives your fraud, FinTech, or ad-verification engine raw, calibrated primitives: `geo.confidence`, `security.confidence`, `security.anonymizer_type`, `connection.connection_type`, and `security_extensions.abuse_velocity_score`.

## Pricing and total cost of ownership (TCO) comparison

Below is a realistic TCO comparison for two common production workloads:

| Workload Scenario | ipscope | MaxMind (GeoIP2) | IPinfo.io | IPLocate.io | ipgeolocation.io |
| --- | --- | --- | --- | --- | --- |
| **Scenario A: 1M API lookups/mo with City Geo + ASN + VPN/Proxy detection** | **$149/mo** (Growth plan: 1M lookups, 100 req/s, Geo + ASN + Company + Privacy included) | ~$300+/mo in GeoIP2 Insights web service credits | ~$163 to $250+/mo (Plus/Max tier required for full privacy + 1M volume) | ~$99 to $150/mo | ~$99 to $199/mo (higher with security credit multiplier) |
| **Scenario B: Self-hosted MMDB + Parquet database with City Geo + ASN + VPN/Proxy** | **$279/mo** (Complete 6-block DB, monthly updates, MMDB + CSV + Parquet included) | ~$650+/mo (requires buying GeoIP2 City + GeoIP2 ISP + GeoIP2 Anonymous IP separately; no Parquet) | ~$1,000+/mo (Enterprise custom database contract) | ~$250 to $400+/mo (no native Parquet) | Separate Geo + Security DB licenses required |

## Migrating to ipscope in under 10 minutes

Whether you are replacing a REST API integration or swapping out a local `.mmdb` file, migrating to ipscope requires minimal code changes:

```typescript
import { IpScopeClient } from '@ipscope/sdk-node';

const client = new IpScopeClient({
  apiKey: process.env.IPSCOPE_API_KEY!,
  cacheMaxEntries: 10_000,
  cacheTtlMs: 60_000,
});

// Single lookup with physical vs. declared country, ASN, and VPN telemetry
const res = await client.lookupIp('198.51.100.7');

if (res.security.is_vpn || res.geo.is_virtual_location) {
  console.log({
    ip: res.ip,
    declaredCountry: res.geo.declared_country_code,
    physicalCountry: res.geo.physical_country_code,
    asn: res.network.asn,
    anonymizer: res.security.anonymizer_type,
    abuseScore: res.security_extensions?.abuse_velocity_score,
  });
}
```

## Frequently asked questions (FAQ)

### Can ipscope `.mmdb` files be opened with standard MMDB readers?
Yes. Every ipscope `.mmdb` database is compiled strictly to the open MMDB binary specification v2.0. You can open ipscope `.mmdb` files using standard open-source MMDB readers in Python, Node.js, Go, Rust, Java, C, or NGINX without changing your reader library.

### Why does ipscope return three country fields instead of one?
Because declared network registration (`declared_country_code`) and physical router location (`physical_country_code`) disagree on millions of leased and anycast IPv4/IPv6 prefixes. Returning `country_code` (best operational country), `declared_country_code`, `physical_country_code`, and `is_virtual_location` gives compliance, tax, and low-latency routing engines complete visibility.

### Which provider is best for offline SIEM and data warehouse enrichment?
**ipscope** is purpose-built for data engineering and SIEM workloads because every database subscription includes Zstandard-compressed Apache **Parquet** (`.parquet`), compressed **CSV** (`.csv.gz`), and **MMDB** (`.mmdb`) artifacts at no additional format fee, complete with pre-computed integer IP bounds for fast range joins in columnar SQL engines and data warehouses.

---

<!-- Article URL: https://ipscope.io/blog/parquet-and-duckdb-for-siem-ip-enrichment -->
---
slug: parquet-for-siem-ip-enrichment
title: "Vectorized IP Intelligence: Joining 1 Billion Log Rows Against ipscope Parquet"
description: "How to use the ZSTD-compressed Apache Parquet artifact included in every ipscope database subscription to enrich massive SIEM, CDN, and firewall logs in DuckDB, ClickHouse, and Spark."
date: "2026-09-28"
readingTime: "15 min read"
category: "Databases"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "Parquet, DuckDB, ClickHouse, SIEM, Analytics, Columnar SQL"
featured: "false"
---

## Why row-by-row `.mmdb` lookups bottleneck analytical SIEM workloads

In real-time edge proxies, API gateways, and transactional services, nothing beats a memory-mapped `.mmdb` binary radix tree. Resolving a single incoming client IP in `500 nanoseconds` via `mmap` is ideal when requests arrive one at a time over the network.

However, data engineering and security operations (SecOps) teams face a completely different access pattern when performing retrospective threat hunting, compliance auditing, or CDN traffic analysis:

- You have **1 billion historical firewall, VPC Flow, or Cloudflare access log rows** stored in Apache Parquet or Iceberg tables on S3/R2.
- You want to enrich every row with `physical_country_code`, `declared_country_code`, `asn`, `is_vpn`, `is_proxy`, and `abuse_velocity_score`, and aggregate the top attacking ASNs over a 30-day window.

If you register a Python or Java User-Defined Function (UDF) inside Spark, Trino, or Snowflake to call an `.mmdb` reader row-by-row across 1 billion records, you break the query engine's vectorized execution pipeline. The engine has to materialize scalar values out of columnar SIMD registers, cross a language runtime boundary 1 billion times, and give up all Parquet row-group min/max zone-map pruning.

That is why **every ipscope database subscription automatically compiles and delivers three synchronized artifact formats** on every release (`Daily`, `Weekly`, or `Monthly`):

1. **`ipscope-*.mmdb`**: Open MMDB v2.0 binary radix tree for real-time edge and application servers.
2. **`ipscope-*.parquet`**: ZSTD-compressed, physical-range-indexed columnar Apache Parquet for vectorized OLAP and SIEM engines (`DuckDB`, `ClickHouse`, `Apache Spark`, `Snowflake`, `BigQuery`, `Trino`, `Polars`).
3. **`ipscope-*.csv.gz`**: RFC 4180 gzipped CSV for traditional relational `COPY` pipelines (`PostgreSQL`, `MySQL`).

This engineering guide details the physical schema of `ipscope-*.parquet`, explains how our compiler embeds pre-computed integer boundary columns (`ip_start_int`, `ip_end_int`) for `O(log N)` row-group skipping, and provides copy-pasteable SQL recipes for DuckDB and ClickHouse.

## Physical layout and schema of `ipscope-*.parquet`

In a nested JSON or `.mmdb` record, fields are grouped into sub-maps (`geo`, `network`, `connection`, `security`, `security_extensions`). For analytical SQL engines, unpacking nested `STRUCT` columns during a 1-billion-row scan wastes CPU cycles.

Our Parquet compiler flattens all licensed intelligence families into top-level, strongly typed physical columns, **physically sorted by `(ip_version, network_start_bin)`** and compressed with `ZSTD (level 9)` + dictionary encoding:

```sql
DESCRIBE SELECT * FROM read_parquet('/var/lib/ipscope/ipscope-complete.parquet');
```

| Column Name | Parquet Logical Type | Intelligence Family | Description & Encoding |
| --- | --- | --- | --- |
| `cidr` | `VARCHAR` | `network` | Canonical CIDR string (`"198.51.100.0/24"`, `"2001:db8::/48"`) |
| `ip_version` | `UTINYINT` (`4` or `6`) | `network` | Address family (`4` for IPv4, `6` for IPv6) |
| `ipv4_start_u32` | `UINTEGER` (`uint32`) | `network` | Inclusive start of IPv4 CIDR as big-endian `uint32` (`NULL` for IPv6) |
| `ipv4_end_u32` | `UINTEGER` (`uint32`) | `network` | Inclusive end of IPv4 CIDR as big-endian `uint32` (`NULL` for IPv6) |
| `country_code` | `VARCHAR` (Dict) | `geo_country` | Effective ISO-3166-1 alpha-2 country code |
| `declared_country_code` | `VARCHAR` (Dict) | `geo_country` | Registered / declared ISO-3166-1 alpha-2 country code |
| `physical_country_code` | `VARCHAR` (Dict) | `geo_country` | Physically verified router termination country code |
| `is_virtual_location` | `BOOLEAN` (RLE) | `geo_country` | `true` when `declared_country_code != physical_country_code` |
| `location_type` | `VARCHAR` (Dict) | `geo_country` | `"physical"`, `"virtual"`, or `"anycast"` |
| `geo_confidence` | `FLOAT` | `geo_country` | Geolocation confidence score (`0.0` to `1.0`) |
| `subdivision` | `VARCHAR` (Dict) | `geo_city` | State / province name (`"Hesse"`, `"Virginia"`) |
| `city` | `VARCHAR` (Dict) | `geo_city` | City name (`"Frankfurt am Main"`, `"Ashburn"`) |
| `latitude` | `DOUBLE` | `geo_city` | WGS84 centroid latitude |
| `longitude` | `DOUBLE` | `geo_city` | WGS84 centroid longitude |
| `accuracy_radius_km` | `UINTEGER` | `geo_city` | Calibrated accuracy radius in kilometers |
| `timezone` | `VARCHAR` (Dict) | `geo_city` | IANA time zone (`"Europe/Berlin"`) |
| `asn` | `UINTEGER` | `network` | Autonomous System Number (`24940`, `13335`) |
| `organization` | `VARCHAR` (Dict) | `network` | Autonomous System / ISP organization name |
| `rir` | `VARCHAR` (Dict) | `network` | Regional Internet Registry (`ARIN`, `RIPE`, `APNIC`, `LACNIC`, `AFRINIC`) |
| `connection_type` | `VARCHAR` (Dict) | `connection` | `"residential"`, `"mobile"`, `"business"`, or `"hosting"` |
| `is_residential` | `BOOLEAN` (RLE) | `connection` | Fixed consumer broadband indicator |
| `is_mobile` | `BOOLEAN` (RLE) | `connection` | Cellular 4G/5G CGNAT indicator |
| `is_business` | `BOOLEAN` (RLE) | `connection` | Enterprise / corporate leased line indicator |
| `is_hosting` | `BOOLEAN` (RLE) | `hosting` | Cloud, VPS, or colocation datacenter indicator |
| `is_vpn` | `BOOLEAN` (RLE) | `anonymity` | Active commercial or protocol-verified VPN exit |
| `is_tor_exit` | `BOOLEAN` (RLE) | `anonymity` | Hourly verified Tor exit relay |
| `is_proxy` | `BOOLEAN` (RLE) | `anonymity` | Open HTTP/SOCKS or residential proxy exit |
| `anonymizer_type` | `VARCHAR` (Dict) | `anonymity` | `"none"`, `"vpn"`, `"proxy"`, `"tor"`, or `"private_relay"` |
| `security_confidence` | `FLOAT` | `anonymity` | Security classification confidence (`0.0` to `1.0`) |
| `reason_codes` | `LIST<VARCHAR>` | `anonymity` | Deterministic evidence codes (`["vpn_handshake_confirmed", ...]`) |
| `abuse_velocity_score` | `UTINYINT` (`0..100`) | `security_extensions` | Rolling time-decayed hostile traffic score |
| `is_known_scanner` | `BOOLEAN` (RLE) | `security_extensions` | Active internet-wide scanner / reconnaissance host |
| `observation_recency_hours` | `UINTEGER` | `security_extensions` | Hours since most recent hostile telemetry hit |

### Why column projection + ZSTD + Run-Length Encoding (RLE) is so fast

In `ipscope-complete.parquet`, boolean columns (`is_vpn`, `is_hosting`, `is_mobile`, `is_virtual_location`) use Parquet **Run-Length / Bit-Packing Hybrid Encoding**, and low-cardinality string columns (`country_code`, `timezone`, `organization`, `connection_type`, `anonymizer_type`) use **Dictionary Encoding**.

Furthermore, when a SIEM query only needs `cidr`, `physical_country_code`, `asn`, `is_vpn`, and `abuse_velocity_score`, the Parquet reader only reads the byte pages for those 5 columns from disk and completely skips reading `city`, `latitude`, `longitude`, `subdivision`, and `timezone`. Across 35.3 million prefixes, projecting 5 columns scans less than **18 MB of compressed bytes**.

## Recipe 1: Instant ad-hoc threat hunting with DuckDB (zero database setup)

[DuckDB](https://duckdb.org/) is an in-process vectorized SQL OLAP engine that reads Parquet files directly from disk or S3/HTTP without loading them into a server first.

### Querying global VPN and scanner concentration by ASN and physical country

```sql
-- Run directly via: duckdb -c "..."
SELECT
  asn,
  organization,
  physical_country_code AS physical_country,
  declared_country_code AS registered_country,
  COUNT(*) AS total_prefixes,
  SUM(CASE WHEN is_vpn THEN 1 ELSE 0 END) AS vpn_prefixes,
  SUM(CASE WHEN is_proxy THEN 1 ELSE 0 END) AS proxy_prefixes,
  ROUND(AVG(abuse_velocity_score), 1) AS avg_abuse_velocity
FROM read_parquet('/var/lib/ipscope/ipscope-complete.parquet')
WHERE (is_vpn = true OR is_proxy = true OR is_known_scanner = true)
GROUP BY
  asn,
  organization,
  physical_country_code,
  declared_country_code
ORDER BY vpn_prefixes DESC, avg_abuse_velocity DESC
LIMIT 15;
```

### Enriching 50 million access log rows in DuckDB using `ipv4_start_u32` range joins

Because the ipscope compiler pre- flattens non-overlapping canonical leaves and stores `ipv4_start_u32` and `ipv4_end_u32` sorted in strictly ascending order, DuckDB's **IEJoin / AsOf Range Join** (`BETWEEN ipv4_start_u32 AND ipv4_end_u32`) resolves millions of log rows in seconds using binary search over Parquet row-group min/max statistics:

```sql
-- Convert dotted-quad IPv4 string to uint32 and join against non-overlapping ipscope leaves
WITH parsed_logs AS (
  SELECT
    request_id,
    timestamp,
    client_ip,
    -- Convert '198.51.100.7' -> 3325256711 (uint32)
    (
      (CAST(split_part(client_ip, '.', 1) AS UBIGINT) << 24) |
      (CAST(split_part(client_ip, '.', 2) AS UBIGINT) << 16) |
      (CAST(split_part(client_ip, '.', 3) AS UBIGINT) << 8)  |
       CAST(split_part(client_ip, '.', 4) AS UBIGINT)
    )::UINTEGER AS ip_u32
  FROM read_parquet('s3://my-secops-lake/Logs/2026-10-28/*.parquet')
  WHERE strpos(client_ip, ':') = 0
),
ipscope_v4 AS (
  SELECT
    ipv4_start_u32,
    ipv4_end_u32,
    cidr,
    physical_country_code,
    declared_country_code,
    is_virtual_location,
    asn,
    organization,
    connection_type,
    is_vpn,
    is_proxy,
    is_tor_exit,
    anonymizer_type,
    abuse_velocity_score
  FROM read_parquet('/var/lib/ipscope/ipscope-complete.parquet')
  WHERE ip_version = 4
)
SELECT
  l.request_id,
  l.timestamp,
  l.client_ip,
  i.cidr AS matched_network,
  i.physical_country_code,
  i.declared_country_code,
  i.is_virtual_location,
  i.asn,
  i.organization,
  i.connection_type,
  i.anonymizer_type,
  i.abuse_velocity_score
FROM parsed_logs l
ASOF JOIN ipscope_v4 i
  ON l.ip_u32 >= i.ipv4_start_u32
WHERE l.ip_u32 <= i.ipv4_end_u32;
```

Look closely at why `ASOF JOIN ... ON l.ip_u32 >= i.ipv4_start_u32 WHERE l.ip_u32 <= i.ipv4_end_u32` is a game-changer for analytical SQL:

- Standard inequality joins (`ON l.ip_u32 >= i.ipv4_start_u32 AND l.ip_u32 <= i.ipv4_end_u32`) often degrade to `O(N * M)` nested-loop comparisons in SQL engines that lack specialized interval trees.
- Because the ipscope compiler outputs a **disjoint, non-overlapping partition of the address space** sorted by `ipv4_start_u32`, finding the prefix containing `l.ip_u32` is mathematically equivalent to finding the single row with the largest `ipv4_start_u32 <= l.ip_u32` (`ASOF JOIN`) and checking `l.ip_u32 <= i.ipv4_end_u32`.
- DuckDB executes `ASOF JOIN` in `O(N log M)` time using vectorized merge/binary search!

## Recipe 2: Sub-second enrichment of 1 billion rows in ClickHouse (`ip_trie` Dictionary)

If your observability or SIEM stack runs on **ClickHouse**, you can load `ipscope-complete.parquet` directly into ClickHouse's native in-memory **`ip_trie` dictionary** structure.

ClickHouse's `ip_trie` layout builds a compact bitwise radix tree in RAM for both IPv4 and IPv6 CIDRs, allowing ClickHouse queries to enrich 1 billion log rows at memory-bandwidth speeds (`> 40 million lookups per second per node`):

```sql
-- 1. Create a staging table backed by your downloaded ipscope Parquet file
CREATE OR REPLACE TABLE ipscope_staging
ENGINE = MergeTree
ORDER BY (ip_version, cidr)
AS SELECT *
FROM file('/var/lib/clickhouse/user_files/ipscope-complete.parquet', Parquet);

-- 2. Create the in-memory ip_trie dictionary over both IPv4 and IPv6 CIDRs
CREATE OR REPLACE DICTIONARY ipscope_dict
(
  cidr String,
  country_code String,
  declared_country_code String,
  physical_country_code String,
  is_virtual_location UInt8,
  city String,
  asn UInt32,
  organization String,
  connection_type String,
  is_vpn UInt8,
  is_proxy UInt8,
  is_tor_exit UInt8,
  anonymizer_type String,
  abuse_velocity_score UInt8
)
PRIMARY KEY cidr
SOURCE(CLICKHOUSE(TABLE 'ipscope_staging'))
LAYOUT(IP_TRIE())
LIFETIME(MIN 3600 MAX 7200);
```

Once `ipscope_dict` is loaded, enriching any IPv4 or IPv6 column in your ClickHouse `http_access_logs` table requires a single vectorized `dictGet` call:

```sql
SELECT
  dictGet('ipscope_dict', 'physical_country_code', toIPv6(client_ip)) AS physical_country,
  dictGet('ipscope_dict', 'declared_country_code', toIPv6(client_ip)) AS declared_country,
  dictGet('ipscope_dict', 'asn', toIPv6(client_ip)) AS asn,
  dictGet('ipscope_dict', 'organization', toIPv6(client_ip)) AS org,
  dictGet('ipscope_dict', 'anonymizer_type', toIPv6(client_ip)) AS anonymizer,
  count() AS request_volume,
  countIf(status_code >= 400) AS error_volume
FROM default.http_access_logs
WHERE event_date >= today() - 7
GROUP BY
  physical_country,
  declared_country,
  asn,
  org,
  anonymizer
HAVING anonymizer != 'none'
ORDER BY request_volume DESC
LIMIT 25;
```

## Recipe 3: Loading `ipscope-*.csv.gz` into PostgreSQL `inet` / `gist` tables

For teams running **PostgreSQL 15+** with the native `inet` and `cidr` data types, every database subscription also includes `ipscope-*.csv.gz`. By indexing the `cidr` column with PostgreSQL's built-in `gist (cidr inet_ops)` operator class, PostgreSQL can execute subnet containment joins (`client_ip <<= i.cidr`) directly inside transactional SQL queries:

```sql
CREATE TABLE IF NOT EXISTS ipscope_blocks (
  cidr cidr PRIMARY KEY,
  ip_version smallint NOT NULL,
  country_code char(2),
  declared_country_code char(2),
  physical_country_code char(2),
  is_virtual_location boolean NOT NULL,
  asn bigint,
  organization text,
  connection_type text,
  is_vpn boolean NOT NULL,
  is_proxy boolean NOT NULL,
  is_tor_exit boolean NOT NULL,
  anonymizer_type text,
  abuse_velocity_score smallint
);

-- Fast GiST radix index for IP containment (<<= and >>=)
CREATE INDEX IF NOT EXISTS idx_ipscope_blocks_gist
  ON ipscope_blocks USING gist (cidr inet_ops);
```

## Benchmark: 100 Million Log Rows Enrichment Across Engines

We benchmarked enriching a 100-million-row CDN access log (`72%` IPv4, `28%` IPv6) on an 8-vCPU (`32 GB` RAM) instance across four execution strategies:

| Enrichment Architecture | Artifact Used | Execution Time (`100M` Rows) | Throughput (`Rows / sec`) |
| --- | --- | --- | --- |
| **Python scalar UDF calling `.mmdb` per row** | `ipscope-complete.mmdb` | `142.0 seconds` | `~704,000 rows/s` |
| **PostgreSQL 16 `gist (cidr inet_ops)` join** | `ipscope-complete.csv.gz` | `38.4 seconds` | `~2.6M rows/s` |
| **DuckDB 1.1 Vectorized `ASOF JOIN` on Parquet** | `ipscope-complete.parquet` | **`4.1 seconds`** | **`~24.4M rows/s`** |
| **ClickHouse `LAYOUT(IP_TRIE())` Dictionary** | `ipscope-complete.parquet` | **`1.9 seconds`** | **`~52.6M rows/s`** |

## Automating daily Parquet updates with ETag and SHA-256 verification

You can pull the latest Parquet build from the same `/v1/mmdb/releases/latest/download` endpoint by passing `?format=parquet`:

```bash
curl -fsSL \
  -H "Authorization: Bearer ${IPSCOPE_DOWNLOAD_KEY}" \
  -o "/var/lib/ipscope/ipscope-complete.parquet.tmp" \
  "https://api.ipscope.io/v1/mmdb/releases/latest/download?format=parquet"

mv -f "/var/lib/ipscope/ipscope-complete.parquet.tmp" "/var/lib/ipscope/ipscope-complete.parquet"
```

## Frequently asked questions (FAQ)

### Do I have to pay extra to get `.parquet` and `.csv.gz` alongside `.mmdb`?
No. Every ipscope database subscription (`IPScope-Country`, `IPScope-Privacy`, `IPScope-Complete`, and any custom SKU built in the Custom Database Builder) automatically includes all three formats (`?format=mmdb`, `?format=parquet`, and `?format=csv`) compiled from the exact same atomic snapshot version.

### Are the CIDR rows in `ipscope-*.parquet` overlapping or non-overlapping?
In our compiled `.parquet` and `.csv.gz` artifacts, the compiler flattens the radix tree into a **disjoint, non-overlapping set of leaf prefixes**. If a `/16` supernet has a `/24` subnet inside it with different city or security metadata, the `/16` is split around the `/24` into exact canonical sub-ranges so that any single IP address matches **at most one row** in the Parquet file. This guarantees that SQL joins never duplicate your log rows!

---

<!-- Article URL: https://ipscope.io/blog/rfc-8805-geofeed-validation-pipeline -->
---
slug: rfc-8805-geofeed-validation-pipeline
title: "Automating RFC 8805 Geofeed Ingestion: Validating Operator Claims Against RPKI"
description: "How network operators publish self-hosted RFC 8805 CSV geofeeds, and how our compiler authenticates RPSL remarks, verifies RPKI ROAs, and filters out spoofed latency claims."
date: "2026-10-04"
readingTime: "13 min read"
category: "Geolocation"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "RFC 8805, RFC 9092, Geofeed, RPKI, Network Operators, BGP"
featured: "false"
---

## Why RFC 8805 and RFC 9092 transformed ISP geolocation

Historically, when a cloud provider, ISP, or enterprise network operator deployed a new `/24` IPv4 subnet or `/48` IPv6 prefix in a new city, updating third-party geolocation databases was a manual, error-prone ordeal. Network engineers had to submit web forms or emails to half a dozen commercial GeoIP vendors and wait weeks for each vendor to manually review and publish the change.

To solve this coordination bottleneck, the IETF published two complementary standards:

1. **RFC 8805 (*A Format for Self-Published IP Geolocation Feeds*)**: Standardizes a machine-readable, UTF-8 CSV format where network operators map their CIDR prefixes to ISO 3166-1 alpha-2 country codes, ISO 3166-2 subdivision codes, and city names.
2. **RFC 9092 (*Finding and Using Geofeed Data*)**: Standardizes how Regional Internet Registry (RIR) database objects (`inetnum`, `inet6num`, and `NetRange` records in RPSL / RDAP) link to a self-published geofeed file via a `remarks: #geofeed https://...` or structured `geofeed:` attribute, and how consumers must validate prefix bailiwick.

While RFC 8805 and RFC 9092 are foundational for modern IP geolocation, **blindly trusting every geofeed CSV found in WHOIS is a critical security and accuracy mistake**. Over the past three years, we have observed everything from expired geofeed domains hijacked by domain drop-catchers to proxy operators deliberately publishing false geofeeds to spoof residential locations.

This article explains the exact syntax of RFC 8805 and RFC 9092, details the three attack vectors against unverified geofeed ingestion, and walks through the four-stage cryptographic and physical verification pipeline built into the ipscope compiler.

## Anatomy of an RFC 8805 CSV feed and RFC 9092 RIR binding

An RFC 8805 geofeed is a UTF-8 text file where each non-comment line contains five comma-separated fields:

```txt
# ip_prefix,alpha2_country,region,city,postal_code
198.51.100.0/24,US,US-VA,Ashburn,
203.0.113.0/24,DE,DE-HE,Frankfurt am Main,
2001:db8:400::/48,NL,NL-NH,Amsterdam,
```

### Field specification per RFC 8805 Section 2.1

| Field Position | Name | Standard / Format | Notes |
| --- | --- | --- | --- |
| `1` | `ip_prefix` | IPv4 or IPv6 CIDR (`a.b.c.d/len` or `v6::/len`) | Mandatory. Must be a valid CIDR block within the operator's RIR allocation. |
| `2` | `alpha2_country` | ISO 3166-1 alpha-2 (`US`, `DE`, `JP`) | Recommended. Uppercase 2-letter country code (or empty if unspecified). |
| `3` | `region` | ISO 3166-2 subdivision code (`US-VA`, `DE-HE`, `GB-LND`) | Optional. Must begin with the matching `alpha2_country` prefix followed by a hyphen. |
| `4` | `city` | UTF-8 city name (`Frankfurt am Main`, `Tokyo`) | Optional. Quoted if the city name contains a comma. |
| `5` | `postal_code` | Postal / ZIP code | Optional. Explicitly deprecated by RFC 8805 for privacy and accuracy reasons; should be left empty with a trailing comma. |

### Binding the CSV to your RIR allocation via RFC 9092

A standalone CSV file on a web server has no authority on its own. Under RFC 9092, the geofeed URL must be referenced inside the authoritative RIR database object (`RIPE`, `ARIN`, `APNIC`, `LACNIC`, or `AFRINIC`) that covers the IP block:

```txt
inetnum:        198.51.100.0 - 198.51.103.255
netname:        EXAMPLE-NET-EU
descr:          Example Cloud Infrastructure GmbH
country:        DE
admin-c:        EC100-RIPE
tech-c:         EC100-RIPE
status:         ASSIGNED PA
geofeed:        https://noc.example.net/geofeed.csv
mnt-by:         EXAMPLE-MNT
source:         RIPE
```

On RIRs that do not yet expose a dedicated `geofeed:` attribute on all legacy object types, operators publish the canonical remark line:

```txt
remarks:        #geofeed https://noc.example.net/geofeed.csv
```

## Three real-world failure and spoofing vectors in public geofeeds

In our daily crawl of over 19,500 public `#geofeed` URLs across all five RIRs, roughly **11.4% of raw geofeed rows fail validation**. Those failures fall into three categories:

### Vector 1: Out-of-bailiwick prefix injection

Suppose Autonomous System `AS64500` owns `192.0.2.0/24` and publishes `remarks: #geofeed https://as64500.example/feed.csv` on its `inetnum` object. Nothing prevents the author of `feed.csv` from adding a row for `8.8.8.0/24` or a competitor's subnet (`198.51.100.0/24`) and claiming it is located in Antarctica.

If a naive parser ingests every CSV referenced anywhere in WHOIS without checking whether each CSV row's `ip_prefix` is a mathematical subnet of the specific `inetnum` / `inet6num` object that referenced that URL, any network operator can overwrite the geolocation of any other network on the internet.

### Vector 2: Expired domain hijacking (`Dangling #geofeed Remarks`)

Smaller ISPs, hosting companies, and startups occasionally rebrand, merge, or shut down without cleaning up old `remarks: #geofeed https://old-brand.example/geofeed.csv` entries in RIPE or ARIN. When `old-brand.example` expires at the domain registrar, malicious actors register the expired domain for $10, stand up an HTTPS server with a valid Let's Encrypt certificate, and serve a manipulated `geofeed.csv` for the still-allocated IP space.

### Vector 3: Intentional geo-spoofing for streaming and ad-fraud arbitrage

Residential proxy aggregators, sneaker-bot proxy providers, and bullet-proof hosting operators routinely lease cheap `/24` datacenter blocks in low-cost colocation facilities (for example, in Frankfurt, Bucharest, or Warsaw) and publish RFC 8805 geofeeds claiming the subnets are physically located in Dallas, Los Angeles, or London. Because they control both the `inetnum` object (via lease assignment) and the HTTPS domain hosting the CSV, standard RFC 9092 bailiwick checks pass completely. Only independent physical network telemetry can catch the lie.

## Inside the ipscope 4-Gate Geofeed Verification Pipeline

Before a single row from an RFC 8805 geofeed is allowed to influence an ipscope dataset snapshot, our compiler passes the feed through four strict verification gates:

```txt
RIR RDAP/RPSL Snapshot (ARIN, RIPE, APNIC, LACNIC, AFRINIC)
   |
   v
[ Gate 1: Strict RFC 9092 Prefix Bailiwick & Longest-Prefix Hierarchy ]
   |
   v
[ Gate 2: RPKI ROA Cryptographic Origin & ASN Ownership Alignment ]
   |
   v
[ Gate 3: TLS 1.3 / Domain WHOIS Age & Content-Delta Quarantine ]
   |
   v
[ Gate 4: Multi-Vantage Speed-of-Light RTT Physical Verification ]
   |
   +---> RTT Compatible? --(Yes)--> Sets declared_country_code AND physical_country_code
   |
   +---> RTT Violated?   --(No)---> Preserves declared_country_code, overrides physical_country_code,
                                    and sets is_virtual_location = true
```

### Gate 1: Strict RFC 9092 bailiwick enforcement

When our crawler extracts `https://noc.example.net/geofeed.csv` from an RIR allocation object covering parent prefix `P`, the parser constructs a strict radix filter scoped exclusively to `P`. Every row in `geofeed.csv` with prefix `C_i` must satisfy two mathematical invariants:

```txt
C_i is a subnet of P   AND   prefix_len(C_i) >= prefix_len(P)
```

Any row where `C_i` falls outside the parent `inetnum` or `inet6num` block `P` is discarded immediately and increments the feed's security violation counter. Furthermore, if a more-specific sub-allocation object `P_sub` exists inside `P` in the RIR database and defines its own `#geofeed` URL, the child object's geofeed takes precedence for `P_sub` per RFC 9092 Section 5.

### Gate 2: RPKI Route Origin Authorization (ROA) cross-checking

To defend against stale RIR remarks and unauthorized sub-allocation edits, the compiler cross-references every candidate prefix `C_i` against the global BGP routing table and the **Resource Public Key Infrastructure (RPKI)** validator cache:

- Does the prefix have an active BGP announcement?
- Is the originating ASN backed by a cryptographically valid RPKI ROA (`RPKI Valid`) signed by the resource holder's RIR certificate?
- If the geofeed domain belongs to an entity that has no relationship to the active RPKI-validated originating ASN (for example, a dangling remark from a previous lessee), the geofeed is demoted to low confidence.

### Gate 3: Domain lifecycle and mass-mutation anomaly quarantine

Before trusting a geofeed HTTPS endpoint, our fetcher inspects:

1. **Transport security**: Must resolve over HTTPS (`TLS 1.2` or `TLS 1.3`) with a valid public WebPKI certificate chain and strict redirect bailiwick (cross-domain redirects from the `#geofeed` URL to third-party domains are flagged).
2. **Domain registration continuity**: If the domain hosting the `#geofeed` URL was recently re-registered after an expiration drop or changed registrar ownership after the RIR `last-modified` date of the `inetnum` object, the feed is automatically quarantined for security review.
3. **Syntax and ISO 3166-2 dictionary validation**: Every country code is validated against ISO 3166-1 alpha-2, and every subdivision code (`US-VA`, `DE-HE`) is validated against the official ISO 3166-2 subdivision registry so typos like `US-VIRGINIA` or `UK-LON` (instead of `GB-LND`) are normalized or rejected cleanly.

### Gate 4: Speed-of-light RTT physical verification

This is where ipscope's dual-country architecture (`declared_country_code` vs `physical_country_code`) shines.

Even when an operator legitimately owns a prefix, passes RFC 9092 bailiwick, and holds a valid RPKI ROA, their self-published geofeed only represents a **declared claim**. Our compiler tests every geofeed row against active minimum-RTT measurements from our global probe fleet:

| Geofeed Claim (`alpha2_country`, `city`) | Measured `min_rtt` from Nearest Local Probe | Compiler Output |
| --- | --- | --- |
| `DE, DE-HE, Frankfurt am Main` | `1.4 ms` from Frankfurt (`fra-01`) | `declared_country_code: "DE"`, `physical_country_code: "DE"`, `is_virtual_location: false` |
| `US, US-NY, New York` | `1.6 ms` from Frankfurt (`fra-01`), `82 ms` from Newark (`ewr-01`) | `declared_country_code: "US"`, `physical_country_code: "DE"`, `is_virtual_location: true` |

Because we preserve the operator's claim in `declared_country_code` while setting `physical_country_code` (and `country_code`) to the physically verified router location, geofeed spoofing can never trick your edge routing or licensing logic.

## Writing a local RFC 8805 validator in Python

If your network engineering team publishes an RFC 8805 geofeed for your own ASN prefixes, we recommend validating the CSV syntax and prefix bailiwick in CI/CD before deploying it to your web server. Here is a zero-dependency Python 3 script using the standard library `ipaddress` and `csv` modules:

```python
import csv
import ipaddress
import re
import sys
from pathlib import Path

ISO_COUNTRY_RE = re.compile(r"^[A-Z]{2}$")
ISO_REGION_RE = re.compile(r"^[A-Z]{2}-[A-Z0-9]{1,3}$")

def validate_geofeed(csv_path: Path, allowed_supernets: list[str]) -> int:
    supernets = [ipaddress.ip_network(cidr, strict=True) for cidr in allowed_supernets]
    errors = 0

    with csv_path.open("r", encoding="utf-8") as fh:
        for line_no, raw_line in enumerate(fh, start=1):
            stripped = raw_line.strip()
            if not stripped or stripped.startswith("#"):
                continue

            row = next(csv.reader([stripped]))
            if len(row) != 5:
                print(f"Line {line_no}: expected 5 CSV columns, got {len(row)}", file=sys.stderr)
                errors += 1
                continue

            prefix_str, country, region, city, postal = [col.strip() for col in row]

            try:
                net = ipaddress.ip_network(prefix_str, strict=True)
            except ValueError as exc:
                print(f"Line {line_no}: invalid CIDR '{prefix_str}': {exc}", file=sys.stderr)
                errors += 1
                continue

            if not any(net.version == sup.version and net.subnet_of(sup) for sup in supernets):
                print(f"Line {line_no}: prefix {net} is out of bailiwick", file=sys.stderr)
                errors += 1

            if country and not ISO_COUNTRY_RE.match(country):
                print(f"Line {line_no}: invalid ISO 3166-1 alpha-2 country '{country}'", file=sys.stderr)
                errors += 1

            if region:
                if not ISO_REGION_RE.match(region) or not region.startswith(f"{country}-"):
                    print(
                        f"Line {line_no}: region '{region}' must match '{country}-XXX' ISO 3166-2 format",
                        file=sys.stderr,
                    )
                    errors += 1

            if postal:
                print(f"Line {line_no}: warning: postal_code ('{postal}') is deprecated by RFC 8805", file=sys.stderr)

    return errors

if __name__ == "__main__":
    err_count = validate_geofeed(
        Path("geofeed.csv"),
        allowed_supernets=["198.51.100.0/22", "2001:db8:400::/40"],
    )
    sys.exit(1 if err_count > 0 else 0)
```

## How network operators can trigger an immediate geofeed sync on ipscope

Our compiler automatically crawls all RIR `#geofeed` remarks daily. However, if you have just re-homed a subnet or published a new RFC 8805 feed and want to trigger an immediate validation and re-probe cycle, you can submit your prefix or geofeed URL programmatically via `POST /v1/feedback`:

```bash
curl -sS -X POST "https://api.ipscope.io/v1/feedback" \
  -H "Authorization: Bearer ${IPSCOPE_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "prefix_or_ip": "198.51.100.0/24",
    "role": "network_owner",
    "issue_type": "wrong_city",
    "requested_correction": {
      "country_code": "DE",
      "subdivision": "Hesse",
      "city": "Frankfurt am Main"
    },
    "evidence_url": "https://noc.example.net/geofeed.csv"
  }'
```

When a `network_owner` correction is submitted with an `evidence_url`, our verification worker immediately runs Gates 1 through 4 against the target prefix and queues verified updates for the next dataset snapshot release (`06:00 UTC`).

## Frequently asked questions (FAQ)

### Should I include postal codes in my RFC 8805 geofeed CSV?
No. RFC 8805 Section 2.1.1.5 explicitly deprecates the 5th column (`postal_code`) because mapping an entire `/24` or `/48` IP block to a single street postal code creates false precision and privacy concerns. Always include the trailing fourth comma so the row has 5 columns, leaving the 5th column empty (for example, `198.51.100.0/24,DE,DE-HE,Frankfurt am Main,`).

### How do I format the `region` column in an RFC 8805 geofeed?
The `region` column (`3rd` column) must use the full **ISO 3166-2** code, which includes the two-letter country prefix, a hyphen, and the one-to-three-character subdivision code (for example, `US-VA` for Virginia, `DE-HE` for Hesse, `CA-ON` for Ontario, or `NL-NH` for North Holland). Writing `VA` instead of `US-VA` violates RFC 8805 Section 2.1.1.3.

### What MIME `Content-Type` header should my web server return for `geofeed.csv`?
Serve your geofeed over HTTPS (`TLS 1.2+`) with `Content-Type: text/csv; charset=utf-8` (or `text/plain; charset=utf-8`), and enable `ETag` and `Last-Modified` response headers so automated crawlers can use conditional `If-None-Match` requests efficiently.

---

<!-- Article URL: https://ipscope.io/blog/streaming-100k-ips-constant-memory-sdks -->
---
slug: streaming-100k-ips-constant-memory-sdks
title: "Enriching 100,000+ IPs in Under 350 KB of RAM with Official IPScope Server SDKs"
description: "How we designed lookupBatchStream across Python, Node.js, Go, Rust, Java, .NET, Ruby, and PHP with bounded O(1) LRU caching, singleflight deduplication, and zero-allocation chunking."
date: "2026-10-10"
readingTime: "15 min read"
category: "Engineering"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "SDKs, Python, Node.js, Go, Rust, Streaming, Performance"
featured: "false"
---

## The memory and socket traps in large-scale IP log enrichment

When a security operations team, data engineering pipeline, or incident response script needs to enrich 100,000 to 1,000,000 IP addresses from an NGINX access log, AWS VPC Flow Log, or Cloudflare log export against a REST API, naive scripts almost always hit two production walls:

1. **Unbounded memory and socket fan-out**: Loading an entire 500 MB log file into memory and firing `Promise.all(ips.map(ip => fetch(...)))` or `asyncio.gather(*tasks)` spawns tens of thousands of concurrent TCP connections, exhausts OS ephemeral ports (`EADDRNOTAVAIL`), triggers `429 Too Many Requests` rate limits, and crashes the worker container with an Out-Of-Memory (`OOMKilled`) exit.
2. **Paying credits and latency for duplicate hot IPs**: Real-world network traffic follows a steep power-law (Zipf) distribution. In a typical 500,000-line web server access log, **fewer than 15% of the IP addresses are unique**. The remaining 85% of lines are repeated requests from the same client sessions, health-checkers, search engine crawlers, and mobile CGNAT gateways. Sending the same IP address to a remote API 400 times wastes lookup credits and multiplies network serialization overhead by 5x to 10x.

When we designed the eight official IPScope server SDKs (`Python`, `Node.js / TypeScript`, `Go`, `Rust`, `Java`, `.NET`, `Ruby`, and `PHP`), we built constant-memory batch streaming and automatic client-side deduplication directly into the core client primitives.

This engineering article walks through the internal architecture of `lookupBatch` and `lookupBatchStream`, explains how our bounded `O(1)` LRU cache and in-flight request coalescing eliminate redundant API calls, and shows production streaming examples in Node.js, Python, Go, and Rust.

## The three architectural guarantees of official IPScope SDKs

Every official IPScope SDK implements the exact same three-stage execution pipeline before a single byte leaves your server over HTTPS:

```txt
Input IP Stream (Iterable / AsyncIterable / Channel / Stream)
   |
   v
[ Stage 1: Lazy 100-IP Chunking Window ]
   |  Pulls at most 100 IPs at a time from the caller's stream
   v
[ Stage 2: Bounded O(1) LRU Cache & Intra-Batch Deduplicator ]
   |  1. Satisfies hot IPs immediately from in-memory LRU cache (0 network calls)
   |  2. Deduplicates repeated IPs inside the current 100-IP window
   v
[ Stage 3: Single POST /v1/ip/batch HTTP/2 Request ]
   |  Sends only the unique cache-miss IPs, populates LRU cache,
   |  and reconstructs the 100-item BatchLookupResponse in original order
   v
Yields BatchLookupResponse to caller (Steady-State Heap < 350 KB)
```

### Guarantee 1: Bounded `O(1)` In-Memory LRU Cache

Every SDK client initializes a thread-safe (or coroutine-safe) Least Recently Used (LRU) cache bounded by `maxCacheEntries` (default `2,048` entries) and `cacheTtlMs` (default `60,000 ms`).

- On every `lookupIp(ip)`, `lookupBatch(ips)`, or `lookupBatchStream(source)` call, the SDK checks the local LRU cache first.
- If `198.51.100.7` was resolved within the last 60 seconds, the SDK returns the cached `IpLookupResponse` immediately in `< 100 nanoseconds` without making a network request or consuming an API credit.
- Because the cache strictly enforces its `maxCacheEntries` cap by evicting the oldest entry in `O(1)` time on insertion, the cache's memory footprint can never grow unboundedly, even if your worker processes billions of unique IPs over months of uptime.

### Guarantee 2: Intra-batch deduplication and order preservation

Suppose a 100-line log chunk contains 60 cached IPs and 40 uncached IPs, and among those 40 uncached IPs, an aggressive scraper IP (`203.0.113.99`) appears 15 times.

Instead of sending all 100 IPs to `POST /v1/ip/batch`, the SDK:

1. Pulls the 60 cached records directly from memory.
2. Collapses the 40 uncached inputs down to the **26 unique IP strings** that actually missed the cache.
3. Sends a single `POST /v1/ip/batch` request containing only those `26` unique IPs.
4. Stores the 26 fresh records into the LRU cache and reassembles the returned `BatchLookupResponse` so the caller receives all 100 records in the exact sequence requested.

On a standard production web log, this two-tier deduplication cuts outbound HTTP payload volume and API credit usage by **75% to 90%** automatically.

### Guarantee 3: Backpressure-aware lazy streaming (`lookupBatchStream`)

Rather than requiring you to load a 2 GB CSV or log file into an array first, `lookupBatchStream` (or `lookup_batch_stream` in Python/Rust/Ruby) accepts a lazy iterator or async generator and yields one `BatchLookupResponse` per 100-IP window.

Because the next 100 lines are not read from disk until your `for await (...)` loop finishes processing the current batch (for example, writing the enriched rows to PostgreSQL, ClickHouse, or an output file), end-to-end backpressure is preserved automatically and steady-state heap usage stays under **350 KB**.

## Streaming 100,000+ IPs in Node.js & TypeScript (`@ipscope/sdk-node`)

In Node.js, combining `node:readline` over a file stream with `client.lookupBatchStream()` lets you enrich arbitrarily large log files in constant memory:

```ts
import { createReadStream, createWriteStream } from 'node:fs';
import { createInterface } from 'node:readline';
import { IpScopeClient } from '@ipscope/sdk-node';

const client = new IpScopeClient({
  apiKey: process.env.IPSCOPE_KEY!,
  maxCacheEntries: 8192,
  cacheTtlMs: 300_000, // 5-minute TTL for log enrichment runs
  timeoutMs: 10_000,
  maxRetries: 3,
});

async function* extractIpsFromLog(filePath: string): AsyncGenerator<string> {
  const rl = createInterface({
    input: createReadStream(filePath, { encoding: 'utf8' }),
    crlfDelay: Infinity,
  });

  for await (const line of rl) {
    const trimmed = line.trim();
    if (!trimmed || trimmed.startsWith('#')) continue;
    // Extract first whitespace-delimited token (e.g. NGINX $remote_addr)
    const firstSpace = trimmed.indexOf(' ');
    const ip = firstSpace === -1 ? trimmed : trimmed.slice(0, firstSpace);
    yield ip;
  }
}

async function enrichAccessLog(inputPath: string, outputPath: string): Promise<void> {
  const out = createWriteStream(outputPath, { encoding: 'utf8' });
  let totalEnriched = 0;

  for await (const batch of client.lookupBatchStream(extractIpsFromLog(inputPath))) {
    for (const record of batch.results) {
      const row = JSON.stringify({
        ip: record.ip,
        country: record.geo.physical_country_code,
        declared_country: record.geo.declared_country_code,
        asn: record.network.asn,
        org: record.network.organization,
        is_vpn: record.security.is_vpn,
        is_proxy: record.security.is_proxy,
        velocity: record.security_extensions?.abuse_velocity_score ?? 0,
      });
      if (!out.write(row + '\n')) {
        await new Promise<void>((resolve) => out.once('drain', resolve));
      }
    }
    totalEnriched += batch.count;
  }

  out.end();
  console.log(`Successfully enriched ${totalEnriched} log lines.`);
}

enrichAccessLog('/var/log/nginx/access.log', '/tmp/enriched.ndjson').catch(console.error);
```

Notice how `out.once('drain', resolve)` cooperates directly with the `for await` generator loop: if the downstream disk or database writer slows down, `lookupBatchStream` pauses pulling from `extractIpsFromLog` immediately.

## Async generator streaming in Python (`ipscope`)

The official Python SDK (`pip install ipscope`) provides both synchronous (`IpScopeClient`) and `asyncio`-native (`AsyncIpScopeClient`) clients backed by persistent HTTP/1.1 and HTTP/2 connection pooling:

```python
import asyncio
import json
import os
from collections.abc import AsyncIterator
from ipscope import AsyncIpScopeClient

async def iter_log_ips(log_path: str) -> AsyncIterator[str]:
    with open(log_path, "r", encoding="utf-8") as fh:
        for line in fh:
            ip = line.strip().split(" ", 1)[0]
            if ip and not ip.startswith("#"):
                yield ip

async def enrich_siem_export(log_path: str) -> None:
    async with AsyncIpScopeClient(
        api_key=os.environ["IPSCOPE_KEY"],
        max_cache_entries=8192,
        cache_ttl_seconds=300.0,
    ) as client:
        flagged_count = 0
        total_count = 0

        async for batch in client.lookup_batch_stream(iter_log_ips(log_path)):
            total_count += batch["count"]
            for item in batch["results"]:
                sec = item["security"]
                ext = item.get("security_extensions") or {}
                if sec["is_vpn"] or sec["is_proxy"] or ext.get("abuse_velocity_score", 0) >= 70:
                    flagged_count += 1
                    print(
                        json.dumps(
                            {
                                "ip": item["ip"],
                                "country": item["geo"]["physical_country_code"],
                                "asn": item["network"]["asn"],
                                "anonymizer_type": sec["anonymizer_type"],
                                "reason_codes": sec["reason_codes"],
                                "abuse_velocity_score": ext.get("abuse_velocity_score", 0),
                            }
                        )
                    )

        print(f"Processed {total_count} IPs ({flagged_count} flagged).")

if __name__ == "__main__":
    asyncio.run(enrich_siem_export("/var/log/auth-ips.txt"))
```

## Context-aware channel streaming in Go (`github.com/badele-llc/ipscope-go`)

In Go (`go get github.com/badele-llc/ipscope-go`), `LookupBatchStream` accepts a `context.Context` and a read-only channel (`<-chan string`), returning a typed result channel (`<-chan StreamBatchResult`) that closes cleanly when the input channel is drained or the `context.Context` is cancelled:

```go
package main

import (
	"bufio"
	"context"
	"fmt"
	"log"
	"os"
	"strings"
	"time"

	ipscope "github.com/badele-llc/ipscope-go"
)

func streamFileIps(ctx context.Context, path string) (<-chan string, error) {
	file, err := os.Open(path)
	if err != nil {
		return nil, err
	}

	out := make(chan string, 256)
	go func() {
		defer close(out)
		defer file.Close()

		scanner := bufio.NewScanner(file)
		for scanner.Scan() {
			ip := strings.TrimSpace(scanner.Text())
			if ip == "" || strings.HasPrefix(ip, "#") {
				continue
			}
			select {
			case out <- ip:
			case <-ctx.Done():
				return
			}
		}
	}()
	return out, nil
}

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
	defer cancel()

	client := ipscope.NewClient(
		os.Getenv("IPSCOPE_KEY"),
		ipscope.WithCache(8192, 5*time.Minute),
	)

	ipChan, err := streamFileIps(ctx, "/var/log/firewall-ips.txt")
	if err != nil {
		log.Fatal(err)
	}

	var total int
	for batchRes := range client.LookupBatchStream(ctx, ipChan, 100) {
		if batchRes.Err != nil {
			log.Fatalf("batch stream error: %v", batchRes.Err)
		}
		total += batchRes.Response.Count
		for _, item := range batchRes.Response.Results {
			if item.Security.IsTorExit || item.Security.IsProxy {
				fmt.Printf("Blocked %s (%s) reasons=%v\n",
					item.IP, item.Security.AnonymizerType, item.Security.ReasonCodes)
			}
		}
	}
	fmt.Printf("Enriched %d IPs cleanly\n", total)
}
```

## Benchmark: Naive `fetch` fan-out vs. IPScope SDK `lookupBatchStream`

We benchmarked enriching a real-world 100,000-line production NGINX access log (`14,820` unique IPv4 and IPv6 addresses across `100,000` log lines) on a standard 2-vCPU Linux container (`Node.js 22`, `50 ms` network RTT to the nearest API edge):

| Execution Strategy | Outbound HTTP Requests | API Credits Consumed | Peak Heap Memory (RSSDelta) | Total Wall-Clock Time |
| --- | --- | --- | --- | --- |
| **Naive per-line `fetch()` (`concurrency = 50`)** | `100,000` (`GET /v1/ip/{ip}`) | `100,000` | `148.4 MB` | `104.2 seconds` |
| **Un-cached 100-IP batch loop** | `1,000` (`POST /v1/ip/batch`) | `100,000` | `19.2 MB` | `6.8 seconds` |
| **IPScope SDK `lookupBatchStream` (`LRU = 8,192`)** | **`162`** (`POST /v1/ip/batch`) | **`14,820`** | **`0.31 MB` (`318 KB`)** | **`1.4 seconds`** |

By combining 100-IP batching with bounded LRU caching and intra-batch deduplication, the official SDK finishes **74x faster**, uses **470x less heap memory**, and reduces API credit consumption by **85.1%**.

## Backend snapshot consistency and atomic pre-debit validation

On the server side, `POST /v1/ip/batch` is engineered with two strict operational guarantees:

1. **Atomic Dataset Snapshot Pinning**: At the very beginning of a batch request, the API worker acquires an atomic pointer reference to the active in-memory Radix/LPM snapshot (`SnapshotLoaderService`). Even if a background dataset hot-reload happens mid-request, all 100 IPs in the batch are resolved against the exact same `dataset_version`.
2. **All-or-Nothing Pre-Debit Validation**: Before debiting a single credit from your workspace quota, the controller validates the syntax of every IP string in the `ips` array (`1` to `100` items). If even one element in the array is malformed (for example, `"999.999.999.999"` or an empty string), the request is rejected immediately with `400 Bad Request` (`INVALID_IP_FORMAT`) and **zero credits are deducted**.

## Installing the official SDK for your stack

All eight official IPScope server SDKs are open-source and published to their canonical package registries:

| Language / Runtime | Package Registry | Installation Command |
| --- | --- | --- |
| **Python (`3.10+`)** | PyPI | `pip install ipscope` |
| **Node.js / TypeScript (`18+`)** | npm | `npm install @ipscope/sdk-node` |
| **Go (`1.22+`)** | pkg.go.dev | `go get github.com/badele-llc/ipscope-go` |
| **Rust (`1.75+`)** | crates.io | `cargo add ipscope` |
| **Java (`17+`)** | Maven | `io.ipscope:ipscope-sdk:1.0.0` |
| **.NET (`8.0+`)** | NuGet | `dotnet add package IpScope.Sdk` |
| **Ruby (`3.1+`)** | RubyGems | `gem install ipscope` |
| **PHP (`8.2+`)** | Packagist | `composer require badele/ipscope-sdk` |

## Frequently asked questions (FAQ)

### How does the SDK handle transient `429 Too Many Requests` or `502/503/504` gateway errors during a long stream?
Every official SDK includes an automatic retry state machine with **exponential backoff and full jitter** (`maxRetries: 2` by default, configurable). When the server returns a `429` or `503` response with a `Retry-After` header, the SDK honors the exact server backoff window before retrying the failed 100-IP chunk transparently inside `lookupBatchStream`.

### When should I use `lookupBatchStream` via the REST API versus downloading the offline `.mmdb` or `.parquet` database?
If your job enriches up to a few million IPs per day from serverless functions, CI/CD workers, or microservices where you do not want to manage local database files, `lookupBatchStream` with the LRU cache enabled is the simplest zero-maintenance choice. If you are enriching hundreds of millions or billions of log rows daily inside a SIEM or data warehouse, we recommend subscribing to an **ipscope Database SKU** and running vectorized SQL joins against our `.parquet` artifact or local `mmap` lookups against our `.mmdb` artifact.

---

<!-- Article URL: https://ipscope.io/blog/sub-microsecond-mmdb-radix-trees -->
---
slug: sub-microsecond-mmdb-radix-trees
title: "Inside the ipscope MMDB Compiler: Packing 35.3M Prefixes into a Zero-Copy Radix Tree"
description: "How we compile unified IPv4 and IPv6 routing tables into open MMDB v2.0 binary artifacts with structural DAG deduplication, compact pointer encoding, and sub-microsecond mmap lookups."
date: "2026-10-20"
readingTime: "15 min read"
category: "Databases"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "MMDB, Radix Tree, Performance, Go, Rust, Binary Format"
featured: "false"
---

## Why local binary radix trees beat network RPCs on high-throughput hot paths

When an edge proxy, Web Application Firewall (WAF), real-time ad-bidding engine, or API gateway processes 50,000 to 200,000 requests per second on a single multi-core machine, every microsecond on the request path counts. Even calling a local Redis or Memcached instance over a loopback socket introduces system call overhead, socket buffer copying, serialization framing, and kernel scheduler jitter that pushes p99 lookup latency to `250 to 800 microseconds`.

By compiling the entire global IPv4 and IPv6 intelligence dataset into an immutable, memory-mapped (`.mmdb`) binary radix tree, your application resolves any IPv4 or IPv6 address in **under 1 microsecond** (`380 to 710 nanoseconds` at p50) using pure pointer arithmetic inside the operating system's `mmap(2)` virtual address space. There are no network sockets, no mutex locks, and zero heap allocations during the tree walk.

This deep dive explains how the ipscope MMDB compiler transforms 35.3 million raw CIDR prefixes and six intelligence families into a compact `.mmdb` file, how binary bit-tries and pointer deduplication work under the hood, and how to tune `.mmdb` lookups in Go and Rust for maximum CPU cache locality.

## Anatomy of the open MMDB v2.0 binary specification

Every `.mmdb` file produced by ipscope adheres strictly to the **MaxMind DB Binary Format Specification v2.0**. Because the wire format is an open standard, you never need a proprietary binary library to read ipscope databases. Standard open-source readers such as `maxminddb-golang` (Go), `maxminddb` (Rust), `node-maxmind` (Node.js), `maxminddb` (Python C-extension), and `ngx_http_geoip2_module` (NGINX C module) open ipscope `.mmdb` files directly without modifying a single line of reader code.

At the byte level, an `.mmdb` file is laid out as three contiguous sections:

```txt
+-------------------------------------------------------------------+
| 1. Binary Search Tree Section                                     |
|    Fixed-width node array (24-bit, 28-bit, or 32-bit records)     |
|    Implicitly indexed from Node 0 (Root) to Node N-1              |
+-------------------------------------------------------------------+
| 16-Byte Null Separator (0x00 * 16)                                |
+-------------------------------------------------------------------+
| 2. Data Section                                                   |
|    Deduplicated UTF-8 strings, IEEE-754 floats, unsigned ints,    |
|    nested maps, arrays, and 2/3/4-byte back-reference pointers    |
+-------------------------------------------------------------------+
| Metadata Start Marker: \xab\xcd\xefMaxMind.com                    |
+-------------------------------------------------------------------+
| 3. Metadata Map Section                                           |
|    node_count, record_size, ip_version (6), database_type,        |
|    build_epoch, languages, description                            |
+-------------------------------------------------------------------+
```

### 1. The Binary Search Tree Section

The search tree is a bitwise radix trie (specifically, a binary trie where each level inspects a single bit of the target IP address from most significant bit `MSB 0` to least significant bit `LSB 31` for IPv4 or `LSB 127` for IPv6).

Rather than storing memory pointers or object headers, the tree is serialized as a flat array of fixed-size nodes. Each node consists of exactly two records:

- **Left record (bit = `0`)**
- **Right record (bit = `1`)**

The bit width of each record is defined by `record_size` in the metadata footer (`24`, `28`, or `32` bits):

| `record_size` | Node Byte Width (`2 * record_size / 8`) | Maximum Addressable `node_count` + Data Offset | When the ipscope Compiler Selects It |
| --- | --- | --- | --- |
| `24-bit` | `6 bytes` per node | `16,777,215` (`2^24 - 1`) | Compact Country-only SKUs (`IPScope-Country-v1`) |
| `28-bit` | `7 bytes` per node | `268,435,455` (`2^28 - 1`) | Mid-sized Privacy and Enterprise custom SKUs (`IPScope-Privacy-v1`, `IPScope-Complete-v1`) |
| `32-bit` | `8 bytes` per node | `4,294,967,295` (`2^32 - 1`) | Ultra-dense unpruned full-table research builds |

In a `28-bit` tree, two 28-bit integers are packed into `7 bytes` using bit-nibbles: the first 3 bytes hold the low 24 bits of the left record, the middle byte (`byte[3]`) splits its high 4 bits for the left record's upper nibble and its low 4 bits for the right record's upper nibble, and the final 3 bytes hold the low 24 bits of the right record. Packing nodes into `7 bytes` instead of `8 bytes` shrinks the search tree section by `12.5%`, saving megabytes of L3 CPU cache.

### How a reader interprets each record value (`value` vs `node_count`)

When a reader inspects bit `i` of the IP address and extracts the corresponding Left or Right integer `R` from the current node, three cases exist:

1. **`R < node_count`**: `R` is the index of the next internal node in the search tree. The reader advances to byte offset `R * node_byte_size` and inspects bit `i + 1`.
2. **`R == node_count`**: Explicit "not found" sentinel (for example, RFC 1918 private space `10.0.0.0/8` or unallocated bogon blocks). The walk terminates immediately in `O(prefix_len)` steps.
3. **`R > node_count`**: Leaf match! The prefix terminates here, and the byte offset of its intelligence payload inside the **Data Section** is computed directly as:

```txt
data_section_offset = (R - node_count) - 16
```

Because `node_count` is known when the file is opened, the reader never has to dereference a leaf node to check whether it is an internal node or a data pointer. A single integer comparison (`R > node_count`) ends the tree walk and yields the exact byte offset in the Data Section.

### Unified IPv4 and IPv6 tree embedding

Every ipscope `.mmdb` file compiles with `ip_version: 6` so a single database file resolves both IPv4 and IPv6 addresses without requiring two separate file descriptors.

To avoid walking 96 redundant zero bits on every IPv4 lookup, standard MMDB readers cache the tree node index reached after walking 96 zero bits (`::/96` or `::ffff:0:0/96`) at startup when `Open()` is called. When you pass an IPv4 address (`32 bits`) to `Lookup()`, the reader starts directly at the cached IPv4 root node and walks at most **24 to 32 steps** (for `/24` to `/32` prefixes), completing the entire search tree traversal in fewer than 30 L1/L3 cache line reads.

## How our compiler shrinks 35.3M prefixes into 141 MB

If you naively serialized 35.3 million JSON-like maps into a binary file without structural optimization, the resulting database would exceed **8.4 gigabytes**. Putting an 8.4 GB file on edge servers would blow past container memory limits and thrash the Linux page cache.

The ipscope MMDB compiler applies three aggressive optimization passes before emitting a single byte to disk:

### Optimization 1: Sibling prefix coalescing (Bottom-Up Trie Pruning)

In raw BGP and RIR telemetry, adjacent `/24` blocks often carry identical intelligence payloads. For example, if a cloud provider announces `198.51.100.0/24` and `198.51.101.0/24` from the exact same Frankfurt data center (`AS24940`), and neither subnet has distinct abuse velocity or scanner flags, keeping two separate `/24` leaves wastes an internal tree node.

During compilation, our bottom-up trie coalescer computes a 128-bit cryptographic structural hash (`XXH3_128`) of the resolved payload at every leaf. In a post-order traversal:

- If `node.left` is a leaf and `node.right` is a leaf, and `hash(node.left.payload) == hash(node.right.payload)`, the compiler collapses `node` itself into a single parent leaf (`cidr_len - 1`) and frees both child nodes.
- This recursive coalescing collapses millions of contiguous residential and cloud sub-allocations into their natural supernets without losing a single bit of intelligence fidelity.

### Optimization 2: Canonical DAG deduplication in the Data Section

Even when two prefixes are geographically separated in the binary search tree (for example, `AS13335` announcing `104.16.0.0/24` and `172.64.0.0/24`), their `network`, `connection`, `security`, and `geo` records are frequently identical or share identical sub-maps.

Instead of emitting duplicate maps into the Data Section, our compiler builds a **Directed Acyclic Graph (DAG)** of every value in the database:

1. **Whole-Record Sharing**: If `12,400` distinct prefixes across the globe resolve to the exact same tuple of `(geo, network, connection, security, security_extensions)`, the compiler writes that top-level map into the Data Section **once**. All `12,400` leaf records in the Binary Search Tree point to the exact same `R = node_count + 16 + offset` integer.
2. **Sub-Map and String Interning**: Even when two prefixes in Frankfurt differ in `network.cidr` or `security_extensions.abuse_velocity_score`, they still share the same `geo` map (`Frankfurt am Main`, `Hesse`, `DE`, `Europe/Berlin`) and the same `network.organization` string. The MMDB v2.0 specification supports a native **Pointer control byte (`type = 1`)** that encodes a 1-byte, 2-byte, 3-byte, or 4-byte offset back into the Data Section.
3. **Frequency-Sorted Offset Placement**: In MMDB v2.0, the smallest pointer representation (`2 bytes` total: 1 control byte + 1 offset byte) can address offsets `0` through `2,047` at the very beginning of the Data Section, and a `3-byte` pointer addresses offsets `2,048` through `526,335`. Our compiler sorts all interned strings and shared sub-maps by **reference frequency** in descending order before laying out the Data Section. The top 500 most referenced strings (`"US"`, `"DE"`, `"residential"`, `"hosting"`, `"none"`, `"America/New_York"`) and top ASN maps are guaranteed to sit in the first `2,048 bytes` of the Data Section, so every reference to them across the entire file costs only **2 bytes**.

## Decoding only what you need in Go (`maxminddb-golang`)

One of the biggest performance advantages of `.mmdb` over JSON or CSV is **lazy partial decoding**. When you pass a Go struct to `db.Lookup(ip, &record)`, the reader walks the binary map keys and only decodes the fields present on your struct, skipping over nested maps you do not need.

```go
package main

import (
	"fmt"
	"log"
	"net"
	"time"

	"github.com/oschwald/maxminddb-golang"
)

// EdgePolicyRecord decodes only the exact fields needed by our edge WAF,
// skipping city coordinates, postal codes, and extended metadata.
type EdgePolicyRecord struct {
	Geo struct {
		CountryCode         string  `maxminddb:"country_code"`
		DeclaredCountryCode string  `maxminddb:"declared_country_code"`
		PhysicalCountryCode string  `maxminddb:"physical_country_code"`
		IsVirtualLocation   bool    `maxminddb:"is_virtual_location"`
		Confidence          float64 `maxminddb:"confidence"`
	} `maxminddb:"geo"`
	Network struct {
		ASN          uint32 `maxminddb:"asn"`
		Organization string `maxminddb:"organization"`
	} `maxminddb:"network"`
	Connection struct {
		ConnectionType string `maxminddb:"connection_type"`
		IsMobile       bool   `maxminddb:"is_mobile"`
	} `maxminddb:"connection"`
	Security struct {
		IsVPN          bool     `maxminddb:"is_vpn"`
		IsTorExit      bool     `maxminddb:"is_tor_exit"`
		IsProxy        bool     `maxminddb:"is_proxy"`
		IsHosting      bool     `maxminddb:"is_hosting"`
		AnonymizerType string   `maxminddb:"anonymizer_type"`
		ReasonCodes    []string `maxminddb:"reason_codes"`
	} `maxminddb:"security"`
}

func main() {
	db, err := maxminddb.Open("/var/lib/ipscope/ipscope-complete.mmdb")
	if err != nil {
		log.Fatalf("failed to open mmdb: %v", err)
	}
	defer db.Close()

	ip := net.ParseIP("198.51.100.7")
	start := time.Now()

	var rec EdgePolicyRecord
	network, ok, err := db.LookupNetwork(ip, &rec)
	if err != nil {
		log.Fatalf("lookup failed: %v", err)
	}
	elapsed := time.Since(start)

	if ok {
		fmt.Printf(
			"Matched=%s Physical=%s Declared=%s ASN=AS%d (%s) VPN=%v Elapsed=%s\n",
			network.String(),
			rec.Geo.PhysicalCountryCode,
			rec.Geo.DeclaredCountryCode,
			rec.Network.ASN,
			rec.Network.Organization,
			rec.Security.IsVPN,
			elapsed,
		)
	}
}
```

Notice the use of `db.LookupNetwork(ip, &rec)`: in addition to decoding the struct fields, `LookupNetwork` returns the exact `*net.IPNet` prefix leaf (`198.51.100.0/24`) where the radix tree walk terminated, using the depth counter `i` from the bit walk at zero extra cost.

## Zero-copy deserialization in Rust (`maxminddb` + `serde`)

In Rust, because the `.mmdb` file is mapped into memory for the lifetime of the `Reader<Mmap>`, you can borrow string slices (`&'a str`) directly from the underlying `mmap` pages without allocating `String` buffers on the heap:

```rust
use maxminddb::Reader;
use serde::Deserialize;
use std::net::IpAddr;

#[derive(Deserialize, Debug)]
pub struct GeoBlock<'a> {
    #[serde(borrow)]
    pub country_code: &'a str,
    #[serde(borrow)]
    pub declared_country_code: &'a str,
    #[serde(borrow)]
    pub physical_country_code: &'a str,
    pub is_virtual_location: bool,
    pub accuracy_radius_km: Option<u32>,
}

#[derive(Deserialize, Debug)]
pub struct SecurityBlock<'a> {
    pub is_vpn: bool,
    pub is_tor_exit: bool,
    pub is_proxy: bool,
    pub is_hosting: bool,
    #[serde(borrow)]
    pub anonymizer_type: &'a str,
}

#[derive(Deserialize, Debug)]
pub struct IpScopeLookup<'a> {
    #[serde(borrow)]
    pub geo: GeoBlock<'a>,
    #[serde(borrow)]
    pub security: SecurityBlock<'a>,
}

pub fn lookup_zero_copy<'a>(
    reader: &'a Reader<memmap2::Mmap>,
    ip: IpAddr,
) -> Result<Option<IpScopeLookup<'a>>, maxminddb::MaxMindDBError> {
    reader.lookup::<IpScopeLookup<'a>>(ip)
}
```

With `#[serde(borrow)]`, `country_code`, `declared_country_code`, `physical_country_code`, and `anonymizer_type` are constructed as fat pointers `(ptr, len)` pointing directly into the read-only OS page cache of the `.mmdb` file. A single core on an AMD EPYC or Apple M-series processor executes over **1.8 million zero-copy lookups per second**.

## Custom field-family pruning: Sizing your `.mmdb` for L3 CPU cache

Modern server CPUs (such as AMD EPYC Genoa/Turin or AWS Graviton4) feature `32 MB to 128 MB` of L3 cache per CCD/socket. When the active working set of your `.mmdb` Binary Search Tree fits entirely inside L3 cache, pointer walks never stall on main DDR5 DRAM.

Not every workload needs all six intelligence families on every node. For example, an edge DDoS scrubber or CDN routing tier might only need `geo_country`, `network`, and `anonymity` (`is_vpn`, `is_tor_exit`, `is_proxy`), while leaving `geo_city` coordinates and `security_extensions` to backend analytics.

When you configure a custom database SKU in the ipscope Custom Database Builder, our compiler strips unselected field families **before** running Stage 1 (Bottom-Up Trie Coalescing). Because removing city-level subdivision splits allows thousands of adjacent `/24` blocks within the same country and ASN to coalesce into `/18` and `/16` supernets, pruning unused families shrinks **both** the Data Section and the Binary Search Tree `node_count`:

| Database SKU Profile | Included Intelligence Families | Tree `record_size` | Uncompressed `.mmdb` Size | p50 Lookup Latency | p99 Lookup Latency |
| --- | --- | --- | --- | --- | --- |
| **`IPScope-Country-v1`** | `geo_country`, `network` | `24-bit` | `~9.4 MB` | `0.38 µs` (`380 ns`) | `0.62 µs` (`620 ns`) |
| **`IPScope-Privacy-v1`** | `geo_country`, `network`, `hosting`, `anonymity` | `28-bit` | `~46.2 MB` | `0.54 µs` (`540 ns`) | `0.89 µs` (`890 ns`) |
| **`IPScope-Complete-v1`** | `geo_country`, `geo_city`, `network`, `connection`, `hosting`, `anonymity`, `security_extensions` | `28-bit` | `~141.4 MB` | `0.71 µs` (`710 ns`) | `1.24 µs` (`1,240 ns`) |

## Frequently asked questions (FAQ)

### Can I open ipscope `.mmdb` files with the standard `maxminddb` libraries in Python, Go, Rust, PHP, Java, and NGINX?
Yes. Every ipscope `.mmdb` artifact is 100% compliant with the open MMDB v2.0 binary specification. Any library or web server module that reads standard `.mmdb` files (`maxminddb-golang`, Rust `maxminddb`, Python `maxminddb`, Node `maxmind`, Java `maxmind-db`, and `ngx_http_geoip2_module`) can load ipscope `.mmdb` files directly. Only the decoded map keys follow the richer ipscope schema (`geo`, `network`, `connection`, `security`, `security_extensions`).

### If I run 16 worker processes on the same Linux host, does opening a 141 MB `.mmdb` file use `16 * 141 MB = 2.25 GB` of physical RAM?
No. When multiple processes open the same inode on disk using `mmap(2)` with `MAP_SHARED` or `MAP_PRIVATE` (read-only), the Linux kernel maps the exact same physical page-cache frames into each process's virtual page table. Across 16 NGINX, Go, or Python worker processes on one host, the physical RAM consumed by the `.mmdb` file is still only **141 MB total**.

### Should I call `mlock(2)` or pre-fault the `.mmdb` pages at startup?
On latency-critical edge proxies where the very first requests after process startup must not incur disk page faults, passing `MAP_POPULATE` (or sequentially reading the file into `/dev/null` during your container readiness probe before marking the pod healthy) pre-faults all `.mmdb` pages into the OS page cache so even cold-start lookups execute in under a microsecond.

---

<!-- Article URL: https://ipscope.io/blog/zero-downtime-atomic-mmdb-hot-reloads-in-production -->
---
slug: zero-downtime-atomic-mmdb-hot-reloads-in-production
title: "Zero-Downtime MMDB Hot-Reloads: Atomic POSIX mmap Swaps in Go, Rust, and Node.js"
description: "How to download, verify SHA-256 checksums, and hot-swap memory-mapped ipscope .mmdb databases in live production processes without SIGBUS crashes or dropped requests."
date: "2026-10-08"
readingTime: "15 min read"
category: "Databases"
author: "Hasan Muzak"
authorRole: "Principal Systems Architect"
tags: "MMDB, Zero Downtime, mmap, Go, Rust, Node.js, DevOps"
featured: "false"
---

## Why overwriting an `.mmdb` file in place crashes production servers with `SIGBUS`

Memory-mapped (`.mmdb`) IP intelligence databases achieve sub-microsecond (`< 1,000 ns`) lookup latencies because the reader maps the underlying file directly into the process's virtual address space using the POSIX `mmap(2)` system call. Instead of allocating 140 MB of private heap objects per worker process, the operating system kernel maps the file's disk blocks into the shared page cache and hands your process a virtual memory pointer (`uint8_t*` / `[]byte`).

However, `mmap(2)` comes with a strict contract between your application and the operating system kernel: **you must never mutate or truncate the underlying filesystem inode while a process has it mapped**.

When an engineer sets up a daily cron job or Kubernetes sidecar to refresh `/var/lib/ipscope/ipscope-complete.mmdb` and writes the naive command:

```bash
# DO NOT DO THIS IN PRODUCTION!
curl -fsSL -H "Authorization: Bearer ${KEY}" \
  -o /var/lib/ipscope/ipscope-complete.mmdb \
  "https://api.ipscope.io/v1/mmdb/releases/latest/download?format=mmdb"
```

Two catastrophic failures happen immediately inside your live Go, Rust, Python, C, or NGINX worker processes:

1. **Fatal `SIGBUS` process crashes on `O_TRUNC`**: When `curl -o /var/lib/ipscope/ipscope-complete.mmdb` opens the existing file for writing, it passes the `O_TRUNC` flag to `open(2)`, which instantaneously truncates the active inode's byte length to `0 bytes` before writing the new network stream. Under the POSIX `mmap` specification, if a thread in your application dereferences a virtual memory address that now lies beyond the truncated end-of-file of the underlying inode, the CPU memory management unit (MMU) faults and the Linux/BSD kernel delivers an uncatchable **`SIGBUS` (Bus Error)** signal, killing your server process immediately.
2. **Torn reads and corrupted pointer walks**: Even while bytes are streaming back into the file over the next few seconds, concurrent threads walking the old binary radix tree read half-written node records and out-of-bounds Data Section offsets, resulting in panics, slice bounds errors, or garbage country codes.

This production engineering guide walks through the exact mechanics of POSIX inodes and `rename(2)`, provides a battle-tested shell/systemd updater with `ETag` caching and cryptographic `SHA-256` verification, and shows lock-free atomic reader swap implementations in **Go (`atomic.Pointer`)**, **Rust (`ArcSwap`)**, and **Node.js**.

## How POSIX inodes, `mmap(2)`, and `rename(2)` actually work

To update a memory-mapped file with zero downtime and zero risk of `SIGBUS`, you must understand the separation between a **directory entry (filename)** and an **inode** on POSIX filesystems (`ext4`, `xfs`, `btrfs`, `zfs`, `APFS`):

```txt
BEFORE RENAME:
  Directory Entry: "ipscope-complete.mmdb"  --->  [ Inode #40112 (141 MB, Old Snapshot) ]
                                                        ^
                                                        | (Active mmap VMA)
                                                  [ Live Go/Rust/NGINX Workers ]

STEP 1: Stage & Verify New File on the SAME Filesystem Mount:
  Directory Entry: ".stage.ipscope.9a8f12"  --->  [ Inode #88304 (142 MB, New Snapshot) ]

STEP 2: Atomic rename(".stage.ipscope.9a8f12", "ipscope-complete.mmdb"):
  Directory Entry: "ipscope-complete.mmdb"  --->  [ Inode #88304 (142 MB, New Snapshot) ]

  Meanwhile, Inode #40112 (Old Snapshot) has link_count = 0 on disk,
  but its kernel reference count > 0 because live workers still hold the old mmap!
  Inode #40112 remains 100% intact in the OS page cache until the last worker calls munmap().
```

Look at what POSIX guarantees when you call `rename(2)` (via `mv -f` on the **same filesystem mount**):

1. `rename(2)` is atomic at the kernel VFS layer. No process opening `/var/lib/ipscope/ipscope-complete.mmdb` will ever observe a missing file or a partially written file; it will either open **Inode #40112** (old) or **Inode #88304** (new).
2. Existing `mmap` regions mapped to **Inode #40112** are completely unaffected by the directory rename! Inode #40112's bytes are never truncated or overwritten. The kernel keeps Inode #40112's pages alive until every process calls `munmap(2)` (`db.Close()`), after which the filesystem automatically frees Inode #40112's blocks.

### The `/tmp` cross-filesystem pitfall (`EXDEV`)

A classic operational bug occurs when a script downloads the temporary file to `/tmp/ipscope.mmdb.tmp` and runs `mv /tmp/ipscope.mmdb.tmp /var/lib/ipscope/ipscope-complete.mmdb`.

On almost all modern Linux distributions, `/tmp` is an in-memory `tmpfs` mount, while `/var/lib/ipscope` lives on the root `ext4` or `xfs` block device. Because an inode number only exists within a single filesystem mount, calling `rename(2)` across two different mounts returns `EXDEV (Invalid cross-device link)`. To hide this error, the `mv` command falls back to **opening the destination file with `O_TRUNC` and copying the bytes**, bringing back the exact `SIGBUS` crash you tried to avoid!

**Rule #1 of atomic `.mmdb` updates**: Always create your staging file in the **exact same directory** (`/var/lib/ipscope/.stage.XXXXXX`) as the target `.mmdb` file so they are guaranteed to share the same filesystem mount.

## Step 1: Production updater script with ETag, SHA-256, and canary validation

Before replacing the live `.mmdb` path, an updater script should verify three things:

1. **Conditional `ETag` check (`304 Not Modified`)**: Avoid downloading 140 MB if your local file already matches the latest release.
2. **Cryptographic `SHA-256` digest verification**: Verify the downloaded bytes against the signed `/v1/mmdb/releases/latest/checksums` manifest.
3. **MMDB Metadata footer sanity check**: Verify that the file ends with the valid `\xab\xcd\xefMaxMind.com` metadata marker before promoting it.

Save the following script to `/usr/local/bin/ipscope-mmdb-sync.sh`:

```bash
#!/usr/bin/env bash
set -euo pipefail

DB_DIR="${IPSCOPE_DB_DIR:-/var/lib/ipscope}"
DB_FILENAME="${IPSCOPE_DB_FILENAME:-ipscope-complete.mmdb}"
TARGET="${DB_DIR}/${DB_FILENAME}"
ETAG_FILE="${DB_DIR}/.${DB_FILENAME}.etag"
API_BASE="${IPSCOPE_API_BASE:-https://api.ipscope.io}"

if [ -z "${IPSCOPE_DOWNLOAD_KEY:-}" ]; then
  echo "ERROR: IPSCOPE_DOWNLOAD_KEY environment variable is not set" >&2
  exit 1
fi

mkdir -p "$DB_DIR"

# Create temporary staging inode in the SAME directory as TARGET
STAGE="$(mktemp "${DB_DIR}/.stage.${DB_FILENAME}.XXXXXX")"
ETAG_TMP="$(mktemp "${DB_DIR}/.etag.${DB_FILENAME}.XXXXXX")"

cleanup() {
  rm -f "$STAGE" "$ETAG_TMP"
}
trap cleanup EXIT

CURL_ETAG_ARGS=()
if [ -f "$ETAG_FILE" ] && [ -f "$TARGET" ]; then
  CURL_ETAG_ARGS=(--etag-compare "$ETAG_FILE")
fi

# 1. Download artifact with conditional ETag support
HTTP_STATUS="$(curl -sS -L -w "%{http_code}" \
  --retry 3 --retry-delay 2 --max-time 300 \
  -H "Authorization: Bearer ${IPSCOPE_DOWNLOAD_KEY}" \
  "${CURL_ETAG_ARGS[@]}" \
  --etag-save "$ETAG_TMP" \
  -o "$STAGE" \
  "${API_BASE}/v1/mmdb/releases/latest/download?format=mmdb")"

if [ "$HTTP_STATUS" = "304" ]; then
  echo "[ipscope-sync] Local ${DB_FILENAME} is already up to date (HTTP 304)."
  exit 0
fi

if [ "$HTTP_STATUS" != "200" ]; then
  echo "ERROR: Download failed with HTTP status ${HTTP_STATUS}" >&2
  exit 1
fi

# 2. Fetch and verify cryptographic SHA-256 digest
EXPECTED_SHA256="$(curl -fsSL \
  --retry 3 --max-time 30 \
  -H "Authorization: Bearer ${IPSCOPE_DOWNLOAD_KEY}" \
  "${API_BASE}/v1/mmdb/releases/latest/checksums?format=text" \
  | awk '/\.mmdb$/ {print $1; exit}')"

if [ -z "$EXPECTED_SHA256" ]; then
  echo "ERROR: Could not parse .mmdb SHA-256 from checksums endpoint" >&2
  exit 1
fi

if command -v sha256sum >/dev/null 2>&1; then
  ACTUAL_SHA256="$(sha256sum "$STAGE" | awk '{print $1}')"
else
  ACTUAL_SHA256="$(shasum -a 256 "$STAGE" | awk '{print $1}')"
fi

if [ "$ACTUAL_SHA256" != "$EXPECTED_SHA256" ]; then
  echo "ERROR: SHA-256 mismatch! expected=${EXPECTED_SHA256} actual=${ACTUAL_SHA256}" >&2
  exit 1
fi

# 3. Flush staging inode pages to disk, set permissions, and perform atomic POSIX rename(2)
chmod 0644 "$STAGE"
sync "$STAGE"
mv -f "$STAGE" "$TARGET"
mv -f "$ETAG_TMP" "$ETAG_FILE"

echo "[ipscope-sync] Atomically promoted new ${DB_FILENAME} (sha256=${ACTUAL_SHA256})."
```

## Step 2: Lock-free hot-reloads in Go (`sync/atomic.Pointer`)

Once `mv -f "$STAGE" "$TARGET"` atomically swaps the directory entry, how should your running Go service switch over to the new inode without blocking 100,000 concurrent HTTP requests per second on a `sync.RWMutex`?

Even an uncontended `sync.RWMutex.RLock()` requires an atomic CPU cache-line write (`readerCount.Add(1)`) on every single lookup, which causes cross-core cache-line bouncing on 64-core servers.

Instead, Go 1.19+ provides `sync/atomic.Pointer[T]`, which performs a **pure atomic memory read (`MOV` on x86_64, `LDAR` on ARM64)** with **zero cache-line invalidation**:

```go
package ipdb

import (
	"fmt"
	"net"
	"os"
	"sync/atomic"
	"syscall"
	"time"

	"github.com/oschwald/maxminddb-golang"
)

type readerSnapshot struct {
	reader *maxminddb.Reader
	inode  uint64
}

type AtomicMMDB struct {
	path    string
	current atomic.Pointer[readerSnapshot]
}

func fileInode(path string) (uint64, error) {
	info, err := os.Stat(path)
	if err != nil {
		return 0, err
	}
	if stat, ok := info.Sys().(*syscall.Stat_t); ok {
		return stat.Ino, nil
	}
	return uint64(info.ModTime().UnixNano()), nil
}

func NewAtomicMMDB(path string) (*AtomicMMDB, error) {
	ino, err := fileInode(path)
	if err != nil {
		return nil, fmt.Errorf("stat mmdb: %w", err)
	}
	r, err := maxminddb.Open(path)
	if err != nil {
		return nil, fmt.Errorf("open mmdb: %w", err)
	}

	// Canary sanity check before accepting reader
	var probe map[string]any
	if err := r.Lookup(net.ParseIP("1.1.1.1"), &probe); err != nil {
		_ = r.Close()
		return nil, fmt.Errorf("canary lookup failed: %w", err)
	}

	m := &AtomicMMDB{path: path}
	m.current.Store(&readerSnapshot{reader: r, inode: ino})
	return m, nil
}

// ReloadIfInodeChanged checks whether the target path points to a new inode.
// If so, it opens the new mmap, validates a canary IP, atomically swaps the pointer,
// and drains the old mmap after a 10-second grace period.
func (m *AtomicMMDB) ReloadIfInodeChanged() (bool, error) {
	newInode, err := fileInode(m.path)
	if err != nil {
		return false, err
	}

	cur := m.current.Load()
	if cur != nil && cur.inode == newInode {
		return false, nil // Same inode, nothing to reload
	}

	nextReader, err := maxminddb.Open(m.path)
	if err != nil {
		return false, fmt.Errorf("open new mmdb inode: %w", err)
	}

	// Canary lookup on new reader before exposing it to live traffic
	var canary map[string]any
	if err := nextReader.Lookup(net.ParseIP("1.1.1.1"), &canary); err != nil {
		_ = nextReader.Close()
		return false, fmt.Errorf("new mmdb failed canary check: %w", err)
	}

	prev := m.current.Swap(&readerSnapshot{
		reader: nextReader,
		inode:  newInode,
	})

	if prev != nil && prev.reader != nil {
		go func(old *maxminddb.Reader) {
			// Allow all in-flight sub-microsecond lookups on the old pointer to complete
			time.Sleep(10 * time.Second)
			_ = old.Close()
		}(prev.reader)
	}

	return true, nil
}

// Lookup executes on the hot path with zero locks and zero allocations.
func (m *AtomicMMDB) Lookup(ip net.IP, result any) error {
	return m.current.Load().reader.Lookup(ip, result)
}
```

Look at why checking `stat.Ino` (`ReloadIfInodeChanged`) is so clean in containerized and Kubernetes environments:

- You do not even need to wire `SIGHUP` signals across container namespaces or worry about `fsnotify` missing directory events on Kubernetes ConfigMap/hostPath mounts.
- A lightweight background goroutine calling `ReloadIfInodeChanged()` once every 15 seconds executes a single `stat(2)` syscall, detects the new inode number immediately after `mv -f` completes, validates the new `.mmdb` with a canary lookup (`1.1.1.1`), swaps the pointer in `1 nanosecond`, and closes the old `mmap` 10 seconds later.

## Step 3: Wait-free hot-reloads in Rust (`arc-swap` + `maxminddb`)

In Rust, because `maxminddb::Reader<Mmap>` owns the memory-mapped region via RAII (`Drop` calls `munmap(2)` automatically when the last reference is dropped!), combining `std::sync::Arc` with the `arc-swap` crate (`ArcSwap`) gives you **automatic reference-counted memory reclamation without even needing a sleep timer**:

```rust
use arc_swap::ArcSwap;
use maxminddb::Reader;
use memmap2::Mmap;
use serde::de::DeserializeOwned;
use std::net::IpAddr;
use std::path::{Path, PathBuf};
use std::sync::Arc;

pub struct HotReloadMmdb {
    path: PathBuf,
    reader: ArcSwap<Reader<Mmap>>,
}

impl HotReloadMmdb {
    pub fn open(path: impl AsRef<Path>) -> Result<Self, maxminddb::MaxMindDBError> {
        let path_buf = path.as_ref().to_path_buf();
        let initial = Reader::open_mmap(&path_buf)?;
        Ok(Self {
            path: path_buf,
            reader: ArcSwap::from_pointee(initial),
        })
    }

    /// Atomically swaps in a freshly memory-mapped `.mmdb` reader.
    /// Any thread currently executing `lookup()` holds an `Arc` guard to the old
    /// `Reader<Mmap>`, so the old `Mmap` is unmapped automatically the instant
    /// the last in-flight lookup guard drops!
    pub fn reload(&self) -> Result<(), maxminddb::MaxMindDBError> {
        let next_reader = Reader::open_mmap(&self.path)?;
        // Validate canary lookup before swapping
        let canary_ip: IpAddr = "1.1.1.1".parse().unwrap();
        let _: Option<serde_json::Value> = next_reader.lookup(canary_ip)?;

        self.reader.store(Arc::new(next_reader));
        Ok(())
    }

    #[inline]
    pub fn lookup<T: DeserializeOwned>(
        &self,
        ip: IpAddr,
    ) -> Result<Option<T>, maxminddb::MaxMindDBError> {
        let guard = self.reader.load();
        guard.lookup::<T>(ip)
    }
}
```

Because `self.reader.load()` returns a wait-free ` arc_swap::Guard<Arc<Reader<Mmap>>>`, the old `Mmap` cannot be unmapped while any thread is still inside `guard.lookup::<T>(ip)`. The moment `self.reader.store(Arc::new(next_reader))` is called and the last in-flight `guard` goes out of scope, Rust's `Drop` implementation calls `munmap(2)` deterministically.

## Step 4: Zero-downtime swaps in Node.js (`maxmind` package)

In Node.js, the V8 JavaScript event loop is single-threaded per worker, which makes reference swapping even simpler: opening the new `Reader` asynchronously and reassigning the module-level variable `activeReader = nextReader` is inherently atomic across event loop ticks, while V8's garbage collector closes the previous buffer once any synchronous calls complete:

```ts
import maxmind, { type Reader, type Response } from 'maxmind';

export class HotReloadMmdbReader<T extends Response> {
  private reader!: Reader<T>;
  private readonly dbPath: string;

  private constructor(dbPath: string, initialReader: Reader<T>) {
    this.dbPath = dbPath;
    this.reader = initialReader;
  }

  public static async open<T extends Response>(dbPath: string): Promise<HotReloadMmdbReader<T>> {
    const initial = await maxmind.open<T>(dbPath);
    return new HotReloadMmdbReader<T>(dbPath, initial);
  }

  public async reload(): Promise<void> {
    const nextReader = await maxmind.open<T>(this.dbPath);
    // Validate canary lookup before replacing live reference
    nextReader.get('1.1.1.1');
    this.reader = nextReader;
  }

  public lookup(ip: string): T | null {
    return this.reader.get(ip);
  }
}
```

## Comparing hot-reload architectures across runtimes and proxies

| Runtime / Proxy | Recommended Reload Pattern | Hot-Path Lookup Overhead | Old Inode Reclamation |
| --- | --- | --- | --- |
| **Go (`maxminddb-golang`)** | `sync/atomic.Pointer[maxminddb.Reader]` + inode `stat(2)` poller or `SIGHUP` | `0 ns` (single atomic pointer load) | Background drain timer (`time.Sleep(10s)` -> `old.Close()`) |
| **Rust (`maxminddb` + `arc-swap`)** | `ArcSwap<Reader<Mmap>>` | `< 2 ns` (wait-free hazard/generation guard) | Instantaneous deterministic RAII `Drop` when last guard exits |
| **Node.js (`maxmind`)** | Async `maxmind.open()` + single-threaded reference reassignment | `0 ns` | Automatic V8 garbage collection |
| **NGINX (`ngx_http_geoip2_module`)** | Atomic `mv -f` on disk followed by `nginx -s reload` | `0 ns` | Old worker processes exit gracefully after finishing active requests |
| **HAProxy / Envoy** | Atomic `mv -f` + admin socket map/filter reload | `0 ns` | Reference-counted epoch reclamation |

## Frequently asked questions (FAQ)

### What happens if the network connection drops halfway through downloading the `.mmdb` update?
Because our updater script downloads into a temporary staging file (`mktemp "${DB_DIR}/.stage.XXXXXX"`) with `set -euo pipefail` and verifies the cryptographic `SHA-256` checksum **before** running `mv -f`, an interrupted download or corrupted file simply exits at the curl or `sha256sum` step and cleans up the staging file via `trap cleanup EXIT`. Your live `/var/lib/ipscope/ipscope-complete.mmdb` file is never touched.

### How much extra RAM is used during the 10-second reload overlap window?
During the brief overlap window when the new inode is mapped and the old inode is draining in-flight requests, both `.mmdb` files reside in the OS page cache. For `IPScope-Country-v1` (`~9.4 MB`), the temporary overlap is `~18.8 MB`; for `IPScope-Privacy-v1` (`~46.2 MB`), `~92.4 MB`; and for `IPScope-Complete-v1` (`~141.4 MB`), `~282.8 MB`. As soon as `old.Close()` (`munmap`) runs and the unlinked old inode's reference count drops to zero, the kernel reclaims 100% of the old inode's page cache and disk blocks immediately.

---