Enriching 100,000+ IPs in Under 350 KB of RAM with Official IPScope Server SDKs

#The memory and socket traps in large-scale IP log enrichment

When a security operations team, data engineering pipeline, or incident response script needs to enrich 100,000 to 1,000,000 IP addresses from an NGINX access log, AWS VPC Flow Log, or Cloudflare log export against a REST API, naive scripts almost always hit two production walls:

1. Unbounded memory and socket fan-out: Loading an entire 500 MB log file into memory and firing Promise.all(ips.map(ip => fetch(...))) or asyncio.gather(*tasks) spawns tens of thousands of concurrent TCP connections, exhausts OS ephemeral ports (EADDRNOTAVAIL), triggers 429 Too Many Requests rate limits, and crashes the worker container with an Out-Of-Memory (OOMKilled) exit.
2. Paying credits and latency for duplicate hot IPs: Real-world network traffic follows a steep power-law (Zipf) distribution. In a typical 500,000-line web server access log, fewer than 15% of the IP addresses are unique. The remaining 85% of lines are repeated requests from the same client sessions, health-checkers, search engine crawlers, and mobile CGNAT gateways. Sending the same IP address to a remote API 400 times wastes lookup credits and multiplies network serialization overhead by 5x to 10x.

When we designed the eight official IPScope server SDKs (Python, Node.js / TypeScript, Go, Rust, Java, .NET, Ruby, and PHP), we built constant-memory batch streaming and automatic client-side deduplication directly into the core client primitives.

This engineering article walks through the internal architecture of lookupBatch and lookupBatchStream, explains how our bounded O(1) LRU cache and in-flight request coalescing eliminate redundant API calls, and shows production streaming examples in Node.js, Python, Go, and Rust.

#The three architectural guarantees of official IPScope SDKs

Every official IPScope SDK implements the exact same three-stage execution pipeline before a single byte leaves your server over HTTPS:

Input IP Stream (Iterable / AsyncIterable / Channel / Stream)
   |
   v
[ Stage 1: Lazy 100-IP Chunking Window ]
   |  Pulls at most 100 IPs at a time from the caller's stream
   v
[ Stage 2: Bounded O(1) LRU Cache & Intra-Batch Deduplicator ]
   |  1. Satisfies hot IPs immediately from in-memory LRU cache (0 network calls)
   |  2. Deduplicates repeated IPs inside the current 100-IP window
   v
[ Stage 3: Single POST /v1/ip/batch HTTP/2 Request ]
   |  Sends only the unique cache-miss IPs, populates LRU cache,
   |  and reconstructs the 100-item BatchLookupResponse in original order
   v
Yields BatchLookupResponse to caller (Steady-State Heap < 350 KB)

Guarantee 1: Bounded O(1) In-Memory LRU Cache

Every SDK client initializes a thread-safe (or coroutine-safe) Least Recently Used (LRU) cache bounded by maxCacheEntries (default 2,048 entries) and cacheTtlMs (default 60,000 ms).

• On every lookupIp(ip), lookupBatch(ips), or lookupBatchStream(source) call, the SDK checks the local LRU cache first.
• If 198.51.100.7 was resolved within the last 60 seconds, the SDK returns the cached IpLookupResponse immediately in < 100 nanoseconds without making a network request or consuming an API credit.
• Because the cache strictly enforces its maxCacheEntries cap by evicting the oldest entry in O(1) time on insertion, the cache's memory footprint can never grow unboundedly, even if your worker processes billions of unique IPs over months of uptime.

Guarantee 2: Intra-batch deduplication and order preservation

Suppose a 100-line log chunk contains 60 cached IPs and 40 uncached IPs, and among those 40 uncached IPs, an aggressive scraper IP (203.0.113.99) appears 15 times.

Instead of sending all 100 IPs to POST /v1/ip/batch, the SDK:

1. Pulls the 60 cached records directly from memory.
2. Collapses the 40 uncached inputs down to the 26 unique IP strings that actually missed the cache.
3. Sends a single POST /v1/ip/batch request containing only those 26 unique IPs.
4. Stores the 26 fresh records into the LRU cache and reassembles the returned BatchLookupResponse so the caller receives all 100 records in the exact sequence requested.

On a standard production web log, this two-tier deduplication cuts outbound HTTP payload volume and API credit usage by 75% to 90% automatically.

Guarantee 3: Backpressure-aware lazy streaming (lookupBatchStream)

Rather than requiring you to load a 2 GB CSV or log file into an array first, lookupBatchStream (or lookup_batch_stream in Python/Rust/Ruby) accepts a lazy iterator or async generator and yields one BatchLookupResponse per 100-IP window.

Because the next 100 lines are not read from disk until your for await (...) loop finishes processing the current batch (for example, writing the enriched rows to PostgreSQL, ClickHouse, or an output file), end-to-end backpressure is preserved automatically and steady-state heap usage stays under 350 KB.

#Streaming 100,000+ IPs in Node.js & TypeScript (@ipscope/sdk-node)

In Node.js, combining node:readline over a file stream with client.lookupBatchStream() lets you enrich arbitrarily large log files in constant memory:

import { createReadStream, createWriteStream } from 'node:fs';
import { createInterface } from 'node:readline';
import { IpScopeClient } from '@ipscope/sdk-node';

const client = new IpScopeClient({
  apiKey: process.env.IPSCOPE_KEY!,
  maxCacheEntries: 8192,
  cacheTtlMs: 300_000, // 5-minute TTL for log enrichment runs
  timeoutMs: 10_000,
  maxRetries: 3,
});

async function* extractIpsFromLog(filePath: string): AsyncGenerator<string> {
  const rl = createInterface({
    input: createReadStream(filePath, { encoding: 'utf8' }),
    crlfDelay: Infinity,
  });

  for await (const line of rl) {
    const trimmed = line.trim();
    if (!trimmed || trimmed.startsWith('#')) continue;
    // Extract first whitespace-delimited token (e.g. NGINX $remote_addr)
    const firstSpace = trimmed.indexOf(' ');
    const ip = firstSpace === -1 ? trimmed : trimmed.slice(0, firstSpace);
    yield ip;
  }
}

async function enrichAccessLog(inputPath: string, outputPath: string): Promise<void> {
  const out = createWriteStream(outputPath, { encoding: 'utf8' });
  let totalEnriched = 0;

  for await (const batch of client.lookupBatchStream(extractIpsFromLog(inputPath))) {
    for (const record of batch.results) {
      const row = JSON.stringify({
        ip: record.ip,
        country: record.geo.physical_country_code,
        declared_country: record.geo.declared_country_code,
        asn: record.network.asn,
        org: record.network.organization,
        is_vpn: record.security.is_vpn,
        is_proxy: record.security.is_proxy,
        velocity: record.security_extensions?.abuse_velocity_score ?? 0,
      });
      if (!out.write(row + '\n')) {
        await new Promise<void>((resolve) => out.once('drain', resolve));
      }
    }
    totalEnriched += batch.count;
  }

  out.end();
  console.log(`Successfully enriched ${totalEnriched} log lines.`);
}

enrichAccessLog('/var/log/nginx/access.log', '/tmp/enriched.ndjson').catch(console.error);

Notice how out.once('drain', resolve) cooperates directly with the for await generator loop: if the downstream disk or database writer slows down, lookupBatchStream pauses pulling from extractIpsFromLog immediately.

#Async generator streaming in Python (ipscope)

The official Python SDK (pip install ipscope) provides both synchronous (IpScopeClient) and asyncio-native (AsyncIpScopeClient) clients backed by persistent HTTP/1.1 and HTTP/2 connection pooling:

import asyncio
import json
import os
from collections.abc import AsyncIterator
from ipscope import AsyncIpScopeClient

async def iter_log_ips(log_path: str) -> AsyncIterator[str]:
    with open(log_path, "r", encoding="utf-8") as fh:
        for line in fh:
            ip = line.strip().split(" ", 1)[0]
            if ip and not ip.startswith("#"):
                yield ip

async def enrich_siem_export(log_path: str) -> None:
    async with AsyncIpScopeClient(
        api_key=os.environ["IPSCOPE_KEY"],
        max_cache_entries=8192,
        cache_ttl_seconds=300.0,
    ) as client:
        flagged_count = 0
        total_count = 0

        async for batch in client.lookup_batch_stream(iter_log_ips(log_path)):
            total_count += batch["count"]
            for item in batch["results"]:
                sec = item["security"]
                ext = item.get("security_extensions") or {}
                if sec["is_vpn"] or sec["is_proxy"] or ext.get("abuse_velocity_score", 0) >= 70:
                    flagged_count += 1
                    print(
                        json.dumps(
                            {
                                "ip": item["ip"],
                                "country": item["geo"]["physical_country_code"],
                                "asn": item["network"]["asn"],
                                "anonymizer_type": sec["anonymizer_type"],
                                "reason_codes": sec["reason_codes"],
                                "abuse_velocity_score": ext.get("abuse_velocity_score", 0),
                            }
                        )
                    )

        print(f"Processed {total_count} IPs ({flagged_count} flagged).")

if __name__ == "__main__":
    asyncio.run(enrich_siem_export("/var/log/auth-ips.txt"))

#Context-aware channel streaming in Go (github.com/badele-llc/ipscope-go)

In Go (go get github.com/badele-llc/ipscope-go), LookupBatchStream accepts a context.Context and a read-only channel (<-chan string), returning a typed result channel (<-chan StreamBatchResult) that closes cleanly when the input channel is drained or the context.Context is cancelled:

package main

import (
	"bufio"
	"context"
	"fmt"
	"log"
	"os"
	"strings"
	"time"

	ipscope "github.com/badele-llc/ipscope-go"
)

func streamFileIps(ctx context.Context, path string) (<-chan string, error) {
	file, err := os.Open(path)
	if err != nil {
		return nil, err
	}

	out := make(chan string, 256)
	go func() {
		defer close(out)
		defer file.Close()

		scanner := bufio.NewScanner(file)
		for scanner.Scan() {
			ip := strings.TrimSpace(scanner.Text())
			if ip == "" || strings.HasPrefix(ip, "#") {
				continue
			}
			select {
			case out <- ip:
			case <-ctx.Done():
				return
			}
		}
	}()
	return out, nil
}

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
	defer cancel()

	client := ipscope.NewClient(
		os.Getenv("IPSCOPE_KEY"),
		ipscope.WithCache(8192, 5*time.Minute),
	)

	ipChan, err := streamFileIps(ctx, "/var/log/firewall-ips.txt")
	if err != nil {
		log.Fatal(err)
	}

	var total int
	for batchRes := range client.LookupBatchStream(ctx, ipChan, 100) {
		if batchRes.Err != nil {
			log.Fatalf("batch stream error: %v", batchRes.Err)
		}
		total += batchRes.Response.Count
		for _, item := range batchRes.Response.Results {
			if item.Security.IsTorExit || item.Security.IsProxy {
				fmt.Printf("Blocked %s (%s) reasons=%v\n",
					item.IP, item.Security.AnonymizerType, item.Security.ReasonCodes)
			}
		}
	}
	fmt.Printf("Enriched %d IPs cleanly\n", total)
}

#Benchmark: Naive fetch fan-out vs. IPScope SDK lookupBatchStream

We benchmarked enriching a real-world 100,000-line production NGINX access log (14,820 unique IPv4 and IPv6 addresses across 100,000 log lines) on a standard 2-vCPU Linux container (Node.js 22, 50 ms network RTT to the nearest API edge):

By combining 100-IP batching with bounded LRU caching and intra-batch deduplication, the official SDK finishes 74x faster, uses 470x less heap memory, and reduces API credit consumption by 85.1%.

#Backend snapshot consistency and atomic pre-debit validation

On the server side, POST /v1/ip/batch is engineered with two strict operational guarantees:

1. Atomic Dataset Snapshot Pinning: At the very beginning of a batch request, the API worker acquires an atomic pointer reference to the active in-memory Radix/LPM snapshot (SnapshotLoaderService). Even if a background dataset hot-reload happens mid-request, all 100 IPs in the batch are resolved against the exact same dataset_version.
2. All-or-Nothing Pre-Debit Validation: Before debiting a single credit from your workspace quota, the controller validates the syntax of every IP string in the ips array (1 to 100 items). If even one element in the array is malformed (for example, "999.999.999.999" or an empty string), the request is rejected immediately with 400 Bad Request (INVALID_IP_FORMAT) and zero credits are deducted.

#Installing the official SDK for your stack

All eight official IPScope server SDKs are open-source and published to their canonical package registries:

#Frequently asked questions (FAQ)

How does the SDK handle transient 429 Too Many Requests or 502/503/504 gateway errors during a long stream?

Every official SDK includes an automatic retry state machine with exponential backoff and full jitter (maxRetries: 2 by default, configurable). When the server returns a 429 or 503 response with a Retry-After header, the SDK honors the exact server backoff window before retrying the failed 100-IP chunk transparently inside lookupBatchStream.

When should I use lookupBatchStream via the REST API versus downloading the offline .mmdb or .parquet database?

If your job enriches up to a few million IPs per day from serverless functions, CI/CD workers, or microservices where you do not want to manage local database files, lookupBatchStream with the LRU cache enabled is the simplest zero-maintenance choice. If you are enriching hundreds of millions or billions of log rows daily inside a SIEM or data warehouse, we recommend subscribing to an ipscope Database SKU and running vectorized SQL joins against our .parquet artifact or local mmap lookups against our .mmdb artifact.