Cards outlined in red are the critical path: get these right and everything else is configuration.
1Nightly edge job CRITICAL BUILD
Runs inside the NWK lake. Applies eligibility filters only (opt-outs, ineligible MDNs/URLs) and writes the raw, filtered result as date-partitioned Parquet. No hashing, tokenization or masking happens on-prem — that work moves into GCP (component 4) so there is one governed place it runs, not two.
Spark/SQL filtering job; output contract (schema registry) versioned with the GCP side; no crypto library to maintain on-prem.
2Transfer: STS + Interconnect CRITICAL CONFIG
Storage Transfer Service agent pools pull the nightly Parquet drop over Dedicated/Partner Interconnect into the GCS bucket. Checksummed, resumable, bandwidth-capped. Ingress to GCP is free — you pay only the circuit.
agent pool on-prem, transfer job with manifest + checksum validation, private Google access, alert on missed transfer window.
3GCS Bucket
The landing place for the raw daily extract: a dedicated, tightly-scoped (not a general-purpose data lake bucket). It holds unmasked data only briefly — service-account-only IAM (no human read access), CMEK encryption at rest, VPC-SC perimeter, and a lifecycle policy that force-deletes objects within ~48 hours of successful processing. This short exposure window plus the perimeter is what makes it safe for raw data to touch GCP at all.
dedicated bucket per environment, IAM bound to the BQ Load job and DLP service accounts only, Pub/Sub event notification for arrival sensing, automatic lifecycle deletion.
4Cloud DLP — de-identification CRITICAL BUILD
The masking step runs as a : , called from the pipeline immediately after landing. Its de-identification templates provide CryptoHashConfig (one-way hash) and CryptoDeterministicConfig (format-preserving, reversible-with-key tokenization) out of the box, wrapped with a Cloud KMS key — no custom tokenization library to write or maintain. The same stage applies the eligibility/whitelist check against the reference table before anything is written to a durable zone.
DLP de-identification template (deterministic encryption for MDN, using a KMS-wrapped crypto key); whitelist reference table in BQ; pipeline step that calls the DLP API per batch.
5BQ Enterprise Slots — category pipeline CRITICAL BUILD
After Cloud DLP masks the raw extract, a BQ Load job materializes it into the lakehouse's curated zone. From there, BQ Enterprise Slots (100 reserved slots, Enterprise Edition at $0.044/slot-hr) run the 5-step Omni Enterprise category pipeline as BigQuery SQL jobs: (1) URL taxonomy lookup, (2) MSISDN identification scan, (3) web day-level aggregation into 5 tables, (4) foot traffic day-level aggregation into 5 tables, (5) category-level rollup. All steps run in a single 4-hour slot window per day ($17.60/day fixed — costs do not scale with data volume). Intermediate tables persist for 12 months with active/long-term BQ storage pricing (~$80/month steady-state). Cloud Composer orchestrates the slot reservation, job sequencing, and SLA alerting.
BQ Enterprise Slots reservation (100 slots, commitment-free or 1-year CUD); 5-table schema for web + foot day-agg intermediate layers; Composer DAG with arrival sensor, DQ gates between steps, retry logic, and slot release on completion.
6BigQuery lakehouse CRITICAL CONFIG
Single analytical home. Zones: (native tables, date-partitioned, clustered on tokenized MDN / geohash — materialized via Cloud DLP + BQ Load job, then processed by the BQ Enterprise Slots category pipeline), (24h aggregates, k-anonymity enforced). The GCP↔on-prem daily match is a plain BQ join on the shared token; the raw zone no longer persists here since raw data lives only briefly in the GCS bucket (component 3). The lakehouse also publishes curated and gold zones to Eureka (component 12) and vendors (component 13) — no separate sharing box in between.
dataset layout per zone/environment, partition expiry, policy tags on sensitive columns, authorized views for consumers, one Analytics Hub listing per consumer (Eureka, each vendor) published straight off this dataset.
7BigQuery Analytics Hub — cross-team ingestion CRITICAL BUILD
Location data, demographics and Maps POI are owned and maintained by a in their own BigQuery project — this pipeline doesn't copy or re-host that data. Instead, that team exposes it as a , and the BQ Enterprise Slots category pipeline (component 5) subscribes to the listing and reads it directly for the join and aggregation steps, zero-copy. There's nothing to build on the source side beyond agreeing the listing's schema and refresh cadence with the owning team; this pipeline only needs a subscription and read access on its own billing project.
subscribe to the owning team's data exchange listing, grant the BQ category pipeline service account read access on the subscribed dataset, agree refresh cadence/SLA with the source team, schema-change alerting.
8Maps POI enrichment
Maps POI reference data (POI-to-lat/long mapping) also lives in the other team's BigQuery project, refreshed there from the Google Maps Places API, and reaches this pipeline through the same Analytics Hub listing as location data and demographics (component 7) — not a separate build here.
confirm Places-licensing terms carry through the shared listing; no ingestion job to write on this side.
9Cloud Composer CRITICAL CONFIG
Owns every production schedule: waits for the on-prem drop, runs DLP de-identification + BQ Load, triggers the BQ Enterprise Slots 5-step category pipeline, runs BQ aggregate SQL, validates, publishes both to Client Delivery and to the Analytics Hub listings. SLAs, retries, backfill, and alerting live here — not in cron scripts.
one DAG per product (daily match, category pipeline, POI refresh, client delivery, Analytics Hub publish), arrival sensors, slot reservation acquire/release steps, SLA-miss alerts to chat/email, backfill runbook.
10Governance & privacy layer CRITICAL CONFIG
Dataplex (catalog, lineage, data quality), Cloud KMS (DLP crypto keys, CMEK), policy tags (column-level access), VPC Service Controls (perimeter so data can't be exfiltrated even by a compromised credential, and so the GCS bucket can never be reached from outside the pipeline's service accounts).
VPC-SC perimeter first (before the GCS bucket goes live), KMS key rotation, Dataplex DQ rules on curated zone, policy-tag taxonomy that also governs Analytics Hub listing eligibility.
11Client Delivery CRITICAL BUILD
The data the operator's hedge-fund clients receive through this path is the 24-hour, k-anonymous gold-zone aggregate — never row-level or tokenized data, no exceptions. Delivered per-fund, once daily: via Storage Transfer Service (GCS→S3 is a native STS job type), or for clients who require it. This is the route for clients without a GCP footprint of their own; clients who do have one are better served by component 13.
STS S3 transfer job per fund (or shared job, per-fund prefix); SFTP push job (Cloud Run + paramiko/similar) for SFTP-only clients; delivery manifest + checksum in the completion notice.
12Eureka Delivery
Eureka builds and operates the client-facing dashboards, so Eureka needs its own access to the lakehouse — via a listing published directly off the lakehouse (component 6), the recommended default: zero-copy, revocable, and able to carry gold (and, where justified, curated) data without any file ever moving. No separate infrastructure to build beyond the listing itself.
Eureka-facing listing on the lakehouse's data exchange (component 6), scoped to gold by default; grant curated access only per documented need.
13Vendor Access
The same Analytics Hub mechanism, extended to who operate their own GCP project — hedge funds, distribution partners, the Omni product line. Rather than standing up a bespoke S3/SFTP feed per vendor, each gets its own revocable listing off the lakehouse (component 6), which can be gold-only or, for vetted partners with a signed data agreement, curated-level. This is the preferred path for any consumer that already lives in GCP; Client Delivery (component 11) remains the fallback for those who don't.
one listing per vendor with its own entitlement and audit trail; default gold-only, curated by exception with sign-off.
14Phase 3: streaming ingestion
When internet browsing data moves in fully: Pub/Sub (events) or Datastream (CDC) stream records directly into BQ via the Storage Write API. Cloud DLP de-identification runs on micro-batches; BQ Enterprise Slots process continuous windows rather than a daily batch. Nightly batch becomes just another window size — the SQL category pipeline is unchanged; only the trigger cadence shifts.
Pub/Sub topic + BQ subscription with DLP inspection, continuous Composer DAG trigger on record count thresholds, backfill of history via STS, dual-run validation before on-prem ETL retirement.
