Cards outlined in red are the critical path: get these right and everything else is configuration.
1Nightly edge job CRITICAL BUILD
Runs inside the NWK lake. Applies eligibility filters only (opt-outs, ineligible MDNs/URLs) and writes the raw, filtered result as date-partitioned Parquet. No hashing, tokenization or masking happens on-prem — that work moves into GCP (component 4) so there is one governed place it runs, not two.
Spark/SQL filtering job; output contract (schema registry) versioned with the GCP side; no crypto library to maintain on-prem.
2Transfer: STS + Interconnect CRITICAL CONFIG
Storage Transfer Service agent pools pull the nightly Parquet drop over Dedicated/Partner Interconnect into the GCS bucket. Checksummed, resumable, bandwidth-capped. Ingress to GCP is free — you pay only the circuit.
agent pool on-prem, transfer job with manifest + checksum validation, private Google access, alert on missed transfer window.
3GCS Bucket
The landing place for the raw daily extract: a dedicated, tightly-scoped (not a general-purpose data lake bucket). It holds unmasked data only briefly — service-account-only IAM (no human read access), CMEK encryption at rest, VPC-SC perimeter, and a lifecycle policy that force-deletes objects within ~48 hours of successful processing. This short exposure window plus the perimeter is what makes it safe for raw data to touch GCP at all.
dedicated bucket per environment, IAM bound to the Dataflow/Dataproc and DLP service accounts only, Pub/Sub event notification for arrival sensing, automatic lifecycle deletion.
4Cloud DLP — de-identification CRITICAL BUILD
The masking step runs as a : , called from the pipeline immediately after landing. Its de-identification templates provide CryptoHashConfig (one-way hash) and CryptoDeterministicConfig (format-preserving, reversible-with-key tokenization) out of the box, wrapped with a Cloud KMS key — no custom tokenization library to write or maintain. The same stage applies the eligibility/whitelist check against the reference table before anything is written to a durable zone.
DLP de-identification template (deterministic encryption for MDN, using a KMS-wrapped crypto key); whitelist reference table in BQ; pipeline step that calls the DLP API per batch.
5Dataflow / Dataproc — join & aggregate CRITICAL BUILD
Implements the remaining pipeline steps — demographic join, POI lat/long mapping, 24-hour aggregation — on the now-tokenized data. is the target: one codebase that runs batch today and streaming in Phase 3, eliminating the current on-prem vs GCP dual build. If today's pipeline is Spark, runs it unchanged as a Phase 1 bridge while the logic is ported to Beam — see the Dataproc vs Dataflow comparison (§03) for the full trade-off.
Phase 1 — Dataproc Serverless job, no rewrite; Phase 1–2 — port to a Beam Flex Template with unit + pipeline tests, dead-letter outputs, data-quality metrics to Cloud Monitoring.
6BigQuery lakehouse CRITICAL CONFIG
Single analytical home. Zones: (native tables, date-partitioned, clustered on tokenized MDN / geohash — output of DLP + Dataflow/Dataproc), (24h aggregates, k-anonymity enforced). The GCP↔on-prem daily match is a plain BQ join on the shared token; the raw zone no longer persists here since raw data lives only briefly in the GCS bucket (component 3). The lakehouse also publishes curated and gold zones to Eureka (component 12) and vendors (component 13) — no separate sharing box in between.
dataset layout per zone/environment, partition expiry, policy tags on sensitive columns, authorized views for consumers, one Analytics Hub listing per consumer (Eureka, each vendor) published straight off this dataset.
7BigQuery Analytics Hub — cross-team ingestion CRITICAL BUILD
Location data, demographics and Maps POI are owned and maintained by a in their own BigQuery project — this pipeline doesn't copy or re-host that data. Instead, that team exposes it as a , and Dataflow/Dataproc (component 5) subscribes to the listing and reads it directly for the join and aggregation step, zero-copy. There's nothing to build on the source side beyond agreeing the listing's schema and refresh cadence with the owning team; this pipeline only needs a subscription and read access on its own billing project.
subscribe to the owning team's data exchange listing, grant the Dataflow/Dataproc service account read access on the subscribed dataset, agree refresh cadence/SLA with the source team, schema-change alerting.
8Maps POI enrichment
Maps POI reference data (POI-to-lat/long mapping) also lives in the other team's BigQuery project, refreshed there from the Google Maps Places API, and reaches this pipeline through the same Analytics Hub listing as location data and demographics (component 7) — not a separate build here.
confirm Places-licensing terms carry through the shared listing; no ingestion job to write on this side.
9Cloud Composer CRITICAL CONFIG
Owns every production schedule: waits for the on-prem drop, runs DLP de-identification, runs Dataflow/Dataproc, runs BQ aggregate SQL, validates, publishes both to Client Delivery and to the Analytics Hub listings. SLAs, retries, backfill, and alerting live here — not in cron scripts.
one DAG per product (daily match, POI refresh, client delivery, Analytics Hub publish), arrival sensors, SLA-miss alerts to chat/email, backfill runbook.
10Governance & privacy layer CRITICAL CONFIG
Dataplex (catalog, lineage, data quality), Cloud KMS (DLP crypto keys, CMEK), policy tags (column-level access), VPC Service Controls (perimeter so data can't be exfiltrated even by a compromised credential, and so the GCS bucket can never be reached from outside the pipeline's service accounts).
VPC-SC perimeter first (before the GCS bucket goes live), KMS key rotation, Dataplex DQ rules on curated zone, policy-tag taxonomy that also governs Analytics Hub listing eligibility.
11Client Delivery CRITICAL BUILD
The data the operator's hedge-fund clients receive through this path is the 24-hour, k-anonymous gold-zone aggregate — never row-level or tokenized data, no exceptions. Delivered per-fund, once daily: via Storage Transfer Service (GCS→S3 is a native STS job type), or for clients who require it. This is the route for clients without a GCP footprint of their own; clients who do have one are better served by component 13.
STS S3 transfer job per fund (or shared job, per-fund prefix); SFTP push job (Cloud Run + paramiko/similar) for SFTP-only clients; delivery manifest + checksum in the completion notice.
12Eureka Delivery
Eureka builds and operates the client-facing dashboards, so Eureka needs its own access to the lakehouse — via a listing published directly off the lakehouse (component 6), the recommended default: zero-copy, revocable, and able to carry gold (and, where justified, curated) data without any file ever moving. No separate infrastructure to build beyond the listing itself.
Eureka-facing listing on the lakehouse's data exchange (component 6), scoped to gold by default; grant curated access only per documented need.
13Vendor Access
The same Analytics Hub mechanism, extended to who operate their own GCP project — hedge funds, distribution partners, the Omni product line. Rather than standing up a bespoke S3/SFTP feed per vendor, each gets its own revocable listing off the lakehouse (component 6), which can be gold-only or, for vetted partners with a signed data agreement, curated-level. This is the preferred path for any consumer that already lives in GCP; Client Delivery (component 11) remains the fallback for those who don't.
one listing per vendor with its own entitlement and audit trail; default gold-only, curated by exception with sign-off.
14Phase 3: streaming ingestion
When internet browsing data moves in fully: Pub/Sub (events) or Datastream (CDC) feed the same Dataflow/Dataproc code in streaming mode, writing via the BQ Storage Write API. Nightly batch becomes just another window size; DLP de-identification and all delivery paths are unchanged.
topic/schema registry, streaming Flex Template deployment, backfill of history via STS, dual-run validation before on-prem ETL retirement.
