Real-time data pipeline on GCP
A streaming data pipeline on Google Cloud: Pub/Sub ingestion, Dataflow streaming transforms, BigQuery as the warehouse, and Looker for dashboards. Backfills come in via Cloud Storage.
Architecture previewReal-time data pipeline on GCP
Recreate this in Raltey
Copy this prompt
Paste this into Raltey's AI agent to generate the same diagram, then tweak it on the canvas.
Prompt
Create a GCP real-time data pipeline architecture. Web and mobile clients publish events to a Pub/Sub ingest topic. Dataflow streaming consumes the topic and writes into BigQuery raw. dbt transforms raw into BigQuery marts. Looker reads marts and serves analysts. Add a separate path: GCS backfill files processed by Dataflow batch into BigQuery raw.
Context
- - Streaming-first; batch backfills are second-class.
- - BigQuery is both the analytics layer and the long-term store.
- - Looker reads BigQuery views, never raw tables.
Patterns
- - Single ingestion path through Pub/Sub.
- - Raw vs. marts split in BigQuery (transform with dbt).
- - Looker only reads marts.
Diagram structure
Nodes and connections
Web appactor
Mobile appactor
Pub/Sub (ingest topic)queue
Dataflow (streaming)service
BigQuery rawdb
BigQuery martsdb
GCS (backfills)storage
Dataflow (batch)service
Lookerservice
Analystactor
Connections
Web appPub/Sub (ingest topic)
Mobile appPub/Sub (ingest topic)
Pub/Sub (ingest topic)Dataflow (streaming)
Dataflow (streaming)BigQuery raw
BigQuery rawBigQuery martsdbt
BigQuery martsLooker
LookerAnalyst
GCS (backfills)Dataflow (batch)
Dataflow (batch)BigQuery raw
Notes from the field
- Use deduplication keys on Pub/Sub to avoid double-counts.
- Set BigQuery table expiration on raw to keep costs predictable.
- For PII, materialize a pseudonymized mart and restrict access.
Frequently asked
When should I add Bigtable to this pipeline?
When you need single-digit-millisecond reads or unbounded scale (e.g., feature stores). BigQuery alone handles most analytics.
Where do alerts fit?
Either inside Dataflow (windowed thresholds) or in a downstream service that subscribes to a secondary Pub/Sub topic populated by Dataflow.
More templates
FlowchartSaaS onboarding email flowFlowchartCustomer support runbook flowFlowchartPayment webhook retry and error flowFlowchartGitHub PR review and merge flowFlowchartLogin with magic link flowArchitectureThree-tier SaaS architecture on AWSArchitectureMicroservices architecture on KubernetesArchitectureNext.js + Vercel + Postgres SaaS stack
Open this template in Raltey
Edit it on the canvas, ask the AI agent to extend it, then share or export.
Start for free