The decision you make before writing any code

Most teams discover their serverless architecture is wrong about six months after they ship it. The Cloud Functions are chaining each other in long sequences. Pub/Sub messages are getting dropped on transient errors. BigQuery costs are climbing every month with no obvious culprit. The system technically works, but it's fragile in ways that weren't obvious when the first function got deployed.

The problem usually isn't the code. It's that someone chose a topology without a framework for making that choice — and GCP's surface area is large enough that the wrong tool is always just one service name away from the right one. Cloud Functions when you needed Cloud Run Jobs. Push subscriptions when you needed pull. Streaming inserts when you needed the Storage Write API.

This lesson is about the pattern layer that sits above the individual service documentation: which architectural shapes work on GCP, why they work, and what breaks when you pick the wrong one. We'll work through five interlocking patterns — from system topology down to BigQuery cost controls — in the order you'd actually encounter them when building a production pipeline.

Pick your spine before you pick your services

The first architectural decision isn't which GCP service to use. It's whether your system is real-time, batch, or hybrid — because that choice determines everything downstream.

Real-time means an event arrives and a result is available within milliseconds to seconds. Batch means work accumulates and gets processed on a schedule or trigger, with results available minutes to hours later. These two spines require fundamentally different infrastructure, and they have fundamentally different cost profiles.

Here's the thing most teams miss: the vast majority of requirements labeled "we need real-time" actually tolerate near-real-time latency of one to thirty seconds. Streaming infrastructure — Pub/Sub consumers, always-warm functions, exactly-once delivery machinery — is significantly more expensive and complex than batch. Before you commit to a real-time spine, force yourself to state the actual SLO in milliseconds. If the honest answer is "thirty seconds is fine," you've just saved yourself a lot of operational complexity.

The production default for most event-driven GCP systems is hybrid: stream events into Pub/Sub for durability, then micro-batch them into BigQuery every thirty seconds using the Storage Write API. You get the latency characteristics of near-real-time with the throughput and cost advantages of batch writes.

The next question after real-time versus batch is coordination style: choreography or orchestration. Choreography means services react to events with no central coordinator — an order event triggers inventory, billing, and analytics independently, each scaling and retrying on its own. Orchestration means a central process, Cloud Workflows in GCP, directs each step explicitly and tracks overall state.

Choreography is simpler for fan-out to independent consumers. Orchestration becomes necessary the moment you have conditional branching, steps that depend on previous outputs, or rollback requirements. The rule of thumb that holds in practice: three or more sequential dependent steps means orchestration wins. Below that threshold, you're probably adding Cloud Workflows overhead to a problem that choreography handles cleanly.

The last topology choice at this level is whether to separate your write and read paths — CQRS. The write path runs API Gateway, then a Cloud Function, then Pub/Sub, then BigQuery as an append-only event log. The read path runs API Gateway, then a Cloud Function, then BigQuery materialized views. This isn't academic architecture astronautics: BigQuery is optimized for analytical reads, not transactional point lookups. Mixing write and read throughput against the same table is how you end up with slot contention that's nearly impossible to diagnose. Separating them lets each path scale independently.

Pub/Sub is not a queue — it's a durable event bus

Most developers model Pub/Sub as a message queue: something goes in, something comes out, you're done. That framing misses what makes it valuable. Pub/Sub is a durable event bus with a seven-day retention window, fan-out to N independent subscribers, and configurable delivery semantics. Understanding those three properties is what separates clean pipeline design from fragile chaining.

Fan-out is the key property for event-driven architectures. When an order-placed event publishes to a topic, Pub/Sub delivers a copy to every subscription — an inventory subscription running as a Cloud Function, a billing subscription running as a Cloud Run service, and an analytics subscription reading directly into BigQuery all receive the same message independently. Each subscriber has its own retry state, its own dead-letter topic, its own scale. The producer knows nothing about the consumers. That decoupling is why fan-out via Pub/Sub is far cleaner than having one function call another directly.

The delivery model choice — push or pull — matters more than it looks. Push subscriptions have Pub/Sub call your HTTP endpoint for each message. This is the simple path and the right default for most Cloud Functions workloads. But push has a ceiling: at high throughput, the overhead of one HTTP call per message becomes the bottleneck. Above roughly five hundred messages per second, switch to pull. A Cloud Run service with a consumer loop that pulls a hundred messages at a time, batch-writes them to BigQuery, and acknowledges all hundred in a single call achieves five to fifty times higher throughput than push for the same CPU allocation.

Pub/Sub's default delivery guarantee is at-least-once: your handler will see duplicate messages. There are two ways to handle this. The first is idempotent handler design — use Pub/Sub's message ID as a deduplication key, store it in Firestore with a twenty-four hour time-to-live, and check on receipt before processing. The latency overhead is two to five milliseconds against a Firestore instance in the same region. The second is enabling exactly-once delivery at the subscription level, which gives you deduplication within a ten-minute window at the cost of ten to a hundred milliseconds of additional latency per message. For most pipelines, idempotent handlers are cheaper and simpler; reserve exactly-once delivery for cases where idempotency is genuinely hard to implement — third-party API calls with no natural deduplication key, for instance.

Ordering is the last Pub/Sub property worth understanding before you need it. By default, message ordering is not guaranteed. You can enable ordering per subscription using ordering keys — messages with the same key are delivered in publish order to a single consumer. The cost is that a single consumer per ordering key serializes processing for that key, reducing throughput. For high-cardinality keys like user ID, the impact is minimal. Avoid ordering for high-throughput pipelines unless it's a hard requirement, because the throughput penalty is real and the debugging complexity when ordering breaks is significant.

Cloud Functions 2nd gen changed the execution model

When Google switched Cloud Functions second generation to run on Cloud Run infrastructure, they changed something fundamental: a single function instance can now handle multiple simultaneous requests. This sounds like an implementation detail. It isn't.

First-generation Cloud Functions handled exactly one request per instance. A burst of a hundred simultaneous requests required a hundred instances, which meant a hundred cold starts if those instances weren't already warm. Second-generation functions support a concurrency setting up to a thousand — so a single warm instance can absorb eighty or a hundred concurrent I/O-bound requests without spinning up new instances. That alone reduces cold start frequency dramatically for functions doing BigQuery queries or outbound API calls.

Cold starts are still real. A new instance takes one to three seconds to initialize for Node.js or Python runtimes, and five to ten seconds for the JVM. For user-facing APIs where cold start latency is unacceptable, set min-instances to one to keep at least one instance warm at all times. For critical low-latency APIs under bursty load, min-instances at three combined with concurrency at eighty handles surge traffic without a cold-start spike. Always enable the CPU boost flag on second-gen functions — it speeds up initialization at no extra cost.

The right concurrency setting depends on what the function does. For I/O-bound functions — BigQuery queries, outbound HTTP calls, Firestore reads — set concurrency to eighty and use a third to one vCPU. For CPU-bound work — image processing, ML inference on CPU — keep concurrency at one and provision the CPU to match. Mixing a high concurrency setting with CPU-bound work causes CPU throttling that shows up as inexplicable latency spikes.

Secret management deserves explicit attention here because the wrong pattern is easy to stumble into. Storing secrets in environment variables set at deploy time makes them visible in Cloud Console and potentially logged. Use Secret Manager instead: mount secrets at runtime so the value is injected by the runtime, not stored in function configuration. Grant each function's service account access to only the specific secrets it needs — not secret accessor at the project level. Use one service account per function for least-privilege isolation. This pattern costs almost nothing operationally and eliminates an entire category of credential leak.

For workloads that exceed Cloud Functions' nine-minute timeout, Cloud Run Jobs are the correct tool, not creative workarounds. Cloud Run Jobs run containerized tasks to completion with configurable parallelism — a hundred task instances, ten at a time, each receiving a task-index environment variable you use to shard input data. Failed tasks retry independently without re-running the full job. The moment your batch exceeds nine minutes or needs more than thirty-two gigabytes of RAM, stop trying to fit it into Cloud Functions and reach for Jobs.

API Gateway is a contract, not just a router

The instinct when deploying Cloud Functions is to call them directly via their generated HTTPS URLs. That works, but it couples your API surface directly to your backend implementation: every URL contains the region, project, and function name. Changing a function's name or region breaks all your clients.

API Gateway solves this by introducing a managed contract layer in front of your functions. You define your API surface in an OpenAPI 2.0 spec, map each path and method to a backend address pointing at a Cloud Function or Cloud Run service, and your clients talk to a stable gateway URL. Multiple functions live behind one domain. The backend can change without touching client configuration.

The spec-as-infrastructure-as-code pattern is how this works cleanly in production: store the OpenAPI spec in version control alongside the function code. A git push triggers Cloud Build, which deploys the functions and updates the API Gateway config in the same pipeline. Rollback means deploying a previous git tag. The whole API surface is auditable through git history.

Rate limiting in API Gateway uses Google Service Control quotas defined in the spec. This is per-API-key quota enforcement — useful for throttling specific partners or tenants — but it isn't real-time burst limiting. For strict per-IP burst limiting, add Cloud Armor policies in front of the gateway's backend URL. The two layers are complementary: API Gateway handles per-key quota, Cloud Armor handles volumetric attacks and IP-level rate limiting.

Authentication in the spec takes one of two forms. API keys are opaque tokens tied to GCP projects, revocable, and logged in Cloud Logging — the right choice for trusted server-to-server calls. For end-user authentication, API Gateway validates JWT signatures using Firebase's public keys before the request reaches your function. No auth code in the function itself: the gateway handles validation and passes the verified JWT through in a forwarded authorization header.

The Backend-for-Frontend pattern is worth naming explicitly because it solves a real problem that appears at scale. Mobile clients need different response shapes than web clients — smaller payloads, different field selections, different aggregations. A single unified API either over-fetches for mobile or requires clients to make multiple calls. The BFF pattern deploys separate API Gateway configurations per client type: one gateway routes to mobile-optimized functions with aggressive payload trimming, another routes to full-detail functions for web. Each gateway has its own API key namespace and quota limits. The overhead is maintaining multiple specs, but it's worth it when client requirements have genuinely diverged.

BigQuery pricing shapes every pattern

BigQuery on-demand charges six dollars and twenty-five cents per terabyte scanned. That number should be in your head whenever you design a BigQuery schema, because every architectural pattern in this section exists to reduce bytes scanned.

The ingest path starts with a choice between legacy streaming inserts and the Storage Write API. Legacy streaming inserts are an HTTP endpoint where rows become queryable within seconds — but they deliver at-least-once semantics, charge per row, and are the older, more expensive option. The Storage Write API is gRPC-based, offers three modes, and is the correct default for any new pipeline.

Committed mode makes rows visible immediately after write with exactly-once semantics within a stream — use this when you're ingesting from Cloud Functions and need freshness. Buffered mode holds rows until you explicitly flush them — the right fit for micro-batching. Pending mode keeps rows invisible until you commit the entire stream — this is how you get all-or-nothing atomicity for batch loads. For the canonical Pub/Sub to Cloud Function to BigQuery pipeline, committed mode with the gRPC client library is the right choice.

Partitioning is the first level of cost control. A partitioned table limits the bytes scanned per query by organizing data into discrete partitions that queries can skip entirely. Ingestion-time partitioning assigns rows automatically to daily or hourly partitions based on insert time — no schema change required. Column partitioning gives you explicit control when you have a business date column like event date that queries always filter on. The critical enforcement mechanism is requiring a partition filter on the table: without it, someone will run an unfiltered query and scan the entire table. With it, queries that accidentally skip the partition column fail at execution time rather than silently scanning everything.

Clustering is the second level. After partitioning narrows the scan to relevant partitions, clustering narrows it further by colocating similar values within each partition. A query filtering on a specific user ID reads only the columnar blocks containing that user's data, not every block in the partition. Add clustering on your high-cardinality filter columns — user ID, session ID, event type — and on columns frequently used in group-by clauses. Clustering is free and degrades gracefully as new data arrives out of order; BigQuery auto-reclusters in the background.

Materialized views are where the cost math becomes dramatic. A dashboard query that aggregates one terabyte of events costs roughly six dollars every time it runs. A materialized view that pre-aggregates those events into a one-megabyte result table costs fractions of a cent for the same read. BigQuery automatically rewrites queries to use materialized views when possible, so existing dashboard queries start hitting the materialized view without any client changes. For partitioned base tables, incremental refresh updates only the partitions that changed — the maintenance cost is proportional to new data volume, not total table size.

For interactive dashboards that need sub-hundred-millisecond query response, BI Engine is the last layer. BI Engine is an in-memory analysis service: reserve capacity in gigabytes of RAM against your project, and queries against cached data return in ten to a hundred milliseconds instead of one to thirty seconds. The trade is cost — approximately five cents per gigabyte-hour of reserved capacity. Size your reservation to the datasets actively queried by dashboards, not your entire BigQuery footprint. Queries that can't be served from BI Engine fall back to standard BigQuery execution transparently, so the failure mode is slower queries rather than errors.

Observability is what makes it all hold together

The five patterns in this lesson — topology selection, Pub/Sub pipeline design, Cloud Functions configuration, API Gateway contracts, and BigQuery cost controls — give you a system that works. Observability is what gives you a system you can actually operate.

Structured JSON logging is non-negotiable. Cloud Logging parses severity, message, and arbitrary fields automatically when you emit valid JSON. Log-based alerts trigger when error-severity messages exceed a threshold. Log sinks export to BigQuery for long-term analysis. Unstructured string logs are useful for reading one line at a time; they're useless for querying at scale.

Beyond infrastructure metrics, emit business metrics as custom Cloud Monitoring time series: events processed per function invocation, BigQuery write latency at the ninety-fifth percentile, the age of the oldest unacknowledged message on your Pub/Sub subscriptions — your pipeline lag indicator — and the undelivered message count on your dead-letter topics. That last one is the canary: when it rises above zero, something is broken upstream and messages are accumulating.

Cost control is observability applied to billing. Set budget alerts at fifty, ninety, and a hundred percent of expected monthly spend per service. Wire the alert Pub/Sub topic to a Cloud Function that can auto-disable non-critical workloads on budget breach — because GCP does not cap spending automatically, and a runaway query or a function in an infinite retry loop can exceed any budget before a human notices. Apply resource labels consistently — team, environment, feature, cost center — to every Cloud Function, BigQuery dataset, Pub/Sub topic, and Cloud Scheduler job. Labels flow into Cloud Billing export to BigQuery, which is how you answer "which feature is this cost center actually paying for."

The unsexy truth: most serverless cost and reliability problems aren't architecture problems — they're instrumentation problems. A system you can't observe is a system you can't debug, and a system you can't debug will eventually surprise you in production in the worst possible way.

The patterns in this lesson cover synchronous APIs and event-driven ingest on GCP. In the next lesson, we'll look at AI Platform and Vertex AI — how to serve models at scale using the same serverless infrastructure, and where the managed AI layer fits into the pipelines you've just built.