A data platform can fail long before the dashboard.
If the wrong ingestion pattern is chosen, teams deal with stale data, duplicates, missed deletes, difficult backfills and pipelines nobody wants to own.
Google Cloud now presents Data Transfer Service, Datastream and Pub/Sub through guided BigQuery Pipelines & Connections workflows. That makes setup easier, but architecture still depends on the behavior of the source.
Actiknow builds integrations, database solutions and BI pipelines. The most maintainable ingestion designs begin by matching the mechanism to the source rather than standardizing every feed on one technology.
Three Different Problems
Data Transfer Service, Datastream and Pub/Sub all move data toward analytics, but they solve different ingestion problems. DTS is primarily managed batch transfer from supported sources. Datastream is designed for change data capture from operational databases. Pub/Sub is an event messaging service for streams produced by applications and systems.
Start With Source Behavior
Ask how the source changes. A SaaS advertising platform may expose daily API extracts. A transactional database changes rows continuously. An application emits events. Those are different source behaviors and should not be forced into one ingestion pattern.
Data Transfer Service
BigQuery Data Transfer Service is attractive when Google provides a managed connector for the source and the business can tolerate scheduled transfer behavior. It reduces custom connector code and operational ownership. Check connector-specific frequency, schema and historical-backfill behavior.
Datastream
Datastream is a CDC-oriented service for capturing database changes. It is appropriate when the warehouse needs inserts, updates and deletes from supported operational databases with lower latency than periodic full extracts.
Pub/Sub
Pub/Sub is appropriate when producers emit events such as clicks, telemetry, application actions or domain events. It decouples producers from consumers and supports streaming architectures. It is not a generic replacement for database CDC.
BigQuery Pipelines and Connections
Google’s 2026 BigQuery Pipelines & Connections experience provides guided workflows for Data Transfer Service, Datastream and Pub/Sub. A unified console experience does not make the underlying ingestion semantics identical.
Batch vs CDC vs Event
Use a simple first classification. Batch SaaS or file-like source: consider DTS. Operational database changes: consider Datastream. Application event stream: consider Pub/Sub. Then evaluate connector support, latency, volume, transformations and recovery.
Latency Requirement
Define the actual freshness SLA. Daily marketing data does not need a streaming architecture. Fraud monitoring may. Lower latency increases operational complexity and cost. Do not pay for real time when the business acts once per day.
Deletes
Deletes are often missed in ingestion design. Determine how each source communicates deletion and how the target should represent it. CDC can capture delete events; batch APIs may expose tombstones or require reconciliation.
Schema Evolution
Sources add columns, change types and introduce new event versions. Define how the pipeline reacts. Should new fields appear automatically, be quarantined or require approval? Schema drift should be monitored.
Initial Load
CDC pipelines usually need an initial snapshot before ongoing changes. Plan the transition carefully so records are neither missed nor duplicated. Document the cutover point.
Ordering
Event systems can introduce ordering considerations. If business logic depends on sequence, define keys and timestamps and handle late events. Do not assume arrival order equals business order.
Exactly Once vs Idempotency
Rather than relying on vague promises of exactly-once behavior, design downstream processing to tolerate retries and duplicates where practical. Use stable event or source keys and deterministic merge logic.
Recovery
Ask how the pipeline recovers after an outage. Can it replay? How far back? Does the source retain changes? Can a batch transfer be rerun safely? Recovery design is part of ingestion architecture.
Backfills
Historical backfills can overload production pipelines or duplicate data. Separate backfill procedures from steady-state ingestion and use controlled date or key ranges.
Transformation Boundary
Keep ingestion responsible for reliable movement and light normalization. Put reusable business transformations into a governed transformation layer. Overloading connectors with business logic makes pipelines difficult to test and replace.
Raw Layer
Land source-aligned data before heavy business transformation when practical. Preserve source identifiers and ingestion metadata. This gives teams a place to reconcile and replay.
Data Quality
Monitor row counts, freshness, duplicates, nulls, key integrity and source-to-target reconciliation. A green connector status does not prove the data is analytically correct.
Observability
Track last successful load, lag, throughput, errors, retries, schema changes and cost. Alert according to business SLA. A pipeline that is two hours late may be critical for operations and irrelevant for monthly reporting.
Ownership
Assign a technical owner and business data owner. The technical owner maintains ingestion. The business owner confirms whether source data and freshness meet business expectations.
Security
Use least-privilege service identities. Protect credentials. Restrict raw datasets. Classify sensitive fields. Data ingestion should not create a broader copy of information than downstream users need.
Cost
Compare service charges, BigQuery compute, storage, network transfer, engineering effort and support. Managed services may cost more per unit than custom code while costing far less to operate.
Multi-Region and Residency
Confirm source and destination regions, network implications and residency requirements. A technically supported connector may still conflict with data-location policy.
Choosing DTS
Prefer DTS when a supported managed transfer matches the source, scheduled freshness is acceptable and minimizing custom operations is valuable.
Choosing Datastream
Prefer Datastream when the source is a supported operational database and change capture, including updates and deletes, is the core requirement.
Choosing Pub/Sub
Prefer Pub/Sub when applications or platforms naturally publish events and consumers need a scalable streaming backbone.
Using More Than One
Enterprises commonly need all three. Marketing SaaS data may arrive through DTS, operational database changes through Datastream and product events through Pub/Sub. Standardize downstream conventions even when ingestion mechanisms differ.
Frequently Asked Questions
Is Data Transfer Service real time?
It is generally oriented toward managed transfers and scheduled ingestion; check the connector’s supported frequency.
Is Datastream the same as Pub/Sub?
No. Datastream focuses on database change capture, while Pub/Sub is a general event messaging service.
Can Pub/Sub ingest database changes?
It can carry events produced by another CDC component, but it does not by itself discover database row changes.
Should every source be streamed?
No. Match freshness to business action.
What should every pipeline record?
Source identifiers, ingestion time, processing status and enough metadata to reconcile and replay.
Conclusion
The right BigQuery ingestion service follows the source and SLA. Use managed batch transfer for supported scheduled sources, CDC for operational database changes and event messaging for application streams. Then standardize raw landing, observability, quality, security and recovery across all of them.
Actiknow’s Custom Solutions practice includes integrations, databases and automation, while its Business Intelligence work covers the analytical layer that consumes the data. If you are designing a BigQuery ingestion architecture, explore Actiknow’s Custom Solutions or discuss your data engineering requirements with Actiknow.
Architecture Review Detail
For each source, document source owner, ingestion mechanism, expected volume, freshness SLA, primary key, delete behavior, schema-change policy, initial-load method, replay window, raw destination, transformation owner and reconciliation test. Run a failure exercise before production: pause the source or connector, restore it, and prove the pipeline catches up without duplicates or missing records. This operational test often reveals more than a successful first load.