Actiknow
Data Engineering

Snowflake Openflow Gen 2: What Changes for CDC Pipeline Operations and Governance?

Explore Snowflake Openflow Gen 2 for PostgreSQL and MySQL CDC, including SQL-managed runtimes, RBAC, configuration versions, monitoring and migration decisions.

Enterprise database infrastructure supporting snowflake openflow gen 2 cdc pipelines
Contents hide

The executive question is not whether Openflow can move data

A change data capture (CDC) pipeline is often judged by a simple question: are records arriving in the warehouse? That is necessary, but it is a weak definition of production success. The harder questions emerge when a database schema changes, a replication slot falls behind, credentials need rotating, a deployment must be recreated, or an auditor asks who can modify a connector.

Snowflake Openflow Gen 2 changes how teams administer those pipelines. It brings deployments, runtimes and supported connectors into Snowflake’s object model, with SQL lifecycle commands, more explicit role-based access and versioned connector configuration. This is a meaningful operational shift, not a promise that CDC itself has become automatic or risk-free.

For a data leader, the opportunity is to manage ingestion more like other governed infrastructure: identifiable owners, controlled releases, documented access and observable service levels. The implementation decision still depends on the source systems, expected change volume, recovery requirements and the maturity of the team’s operations.

What is available as of November 2026?

Snowflake introduced second-generation Openflow in public preview on August 31, 2026. Deployment and runtime SQL objects subsequently became generally available on AWS and Azure, with Google Cloud support announced on September 11. The important distinction is that Gen 2 connector configuration, including the setup wizard and SQL/stage configuration, remains in public preview. Do not describe the entire Gen 2 connector stack as generally available.

The currently documented Gen 2 catalog connectors are PostgreSQL CDC and MySQL/MariaDB CDC. Other sources may be supported through Gen 1 connectors, including on Gen 2 runtimes, but that does not make their configuration a Gen 2 connector feature. Snowflake’s Openflow Gen 1 and Gen 2 comparison explains the coexistence and permission differences.

New Openflow deployments use Gen 2. Existing Gen 1 deployments and their connectors do not disappear, and teams should not treat a migration as a prerequisite for keeping a functioning pipeline operational. Check current account entitlements, region, deployment model, connector catalog and release notes before committing a production architecture.

The broader Snowflake release history matters because a status such as “GA” applies to a specific capability and date, not every associated feature. Record which objects and connectors the project actually depends on and which of them remain preview capabilities.

What changes from Gen 1 to Gen 2?

1. Deployments and runtimes become manageable Snowflake objects

Gen 1 administration leaned heavily on the Openflow UI and runtime integrations. Gen 2 exposes deployments and schema-scoped runtimes as SQL objects. Teams can use supported CREATE, ALTER, SHOW, DESCRIBE and DROP operations, alongside the UI, for lifecycle management.

That matters when environments need to be reproducible. Instead of relying solely on a sequence of manual clicks, a platform team can define deployment conventions, validate required grants and track changes in a repository. Infrastructure as code becomes more practical, but teams must still handle networking, runtime sizing, source permissions and secrets.

A Gen 2 runtime can host Gen 1 and Gen 2 connectors. This gives organizations a transition path when their preferred source connector is not yet offered as a Gen 2 catalog entry. The connector generation still determines how it is configured and operated.

2. Access control becomes more granular

Gen 2 uses Snowflake role-based access control (RBAC) for deployment, runtime and connector objects. This can separate people who inspect pipeline health from those authorized to operate or change the infrastructure. A deployment privilege does not automatically give visibility into every runtime; access to the runtime and its containing database and schema must be considered separately.

The execution identity is another distinct control. The runtime’s EXECUTE_AS_ROLE determines access used while connector workloads run. A person who can start a connector should not automatically have unrestricted access to every destination table, secret or external endpoint.

Do not copy a generic GRANT script into production without reviewing the current privilege model. Snowflake’s documentation also warns that transferring OWNERSHIP of Gen 2 Openflow objects can disrupt underlying functionality. Treat ownership transfers as a tested administrative operation, not a routine cleanup step.

3. Connector configuration becomes versioned

Supported Gen 2 connectors store configuration in versioned file-based entities rather than relying only on mutable canvas settings. Teams can prepare a live draft, validate it, commit a default version, and retain the ability to revert configuration changes. A draft connector cannot start until a configuration is committed.

Version controlled connector configuration documents prepared for a controlled release

The practical benefit is traceability: a change to source host, table selection or runtime settings can be reviewed as a controlled release. The limitation is equally important: reverting configuration does not automatically reverse data already written, restore lost source logs or undo an incorrect downstream transformation.

A Git repository should capture approved configuration templates and release decisions. Secrets should not be embedded in those files. Snowflake supports secret references in connector configuration, and access to referenced secrets must be granted to the runtime execution role.

4. The management interface changes, not the fundamental CDC risks

Gen 2 connector setup can use a guided wizard or SQL and stage commands. For Gen 2 connectors, the runtime canvas is read-only for configuration; changes belong in the connector management interfaces. Teams accustomed to editing NiFi processors directly need to adjust their runbooks.

This is a control-plane improvement. It does not eliminate source database load, log-retention requirements, schema evolution, network outages, partial failures or the need to reconcile the target with the source. Those remain core engineering responsibilities.

A reference architecture for governed CDC

Consider a transactional PostgreSQL database that supports customer orders and payments. The objective is to feed analytical tables in Snowflake without repeatedly querying the live application for reports.

A sensible architecture has six logical layers: the source database and its change log; source-specific CDC prerequisites; network and authentication controls; an Openflow deployment, runtime and connector; landing and change-history tables in Snowflake; and downstream transformation, quality and consumption models.

Database infrastructure representing the source systems and change data capture architecture

The source team owns replication permissions, publication or binlog settings, source capacity and change-log retention. The platform team owns deployment configuration, networking, secrets, runtime sizing and monitoring. The data engineering team owns target schemas, reconciliation, transformations and consumer contracts. The business owner defines acceptable freshness and the consequences of missing or duplicated changes.

Keep raw replicated data separate from curated business metrics. A pipeline can be technically healthy while a reporting model is wrong because it interprets deletes incorrectly, joins incompatible versions of a record, or counts a replayed event twice.

For teams building reporting from operational databases, Actiknow’s business intelligence consulting and data integration services address the broader design problem: turning source data into governed models and usable reports. The ingestion mechanism is one component of that architecture.

PostgreSQL and MySQL CDC: operational differences that matter

Source setup is not a checkbox

PostgreSQL CDC commonly relies on logical replication and replication slots. Operators must account for publication configuration, replica identity, permissions, snapshot behavior and the accumulation of write-ahead logs when consumers lag. A connector that stops reading changes can become a storage or availability problem for the source database.

MySQL and MariaDB CDC use database change-log mechanisms with their own prerequisites, retention behavior and permissions. A successful initial snapshot is not proof that ongoing change capture will survive a restart, failover or extended outage. Validate the supported database versions and configuration for the exact connector being deployed.

For either source, document the change-log retention window and the maximum tolerable outage. If an interruption lasts longer than the source retains necessary change history, a restart may require a new snapshot or other recovery procedure. The relevant test is not simply whether the connector reconnects, but whether all required changes are recovered without silent gaps.

Initial snapshots and incremental changes need separate acceptance criteria

Initial load is a bulk data movement problem. Incremental CDC is a continuous ordering and correctness problem. Test both. During the initial snapshot, confirm table coverage, row counts, representative values, primary-key behavior and source impact. Then generate inserts, updates and deletes while the pipeline runs and while it is deliberately interrupted.

For transactional applications, verify that downstream tables represent the intended business state. A sequence of updates can arrive in ways that are operationally correct but surprising to a report that assumes every row is a unique order. Decide whether consumers need a latest-state table, a change journal or both.

A representative test matrix should include: a table with a primary key, a table with unusual identity settings, nullable fields, large rows, deletes, a new column, a source restart, a connector restart, and a backlog that exceeds normal peak traffic. Add transaction-boundary tests for workflows that update multiple related tables.

Schema evolution requires explicit ownership

When a source team adds, renames or changes a column, the ingestion pipeline and the analytical model can react differently. Some changes may flow through automatically, some may require intervention, and others can break downstream SQL or semantic definitions. Do not assume that successful capture implies compatibility.

Define a schema-change agreement: who announces the change, how much notice is required, which changes are backward-compatible, how the target is validated, and who approves a breaking change. Versioned connector configuration is helpful, but it is not a substitute for a source-to-report contract.

A practical SQL-managed connector workflow

Snowflake documents a SQL route for supported Gen 2 connectors. The example below illustrates the lifecycle, not a complete production script. Object names, grants, secrets, network rules and connector-specific config.json fields must be established first.

Example:

USE DATABASE INGESTION_ADMIN;

USE SCHEMA OPENFLOW;

SHOW OPENFLOW CONNECTOR DEFINITIONS;

CREATE OPENFLOW CONNECTOR ORDERS_CDC

  IN RUNTIME ORDERS_RUNTIME

  FROM DEFINITION OPENFLOW_POSTGRES_CDC

  DISPLAY_NAME = ‘Orders PostgreSQL CDC’;

The newly created connector starts in a stopped draft state. The team then obtains its live configuration from the connector’s versioned stage, edits the approved properties, uploads the revised config.json using a client that supports stage GET and PUT, validates the configuration and commits it.

After a successful commit, the documented lifecycle includes:

ALTER OPENFLOW CONNECTOR ORDERS_CDC COMMIT;

ALTER OPENFLOW CONNECTOR ORDERS_CDC START;

For a later change, create a live version from the last committed configuration, review the differences, then COMMIT or ABORT. These commands are intentionally separated from the full configuration workflow: the file upload, secret references, privileges and validation are required steps, not optional details.

The setup wizard is often the safer first implementation path because it helps validate configuration. Teams can then use the resulting configuration as a reference for repeatable deployments. Snowflake’s Configure a Gen 2 connector with SQL documentation describes the exact command and stage workflow; confirm it against the release supported in your account.

RBAC: define four operational responsibilities

An effective permissions model starts with duties rather than a list of available grants.

The platform administrator provisions deployments, networking, runtime capacity and the execution role. The connector operator can start, stop and inspect authorized connectors, but should not automatically administer every database. The data engineer owns target tables, transformations and reconciliation checks. The auditor or read-only operator can inspect configuration history, access assignments and pipeline health without changing production state.

Secure access control for role based permissions in enterprise data infrastructure

In a small team, individuals may hold more than one role. The important control is that powerful access is deliberate, reviewable and preferably time-bounded. Record who approves connector changes, who can alter source credentials and who can terminate a runtime.

Grant the minimum privileges required on the target database and warehouse. Restrict access to secrets and external access integrations. For Snowflake-managed deployments, source connectivity may require a network rule and external access integration; for bring-your-own-cloud deployments, the networking model differs. Validate the relevant path rather than assuming the same checklist applies to both.

A quarterly review should answer three questions: which identities can modify a production connector, which roles can access its credentials and destination data, and which permissions are no longer needed?

Observability: measure business freshness, not only process status

A connector that reports “running” can still deliver stale, incomplete or unusable data. Monitoring must span the source, the ingestion runtime, the destination and the business model.

Track connector state, error counts, queue pressure, processing throughput and restart events. Also track source log retention headroom, time since the latest captured change, time since the latest applied change, table-level row or key discrepancies, and the age of the latest business-relevant event. A quiet source table should not be falsely classified as failed solely because it has no new rows.

Precision monitoring instruments representing cdc pipeline freshness and operational measurements

Snowflake provides an Openflow Connectors Dashboard for connector health and ingestion status. Pair it with SQL-based checks against destination tables and, where possible, source-side replication metrics. The dashboard and the data-quality checks answer different questions.

Define a freshness service level by use case. An hourly finance dashboard may tolerate more delay than an operational exception queue. Express it as a business commitment, such as “95% of eligible source changes appear in the curated model within the agreed interval,” and verify that the measurement captures both source and destination timestamps.

Alerting needs an owner and a response path. A lag alert without a runbook simply transfers the uncertainty to whoever is on call.

Recovery and change management: what a production runbook must include

Every CDC pipeline needs a recovery decision tree. Start with the failure class: source unavailable, credentials rejected, network blocked, runtime unhealthy, connector misconfigured, destination write failing, schema incompatible, or change history missing.

For a short outage, operators may restore connectivity and let the connector drain its backlog. For a prolonged outage, first check whether the required source logs are still available. If they are not, plan a controlled resnapshot or other reconciliation strategy rather than assuming replay is complete.

A connector configuration rollback and a data recovery are separate actions. Restoring a prior config.json can undo an erroneous setting, but it does not remove incorrect records already published to Snowflake. Data correction may require replay, table rebuilds, quarantining bad batches or adjusting downstream models.

Before deploying a change, capture the current configuration version, current ingestion checkpoint where applicable, source and target counts, owner approvals, maintenance window and rollback conditions. After deployment, test a representative business query and compare target values with the source.

Be especially careful with destructive lifecycle commands. Gen 2 runtime termination can drain in-flight data; forced termination can purge queues when data loss is accepted. Do not treat FORCE as an ordinary restart mechanism.

Planned data center maintenance representing cdc pipeline recovery and controlled change management

Migration: when to move a Gen 1 environment

A Gen 1 pipeline that meets freshness, security and recovery requirements does not need an emergency rewrite. Migration is justified when SQL-managed operations, granular RBAC, reproducible environments or connector versioning solve a real operational problem.

Start with an inventory: deployments, runtimes, connector generations, source versions, secrets, network dependencies, service accounts, destination tables, downstream jobs and monitoring. Identify which components can migrate and which must remain on Gen 1 connector configuration.

Gen 2 runtimes can support Gen 1 connectors, so deployment/runtime modernization and connector modernization can be staged separately. Follow Snowflake’s supported migration procedure rather than attempting to recreate every object from memory. Verify privileges after migration because visibility and object scopes differ.

Use a parallel validation window when feasible. Compare freshness, record counts, deletes, schema handling and source resource consumption before redirecting downstream consumers. A cutover should have an explicit rollback owner and an agreed point after which reconciliation is required.

Cost and operating model: what changes financially?

Gen 2 does not make ingestion free. Total cost depends on deployment model, runtime capacity and uptime, source activity, data volume, Snowflake ingestion and processing, storage, monitoring and the labor needed to operate the pipeline.

Model at least three workloads: normal weekday activity, peak transactional periods and recovery after a multi-hour interruption. The third case often exposes sizing assumptions that look reasonable during steady state. Compare the cost of maintaining headroom with the cost and business impact of delayed recovery.

Account for engineering effort as well as infrastructure. SQL-managed objects and configuration-as-code can reduce undocumented manual changes, but the organization still needs release pipelines, testing, access reviews and incident ownership. These are investments in control, not guaranteed savings.

If the business primarily needs a nightly snapshot, a simpler batch integration may be more economical and easier to support than continuous CDC. Choose CDC because the freshness requirement justifies its operational complexity, not because it is a newer connector option.

A 30-day evaluation plan

Week 1: define the business contract

Select one source and two or three representative tables. Document consumers, freshness objectives, delete semantics, acceptable outage, source change-log retention and data ownership. Inventory existing pipelines and identify a baseline for reconciliation.

Week 2: establish a controlled environment

Provision the appropriate Gen 2 deployment and runtime, using the documented region and deployment-type prerequisites. Create separate administrative and execution roles, network controls, secret references and destination objects. Verify that access is no broader than necessary.

Week 3: test the connector and failure scenarios

Configure the PostgreSQL or MySQL/MariaDB Gen 2 connector in a nonproduction environment. Complete an initial snapshot, validate incremental inserts/updates/deletes, change a noncritical schema element, rotate a credential, restart the connector and simulate a backlog. Record the observed recovery behavior.

Week 4: decide with evidence

Review reconciliation results, lag distributions, source overhead, runtime consumption, supportability and preview-feature exposure. Approve production only if there is an accountable owner, tested runbook, defined monitoring, rollback path and a business sign-off on data correctness.

The deliverable is not just a working connector. It is an operational design package: architecture diagram, roles and grants, approved configuration, validation evidence, alert thresholds, cost assumptions and incident procedure.

Executive go/no-go scorecard

Use a simple decision table with five dimensions. Each dimension must be marked Pass, Conditional or Fail, with an owner and evidence.

Business correctness: Have inserts, updates, deletes and a representative business metric been reconciled?

Freshness: Has the pipeline met the agreed lag threshold under normal and recovery loads.

Recoverability: Has the team tested interruption, restart and missing-log decisions?

Security: Are secrets, runtime execution access and operator privileges reviewed?

Operability: Can an on-call engineer diagnose and recover the pipeline using documented steps?

A conditional launch should identify a compensating control and a deadline. For example, a preview connector might be accepted for a low-criticality workload with a fallback batch extract and daily reconciliation. A critical financial feed with untested delete handling should not be accepted simply because the dashboard appears current.

Frequently asked questions

1. Is Snowflake Openflow Gen 2 generally available?

Gen 2 deployment and runtime SQL objects are generally available on AWS, Azure and GCP. Gen 2 connector configuration and its currently documented Gen 2 connectors remain in public preview as of November 2026. Check the current Snowflake release notes before production adoption.

2. Does Gen 2 replace Apache NiFi?

Openflow still uses NiFi-based data movement concepts. Gen 2 changes how resources and supported connectors are represented and managed. It is not accurate to say that the underlying CDC operational concerns disappear.

3. Can a Gen 2 runtime run a Gen 1 connector?

Yes. Snowflake documents coexistence of Gen 1 and Gen 2 connectors on Gen 2 runtimes. A Gen 1 connector continues to follow its own configuration and operational model.

4. Which Gen 2 CDC connectors are documented?

PostgreSQL CDC and MySQL/MariaDB CDC are the documented Gen 2 catalog connectors. Availability of a source in the broader Openflow catalog does not automatically mean its Gen 2 configuration is supported.

5. Can connectors be deployed entirely with SQL?

Supported Gen 2 connectors can be created and managed with SQL, but configuration also uses versioned stage files and connector-specific prerequisites. A complete deployment requires privileges, networking, secrets, validation and lifecycle operations.

6. Does RBAC protect the source database automatically?

No. Snowflake RBAC controls Snowflake-side objects and execution permissions. Source-side database accounts, network controls, replication privileges and secret management must be configured and reviewed separately.

7. What happens when the CDC connector is offline?

The outcome depends on outage duration, source log retention, connector state and recovery behavior. Short interruptions may be recoverable by replaying retained changes; missing history can require a resnapshot or other reconciliation.

8. Is configuration rollback the same as data rollback?

No. Reverting a connector version changes configuration. It does not undo data already delivered or guarantee that a source log gap can be reconstructed.

9. How should a business measure CDC success?

Measure the correctness and timeliness of the business data consumed, not only connector uptime. Use freshness, reconciliation, incident recovery, access control and cost measures together.

10. Should every existing Gen 1 pipeline migrate immediately?

No. Prioritize migrations where the operational benefits justify the change and where connector compatibility, rollback and validation are established. Stable existing pipelines can be assessed and migrated in stages.

The decision for data leaders

Snowflake Openflow Gen 2 makes CDC infrastructure easier to express as governed, reviewable objects. That is valuable when organizations are scaling from a handful of manually managed feeds to a portfolio of production integrations. The biggest gain is the opportunity to standardize ownership, access, releases and recovery.

The key limitation is that management maturity cannot be purchased with a connector. Source change-log behavior, data correctness, schema contracts and business-level monitoring still determine whether a CDC pipeline can be trusted.

If you are evaluating Openflow alongside existing integrations, Actiknow’s custom integration and database solutions can help frame the source-to-warehouse architecture, while the team can discuss reporting and governance requirements through the Actiknow contact page. Start with one representative workload, a documented operating model and a measurable acceptance test.

Technical references and verification notes

The platform statements above were checked against Snowflake’s Openflow Gen 1 and Gen 2 documentation, Configure a Gen 2 connector with SQL guide, and September 11, 2026 GCP availability release note. Feature maturity, supported connectors and privileges can change, so verify these official resources again before implementation. This article provides a decision framework, not a guarantee of compatibility or a substitute for testing in your own account.