Backfilling a year of records or keeping thousands of them in sync used to mean workarounds and hard-to-find failures. It doesn't anymore. We shipped native support for large data syncs. Here's the problem it solves and how it works.
Syncing a large volume of records is one of those problems that nearly everyone building serious integrations eventually hits. Wiring up an ongoing sync (say, pushing Salesforce accounts into your product as they're created) is straightforward. The trouble starts when a customer also wants to backfill every account from the past year, keep tens of thousands of records in sync bi-directionally, or refresh a large source system. That's when things go sideways.
We just shipped native support for this, so high-volume backfills and syncs will work without the workarounds much of the market still relies on.
What is a large data sync?
A large data sync is the movement of a high volume of records between two systems in a single logical job. That commonly means an initial backfill when an integration is first deployed, a periodic bulk refresh, or an ongoing bi-directional sync that keeps thousands of records or more current between two systems. It's distinct from routine event-by-event syncing precisely because the volume is large enough to break the assumptions most integration tooling is built on.
Why large data syncs break integrations
It's tempting to think the hard part is moving the data. It isn't. Moving records is easy. The hard part is everything around it.
Moving a large volume of records reliably means you can't run the job as one long execution. Consider a 100,000-record backfill at 10 records per second: that's roughly three hours of continuous processing, and if it fails two hours in, you're starting over from scratch. High-volume work has to be broken into smaller units so it can be retried, resumed, throttled, and monitored. That's true on any platform, and it's where the real difficulty starts.
Break the job up yourself, though, and you're left with a stack of problems: splitting the dataset into pieces that each fit the limit, tracking where you left off so work can resume, handling the batch that fails at 40,000 of 80,000 records, staying under the third-party API's rate limits, and monitoring all of it when your one "sync" is now a dozen different things.
The usual answers (recursive flows, fanning work out across sibling flows, and hand-rolled continuation tokens) are insufficient. They work until a bigger customer shows up with a bigger dataset, and then they fail in ways that are hard to debug, because the failure is scattered across a tree of parent and child executions. Worse, these workarounds often pass all the regular QA but break months later in production, right when a customer is counting on them.
This is where a platform earns its place. Handling all of that (infrastructure) is the real work.
Watch: why large data syncs break (and how we fixed it)
Brian Munz, Developer Advocate at Prismatic, walks through the problem and the fix in about three minutes:
How Prismatic handles large data syncs
Instead of pushing all of that complexity into flow logic, we moved it into the platform. You describe how to page through the records at the trigger, and the platform handles the batching from there.
A few things came out of that design:
- One trigger covers both jobs. The same trigger handles the initial backfill of all historical data when an instance deploys and the ongoing changes after that, and it shares the same downstream logic. You configure how to identify records, how to paginate, and how many records to process per batch, rather than building and maintaining separate flows.
- The whole sync is one logical execution, with batch-level visibility. A large sync rolls up into a single execution, and you can see exactly which batch failed and where, with no need to reconstruct state from a scatter of child executions.
- Concurrency controls are built-in. Run batches in parallel for throughput without triggering a third-party API's rate limit. Set how many batches run at once and the platform respects it.
- It's opt-in. Nothing changes in your existing integrations until you build with it. Adopting large data syncs is a deliberate choice you make on a given integration, not a passive upgrade.
Each batch runs as a standard execution, within the platform's usual 15-minute window. That's by design: small, bounded units of work are what make retries, resumes, and concurrency possible. An individual batch still needs to fit that window, but that constraint is exactly what keeps large syncs reliable.
Availability
Large data syncs are live today for code-native integrations, where builders get direct access to the batching primitives in the Spectral SDK. Support for low-code integrations and the embedded workflow builder is coming soon.
Why it matters
The question B2B SaaS prospects regularly ask SaaS vendors with integrations is some version of "Can you handle our volume?" Until now, the answer across much of the category has been "Yes, with some workarounds." But those workarounds come with their own, non-trivial risks. Native large data syncs allow you to answer the question with a clean “Yes.”
This is a good example of what we mean when we say Prismatic handles what breaks at scale: the platform abstracts the hard parts of large data syncs, so your team can ship integrations instead of maintaining plumbing.
Get started
The feature is documented, with a working Salesforce example, in the bulk data sync guide. You can also see the release details in the changelog. If you build in code-native, you can start using it today.
Common questions
Question: What is a large data sync?
Answer: A large data sync moves a high volume of records between two systems in one logical job: typically an initial backfill, a periodic bulk refresh, or an ongoing bi-directional sync involving huge numbers of records.
Question: Why do large data syncs break integrations?
Answer: Because a high-volume job can't reliably run as one long execution. A 100,000-record backfill at 10 records per second is about three hours of continuous work, and a failure partway through means starting over. The job has to be split into smaller units, which creates the real difficulty: chunking the dataset, tracking a cursor to resume, handling partial failures, staying under API rate limits, and monitoring it all as one job.
Question: How does Prismatic handle large data syncs?
Answer: You define how to page through records at the trigger, and the platform automatically decomposes the dataset into bounded batches and rolls them back up into a single execution with batch-level visibility. One trigger handles both the initial backfill and ongoing changes, concurrency controls let batches run in parallel without exceeding rate limits, and the whole thing is opt-in.
Question: How long can each batch run?
Answer: Each batch runs as a standard execution within the platform's 15-minute window. Batching exists so large jobs never depend on one long-running execution: work is split into bounded units that can be retried, resumed, and run concurrently.
Question: Is it available in low-code and the Embedded Workflow Builder?
Answer: Large data syncs are available today for code-native integrations. Low-code and embedded workflow builder support are coming soon.
Question: Do I need to change my existing integrations?
Answer: Only if you want the feature. Large data syncs are opt-in and builder-driven, so existing integrations keep working unchanged and won't gain batching until it is explicitly added.
