“Real time” sounds automatically better than batch, but ingestion architecture is a tradeoff among freshness, volume, source capability, downstream latency, operating cost, and failure recovery. A customer attribute needed for an overnight propensity model does not need sub-second delivery. A fraud or service-intervention signal can become useless if it arrives hours later.

For the Salesforce Data 360 Consultant, the practical skill is choosing the right ingestion mode for each data product rather than forcing every source into the same pattern. Data 360 supports scheduled/batch data streams, streaming-capable connections, and programmatic ingestion patterns. The right design starts with a service-level requirement for freshness and then works backward to source behavior and cost.

Define the freshness requirement before choosing a connector

Ask how quickly a source change must influence a business decision. “As fast as possible” is not a requirement. A marketing audience refreshed every morning may be satisfied with hourly or daily ingestion. A customer-service agent who needs to see a just-completed order may need much lower latency. A real-time web interaction can demand an even faster event path.

Freshness has several components: how quickly the source emits or exposes a change, how often Data 360 ingests it, how quickly downstream transforms or identity processes update, and how quickly a segment, insight, or activation consumes the result. Improving ingestion latency alone does not guarantee real-time business behavior.

Document an end-to-end target such as “eligible purchase events are available to the decisioning use case within two minutes under normal conditions.” That statement can be tested. “Use streaming” cannot.

Batch ingestion favors predictable volume and recoverability

Batch processing collects changes and moves them on a schedule or through a bulk operation. It is well suited to large historical loads, periodic exports, systems that cannot emit events, and use cases where freshness is measured in hours rather than seconds.

Batch also simplifies reconciliation. A team can reason about a bounded file or extraction window, validate record counts, rerun a failed period, and compare source and destination totals. For slowly changing reference data, catalog information, monthly financial attributes, or backfills, that operational simplicity can matter more than latency.

Refresh mode affects efficiency. A full refresh repeatedly processes the entire relevant data set, while incremental or upsert behavior focuses on changed records when the connector supports it. Full refreshes can be necessary for some sources and recovery scenarios, but they create more work as data volume grows. Design the refresh strategy around how reliably the source exposes changes.

Streaming ingestion favors event freshness and continuous change

Streaming processes smaller units of change continuously or in very small micro-batches. It is appropriate when a business action loses value quickly: a cart event, service interaction, product usage signal, consent change, or operational event that should affect a near-term decision.

Streaming does not eliminate batch behavior from the architecture. Initial loads often require a baseline batch, and full refresh or recovery scenarios can still use batch processing. The steady-state path then processes eligible changes with lower latency. Salesforce’s CRM connector documentation, for example, distinguishes scheduled batch behavior from eligible streaming processing and notes that an initial/full refresh can still be batch-oriented.

A good design therefore separates bootstrap from steady state. Load the historical state efficiently, establish a checkpoint, and then process changes. Treat the change stream as a continuation of the mastered state, not as a substitute for it.

Choose between bulk and streaming API patterns deliberately

Programmatic ingestion can also use bulk or streaming patterns. Bulk ingestion is appropriate when an upstream application can accumulate a meaningful volume and send it at defined intervals. Streaming ingestion is better when individual events or small micro-batches need to arrive continuously.

The source application must be able to preserve ordering and retry behavior appropriate to the use case. Network failures, throttling, duplicate delivery, and partial errors are normal integration conditions. Include stable source identifiers, timestamps, idempotent processing where possible, and a replay or reconciliation strategy so that recovery does not depend on guessing which records were missed.

Do not create a fragile “real-time” pipeline that has no backfill mechanism. A mature streaming design normally includes a durable source of truth or log from which missing events can be reprocessed.

Credits and billing can change the architecture decision

Data 360 consumption should be part of design, not an afterthought. Current Salesforce documentation distinguishes internal data pipelines, external batch pipelines, and external streaming pipelines, and the exact billing behavior depends on the organization’s license and active consumption card. Some usage can be treated differently under Data Services, Flex Credits, Data 360 Profiles, or other entitlements.

That means architects should not memorize one universal statement such as “batch is free” or “streaming always costs more.” Instead, inspect the org’s Digital Wallet and license model, identify the usage types generated by the proposed connector, and estimate volume and frequency. Contract terms and Salesforce pricing models can change independently of the technical interface.

Optimization follows the workload. Reduce unnecessary full refreshes, avoid ingesting fields and records that have no downstream use, lower schedule frequency when freshness does not justify it, and reserve continuous streaming for signals that actually need it. Cost-aware design is not about making every pipeline slower; it is about paying for freshness where freshness creates value.

Ingestion latency is only the first stage of activation latency

Suppose a purchase event arrives in Data 360 in seconds. The business may still wait for mapping, transformation, identity resolution, calculated insights, segmentation, or activation before the event changes the customer experience. Each step can have its own processing model and schedule.

For an activation use case, map the complete path: source event → ingestion → data model object → transformation or calculated insight → identity/unified profile if required → segment or real-time decision layer → destination. Then assign a latency expectation to each stage. If the segment publishes every few hours, changing ingestion from hourly batch to streaming will not create a real-time campaign.

Conversely, a use case may not need identity resolution at all. A device or session event could be acted on using a known identifier before a full customer profile is reconciled. Architecture should be driven by the decision being made, not by a blanket preference for the most comprehensive pipeline.

Design for source-specific behavior

Different sources expose change differently. Some provide change data capture or native event streams. Others offer only scheduled exports. Cloud storage connectors may rely on new files and predetermined refresh schedules. SaaS connectors may support incremental extraction but not true streaming for every object. The same platform can therefore contain a mixture of ingestion modes.

Document the contract for each stream: source owner, object or event type, primary key, change indicator, expected volume, refresh mode, normal frequency, maximum acceptable delay, deletion behavior, schema-change process, and replay strategy. This turns a collection of connectors into an operable data architecture.

Schema evolution deserves special attention. A source that adds a field or changes semantics can break mapping or silently change downstream behavior. Real-time pipelines amplify mistakes faster, so they need disciplined schema governance rather than less governance.

Use observability to distinguish “late” from “wrong”

When users report stale data, measure each stage before changing ingestion mode. Check when the source record changed, when it was made available to the connector, when Data 360 ingested it, when downstream processing completed, and when the target application received the result. The delay may be upstream extraction, ingestion, identity processing, segmentation, or activation.

Monitor record counts, failed jobs, processing history, lag, retry volume, schema errors, and unexpected full refreshes. A healthy streaming pipeline can still deliver bad outcomes if source timestamps are wrong or events are duplicated. A healthy batch job can still miss the business SLA if the schedule is misaligned with activation.

Operational dashboards should therefore represent data freshness as a chain of timestamps, not a binary “job succeeded” indicator.

Validate deletes, late events, and replay behavior

Freshness is not only about new and updated records. Determine how a source represents deletion, deactivation, consent withdrawal, and corrected history. A batch connector may communicate a deletion through a tombstone, a full-refresh comparison, or a source-specific flag. A streaming source may publish a dedicated delete event. If the destination never receives that semantic change, the data can be fast and still be wrong.

Late and out-of-order events require a policy as well. Event time and ingestion time are not always the same. A mobile device can reconnect after hours offline, or an upstream queue can replay older events after recovery. Preserve source timestamps and stable identifiers so downstream logic can distinguish a genuinely new change from a delayed delivery. Test replay before production rather than assuming that retries will always arrive once and in order.

Use hybrid ingestion instead of forcing one mode everywhere

Most enterprise implementations are hybrid. Large historical and reference data arrives in batch. High-value behavioral signals stream. Some CRM objects move incrementally on a frequent schedule, while a subset of eligible changes uses lower-latency processing. Backfills and recovery use bulk paths even when normal operation is streaming.

The Salesforce design principle is to make that mix intentional. Choose batch when bounded processing, scale, and recoverability dominate. Choose streaming when event freshness materially changes a decision. Then verify that downstream identity, insight, segmentation, and activation stages can actually preserve the freshness you paid to create.

A successful ingestion strategy is not the one with the lowest theoretical latency. It is the one that meets the business freshness target reliably, explains its cost, survives failure, and can prove where time is spent from the source system to the final customer experience.