Surviving flaky connectors: chunked sync and idempotent pipelines

External third-party data sources — enterprise wikis, note repositories, cloud drives, and feeds — are inherently unpredictable. They hit rate limits, drop TCP connections mid-transfer, and fail on massive paginated payloads.

An AI memory sync protocol must make an unreliable external source look perfectly resilient to the rest of the platform. Here is how bounded chunking and cursor-lock protocols accomplish that.

The Bounded Chunk Protocol

If a sync engine attempts to import 10,000 documents in a single HTTP transaction, any transient network dropout on item 9,999 destroys the entire batch.

To eliminate this vulnerability, the sync protocol follows three core design invariants:

1. Bounded Batch Size: Data is ingested in discrete packets (e.g. 25–50 items per payload).
2. Intermediate Heartbeat: Non-final packets write records to storage and touch the connection timestamp to signal healthy progress, preventing stale-sync alarms.
3. Atomic Finality: The sync cursor is committed only when the final chunk is acknowledged with an isFinal flag.

High-Level Algorithm

function IngestConnectorStream(sourceReader, batchSize):
    lastAcknowledgedCursor = sourceReader.getSavedCursor()
    
    while sourceReader.hasMore():
        itemsBatch, nextCursor, isFinal = sourceReader.readBatch(batchSize)
        
        # Step 1: Write items idempotently to the memory store
        StorageEngine.writeDocuments(itemsBatch)
        
        # Step 2: Refresh connector heartbeat
        ConnectorRegistry.touchHeartbeat()
        
        # Step 3: Only the final batch advances the persistent sync cursor
        if isFinal:
            ConnectorRegistry.commitCursor(nextCursor)
            ConnectorRegistry.setSyncStatus(STATUS_COMPLETED)

Architectural Guarantees

• Zero-Loss Crash Recovery: If a worker crashes midway through chunk 40 of 100, the previous 39 chunks are safely stored, and the subsequent sync attempt picks up from the last acknowledged checkpoint.
• Idempotent Ingestion: Re-processing an existing chunk overwrites or deduplicates records seamlessly, avoiding duplicate memory clutter for the user.