Reliable data imports: batches, retries and deduplication
Fast imports are not necessarily correct. Stable source keys, atomic operations and resumable checkpoints should be designed together.
Start from stable source identity
Suppose orders are copied from a transactional system into a reporting database. Source OrderId should map to a stable unique target key. Running the same import twice must not double the number of orders. A unique constraint and explicit source identity are safer than fuzzy matching on names or dates that happen to look similar.
Batch size and checkpoints
Split work into bounded batches, selecting sizes based on measured API limits, transaction duration and memory. Persist a checkpoint only after its associated batch has been committed. When a worker crashes and retries a batch, the upsert must not create duplicates. Where possible, updating the target rows and checkpoint in one transaction provides a stronger recovery guarantee.
Test the failures, not just the happy path
Submit the same OrderId twice, inject an invalid value into a batch, and simulate a connection loss during a write. Measure committed rows, rejected rows and retried batches separately. Compare transaction amounts, required fields and source-to-target keys rather than trusting row count alone. Never report silent failures as successful imports.
Maintenance lessons from 2021
This newly written article revisits import problems relevant to 2021. A separate error queue does not solve problems when nobody monitors it. Start with bounded retries, clear operational status and a safe rollback strategy. Use measured traffic and database costs when choosing concurrency instead of increasing every limit blindly.
Comments
No comments yet. Be the first to contribute.
✎ Write a comment