TRENDING

TECHNOLOGY NEWS FROM AROUND THE WORLD

→
Reliable data imports: batches, retries and deduplication
Software Engineering · 2021
Software Engineering2021

Reliable data imports: batches, retries and deduplication

Fast imports are not necessarily correct. Stable source keys, atomic operations and resumable checkpoints should be designed together.

Yazılımcının GünlüğüPublished: Nov 16, 20263 min read

Start from stable source identity

Suppose orders are copied from a transactional system into a reporting database. Source OrderId should map to a stable unique target key. Running the same import twice must not double the number of orders. A unique constraint and explicit source identity are safer than fuzzy matching on names or dates that happen to look similar.

Batch size and checkpoints

Split work into bounded batches, selecting sizes based on measured API limits, transaction duration and memory. Persist a checkpoint only after its associated batch has been committed. When a worker crashes and retries a batch, the upsert must not create duplicates. Where possible, updating the target rows and checkpoint in one transaction provides a stronger recovery guarantee.

Test the failures, not just the happy path

Submit the same OrderId twice, inject an invalid value into a batch, and simulate a connection loss during a write. Measure committed rows, rejected rows and retried batches separately. Compare transaction amounts, required fields and source-to-target keys rather than trusting row count alone. Never report silent failures as successful imports.

Maintenance lessons from 2021

This newly written article revisits import problems relevant to 2021. A separate error queue does not solve problems when nobody monitors it. Start with bounded retries, clear operational status and a safe rollback strategy. Use measured traffic and database costs when choosing concurrency instead of increasing every limit blindly.

0 reads

Comments

No comments yet. Be the first to contribute.

✎ Write a comment

Your comment appears after submission; spam and duplicate submissions are blocked.