2.2 Large-Scale API Data Transfer

Move a large amount of data in small, controlled pieces without overwhelming either system.

Imagine that your application needs to copy product information from a partner's API into your own system. The partner has only 20 products when you begin testing, so one request works:

Your application -> GET /products -> partner API -> 20 products

Six months later, the partner has two million products. Asking the API to return all of them in one response is no longer practical. The response may use too much memory, take so long that it times out, or put heavy load on both systems.

This is not specifically a database problem. It is a communication problem between two systems: how can the caller move a large amount of data without requesting or sending everything at once?

We will use a commerce catalog as the running example because products and inventory make the numbers concrete. The same techniques apply when moving customer records, support tickets, documents, payment transactions, or any other large collection.