Coding Question 40 minutesMedium
Resumable Customer Backfill
Copy a paginated customer dataset into a warehouse without losing progress or duplicating rows.
Question
Implement backfill in backfill.py. Copy every customer from the paginated source into the warehouse, persist progress safely, and ensure a retry after interruption neither loses nor duplicates rows.
Function parameters
- source
- A SourceApi object that returns paginated customer dictionaries and opaque cursors.
- warehouse
- A Warehouse object that upserts transformed customer rows.
- checkpoint
- A Checkpoint object that loads and saves the cursor from the last durable page.
Inputs and API responses
Source API
| Entity | Fields |
|---|---|
| Customer | id, email, plan, updated_at |
| Page | data, next_cursor |
- Cursors are opaque strings.
- Maximum page size is 3.
Warehouse
| Entity | Fields |
|---|---|
| CustomerRow | source_id, email, plan, source_updated_at |
| Checkpoint | cursor |
- upsert uses source_id as its key.
- Checkpoint state survives function calls.
Commit ordering
- Upsert the entire page before saving its next_cursor
- Resume from the stored cursor
- Never derive or increment an opaque cursor
Requirements
- Call source.list_customers(limit=3, cursor=checkpoint.get()).
- Transform source fields to the warehouse schema.
- Upsert each page before updating the checkpoint.
- Continue until next_cursor is None.
- A retry must not duplicate rows.
- New source rows must be picked up by a later complete run.
- If the checkpoint advances before the page is durable, a crash loses data. If no checkpoint is stored, every retry starts from page one.
Editable
Test output
Run the challenge to test failure, checkpointing, and resume.