The type of data contained in a page
Function to fetch individual pages; called with zero-based page indices
Invoked for each non-null page with the page data and its index; may be called out-of-order relative to page indices
A promise that resolves when all pages have been fetched and processed
Concurrency: All pages are fetched in parallel with no concurrency limit. It is highly recommended to use rateLimited or similar rate-limiting utilities to wrap the fetcher function to avoid overwhelming the server.
Dynamic page counts: The function starts by fetching page 0. If subsequent pages
report a higher num_pages, additional pages are automatically fetched. If num_pages
decreases during execution, previously started fetches may continue.
If any fetch fails, the promise rejects immediately with the error from the failed fetch. Other in-flight fetches may continue running in the background, but their results are ignored and their callbacks will not be invoked.
await forEachPage(
async (page) => ({ num_pages: 5, page: await fetchUsers(page) }),
(users, index) => console.log(`Page ${index}:`, users)
);
// If page 0 returns num_pages=2, but page 1 returns num_pages=3,
// page 2 will be automatically fetched
await forEachPage(fetcher, callback);
const limitedFetcher = rateLimited(fetcher, 100);
await forEachPage(limitedFetcher, processPage);
fetchPages for an async generator alternative
Fetches multiple pages in parallel and invokes a callback for each non-null page.