Merge pull request #4 from treosh/batch-api-limits

Batch API limits

.gitignore

@@ -1,2 +1,3 @@
 dist
+result
 node_modules

README.md

@@ -1,19 +1,19 @@
 # crux-api
 
-> A [Chrome UX Report API](https://developers.google.com/web/tools/chrome-user-experience-report/api/reference) wrapper that handles errors and provides types.
+> A [Chrome UX Report API](https://developers.google.com/web/tools/chrome-user-experience-report/api/reference) wrapper that supports batching, handles errors, and provides types.
 
 **Motivation**: [CrUX API](https://web.dev/chrome-ux-report-api/) is a fantastic tool to get RUM data without installing any script.
-While using the API in [Treo](https://treo.sh/), we discovered a few cases that require extra code: error responses, not found entries, API limits, URLs normalization, TypeScript notations. The `crux-api` library makes it easy to work with CrUX API by handling errors and providing TypeScript support.
+While using the API in [Treo](https://treo.sh/), we discovered a few complex cases like API errors, rate limits, not found entries, a complicated multipart response from the batch API, URLs normalization, and TypeScript notations. We decided to build the `crux-api` library to makes it easier to work with the CrUX API.
 
 **Features**:
 
-- A tiny (450 byte) wrapper for [Chrome UX Report API](https://developers.google.com/web/tools/chrome-user-experience-report/api/reference);
+- A tiny (450 bytes) wrapper for [Chrome UX Report API](https://developers.google.com/web/tools/chrome-user-experience-report/api/reference);
 - [Batch API support](https://developers.google.com/web/tools/chrome-user-experience-report/api/guides/batch) for up to 1000 records per one request;
-- TypeScript support for [options and responses](https://developers.google.com/web/tools/chrome-user-experience-report/api/reference/rest/v1/records/queryRecord);
+- TypeScript notations for [options and responses](https://developers.google.com/web/tools/chrome-user-experience-report/api/reference/rest/v1/records/queryRecord);
+- Isomorphic: works in a browser and node.js;
 - Returns `null` for the `404 (CrUX data not found)` response;
-- Handles the `429 (Quota exceeded)` response with automatic retries;
+- Automatic retry when hits the API rate limits: `429 (Quota exceeded)`;
 - URL normalization helper to match the CrUX API index;
-- Isomorphic: works in a browser and node.js;
 
 ## Usage
 
@@ -33,6 +33,20 @@ const res1 = await queryRecord({ url: 'https://www.github.com/' }) // fetch all
 const res2 = await queryRecord({ url: 'https://www.github.com/explore', formFactor: 'DESKTOP' }) // fetch data for desktop devices
 ```
 
+Use the [CrUX Batch API](https://developers.google.com/web/tools/chrome-user-experience-report/api/guides/batch) to combine up to 1000 requests and get results in less than 1 second:
+
+```js
+import { createBatch } from 'crux-api/batch'
+const batch = createBatch({ key: CRUX_API_KEY })
+
+const records = await batch([
+  { url: 'https://github.com/', formFactor: 'MOBILE', effectiveConnectionType: '4G' },
+  { url: 'https://github.com/marketplace', formFactor: 'DESKTOP' },
+  { url: 'https://www.github.com/explore', formFactor: 'TABLET' },
+  // ... up to 1000 records.
+])
+```
+
 Fetch origin-level data in node.js using [`node-fetch`](https://www.npmjs.com/package/node-fetch):
 
 ```js
@@ -81,31 +95,16 @@ Result is `null` or an `object` with [queryRecord response body](https://develop
 }
 ```
 
-Use the [CrUX Batch API](https://developers.google.com/web/tools/chrome-user-experience-report/api/guides/batch) to combine multiple requests into one:
-
-```js
-import { createBatch } from 'crux-api/batch'
-const batch = createBatch({ key: CRUX_API_KEY })
-
-const records = await batch([
-  { origin: 'https://github.com' }
-  { url: 'https://github.com/marketplace', formFactor: 'DESKTOP' },
-  { url: 'https://github.com/', formFactor: 'MOBILE', effectiveConnectionType: '4G' },
-  { url: 'https://www.github.com/explore', formFactor: 'TABLET' },
-])
-```
-
 ## API
 
 ### Single Record Request
 
-#### createQueryRecord(createQueryOptions)
+#### createQueryRecord(createOptions)
 
 Returns a `queryRecord` function.
 
-- _createQueryOptions.key_ (**required**) - CrUX API key, use https://goo.gle/crux-api-key to generate a new key;
-- _createQueryOptions.fetch_ (optional, default: `window.fetch`) - pass a [WHATWG fetch](https://github.com/whatwg/fetch) implementation for a non-browser environment;
-- _createQueryOptions.maxRetries_ (optional, default: 5) and **options.maxRetryTimeout** (optional, default: 60000) - retry limit after `429` error and the maximum time to wait for a retry.
+- _createOptions.key_ (**required**) - CrUX API key, use https://goo.gle/crux-api-key to generate a new key;
+- _createOptions.fetch_ (optional, default: `window.fetch`) - pass a [WHATWG fetch](https://github.com/whatwg/fetch) implementation for a non-browser environment;
 
 #### queryRecord(queryOptions)
 
@@ -120,46 +119,50 @@ Returns a Promise with a raw [`queryRecord` response](https://developers.google.
 ```js
 import { createQueryRecord } from 'crux-api'
 
-// disable retries, throw 429 error, similar to 400 and 404
-const queryRecord = createQueryRecord({ key: process.env.CRUX_API_KEY, maxRetries: 0 })
-
+const queryRecord = createQueryRecord({ key: process.env.CRUX_API_KEY })
 const res = await queryRecord({
   url: 'https://github.com/marketplace?type=actions',
   formFactor: 'DESKTOP',
   effectiveConnectionType: '4G',
 })
+
+// res -> URL-level data for https://github.com/marketplace
 ```
 
 ### Batch Request
 
-#### createBatch(createBatchOptions)
+`crux-api/batch` uses the [CrUX Batch API](https://developers.google.com/web/tools/chrome-user-experience-report/api/guides/batch), which allows combining 1000 calls in a single batch request. It's a separate namespace because the API is different, and it's bigger (850 bytes) due to the complexity of constructing and parsing multipart requests.
 
-Returns a `batch` function.
+_Note_: A set of `n` requests batched together counts toward your usage limit as `n` requests, not as one request. That's why the sometimes a batch response contains `429` responses. But the `crux-api` automatically retries these responses, aiming always to return the data you need.
 
-- _createBatchOptions.key_ (**required**) - CrUX API key;
-- _createBatchOptions.fetch_ (optional, default: `window.fetch`) - a [WHATWG fetch](https://github.com/whatwg/fetch) polyfill;
+#### createBatch(createOptions)
+
+Accepts the same [`createOptions` as the `createQueryRecord`](#createqueryrecordcreateoptions) and returns a `batch` function.
 
 #### batch(batchOptions)
 
-An array of [queryRecord](#queryrecordqueryoptions) options.
+Accepts an array of [`queryRecord`](#queryrecordqueryoptions) options and returns an array with an exact position for each record. If the record doesn't exist in CrUX index, the value set to null. If some requests hit rate-limit, `batch` will retry them after a short timeout.
 
 ```js
-import nodeFetch from 'node-fetch'
 import { createBatch } from 'crux-api/batch'
+import nodeFetch from 'node-fetch'
 
 const batch = createBatch({ key: process.env.CRUX_KEY, fetch: nodeFetch })
 const res = await batch([
   { origin: 'https://example.com' },
   { url: 'https://github.com/', formFactor: 'DESKTOP' },
   { origin: 'https://fooo.bar' },
 ])
-console.log(JSON.stringify(res, null, '  '))
+
+// res[0] -> origin-level data for https://example.com
+// res[1] -> URL-level data for https://github.com/ on desktop devices
+// res[2] -> null (invalid origin that not found in the CrUX index)
 ```
 
 ### normalizeUrl(url)
 
 Normalize a URL to match the CrUX API internal index.
-It is a URL's `origin` + `pathname` ([source](https://github.com/treosh/crux-api/blob/main/src/index.js#L81)).
+It is a URL's `origin` + `pathname` ([source](./src/index.js#76)).
 
 ```js
 import { normalizeUrl } from 'crux-api'

batch/src/index.js

@@ -1,14 +1,13 @@
-/** @typedef {{ key: string, fetch?: function }} CreateBatchOptions */
-/** @typedef {import('../../src').QueryRecordOptions[]} BatchOptions */
-/** @typedef {(import('../../src').SuccessResponse | import('../../src').ErrorResponse)[]} BatchResponse */
-
+import { retryAfterTimeout } from '../../src/retry'
 const boundary = 'BATCH_BOUNDARY'
 
 /**
  * Create batch interface for CrUX API.
  * https://developers.google.com/web/tools/chrome-user-experience-report/api/guides/batch
  *
- * @param {CreateBatchOptions} createOptions
+ * @typedef {{ options: import('../../src').QueryRecordOptions, result: import('../../src').SuccessResponse | null | undefined }[]} BatchValues
+ *
+ * @param {import('../../src').CreateOptions} createOptions
  */
 
 export function createBatch(createOptions) {
@@ -17,20 +16,50 @@ export function createBatch(createOptions) {
   return batch
 
   /**
-   * @param {BatchOptions} batchOptions
-   * @return {Promise<BatchResponse>}
+   * @param {import('../../src').BatchOptions} batchOptions
+   *
    */
 
-  async function batch(batchOptions) {
-    const body = generateBatchBody(batchOptions, key)
-    const res = await fetch('https://chromeuxreport.googleapis.com/batch/', {
-      method: 'POST',
-      headers: { 'Content-Type': `multipart/mixed; boundary=${boundary}` },
-      body,
-    })
-    const text = await res.text()
-    if (res.status !== 200) throw new Error(`Invalid batch response: ${text}`)
-    return parseBatchResponse(text)
+  function batch(batchOptions) {
+    const batchValues = /** @type {BatchValues} */ (batchOptions.map((options) => ({ options, result: undefined })))
+    return batchRequest(1)
+
+    /**
+     * @param {number} retryCounter
+     * @return {Promise<import('../../src').BatchResponse>}
+     */
+
+    async function batchRequest(retryCounter) {
+      const body = generateBatchBody(batchValues, key)
+      const res = await fetch('https://chromeuxreport.googleapis.com/batch/', {
+        method: 'POST',
+        headers: { 'Content-Type': `multipart/mixed; boundary=${boundary}` },
+        body,
+      })
+      const text = await res.text()
+      if (res.status !== 200) throw new Error(`Invalid batch response: ${text}`)
+      const results = parseBatchResponse(text)
+      results.forEach(({ index, json }) => {
+        if (!json) {
+          throw new Error('Empty result')
+        } else if (json.error) {
+          const { error } = /** @type {import('../../src').ErrorResponse} */ (json)
+          if (error.code === 404) {
+            batchValues[index].result = null
+          } else if (error.code !== 429) {
+            throw new Error(JSON.stringify(error))
+          }
+        } else {
+          batchValues[index].result = json
+        }
+      })
+      const rateLimitedRequests = batchValues.filter(({ result }) => result === undefined)
+      if (rateLimitedRequests.length) {
+        console.log('Rate-limit #%s: %s/%s', retryCounter, rateLimitedRequests.length, results.length)
+        return retryAfterTimeout(retryCounter, () => batchRequest(retryCounter + 1))
+      }
+      return batchValues.map(({ result }) => /** @type {import('../../src').SuccessResponse | null} */ (result))
+    }
   }
 }
 
@@ -53,24 +82,27 @@ export function createBatch(createOptions) {
  *
  * --BATCH_BOUNDARY--
  *
- * @param {BatchOptions} batchOptions
+ * @param {BatchValues} batchValues
  * @param {string} key
  */
 
-function generateBatchBody(batchOptions, key) {
-  const strOpts = batchOptions.map((queryRecordOptions, index) => {
-    return `--${boundary}
+function generateBatchBody(batchValues, key) {
+  let str = ''
+  batchValues.forEach(({ options, result }, index) => {
+    if (result !== undefined) return
+    str += `--${boundary}
 Content-Type: application/http
 Content-ID: ${index + 1}
 
 POST /v1/records:queryRecord?key=${key}
 Content-Type: application/json
 Accept: application/json
 
-${JSON.stringify(queryRecordOptions, null, '  ')}
+${JSON.stringify(options, null, '  ')}
+
 `
   })
-  return strOpts.join('\n') + `\n--${boundary}--`
+  return `${str}\n--${boundary}--`
 }
 
 /**
@@ -208,7 +240,7 @@ ${JSON.stringify(queryRecordOptions, null, '  ')}
  */
 
 function parseBatchResponse(text) {
-  const res = /** @type {BatchResponse} */ ([])
+  const results = /** @type {{ index: number, json: any }[]} */ ([])
   let index = /** @type {number | null} */ (null)
   let contentBody = ''
   for (const line of text.split('\n')) {
@@ -220,14 +252,10 @@ function parseBatchResponse(text) {
       contentBody += line
     }
     if (index && contentBody && line.startsWith('}')) {
-      res[
-        index - 1
-      ] = /** @type {import('../../src').SuccessResponse | import('../../src').ErrorResponse} */ (JSON.parse(
-        contentBody
-      ))
+      results.push({ index: index - 1, json: JSON.parse(contentBody) })
       index = null
       contentBody = ''
     }
   }
-  return res
+  return results
 }

package.json

@@ -1,7 +1,7 @@
 {
   "name": "crux-api",
-  "version": "1.1.0-beta1",
-  "description": "A Chrome UX Report API wrapper that handles errors and provides types.",
+  "version": "1.1.0",
+  "description": "A Chrome UX Report API wrapper wrapper that supports batching, handles errors, and provides types.",
   "repository": "https://github.com/treosh/crux-api",
   "bugs": "https://github.com/treosh/crux-api/issues",
   "license": "MIT",
@@ -59,7 +59,7 @@
       "path": "./src/index.js"
     },
     {
-      "limit": "500B",
+      "limit": "850B",
       "path": "./batch/src/index.js"
     }
   ],