Overview
What is HTTP Fetch: HTTP Fetch is Datagrid's own integration for making outbound HTTP requests to API endpoints and bringing the response back into Datagrid, for use when a target system has no dedicated Datagrid integration. The request/response model follows the same idea as the Fetch Standard, an open web standard maintained by WHATWG, though the same underlying concept appears across platforms under different names.
How to integrate HTTP Fetch with Datagrid
Use HTTP Fetch when Datagrid agents need live records from a REST API or a partner system that lacks a dedicated Datagrid integration, or when an existing integration doesn't cover the required workflow. Set up the integration by configuring the request, authenticating the target API, testing the response shape, and defining how Datagrid should process each returned record.
Configure the request endpoint
Start with a single endpoint and a narrow query before running against a full dataset. This keeps the request easy to test and makes the target API's pagination behavior visible.
Open Integrations in Datagrid and add the HTTP Fetch integration.
Enter the target endpoint URL and select the HTTP method: GET, POST, PUT, or DELETE.
Configure authentication for the target API.
Add custom headers and query parameters required by the target API.
Set the request body format for POST and PUT calls: JSON, form-urlencoded, or multipart/form-data.
Configure pagination if the target API returns multiple pages.
Test with a single request before running against a full dataset.
When target API documentation requires paginated requests with headers and query parameters, use a configuration pattern like this:
method: GET
url: <target-endpoint-url>
headers:
Authorization: Bearer <token>
Accept: application/json
query_parameters:
limit: 100
start: 0
body_format: none
pagination:
mode: cursor_or_offset
next_pointer: <next-page URL or cursor field>For POST and PUT calls, set the body format explicitly and pass only the fields the target API accepts:
{
"external_id": "<record-id>",
"status": "complete",
"notes": "<agent-generated summary>"
}Authenticate the target API
The integration handles common auth patterns for target APIs. Supported patterns include Basic auth, Bearer token, Header auth with an API key, Digest auth, Query auth, OAuth 1.0, and OAuth 2.0. Datagrid's own integrations use these same patterns: Freshdesk uses HTTP Basic auth, and QuickBooks uses the OAuth 2.0 authorization code grant. Datagrid stores the authentication configuration and applies the selected method during integration runs. Match the target API's documented method exactly.
Use redacted values when testing authentication syntax:
auth_examples:
bearer_token:
headers:
Authorization: Bearer <token>
api_key_header:
headers:
X-API-Key: <api-key>
basic_auth:
username: <username>
password: <password>Sync response data into Datagrid
The integration imports responses as the primary direction, pulling data from an external endpoint into Datagrid. It also writes outbound for target APIs that accept POST and PUT. Formats include JSON, XML, CSV, multipart/form-data, and form-urlencoded. Sync can run by schedule or trigger, with polling for endpoints that require repeated checks.
A JSON response shape can look like this:
{
"items": [
{
"id": "<record-id>",
"status": "open",
"updated_at": "<timestamp>"
}
],
"next_cursor": "<cursor-value>",
"has_more": true
}For paginated target APIs, configure HTTP Fetch to follow the next-page pointer until no cursor remains. If operators do not configure pagination, Datagrid imports only the first response page from the target API and silently misses the remaining data. Before production workflows, review Datagrid's quickstart for detailed integration setup patterns.
Why use HTTP Fetch with Datagrid
HTTP Fetch is useful when mission-critical workflow data is behind a REST endpoint and operators need that data in Datagrid, especially when a dedicated integration or webhook path does not cover the workflow.
Reach REST API records directly: HTTP Fetch connects AI agents to internal and partner endpoints, enabling live records to enter Datagrid workflows.
Bridge systems with scheduled polling: Agents poll REST endpoints on a schedule, the same fetch pattern Datagrid uses for Hilti ON!Track and other polling-only systems.
Parse structured responses automatically: Agents extract fields from JSON, XML, CSV, and other structured payloads and map them to workflow logic.
Match target API authentication: Configure the documented authentication method for the endpoint, including Basic, Bearer, API key, or OAuth.
Read and write in a single workflow: Agents fetch records, process them, and either write results back to the source or forward them to other integrations.
Combine fetched data with other sources: Agents cross-reference API responses against other business records for enrichment and reporting.
What you can build with HTTP Fetch Datagrid integration
Teams use HTTP Fetch when the work depends on data that changes outside Datagrid.
Scheduled extraction from any REST API: Configure the integration to pull records from an endpoint on a set interval, then process them in Datagrid. This mirrors how Oracle Primavera Cloud syncs Activities, Projects, and WBS data with pagination through start and limit parameters.
Data enrichment pipelines: Agents fetch fields from an external API and fill gaps in existing records. For Salesforce lead enrichment, agents cross-reference Lead records with additional sources to complete company size, industry, and job title fields.
Cross-platform sync workflows: Agents fetch data from one system and write it to another. In a closed-loop invoicing pattern, an agent reads deal line items and creates an invoice in QuickBooks when a deal closes.
Automated reporting from live API data: Agents pull stats and metrics via fetch calls, calculate variances, and distribute the results. A Freshdesk SLA dashboard extracts ticket stats and SLA configs, calculates breach rates and then pushes results to BI tools.
Resources and documentation
Datagrid quickstart: How developers start with integrations, connections, webhooks, and tools.
Datagrid connectors API: Lists available connectors before teams choose HTTP Fetch or a dedicated integration.
Fetch Standard: The open-web behavior of HTTP fetch requests and responses.
Freshdesk integration: Shows a Datagrid workflow pattern that uses HTTP Basic auth.
QuickBooks integration: Shows a Datagrid workflow pattern that uses the OAuth 2.0 authorization code grant.
Oracle Primavera Cloud integration: Shows pagination through start and limit parameters.
Frequently asked questions
What authentication methods does the HTTP Fetch integration support?
Match the method documented by the target API. Common configurations include Basic auth, Bearer tokens, API-key headers, Digest auth, query auth, OAuth 1.0, and OAuth 2.0. Datagrid stores the selected authentication settings for integration runs.
How do I handle paginated API responses?
Configure HTTP Fetch to follow the next-page pointer returned by the target API. Common modes include updating a page or offset parameter in each request, or following a next-page URL from the response body. Check for a next cursor or next page URL; some APIs use a has_more flag instead. Loop until the target API returns no pointer.
What request methods can the HTTP Fetch integration send?
Use GET for reads, POST and PUT for writes, and DELETE for removals. Set the request body format for POST and PUT calls to JSON, form-urlencoded, or multipart/form-data. Test with a single request before running against a full dataset.
What common HTTP errors should I expect, and how do I fix them?
Common HTTP errors follow standard status codes. A 400 means the target API received an invalid query parameter or a malformed array. A 403 means authentication failed; review the credentials and scopes for the target API. A 404 means Datagrid called an invalid endpoint URL. Invalid JSON usually indicates formatting errors, such as missing quotes or extra commas.
Similar integrations
Zyte: Use HTTP Fetch to call Zyte's scraping API and ingest structured extraction results into agentic data workflows.
Amazon AWS S3: Stage HTTP-fetched files and API responses in S3 to trigger file-based ingestion and downstream AI processing.
Snowflake: Write agent-processed records fetched via HTTP Fetch into Snowflake for analytics, joins, and scheduled reporting.
Google Sheets: Push API responses fetched with HTTP Fetch into Google Sheets for lightweight review, manual edits, or downstream automations.
Azure Blob Storage: Route fetched documents and binary responses into Azure Blob Storage to trigger document extraction and processing pipelines.
PostgreSQL: Load JSON responses retrieved by HTTP Fetch into PostgreSQL for relational enrichment, indexing, and cross-platform synchronization.