Overview
What is Amazon AWS S3: Amazon AWS S3 is a cloud-based object storage service built for scalability, durability, and security. It stores data as objects in buckets, organized by key name prefixes rather than traditional folder hierarchies.
How to integrate Amazon AWS S3 with Datagrid
Amazon AWS S3 stores the project files teams depend on, including project records, data exports, sensor readings, and processed reports. Datagrid's agentic AI agents connect to S3 and automatically execute workflows on that data. When files are imported from a bucket, whether on a schedule, on demand, or through an AWS event-driven pattern, agents extract fields, cross-reference records, validate quality, and push structured results into your CRM, ERP, or project management tools. This removes manual downloads and copy-paste pipelines.
The setup follows these steps:
Connect Amazon AWS S3 App
Creating a dataset from the Amazon AWS S3 integration involves selecting the specific bucket and data direction you want to configure:
Open your Datagrid workspace and click + Create → Connect Apps.
Search for the Amazon AWS S3 integration in the integration catalog.
Enter your AWS credentials, access key ID and secret access key for an IAM user with appropriate S3 permissions.
Specify the target S3 bucket name and AWS Region.
Configure the data direction to import from S3, export to S3, or both.
Test the connection to verify Datagrid can list objects in the specified bucket.
Click Save to save the configuration and assign the integration to your active workflows.
Prerequisites
To configure the Amazon AWS S3 integration, you will need:
An AWS account with an IAM user or role that has appropriate S3 permissions scoped to the target buckets.
An access key ID and secret access key for the IAM user.
The target S3 bucket name and AWS Region.
For cross-account setups, a bucket policy on the target bucket, because identity-based policies alone are not sufficient for cross-account access.
Authenticate with AWS IAM
Datagrid authenticates to S3 with AWS-native access controls. Permissions define which buckets and objects Datagrid can read or write, and those permissions should match the scope of the workflow you want agents to execute.
Datagrid authenticates to S3 using IAM identity-based policies. These are permissions explicitly granted to IAM users, roles, or services to access S3 resources. AWS Signature V4 handles request signing automatically when using AWS SDKs. S3 does not support HTTP Basic Authentication. For cross-account setups, apply a bucket policy to the target bucket, as identity-based policies alone are not sufficient for cross-account access.
Configure your data sync settings
After access and authentication are in place, define what Datagrid should sync, when it should run, and what object data the workflow should read or write. This is the operating model for the integration.
Sync direction: Bidirectional, read from S3 and write to S3
Data objects synced: Objects (files) within specified buckets, including metadata (key name, ETag, storage class, last modified date, user-defined tags)
Supported formats: CSV, JSON, and PDF; additional formats such as Parquet and ORC may be supported where enabled in your workspace
Sync trigger: On-demand, scheduled imports, or automation-triggered exports
You can use the following configuration pattern when setting up the integration in Datagrid:
{
"source": "Amazon AWS S3",
"objects": ["Objects (files)", "Metadata"],
"sync_direction": "bidirectional",
"supported_formats": ["CSV", "JSON", "PDF"],
"sync_trigger": "on-demand | scheduled | event-driven",
"downtime_windows": "configured in Datagrid scheduling settings"
}
This structure gives project teams a clear way to move from raw files in S3 to validated outputs in business systems. It also keeps S3 available as both an intake layer and a durable destination for processed records.
Advanced: event-driven setup with AWS Lambda or EventBridge
For teams that want Datagrid workflows to trigger automatically when new files land in S3, AWS provides S3 event notifications that fire on object creation, deletion, or modification. These events route to AWS-native destinations such as Lambda, SNS, SQS, or EventBridge. In practice, this means wiring S3 events to an intermediary such as AWS Lambda or EventBridge, which then calls Datagrid's workflow endpoint.
This pattern is fully feasible but requires an engineer comfortable with AWS event infrastructure. Without it, teams can still run scheduled or on-demand imports to process bucket contents.
Why use Amazon AWS S3 with Datagrid
Teams that already store operational files in S3 can turn that storage layer into an execution layer for cross-system workflows. Datagrid's AI agents read what lands in a bucket, apply the workflow logic you configure, and route the result where work needs to happen next.
These are the main workflow advantages:
Turn storage into workflow inputs: S3 can become the starting point for agentic AI workflows. Datagrid's AI agents process imported files and push results into connected tools without manual intervention.
Bidirectional data movement: Read raw files from S3 for agentic AI processing and write transformed, structured outputs back to S3 as a durable staging layer or audit trail.
Cross-platform data routing: Datagrid connects S3 workflows to connected business tools documented in the Datagrid Amazon AWS S3 integration documentation. Agents extract data from S3 files and route it directly to CRMs, ERPs, project management platforms, and communication tools.
Autonomous project file processing: AI agents interpret unstructured S3 files such as contracts, invoices, submittals, and specs, extract fields, validate data quality, and flag exceptions for human review.
Format-agnostic ingestion: S3 stores any file type without restriction. Datagrid agents handle CSV, JSON, and PDF from S3 today, with additional format support available depending on workspace configuration.
Scalable data lake integration: Connect S3's multi-tier data lake architecture to Datagrid's workflows for automated tier promotion, quality checks, and cross-system synchronization.
What you can build with Amazon AWS S3 Datagrid integration
Automated project document processing: When a contractor uploads a submittal PDF to an S3 bucket, a Datagrid AI agent can process that file through a configured import or event-driven workflow, extract key fields such as vendor name, spec references, and product data, cross-reference them against project requirements stored in a connected system, and flag discrepancies before a project manager opens their inbox.
S3-to-CRM data sync for field operations: Field teams capture inspection data, photos, or sensor readings that land in S3.
Multi-source data consolidation for reporting: Data exports from multiple SaaS tools, including accounting systems, HR platforms, and analytics dashboards, accumulate in S3 as CSV or JSON files.
Reverse ETL from S3 to operational tools: After an agentic AI transformation pipeline writes clean, structured data to an S3 output layer, Datagrid agents pick up the file, perform field mapping to the destination system's schema, run deduplication and type checks, and write records to connected business tools using upsert logic.
Resources and documentation
Amazon S3 User Guide - official AWS documentation covering buckets, objects, storage classes, and security
Amazon S3 REST API reference - API version, operations index, and request formats
S3 event notifications setup guide - step-by-step configuration for event-driven triggers
S3 access management documentation - IAM policies, bucket policies, access points, and cross-account permissions
Getting started with Amazon S3 - onboarding guide and data processing tutorial index
For Datagrid support, you can use the email: support@datagrid.ai
Website: https://www.datagrid.ai
Request an endpoint: Don't see endpoints you're looking for? We're always happy to make new endpoints available.
Frequently asked questions
How does Datagrid authenticate with Amazon AWS S3?
Datagrid connects to S3 using IAM identity-based credentials, an access key ID and secret access key tied to an IAM user or role with permissions scoped to the target buckets. S3 does not support HTTP Basic Authentication. For cross-account access, AWS requires a bucket policy or S3 Access Points in addition to IAM credentials. Pre-signed URLs offer an alternative for time-limited access, with a maximum validity of 7 days when generated via SDK or CLI. For enhanced security, AWS best practice recommends using IAM roles rather than long-lived access keys where your deployment model supports it.
What file formats can Datagrid process from S3?
S3 itself is format-agnostic object storage, which means any file type can be stored and retrieved. Datagrid agents ingest common file formats from S3, including CSV, JSON, and PDF. For analytics-optimized workflows, columnar formats like Parquet and ORC deliver better query performance and may be supported depending on your workspace configuration. AWS Glue supports format conversion between CSV, JSON, Parquet, ORC, Avro, and Apache Iceberg within S3-backed data pipelines.
Is the S3 integration bidirectional?
Yes. The Datagrid S3 integration supports both reading data from S3 into Datagrid and exporting transformed data back to S3. Agents can pull a raw CSV from S3, process it, and write the structured output to a different S3 prefix or bucket. Agents can also push that result to another connected tool.
Can Datagrid trigger workflows automatically when new files land in S3?
S3 natively supports event notifications that fire when objects are created, deleted, or modified. These events route to AWS-native destinations such as AWS Lambda, Amazon SNS, Amazon SQS, or Amazon EventBridge. In practice, this means wiring S3 events to an intermediary such as AWS Lambda or EventBridge, which then calls Datagrid's workflow endpoint. This is an advanced configuration that requires familiarity with AWS event infrastructure. Without it, teams can still run scheduled or on-demand imports to process bucket contents.
What S3 data does Datagrid have access to?
Datagrid accesses the objects (files) and their associated metadata within the S3 buckets you authorize during setup. Object metadata includes system-defined fields like key name, ETag (content hash), storage class, size, and last modified date, plus any user-defined key-value metadata you attach to objects. Access is scoped by the IAM permissions granted to the credentials you configure.
Similar integrations
Amazon Redshift: AWS analytics service that queries and loads data directly from S3, often the next step after S3-based staging.
Amazon RDS: AWS's managed relational database service, frequently paired with S3 in pipelines where structured database records complement S3's unstructured file storage.
Browse by category
Cloud Storage
Data Warehouse