aws--agent-toolkit-for-aws/plugins/aws-data-analytics/skills/ingesting-into-data-lake/references/local-upload.md

Local File Upload

Upload files from the local filesystem to S3, with optional ingestion into a table.

Workflow

1. Determine Intent

First, check the source path. If the user provides an S3 URI (e.g., s3://...) as the source, stop and use s3-files.md instead. This workflow is for local files only.

Parse the user's request to route:

• Upload only? ("put this in S3", "upload my file", "move to AWS") -> Path A
• Upload + make queryable? ("load this into a table", "ingest this CSV", "make it queryable") -> Path B

If ambiguous and the file is structured (CSV, JSON, Parquet, TSV, Avro, ORC), ask: "Do you want this queryable as a table, or just stored in S3?"

2. Discover Local Data

1. Validate path: Confirm the file or directory exists and is readable
2. Detect format: Infer from extension (.csv, .json, .parquet, .tsv, .avro, .orc) or ask
3. Check size: ls -lh for files, du -sh for directories
4. For structured files, peek at content:
   • CSV/TSV: head -5 to check headers, delimiter, encoding
   • JSON: head -20 to check structure (records vs. arrays)
   • Parquet/Avro/ORC: note format, skip content peek

Encoding check (CSV/TSV/JSON only):

file --mime-encoding <path>

If not UTF-8 or ASCII, warn the user before upload. Non-UTF-8 files can cause downstream parsing failures.

3. Choose S3 Destination

1. Ask for target bucket or list available buckets:

   aws s3 ls
   

2. Suggest prefix structure: s3://<bucket>/<domain>/<dataset>/<filename>

3. Confirm with user before uploading

Default: preserve original filename. Override: user specifies a different key.

4. Upload

Single file -- check for existing objects before uploading (aws s3 cp silently overwrites):

aws s3 ls s3://<bucket>/<prefix>/<filename>

If the object exists, warn the user and get explicit confirmation before proceeding.

Directory -- check for existing objects before syncing. Use a bounded existence check to avoid enumerating every object under the prefix (which can be very slow on large prefixes):

aws s3api list-objects-v2 --bucket <bucket> --prefix <prefix>/ --max-items 1

If the result contains any Contents, objects exist and the user should be warned before proceeding. aws s3 sync skips unchanged files but overwrites modified ones without prompting.

Single file upload:

aws s3 cp <local-path> s3://<bucket>/<prefix>/<filename>

Directory upload:

aws s3 sync <local-dir> s3://<bucket>/<prefix>/

For files over 8 MB, aws s3 cp uses multipart upload automatically. No special flags needed.

Verify upload:

aws s3 ls s3://<bucket>/<prefix>/<filename>

5. Route Based on Intent

Path A: Upload Only

Report results and stop:

• S3 URI of uploaded file(s)
• File size and format
• Example command to download: aws s3 cp s3://... .

Path B: Upload + Table Ingestion

After upload completes, continue with the s3-files.md workflow using:

• S3 path where data was uploaded
• Detected file format
• Row/size estimate
• Encoding (if checked)

Do not reimplement schema inference or table creation -- follow the S3 files workflow for those steps.

Gotchas

• aws s3 cp silently overwrites existing S3 objects. Always check first.
• aws s3 sync skips unchanged files but overwrites modified ones without prompting. Check destination before syncing directories.
• CSV files with mixed encodings (e.g., Latin-1 headers, UTF-8 body) upload fine but break downstream parsing. Always check encoding for text formats.
• Large uploads on slow connections can time out. For files over 5 GB, suggest running the upload in a screen or tmux session.
• Compressed files (.gz, .zip): upload as-is for Path A. For Path B, note the compression so the S3 files workflow can handle decompression.

Troubleshooting

Error	Cause	Fix
upload failed: ... An error occurred (AccessDenied)	No write permission to target bucket	Check IAM policy or bucket policy allows s3:PutObject
The user-provided path ... does not exist	Typo in local path	Verify path with ls
fatal error: An error occurred (NoSuchBucket)	Bucket does not exist	List buckets with aws s3 ls and pick an existing one
Upload hangs or is very slow	Large file on slow connection	Check file size, suggest tmux/screen, verify network

References

• upload-options.md -- Compression, multipart thresholds, sync vs cp tradeoffs