Use include and exclude rules with S3 sync

Part 19 of AWS from Zero. Today we make sync selective.

What we are learning

--exclude removes matching paths from a transfer. --include adds matching paths back. The order of these rules matters.

Before you run it

REGION="ap-south-1"
BUCKET="aws-zero-sync-filters-12345"
 
aws s3api create-bucket \
  --bucket "$BUCKET" \
  --region "$REGION" \
  --create-bucket-configuration LocationConstraint="$REGION"
 
mkdir sync-demo
echo "first note" > sync-demo/one.txt
echo "second note" > sync-demo/two.txt
echo "temporary data" > sync-demo/cache.tmp
echo '{"ready":true}' > sync-demo/config.json

Preview the command

aws s3 sync sync-demo/ "s3://$BUCKET/demo/" \
  --exclude "*" \
  --include "*.txt" \
  --dryrun

--dryrun shows what would happen without uploading anything.

Run the sync

aws s3 sync sync-demo/ "s3://$BUCKET/demo/" \
  --exclude "*" \
  --include "*.txt"

Inspect the result

aws s3api list-objects-v2 \
  --bucket "$BUCKET" \
  --prefix "demo/" \
  --query "Contents[].Key" \
  --output table

Only the two .txt files should appear.

One tiny variation

Include JSON files as well:

aws s3 sync sync-demo/ "s3://$BUCKET/demo/" \
  --exclude "*" \
  --include "*.txt" \
  --include "*.json" \
  --dryrun

Common mistake

This order does not produce the same selection:

--include "*.txt" --exclude "*"

Filters are evaluated in order, and later matching rules can override earlier ones. Start by excluding everything, then include the patterns you want.

Cleanup

aws s3 rm "s3://$BUCKET/" --recursive
aws s3api delete-bucket --bucket "$BUCKET" --region "$REGION"
rm -rf sync-demo

Next, we will delete one object and verify that it is gone.

AWS CLI sync reference