AWS S3 Cp [Copy] Command Overview with Examples

Effectively working with S3 requires moving data in and out of S3 buckets efficiently, supporting use cases such as web hosting, content distribution, backups, archiving, media storage and streaming, and more. In this article, we will explore how to use the aws s3 cp command to transfer data between your local filesystem and the S3 buckets.

We will cover:

1. What is aws s3 cp and what does it do?
2. Examples: How to use aws s3 cp command
3. How to use aws s3 cp recursive command flag
4. Uploading a local file stream to S3
5. Downloading an S3 object as a local file stream
6. Uploading to an S3 access point
7. Downloading from an S3 access point
8. aws s3 cp vs sync

Before learning about the aws s3 cp command, install the AWS CLI. If it is not already installed, here’s a quick guide to help you get started.

The aws s3 cp command allows you to copy files to and from Amazon S3 buckets. It is used for uploading, downloading, and moving data efficiently in and across AWS S3 storage environments.

Below is the syntax of the cp command:

The cp command is straightforward, requiring only the source and a target optionally followed by options.

Note: The source and destination cannot both be local. i.e., the aws cp command cannot be used to copy files to and from your local filesystem.

With that out of the way, let’s learn the common use cases of copying files to and from an S3 bucket.

Example 1: Copying a local file to S3

To copy a single file from the current directory to an S3 bucket, simply mention the filename followed by the name of the S3 bucket. 

Remember, the bucket name must be prefixed with s3://.

If you want to copy a file with a different name, simply add its name to the destination path. For example, the following command copies file1.txt as robot.txt to the aws-s3-cp-tutorial bucket:

Example 2: Copying an object from S3 one bucket to another

To copy a file from one S3 bucket to another, replace the source with the name of the source S3 bucket followed by the path to the file and the destination with the name of the destination S3 bucket where the file is to be copied. 

Remember to prefix both the source and the destination bucket names with s3://.

If you want to copy the file with a different name, add the desired file name to the destination S3 bucket path:

The aws s3 cp command can also be used to rename files within an S3 bucket. Set the same bucket as both the source and the destination and add the new file name to the destination path. 

Let’s see an example:

Example 3: Downloading (copying) a file from S3 to local

Downloading files from S3 is nothing but the same as copying files from an S3 bucket to your machine.

For example, to download the robot.txt file from the aws-s3-cp-tutorial bucket, we use the aws s3 cp command and replace the source with the s3 bucket name followed by the path to the file and the destination with the desired location on your machine where you want to download the file.

If you want to download the file with a different name, simply add the new file name to the destination path:

With the fundamentals in place, let’s explore how to extend the capabilities of the aws s3 cp command by learning how to use option flags.

The aws s3 cp command can handle various use cases, from copying multiple files to applying access control lists (ACLs) and much more. By incorporating flags with the base aws s3 cp command, we can unlock the additional functionalities and cater to the advanced use cases.

Below are some of the important flags that often come in handy:

We will look at these individually to understand when and where to use them.

1. Copy multiple files ( — recursive)

Local to s3

To copy all files in a directory, use the --recursive flag. For instance, to copy all files from the current directory to the aws-s3-cp-tutorial s3 bucket, use the following command:

The --recursive flag also copies files from any sub-directories. 

For example, if the directory structure is as shown below, the same directory structure would be replicated in the S3 bucket.

Note that the files are being copied at the sub-directory level.

S3 to local

Similarly, the --recursive flag can be used to copy everything from an S3 bucket to the local file system, including sub-directories:

s3 to s3

The --recursive flag works similarly when copying files from one S3 bucket to another.

2. Exclude and include specific files (–exclude and  –include)

When copying multiple files, the s3 cp command allows selecting specific files to include or exclude in the copy operation.

The --exclude flag enables the exclusion of certain files from the copy operation. The --include flag lets you include specific files in the copy operation, often used in conjunction with the --exclude flag. 

Let’s explore them with examples, keeping in mind the following directory structure for the working directory:

Excluding files

To exclude a particular file, use the --exclude flag followed by the name of the file to be excluded. 

For example, to exclude the random.txt file when copying the entire current directory, execute the following command:

As expected, only the file1.txt and robot.txt files are copied, excluding random.txt 

Including a specific file

To include a specific file while excluding others, use the following command. First, ignore all files with the --exclude flag followed by including only the random.txt file:

As expected only the random.txt file is copied.

Note: The order of flags is crucial in determining the final operation. For instance, switching the positions of the --include and --exclude flags alters the outcome as well.

In this case, no operation is performed as all files, including the one explicitly included, are excluded by the last --exclude flag.

3. Preview the changes made by aws s3 cp (–dryrun)

Sometimes, s3 cp operations can get complex, and you might be unsure of the expected changes. Or you might just want to double-check the changes before applying. 

In these cases, you can use the --dryrun flag. As the name suggests, it allows you to preview the changes before committing them. Simply append the --dryrun flag to any command to see the preview.

Let’s see the output of the command we ran earlier after appending the --dryrun flag.

Upon execution, the command previews the output of uploading ./random.txt to s3://aws-s3-cp-tutorial/random.txt, enabling us to verify the expected results before making any changes. 

4. Access control using ACLs (–acl)

ACLs (Access Control Lists) are crucial in managing access to S3 buckets and the objects they contain. With the aws s3 cp command, you can set Canned ACLs using the --acl flag, which accepts a range of values including private, public-read, public-read-write, authenticated-read, aws-exec-read, bucket-owner-read, bucket-owner-full-control and log-delivery-write .

Note: To use the --acl flag, the s3:PutObjectAcl permission must be included in the list of actions for your IAM policy. You can verify this using the following command:

The output should resemble the following:

Setting the Access Control List (ACL) on files being copied to an S3 object

To grant public read access to the files being copied to the S3 bucket, use the --acl flag to apply the public-read ACL on the file.

Public read access is granted to all users.

If you see the error below, make sure that the bucket allows setting ACLs for public access. Alternatively, you can try out a different canned ACL, which is not public.

5. Set fine-grained grants on the files being copied (–grants)

Grants allow managing fine-grained access control in S3. The following cp command demonstrates the use of the --grants flag to grant read access to all authenticated users:

Result:

Additionally, it’s possible to apply multiple grants simultaneously. The following cp command grants read access to all authenticated users identified by a URI and full control to a specific user identified by their email address:

As expected read access is granted to all authenticated users and the user associated with the canonical ID shown in the above picture.

6. Specify the storage class for the files being copied (–storage-class)

To copy a file with a specific storage class, we can use the --storage-class flag. The accepted values for the storage class are STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | GLACIER_IR. STANDARD is the default storage class.

In the example below, file1.txt is copied to the aws-s3-cp-acl-tutorial bucket using the REDUCED_REDUNDANCY storage class:

As expected file1.txt is stored with storage class REDUCED_REDUNDANCY.

The cp command supports uploading a local file stream from standard input to an s3 bucket. The example below demonstrates copying from the standard input to the stream.txt file in the destination S3 bucket.

It’s important to note that when uploading a local file stream larger than 50GB, the --expected-size option must be provided, or the upload may fail when it reaches the default part limit of 10,000. 

For example, for a file stream of 51GB, we can set the --expected-size as follows.

Similarly to uploading, we can download files from S3 as a local file stream. For example, the command below downloads the stream.txt file from an S3 bucket as a stream to the standard output.

Note: Downloading as a stream is not currently compatible with the --recursive parameter.

Access points are named network endpoints attached to S3 buckets. An Access Point alias provides the same functionality as an Access Point ARN and can be substituted for an S3 bucket name for data access.

The following command uploads file1.txt file via the access point access-point-cp-tutorial to S3:

Note: To successfully copy files to an S3 bucket using an access point, ensure that the access point policy allows the s3:PutObject action for your principal as shown below:

Access points can also be used to download files, provided the access point policy allows the s3:GetObject action. The following command downloads file1.txt using the access-point-cp-tutorial access point to local:

As we approach the end of the article, one last thing to learn is how aws s3 cp is different from aws s3 sync.

The difference between aws s3 cp and aws s3 sync lies in their behavior when copying files:

aws s3 sync recursively copies new and updated files from the source directory to the destination. It does not copy existing unchanged files and only creates folders in the destination if they contain one or more files. aws s3 cp --recursive on the other hand copies all files and folders from the source to the destination, overwriting any existing files. However, it does not delete any files from the destination that no longer exist in the source.

When using aws s3 sync with the --delete flag, it deletes any files from the destination that have been deleted from the source.

In summary, the sync command is more efficient when you want the destination to reflect the exact changes made in the source, while cp is more suitable when you simply want to copy and overwrite files to the destination.

The aws s3 cp is a robust command for transferring data to and from S3 buckets, providing a wide range of capabilities from recursive copying to applying ACL(s). Mastering the aws s3 cp command can significantly boost productivity when working with S3.

In the meantime, go ahead and learn how a platform like Spacelift can help you and your organization fully manage cloud resources within minutes.

Spacelift is a CI/CD platform for infrastructure-as-code that supports tools like Terraform, Pulumi, Kubernetes, and more. For example, it enables policy-as-code, which lets you define policies and rules that govern your infrastructure automatically. You can even invite your security and compliance teams to collaborate on and approve certain workflows and policies for parts that require a more manual approach. You can check it for free by creating a trial account or booking a demo with one of our engineers.