Skip to content
This repository was archived by the owner on May 26, 2021. It is now read-only.

S3

Jump to bottom
Kevin Zhuang edited this page Jul 26, 2020 · 2 revisions

All S3 related actions support alternate profile but not region due to S3 being a "global" service. You could either configure them through the fzfaws.yml config file or you could pass in command line options. The command line option take higher precedence and will override the config file settings. For more information about priority, checkout here.

fzfaws s3 upload --profile root

Configure fzfaws.yml to set default profile for all S3 actions, this will override the global settings of profile. In the example below, the default profile will be used for all other services, but S3 service will use the root profile by default.

global:
  profile: default
  region: us-east-1
services:
  s3:
    profile: root

Uploading files to S3

Reference individual option flag through fzfaws s3 upload --help.

Without any arguments:

  1. Select files from current directory (support multi selection)
  2. Select a bucket
  3. Select a path in bucket
  4. Files will be uploaded to the selected bucket path
fzfaws s3 upload

To list files from home directory, you could use the --root flag. Note: it is extremely slow and resource intensive if you don't have fd installed to search from home directory, install fd before using this flag.

When using fd, you can also include the --hideen flag to include searching for hidden files.

fzfaws s3 upload --root --hidden

fzfaws support configuration of object settings during upload, you can configure: ACL, StorageClass, Encryption, Tags, Metadata. A menu will be prompt to ask you which options you want to configure, use TAB to multi select. The --extra as in extra settings, the naming could probably be better...

fzfaws s3 upload --extra

The demo below demonstrates an upload operation to S3 bucket, uploading hidden files from home directory, the preview window on the right list the objects in the current S3 path. demo

Directories

Directories does not allow multi selections, if you wish to upload or configure multiple directories at once, use --exclude and --include globbing to achieve it.

To upload an entire directory, same as aws-cli, you will need to pass in the --recursive flag to indicate a recursive operation. fzfaws also provides --exclude and --include flag for some extra filtering capabilities. Both of them takes a number of bash globbing pattern, and --exclude will always be processed before --include.

When using --recursive together with --extra flag, the extra configuration applies to all files uploaded (i.e. if you set a different StorageClass, all items in the folder gets that StorageClass)

# just upload a directory
fzfaws s3 upload --recursive

# Uploading the current directory to S3 bucket, but we don't want to upload the .git folder
fzfaws s3 upload --recursive --path $PWD/ --exclude ".git/*"

# Only upload the "fooboo" and "helloworld" directory under current directory
fzfaws s3 upload --recursive --path $PWD/ --exclude "*" --include "fooboo/*" "helloworld/*"

If you have aws-cli installed, you could use the --sync flag to leverage it's sync functionality. The --exclude and --include flag also applies it's functionality during sync.

# select a directory through fzf and sync the directory with a S3 bucket without the .git folder
fzfaws s3 upload --sync --exclude ".git/*"

Downloading files from S3

Reference individual option flag through fzfaws s3 download --help.

Without any arguments:

  1. Select a bucket
  2. Select objects (support multi selection)
  3. Select a download location under current directory
  4. Objects will be downloaded to the selected location
fzfaws s3 download

To select a location for download from the home directory, use the --root flag. Note: it is extremely slow and resource intensive if you don't have fd installed to search from home directory, install fd before using this flag.

When using fd, you can also include the --hideen flag to include searching for hidden files.

fzfaws s3 download --root --hidden

Versioning enabled buckets

You can also download versions of an object in S3 buckets that has versioning turned on. This flag does not work with --recursive flag. The red colored item are the item with "delete marker" associated. Versions supports multi selection.

fzfaws s3 download --version

Directories

Downloading directories have the same mechanics as uploading directories, using --recursive flag while also support --exclude, --include and --sync flag, consult directories section in #uploading-files-to-s3.

The demo below demonstrates a recursive download operation to a folder under current directory, the preview window on the right list the objects in the current S3 path. demo

Transferring objects between buckets

Reference individual option flag through fzfaws s3 bucket --help.

Without any arguments:

  1. Select a source bucket
  2. Select source objects (support multi selection)
  3. Select destination bucket
  4. Select destination path
  5. Objects will be transferred
fzfaws s3 bucket

Versioning enabled buckets

You can also transfer versions of an object between S3 buckets if the source bucket has versioning turned on. This flag does not work with --recursive flag. The red colored item are the item with "delete marker" associated. Versions supports multi selection.

fzfaws s3 bucket --version

Preserving attributes

By default, when objects are moved to other buckets, other buckets settings will apply to the object. Optionally, you could also choose to preserve all objects details when transferring, including ACL, StorageClass, Encryption, Metadata, Tags.

Note: --preserve does support cross region transfer, however if you are using KMS encryption, it will fall back to SSE-S3 encryption.

fzfaws s3 bucket --preserve

"Folders" (also known as common prefix)

Transferring recursively have the same mechanics as uploading directories, using --recursive flag while also support --exclude, --include and --sync flag, consult directories section in #uploading-files-to-s3.

Update objects details

Reference individual option flag through fzfaws s3 object --help.

You could use fzfaws to update most of the objects details. The supported attributes by default are: ACL, StorageClass, Encryption, Metadata, Tags.

Without any arguments:

  1. Select a bucket
  2. Select objects to update (support multi selection)
  3. Select which attributes to update (support multi selection)
  4. Going through each of the selected attributes and input their new value
  5. Objects will be updated
fzfaws s3 object

You also update the name of an object. This will preserve all objects details comparing to a normal aws s3 cp s3://bucket1/oldename s3://bucket1/newname.

fzfaws s3 object --name

Versioning enabled buckets

You can also configure specific versions attributes of an object. With --version flag, the available attributes for update will be limited to Tags and ACL because updating all other attributes will create a new version.

fzfaws s3 object --version

# updating attributes of all versions
fzfaws s3 object --allversion

Operating on versioning enabled bucket it is recommended to always use the --version flag even if you just want to update the latest version attributes. Because updating a versioned objects StorageClass, Encryption and Metadata will always create a new version. Using the --version flag will limit the default menu to only show Tags and ACL.

It is also not recommended to update name of on versioning enabled buckets because it doesn't really rename, it will just create a new object with the new name and leaving the old objects and it's versions associated with a delete marker.

Recursive operation

You can recursively update attributes of an entire "folder"(common prefix), same as uploading directories to S3, it takes --exclude and --include for extra filtering.

fzfaws s3 object --recursive

Deleting objects from S3

Reference individual option flag through fzfaws s3 delete --help.

Without any arguments:

  1. Select a bucket
  2. Select objects to delete (support multi selection)
  3. Objects will be deleted

If versioning is enabled, a delete marker will be associated.

fzfaws s3 delete

Recursive operation

You can recursively delete objects of an entire "folder"(common prefix), same as uploading directories to S3, it takes --exclude and --include for extra filtering.

fzfaws s3 delete --recursive

Versioning enabled buckets

fzfaws provides a few useful commands to handle deletion of versioned objects. The red colored objects are objects with delete marker associated.

When deleting a versioned objects, only a delete marker is associated with the object, we are not really deleting the objects. Commands below will delete the selected version instead of associating a delete marker, you can even delete a delete marker. Support multi selection on versions.

fzfaws s3 delete --version

You can run a clean up commands to delete all older versions of an object, meaning that only the latest version will be retained. The --version flag is optional here as --clean will automatically apply --version.

fzfaws s3 delete --clean

# You could even do it recursively
fzfaws s3 delete --clean --recursive

You can also cleanly delete a versioned object, meaning delete all versions including the current version. The --version flag is optional here as --allversion will automatically apply --version.

fzfaws s3 delete --allversion

# You could even use this to nuke an entire versioned bucket
fzfaws s3 delete --allversion --recursive

MFA protection

You could also delete objects from MFA protected bucket.

fzfaws s3 delete --mfa [device serial number] [the MFA 4 digit code]

The above operation doesn't support any type of multiple operations because the MFA code expires after first operation.

Generating presign URLs

Reference individual option flag through fzfaws s3 presign --help.

Without any arguments:

  1. Select a bucket
  2. Select objects (support multi selection)
  3. A presign GET URL will be generated with 3600 seconds of expiration
fzfaws s3 presign

Edit the fzfaws.yml config file to set a different default expiration.

services:
  s3:
    default_args:
      presign: --expires 7200

Versioning enabled buckets

You can also generate presign GET URL for versions of objects. Support multi select on versions.

fzfaws s3 presign --version

Listing S3 information

Checkout the dedicated section for extending fzfaws functionalities which explains how and why for the ls commands. The ls command will just print out information without doing any actions. Consult the help manual to see the available options.

fzfaws s3 ls --help

Skipping some prompts

Sometimes if you remember a partial part of the objects path and its relatively short, it may be beneficial to skip some fzf prompt to speed things up a little bit. The format is a bit strict:

  • Specify just the bucket: fzfaws s3 upload --bucketpath bucket1/
  • Specify a "folder" in S3: fzfaws s3 download --bucketpath bucket1/folder1/
  • Specify a object in S3: fzfaws s3 bucket --bucketpath bucket1/object

This --bucketpath applies to all operations that contains the --bucketpath flag not just fzfaws s3 bucket exclusively. If you wish to use a S3 access point, you will have to use the --bucketpath flag, fzfaws s3 bucket --bucketpath accesspoint/.

fzfaws s3 bucket --bucketpath bucket1/hello.txt --to bucket2/

Same applies to local file/directory path, you can use the --path to specify local file path. This is especially useful if you are using fd, sometimes you may want to upload a file type that is ignored by .ignore and during the file listing you couldn't find the file to upload.

fzfaws s3 upload --path foo.png hello.jpeg lol.txt
Clone this wiki locally