Skip to content

Added method for concatenating S3 buckets. Fixed UnsupportedOperation… #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Added method for concatenating S3 buckets. Fixed UnsupportedOperation… #80

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion src/aws/sdk/s3.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
com.amazonaws.services.s3.model.AbortMultipartUploadRequest
com.amazonaws.services.s3.model.CompleteMultipartUploadRequest
com.amazonaws.services.s3.model.UploadPartRequest
com.amazonaws.services.s3.model.CopyPartRequest
java.util.concurrent.Executors
java.io.ByteArrayInputStream
java.io.File
Expand Down Expand Up @@ -232,7 +233,7 @@
[{cred :cred bucket :bucket key :key upload-id :upload-id e-tags :e-tags}]
(.completeMultipartUpload
(s3-client cred)
(CompleteMultipartUploadRequest. bucket key upload-id e-tags)))
(CompleteMultipartUploadRequest. bucket key upload-id (java.util.ArrayList. e-tags))))

(defn- upload-part
[{cred :cred bucket :bucket key :key upload-id :upload-id
Expand All @@ -249,6 +250,20 @@
(.setPartSize ^long (min part-size (- (.length file) offset)))
(.setFile file)))))

(defn- copy-part
[{cred :cred upload-id :upload-id source-bucket :source-bucket source-key :source-key
target-bucket :bucket target-key :key part-num :part-num}]
(.getPartETag
(.copyPart
(s3-client cred)
(doto (CopyPartRequest.)
(.withDestinationBucketName target-bucket)
(.withDestinationKey target-key)
(.withSourceBucketName source-bucket)
(.withSourceKey source-key)
(.withUploadId upload-id)
(.withPartNumber part-num)))))

(defn put-multipart-object
"Do a multipart upload of a file into a S3 bucket at the specified key.
The value must be a java.io.File object. The entire file is uploaded
Expand Down Expand Up @@ -277,6 +292,22 @@
(throw ex))
(finally (.shutdown pool)))))

(defn concat-keys
"Efficiently concatenates keys on S3 using multi-part upload API.
Please note that each part must be at least 5mb in size."
[cred target-bucket target-key source-bucket & source-keys]
(let [upload-id (initiate-multipart-upload cred target-bucket target-key)
upload {:upload-id upload-id :cred cred :bucket target-bucket :key target-key :source-bucket source-bucket}]
(try
(loop [part-num 1 source-keys source-keys e-tags '()]
(if (seq source-keys)
(recur (inc part-num) (next source-keys)
(conj e-tags (copy-part (assoc upload :source-key (first source-keys) :part-num part-num))))
(complete-multipart-upload (assoc upload :e-tags e-tags))))
(catch Exception ex
(abort-multipart-upload upload)
(throw ex)))))

(extend-protocol Mappable
S3Object
(to-map [object]
Expand Down