Streaming Uploading to S3

[Steve Frank]

I need to copy large files from a customer's bucket to ours. I implemented
code to download file in chunks, but would now also like to upload in
chunks so I can bypass having to store in a temp file locally first.

File.open(ltmpfilepath, mode='wb') do |ltmpfile|
remoteDir.files.get(rfilename) {|chunk, remaining, total|
ltmpfile.write chunk
#Instead of writing to a temp file above, i want to pass this along
to put_object
}
end

I know that put_object takes in a string or file, so I suspect I can pass
in some other object that implements an expected interface, just not sure
which. (I would solve this with streams in the c# world)

Thanks!

-Steve

[Benjamin Manns]

Steve,

You may be able to do this entirely on Amazon's side by giving your account
read access to your client's S3 bucket, and running:

storage = Fog::Storage.new(provider: 'AWS', credentials...)
directory = storage.directories.new(key: client_bucket_name)
directory.files.each do |file|
storage.copy_object(client_bucket_name, file.key, your_bucket_name,
file.key, 'x-amz-metadata-directive' => 'COPY')
end

This will do all the copying on AWS instead of having to download each file
locally, which is much faster and consumes less bandwidth, but the
permissions stuff can be a pain.

Ben

Post by Steve Frank
I need to copy large files from a customer's bucket to ours. I implemented
code to download file in chunks, but would now also like to upload in
chunks so I can bypass having to store in a temp file locally first.
File.open(ltmpfilepath, mode='wb') do |ltmpfile|
remoteDir.files.get(rfilename) {|chunk, remaining, total|
ltmpfile.write chunk
#Instead of writing to a temp file above, i want to pass this
along to put_object
}
end
I know that put_object takes in a string or file, so I suspect I can pass
in some other object that implements an expected interface, just not sure
which. (I would solve this with streams in the c# world)
Thanks!
-Steve
--
You received this message because you are subscribed to the Google Groups "ruby-fog" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out<https://app.yesware.com/tl/215cd8db50bd0c97a50bc1ae44e9e6832a3d0b6c/5897caefeffe816003b17ce7ab07b9d5/f27120e93443a16141d0fc4788847809?ytl=https%3A%2F%2Fgroups.google.com%2Fgroups%2Fopt_out>
.

[geemus (Wesley Beary)]

It think you could probably manage it via two connections (one down, one
up) both with a pipe/stringio. Not saying that would necessarily be easy
though. Giving temp permissions to write into your bucket directly as
Benjamin describes might be a lot easier to deal with/maintain if that is
feasible.

Post by Benjamin Manns
Steve,
You may be able to do this entirely on Amazon's side by giving your
storage = Fog::Storage.new(provider: 'AWS', credentials...)
directory = storage.directories.new(key: client_bucket_name)
directory.files.each do |file|
storage.copy_object(client_bucket_name, file.key, your_bucket_name,
file.key, 'x-amz-metadata-directive' => 'COPY')
end
This will do all the copying on AWS instead of having to download each
file locally, which is much faster and consumes less bandwidth, but the
permissions stuff can be a pain.
Ben

Post by Steve Frank
I need to copy large files from a customer's bucket to ours. I
implemented code to download file in chunks, but would now also like to
upload in chunks so I can bypass having to store in a temp file locally
first.
File.open(ltmpfilepath, mode='wb') do |ltmpfile|
remoteDir.files.get(rfilename) {|chunk, remaining, total|
ltmpfile.write chunk
#Instead of writing to a temp file above, i want to pass this
along to put_object
}
end
I know that put_object takes in a string or file, so I suspect I can pass
in some other object that implements an expected interface, just not sure
which. (I would solve this with streams in the c# world)
Thanks!
-Steve
--
You received this message because you are subscribed to the Google Groups
"ruby-fog" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out<https://app.yesware.com/tl/215cd8db50bd0c97a50bc1ae44e9e6832a3d0b6c/5897caefeffe816003b17ce7ab07b9d5/f27120e93443a16141d0fc4788847809?ytl=https%3A%2F%2Fgroups.google.com%2Fgroups%2Fopt_out>
.

--
Benjamin Manns
[phone]
--
You received this message because you are subscribed to the Google Groups "ruby-fog" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.