OpenStack large file support

[Ben McCann]

Hi,

Does the OpenStack storage have any support for large files?
http://docs.openstack.org/trunk/openstack-object-storage/developer/content/large-object-creation.html

Thanks,
Ben

[Kyle Rames]

Hi Ben,

The OpenStack provider does have support for large file upload, however, it
requires you to read and upload the file in 5 GB chunks. I have fixed this
issue in the Rackspace provider ( (the Rackspace Cloud is based on
OpenStack).

You can read about the Rackspace fix in this pull request
-> https://github.com/fog/fog/pull/1844

I should have some time to get this fix into OpenStack sometime tomorrow.

Hope that helps.

Kyle

Post by Ben McCann
Hi,
Does the OpenStack storage have any support for large files?
http://docs.openstack.org/trunk/openstack-object-storage/developer/content/large-object-creation.html
Thanks,
Ben

[Ben McCann]

Great, that's fantastic! I've actually been using the Rackspace provider
with other OpenStack services and it works just fine, so I'm pretty happy
to have it in the Rackspace provider, but definitely having it in the
OpenStack provider would be nicer. My main issue now is that when I do a
bunch of large uploads, one will often fail and there's no way to increase
the number of retries that I'm aware of. Do you know if there's anyway to
make the uploads more resilient by setting an arbitrary number of retries?

Thanks,
Ben

Post by Kyle Rames
Hi Ben,
The OpenStack provider does have support for large file upload, however,
it requires you to read and upload the file in 5 GB chunks. I have fixed
this issue in the Rackspace provider ( (the Rackspace Cloud is based on
OpenStack).
You can read about the Rackspace fix in this pull request ->
https://github.com/fog/fog/pull/1844
I should have some time to get this fix into OpenStack sometime tomorrow.
Hope that helps.
Kyle

Post by Ben McCann
Hi,
Does the OpenStack storage have any support for large files?
http://docs.openstack.org/**trunk/openstack-object-**
storage/developer/content/**large-object-creation.html<http://docs.openstack.org/trunk/openstack-object-storage/developer/content/large-object-creation.html>
Thanks,
Ben
--

You received this message because you are subscribed to a topic in the
Google Groups "ruby-fog" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ruby-fog/EFlDibChN-o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/groups/opt_out.

[Kyle Rames]

I actually meant to look into this today, but I got pulled off on something
else!

I believe you should be able to specify an etag with your uploaded segment.
If the etag you specified doesn't match the etag OpenStack computes, it
should throw an exception. Let me dig into that a bit more for you.

Kyle

Post by Ben McCann
Great, that's fantastic! I've actually been using the Rackspace provider
with other OpenStack services and it works just fine, so I'm pretty happy
to have it in the Rackspace provider, but definitely having it in the
OpenStack provider would be nicer. My main issue now is that when I do a
bunch of large uploads, one will often fail and there's no way to increase
the number of retries that I'm aware of. Do you know if there's anyway to
make the uploads more resilient by setting an arbitrary number of retries?
Thanks,
Ben

Post by Kyle Rames
Hi Ben,
The OpenStack provider does have support for large file upload, however,
it requires you to read and upload the file in 5 GB chunks. I have fixed
this issue in the Rackspace provider ( (the Rackspace Cloud is based on
OpenStack).
You can read about the Rackspace fix in this pull request ->
https://github.com/fog/fog/pull/1844
I should have some time to get this fix into OpenStack sometime tomorrow.
Hope that helps.
Kyle

Post by Ben McCann
Hi,
Does the OpenStack storage have any support for large files?
http://docs.openstack.org/**trunk/openstack-object-**
storage/developer/content/**large-object-creation.html<http://docs.openstack.org/trunk/openstack-object-storage/developer/content/large-object-creation.html>
Thanks,
Ben
--

You received this message because you are subscribed to a topic in the
Google Groups "ruby-fog" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ruby-fog/EFlDibChN-o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/groups/opt_out.

--
about.me/benmccann

[Kyle Rames]

Hi Ben,

I just submitted a pull request to allow you to use blocks when uploaded
files. Sorry about the delay!

https://github.com/fog/fog/pull/1958

I am also going to look into creating a sample that will help validate
larger uploads.


Kyle

Post by Kyle Rames
I actually meant to look into this today, but I got pulled off on
something else!
I believe you should be able to specify an etag with your uploaded
segment. If the etag you specified doesn't match the etag OpenStack
computes, it should throw an exception. Let me dig into that a bit more for
you.
Kyle

Post by Ben McCann
Great, that's fantastic! I've actually been using the Rackspace provider
with other OpenStack services and it works just fine, so I'm pretty happy
to have it in the Rackspace provider, but definitely having it in the
OpenStack provider would be nicer. My main issue now is that when I do a
bunch of large uploads, one will often fail and there's no way to increase
the number of retries that I'm aware of. Do you know if there's anyway to
make the uploads more resilient by setting an arbitrary number of retries?
Thanks,
Ben

Post by Kyle Rames
Hi Ben,
The OpenStack provider does have support for large file upload, however,
it requires you to read and upload the file in 5 GB chunks. I have fixed
this issue in the Rackspace provider ( (the Rackspace Cloud is based on
OpenStack).
You can read about the Rackspace fix in this pull request ->
https://github.com/fog/fog/pull/1844
I should have some time to get this fix into OpenStack sometime tomorrow.
Hope that helps.
Kyle

Post by Ben McCann
Hi,
Does the OpenStack storage have any support for large files?
http://docs.openstack.org/**trunk/openstack-object-**
storage/developer/content/**large-object-creation.html<http://docs.openstack.org/trunk/openstack-object-storage/developer/content/large-object-creation.html>
Thanks,
Ben
--

You received this message because you are subscribed to a topic in the
Google Groups "ruby-fog" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ruby-fog/EFlDibChN-o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/groups/opt_out.

--
about.me/benmccann

[Ben McCann]

That's awesome! No need to be sorry, that's a great turnaround :-)

I saw that you and and some others were working on making RackSpace based
off OpenStack (here <https://github.com/fog/fog/issues/1733> and
here<https://github.com/fog/fog/pull/1884>).
I'd love to see these efforts make their way in as well as that looks like
a great idea to me.

-Ben

Post by Kyle Rames
Hi Ben,
I just submitted a pull request to allow you to use blocks when uploaded
files. Sorry about the delay!
https://github.com/fog/fog/pull/1958
I am also going to look into creating a sample that will help validate
larger uploads.
Kyle

Post by Kyle Rames
I actually meant to look into this today, but I got pulled off on
something else!
I believe you should be able to specify an etag with your uploaded
segment. If the etag you specified doesn't match the etag OpenStack
computes, it should throw an exception. Let me dig into that a bit more for
you.
Kyle

Post by Ben McCann
Great, that's fantastic! I've actually been using the Rackspace provider
with other OpenStack services and it works just fine, so I'm pretty happy
to have it in the Rackspace provider, but definitely having it in the
OpenStack provider would be nicer. My main issue now is that when I do a
bunch of large uploads, one will often fail and there's no way to increase
the number of retries that I'm aware of. Do you know if there's anyway to
make the uploads more resilient by setting an arbitrary number of retries?
Thanks,
Ben

Post by Kyle Rames
Hi Ben,
The OpenStack provider does have support for large file upload,
however, it requires you to read and upload the file in 5 GB chunks. I have
fixed this issue in the Rackspace provider ( (the Rackspace Cloud is based
on OpenStack).
You can read about the Rackspace fix in this pull request ->
https://github.com/fog/fog/**pull/1844<https://github.com/fog/fog/pull/1844>
I should have some time to get this fix into OpenStack sometime tomorrow.
Hope that helps.
Kyle

Post by Ben McCann
Hi,
Does the OpenStack storage have any support for large files?
http://docs.openstack.org/**trun**k/openstack-object-**storage/**
developer/content/**large-**object-creation.html<http://docs.openstack.org/trunk/openstack-object-storage/developer/content/large-object-creation.html>
Thanks,
Ben
--

You received this message because you are subscribed to a topic in the
Google Groups "ruby-fog" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/ruby-fog/EFlDibChN-o/**unsubscribe<https://groups.google.com/d/topic/ruby-fog/EFlDibChN-o/unsubscribe>
.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
.

--
about.me/benmccann

--

You received this message because you are subscribed to a topic in the
Google Groups "ruby-fog" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ruby-fog/EFlDibChN-o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/groups/opt_out.