Topic: Large files > 2GB

Hi,
How is the support for > 2GB ?
I know flash has 32bit issues and Google gears on OSX has 32 bit issues.

I have not tested Silverlight or Browserplus, does anyone have any experience ?

Ben

Re: Large files > 2GB

I think most runtimes should work if you use the chunking feature. HTML5 doesn't work since it will allocate the whole 2GB file into memory before uploading.

Re: Large files > 2GB

Google gears does not work on OSX due to 32bit ness, even with slicing /chunking
Flash was very 32 bit as well for file access ( has this changed ? )
Silverlight have not checked
Browserplus have not checked
HTML5 does not have slice / chunking /Blob  ( yet fingers crossed ) in any of the implementation yet. http://dev.w3.org/2006/webapi/FileAPI/#dfn-Blob

Re: Large files > 2GB

OK I have been doing some testing, only on OS X ( snow leopard 10.6.2 ) so far and I thought I would share my results, I know this is not pluploads fault.

I have downloaded the example from the green button above to the right.

I have a file called biglump (~4GiB)
4246405272  biglump.mov
Using the example file examples/queue_widget.html ( with the slight change to the size of the chunks, increased from 1MB to 40MB )

PHP 5.3.0 (cli) (built: Jul 19 2009 00:34:29)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2009 Zend Technologies

Firefox version 3.5.8, with gears (version 0.5.36.0) , Silverlight 3 (3.0.50106.0),

Gears stops as expected at 51% ( internal error caused unsigned int / long problem )
Silverlight stops at 50 % ( guess similar issue ) do not know that much about c#, I am guessing the plugin is not 64 bit clean ( will look into this )
HTML5 eats the machine as Firefox will try and read the whole file into memory
HTML4 nothing appears to happen ( no data transfer )
Browserplus reports the filesize as -48562024, which I am guessing is not a good sign :-)

Safari 4 Version 4.0.5 (6531.22.7)
Flash reports the file size as -48562024 b
Gears does not work ( not supported on safari 4)
Silverlight  Stops with error at 50 % see firefox errors
Browserplus reports the file size as -48562024 b
HTML5 looks like it might work and does not chew through all of your memory
HTML4  looks like it might work and does not chew through all of your memory no progress though


Firefox Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6
Flash reports the file size as -48562024 b
Gears as expected at 51% ( internal error caused unsigned int / long problem )
Silverlight stops at 50 % see errors abov
Browserplus reports the file size as -48562024 b
HTML5 eats the machine as Firefox will try and read the whole file into memory
HTML4 not tested


Much as I hate to say it, how come there is no Java runtime ?

Ben

Re: Large files > 2GB

Hmm, such huge file sizes seems to break their internal numbers. I guess they are int based not long based. We have tested the app with large files like 200MB and no issues there.

6 (edited by ben_roeder 2010-03-25 23:41:34)

Re: Large files > 2GB

Yes 200 MB files will, I was looking for something to upload much larger files, in a cross platform unified way :-(
I have had success with java, is there a reason why you do not have a java runtime ?

Re: Large files > 2GB

Is there any plans to increase the compatibility of Plupload with >200MB files?

I've been testing an implementation of this where users regularly upload 1GB+ files, and have been having problems with Flash and HTML5 uploading anything bigger then a 200MB file.

Re: Large files > 2GB

Our hands are tied in HTML5 since it loads the whole file into memory there is no chunking support. However WebKit has support in their trunk now so we might support it soon.

You should be able to upload pretty large files with Flash, Gears and Silverlight since we load the files into smaller chunks. And I've tested them with files larger than 200MB. But as stated earlier in this thread files larger than 4GB will result in int overflows.

We have been thinking about making a Java runtime as well. But we are currently unsure if we can make a Java applet transparent on top of a button as we can for the other runtimes. So if someone with more up to date Java experience would like to help us answer questions like that we could start sooner. Last time I wrote a Java applet was back in the nineties.

Re: Large files > 2GB

Hi Spocke,

I was wondering if you have done more than just think about a Java runtime to support the large file uploads?

kind regards,
Jakob

Re: Large files > 2GB

Hi,

i am using mollify and mollify is using plupload. Plupload is a really nice project. But to be honest i think its going into the totally wrong direction. I have even set up plupload myself just to test the upload problems with big files myself to not blame mollify or not to make any config errors. I am using the flash uploader because it is the only one that is available everywhere. The flash uploader has severe problems with big files and we are not speaking about 2GB. 1.9GB reliably uploads would really be a joy. I am speaking about 400MB.
The biggest problem is that the uploader uploads the same file fine sometimes and sometimes not. It does not even provide an error message. It sits there and nothing happens. You implement more and more very nice features but the basic feature, to upload a big file using chunking to circumvent server side problems (php.ini) does not always work reliable. It is quite clear that "big" means something else for different people. You have a lot of features and a lot of upload modules and have done a lot of great work.
The last test i made before i gave up was to split up an 900MB file into 200MB files and upload them together. The first one has been uploaded like a charm but plupload stucked at the second. I know that some people were able to upload quite big file. I was able too. 600MB no problem. I have tested it 3 times. The first time it failed it faild on an normal user.

Re: Large files > 2GB

Hi all,

I've been working on a Java runtime for unlimited sized resumable streamed uploads using plupload.

The fork is here:
http://github.com/jakobadam/plupload

And the readme file describes how to build the Java applet.

I hope you will try it out and give me some feedback.

kind regards,
Jakob

Re: Large files > 2GB

ben_roeder wrote:

Hi,
How is the support for > 2GB ?
I know flash has 32bit issues and Google gears on OSX has 32 bit issues.

I have not tested Silverlight or Browserplus, does anyone have any experience ?

Ben

I have tested silverlight and it works nicly. But I have not tested Browserplus. For this reason I have no idea about it.

Re: Large files > 2GB

ben_roeder, you've put quite an effort into your testings. I'm going to do some tests here too. Will try to sort that out.

jakobdam! that Java runtime looks very promising! I need to look into it deeper. I think we will contact you very soon. Do you have it running on production site somewhere? I saw you got some examples on github, but I wouldn't mind to see it in real action smile

If you want to see your issue fixed, do not report it here, do it on - GitHub.

Re: Large files > 2GB

We have had similar problems running the flash runtime for files at 1gb with chunking enabled.  It will get up to ~50% and flash will crash.

Hoping we can resolve this, but for now, since this uploader is in an admin, we are going to test this with silverlight.

Re: Large files > 2GB

Huge files and flash will not work until Adobe gets their thumb out and does a better implementation of file upload. The current official FileReference.upload method is full of bugs and doesn't support chunking or any advanced options for that matter. Our workaround that uses UrlStream loads the entire file into RAM before uploading and this is the only way to do it since Flash doesn't have the ability to read portions of a file either.

So if you want to upload huge files like 2GB I recommend using other runtimes than Flash.

Re: Large files > 2GB

I have the same problem.
I relay like this project, but I have problems even at 160 MB video file.
The flash sometimes uploads it good, sometimes stops and the exclamation mark icon is appearing in the uploadlist near the uploading file.
The silverlight the same. Sometimes uploads it, other times crashes the browser.
The html5 runtime sometimes avoid some chunks. For example there are 32 chunks and he takes only 25, and finishes the uploading but the file is not complete.
The main problem with these cases is the volatile nature, they happen just sometimes.
I use only your examples, I have not integrated yet into my system, so I don't think that I missed something.
Could you please give me a point how to proceed, because I like very much this project.
Thanks

Re: Large files > 2GB

I would guess the memory is the problem. Flash and HTML5 will load the file into RAM then upload it.

18 (edited by MikeCa 2011-01-18 18:21:46)

Re: Large files > 2GB

Spocke: Are you certain that html5 will load the entire file into memory first?  I am using html5 in Chrome on Debian Lenny and it does not appear to load the entire file into memory before uploading.

Also, I don't see much discussion around using the 'urlstream_upload = false' feature to tell Flash to not load the entire file.  Is there a reason this won't work? (I understand you need to pass a session cookie to keep auth alive)

Edit: Ok, looks like I misunderstood how to use 'urlstream_upload'.  If it defaults to true, shouldn't this be saving us from Flash reading the entire file at first?  My browser appears to be reading in the whole file, regardless of my 'urlstream_upload' setting.

Re: Large files > 2GB

No, flash will read the whole file in memory anyway. There's currently no way to avoid this. From wide-spread runtimes at the moment only SilverLight does comparably good job. In my tests it easily took on 1.72 gb file.

And we are going to add Java runtime, some time soon.

If you want to see your issue fixed, do not report it here, do it on - GitHub.

Re: Large files > 2GB

I have completed a review of the silverlight code and converted to 64 bit integers and now it supports files > 2gb just like the java applet.

Tested with 5.5gb file.

https://github.com/JProgrammer/plupload

Re: Large files > 2GB

thanks jpprogrammer, I was stumped with which to use for 2gb upward, most HD videos are larger and need the larger capacity, let me know how large you get to, will post when tested myself

[url=http://www.harleystreetplasticsurgery.info]breast augmentation[/url]
[url=http://www.harleystreetplasticsurgery.info]tummy tuck[/url]

Re: Large files > 2GB

I tested with 5.5Gb file that successfully uploaded on a test server locally.

Re: Large files > 2GB

I have uploaded a compiled xap if you can test and let me know how you go.

http://rapidshare.com/files/445765758/p … rlight.xap

Re: Large files > 2GB

Try upgrading your memory and uploading will be smooth.

[url=http://efttappingtechniques.com/world-tapping-summit/2011-tapping-world-summit/]2011 Tapping World Summit[/url]

Re: Large files > 2GB

ben_roeder wrote:

Hi,
How is the support for > 2GB ?
I know flash has 32bit issues and Google gears on OSX has 32 bit issues.

I have not tested Silverlight or Browserplus, does anyone have any experience ?

Ben

MASSIVE .mov file!  How about you compress it to a more usable size?

Josh Rockman
www.RockmansCreativeMedia.com.au