Topic: Not working reliably direct to S3 silverlight & flash runtimes

Hi,

After much work & testing to get PLUpload direct to s3 set up I am experiencing a great many failures.

There are no symptoms to be found in Fiddler. It is almost as if no request is sent to S3. This happens without rhyme or reason.

We have tested now on 5 machines in 2 states and the only symptom is the progress will go very quickly from 1% - 100% and then give the exclamation point with an I/O error message.

It is also runtime affected. It almost always fails with silverlight although I get an HTTP 200 to request my client access policy from S3.

It has only worked correctly on one machine for Silverlight using IE9.

The site is SSL enabled if that might make any difference, but the fact it sometimes works on one machine in one browser, then an hour or two later fails, or that switching to flash as the primary runtime will get it working for awhile only to fail again has me baffled.

I use flajaxian so I know it is not Flash as flajaxian performs flawlessly direct to S3.

I had really wanted to use PLUpload and had spent at least 200+ hours getting it to work but I am unable to debug it.

The fact it works sometimes and not others on some machines sometimes and others other times makes it extremely time consuming and frustrating to test as their are no discernible http errors or requests coming out of Fiddler.

I cannot add snapshots here or I would. I am not sure if anyone ever tested this in asp.net c# or from a secure site, but I would say after rigorous coding and testing that this product is not ready for .NET/SSL direct to S3 in my experience.

I can share my source code but other than that I am now prepared to abandon my hopes of using this product unless a solution can be offered.

Thanks very much in advance Davit for reading this.

Best,

Cary

Re: Not working reliably direct to S3 silverlight & flash runtimes

Hi,

I tracked the problem down. It is a compendium of a few variables but I hope this help someone in the future should they ever encounter a similar issue(s).

First I should say even when things work right Silverlight runtime races through the progress %age from 1% to a very high percentage near 95-99% and then just hovers there with no feedback.

With a 150 meg file it could be stuck at say 97% for 40-90 minutes depending upon your connection.

This ONLY happens in Chrome & Firefox. IE9, perhaps Microsoft created Silverlight runs through the % properly. So where Chrome/FF will race to a very high percentage and then appear to stall, even though perfmon shows the upload is running, a user would easily become confused and frustrated.

So I have made Silverlight my second runtime, i.e. 'Flash, Silverlight' instead of 'Silverlight, Flash' as I had hoped to do to bypass the Flash 2GB limit.

Now as to why it was working in some browsers, sometimes is as follows:

The url: attribute of the plugin.

As mentioned my site is SSL.

As of late I had begun to take all my http: & https: references out & just use "//somedomain/somefile.ext".

This works in IE always & in other browsers sometimes.

Since I was ready to give up it suddenly occurred to me to try it again so I set the dev site to go to https://mybucket.s3.amazonaws.com so I wouldn't have to keep remembering to change http to https when I deploy to QA.

Surprisingly it works when I use https://s3.amazonaws.com/mybucket in my dev site (i.e. non-https) but NOT when I use https://mybucket.s3.amazonaws.com, IN SOME BROWSERS.

Anyway there were other permutations tried, and I test in FF, Chrome, IE, Safari & also on Mac but to make a long story short here's what worked:

For https site url should be: https://mybucket.s3.amazonaws.com but again, this only works properly with Flash in my testing, Silverlight will run up to 97% off the bat. Unless you are in IE.

For dev http://mybucket.s3.amazonaws.com, but again, same SL issue.

I have tried many combinations & was not sure it was the URL as there are many variables in my site including JS intervals, IFrames, webservices for policy signing, etc. so while I may not be 100% certain, right now I have 300 Meg uploads running on 2 windows boxes & a macbook in every browser successfully for past 20 minutes without any sign of an issue.

Perhaps Silverlight & Chrome/FF vis a vis S3 have issues that Flash does not.

I long for the day IE9 & Safari support HTML5 uploads as I have a direct to S3 html5 uploader ready the moment my JS features check registers HTML5Upload=TRUE and then my html5 uploader will automagically work in IE9 & Safari and my plugin troubles will be a thing of the past.

At any rate I have to say all my long nights and days spent to get PLUpload working direct to S3, and with JQuery Themeswitching paid off (if this test I'm running succeeds, which I think it will smile)) and I do like the PLUploader very much.

My only wish is out of the box it could calculate an ETA for completion of the job although all the raw materials are there, I'm just too tired of direct to S3 uploading to write any more code for now smile)

Thanks for a great product and I hope this was my last issue.

smile)

Cary Abramoff, happy to be PLUploading direct to the cloud!

Re: Not working reliably direct to S3 silverlight & flash runtimes

Final Post Script: Even with it all sorted out for every 10 files or so I upload one will fail. As this only happens intermittently, with Flash runtime and has never happened in 3 months of testing with Flajaxian, I can only assume that it is not quite as robust for my situation as I had hoped.

Thanks again!

Cary

Re: Not working reliably direct to S3 silverlight & flash runtimes

Cary smile I think you already can start write memoirs about you and plupload (200+ hours is quite a time) smile Ok, joke.

SSL and Plupload currently do not play nice together. Actually in some cases they do, for example Flash can be forced into the compatible mode using urlstream_upload: true, but then you will lose upload progress indication. So it's kinda not a robust solution anyway.

Next major release, which is under heavy development now, will bring support for chunked upload to S3. Until then you should probably use direct upload to S3 with Plupload only to http:// or with relatively small files ( < 5mb).

If you want to see your issue fixed, do not report it here, do it on - GitHub.

Re: Not working reliably direct to S3 silverlight & flash runtimes

Hi Davit,

It's a good joke sir. But I have learned a lot about jquery plugins and many other things.

Here are the final results of an all night upload test with PLUpload Flash.

FireFox 4.0 & 3.6 are not reliable with big or small files. In Mac, Windows Server 2008, & Windows Vista the following occurred.

Vista & 08 Server FF: 2 files uploading 102mb & 193MB.
193MB fails as 2nd file in queue. On Vista it made it 2% and 08 Server 46%.

Same test on Vista/08 succeeded in IE9 on Server 08 & IE8 Vista. Also worth re-mentioning Silverlight works with IE properly on IE. So what is I find quite odd is ever since I began coding Ajax in 2006/2007 IE was usually the culprit with all manner of nasty issues where at the time FF was far superior. Then came Chrome and I also started writing cross-browser/platform sites and got to know Safari. IE became my least loved browser that would make a 200+ hour investments look like a nanosecond. And yet it handles the direct to S3 plupload plugins very well compared to FF.

Further that is unusual as Chrome / Safari use webkit and IE/FF usually seem to be on par when I break down my browser issues. But FF & IE have broken ranks here for sure and FF does not like the plugins.

Further this is my 4th direct to S3 uploader. I also have an html5 direct to s3 upload I spent 200+ hours on, Flajaxian was easily good for 175 hours, and I am currently about to start finishing up integrating the Aurigma direct to S3 uploader.

My uploaders run in floating divs to simulate windows and have to update SQL Server via web services so alot of that time is also spent on integration in all honesty with the larger system.

I say this because Aurigma which uses ActiveX for IE and Java for all other browsers is an interesting case too. Aurigma, I learned chokes on FF for mac in simply rendering itself in a div. On safari mac it flickers and then when floating div is moved it disappears entirely leaving whitespace in its wake.

It turns out this is due to Mac & Java I am told, although I tried to learn Java in 97 and did not like the lack of GUI controls and went to ASP instead, so I am no Java expert. At any rate I say all this because FF now has issues in Mac/Window with PLUpload, and Aurigma uploader.

The HTML5 uploader works perfectly always, and the flajaxian code works perfectly always. The flajaxian code is very old and is not maintained although it is free. I was able to migrate it to .net 4.0 and I had tried to learn flash and modify it but I like Flash coding far less than Java smile)

The flajaxian code however from a Javascript/C#/ and what I could learn in 2 weeks of Flash is very well coded from a best practices/efficiency point of view but again it is years old now in the middle of 2011.

I think the solution is many solutions if you support many browsers, at least until HTML5 File API is supported by IE & Safari.

I hope this all helps someone else. My asp.net code that works is available if people look through my now 28 posts.

So this is it until the next release.

Can you give me an indication when it is to be released as the site I am coding is still 3 months away from completion of development. There's time if it can work in Firefox!

If not I will most like detect FF and then not render the button to launch PLUpload FF but will most likely leave it available in Safari, IE & Chrome.

By the way again, size does not matter with FF. It can happen anywhere in the queue at any size. And in a large queue if it fails on one it can keep going and successfully upload the rest.

I would say whoever could really successfully get multipart uploading with resumability, the way the old napster used to work, would own the S3 cloud which I think now has going on 2 billion items uploaded.

That makes McDonald's look pretty bad considering it's only a few years old.

Sayonara, Buenos Noches, Arrivadurci, Peace & Good Tidings to those who dare to direct upload to S3 and save their EC2 instances from the bottleneck of the millennia!

Cary Abramoff
smile)

Re: Not working reliably direct to S3 silverlight & flash runtimes

Cary, do you have your failing test case online somewhere? I would like to take a look.

Next Plupload should come out within next 3 months - that's the plan at least smile

If you want to see your issue fixed, do not report it here, do it on - GitHub.

Re: Not working reliably direct to S3 silverlight & flash runtimes

Hi Davit,

Three months sounds good as my site will not be live for apx. 3-6 months.

Unfortunately, our site won't be available to anyone else to play with until we go into official beta testing in several months sad(

I'll supply a link when we are in beta and you can see how we've implemented PLUpload. Perhaps some of your forum members will even be interested in being beta testers as our site is a generic cloud productivity suite (file storage, and alot of other stuff I cannot talk about smile))

A happy note though, after my final changes our newest test box which is a new dell 64 bit box with a very fast connection had the following results:

excerpts QA #11:

IE - Main and SS succeeded without issue.  All 11 attachments made it into the item and the PLupload window stayed put.

FF and Chrome (tabbed SS) - uploaded perfect but this morning the SS tabs must have reloaded as the tab now showed the main page - indicating a refresh event.  clicking on the paperclip showed all uploads presents

Safari - SS test perfect - upload window still active, all files in

Safari Main - had an error - it timed out.  see snap.  BUT - the PL up window indicated 11/11 uploaded.  click off the error message and the main window showed 10 fles in the att tab container.  click the paper clip to wake it up and the 11th file popped into the attachment tab and all were working fine (needed a refresh for the 11th bad boy)

SO - end result - 66/66 files made it to the items  And the fastest upload completed by 11:11 and the slowest by 11:15 - hows that for consistency.


PS- Each test had 10 3-6 mb music files & 6 100+MB zip files. Thus each test was over 1/2 a gig of direct to S3 uploads.


Thanks again & I look forward to your next release. Hopefully it will "plug & play!"

Cary

Re: Not working reliably direct to S3 silverlight & flash runtimes

Quick Postscript to PLUpload & S3.
Tonight my partner & I spun up plupload on all 4 major browsers on a total of 8 machines in 2 offices.

this included mac, netbooks, laptops, servers, dell vista boxes, etc.

I am awaiting his results but in main development office as my 10 down/ 100 up dedicated DSL circuit got tapped out PLUpload was the first casualty of war in every instance.

Flajaxian could slow to a crawl as could html5 without dropping files and failing.

PLUpload invariably hits a bump and drops the files.

These are all 80+ meg files. The test is with about 30 gigs worth of files.

My partner who uses comcast in Florida gets much better results as of yesterday but we never tried a full bore upload test using every uploader at once with the same payload.

My experience tells me something is happening inside its inner mechanism making it less fault tolerant than my other 2 uploaders.

Again this is with SSL for whoever may read this and it is going straight to the cloud.

If the next version were more fault tolerant it would be awesome indeed.

However the odds of my customers trying a test like this are very slim indeed but this is the big test I've been waiting months to run.

By the by, using perfmon to monitor network, cpu & physical memory resources reveals that it really all boils down to the connection you are running more than anything. If the connection is blazing with mad bandwidth to spare even a relatively weak machine will get the files uploaded.

And I do believe this is my 30th & last post till the next version.

I really do believe this uploader has massive promise for its next version and I can say after 4 months of relentless direct to S3 due diligence that direct to s3 uploaders are a (well I cannot say it without offending those more sensitive souls) smile)

Peace in the multiverse!

Cary