Topic: Retry if error

I am looking to switch to plupload from swfupload.  I am uploading massive video files to Amazon S3.  I would like to break the files into chunks.  However, if there is an error, such as IO error due to bad connection, I'd like a way to retry the failed chunk X number of times.  Otherwise I don't see that much use in having chunks in the first place.

Can this be achieved with plupload?

Thanks!

Re: Retry if error

The current implementation doesn't have support for retrying the upload of chunks. We added chunk support so that the server doesn't time out or block to large uploads mainly PHP. But I guess it could be used as a reliability feature as well.

Re: Retry if error

Thanks for the response.  Yeah, auto retry would be killer for me.  My users are uploading videos up to 2gb and failures are so common.  Right now I use swfupload and post directly to amazon's s3 servers.  Failures are so common dude it's ridiculous and is making me look bad.  I have put together some form of band aid this week but the right fix is chunking with retries. 

I am still debating whether to modify swfupload, fork plupload, or do it myself.  What do you think?

Re: Retry if error

I would fork plupload since it uses more technologies than swfupload and it also have basic chunking built in. Chunking is also kind of possible in the latest Chrome HTML5 runtime so in the future it will be available for Safari since it's in the WebKit trunk. But I haven't had the time to develop this feature and since it's currently in a beta browser it's no rush.

Re: Retry if error

Steve,

Did you fork Plupload to add the retry feature? Could you point me towards an alternate solution?

Thanks

Re: Retry if error

Gears can do this very easily, HTML5 File API doesn't work yet. I've tried implementing this myself before switching to PLUpload, in FF4 beta, but the Blob.slice methods combined with FileReader and XHR don't play nice, the images get all fubar when put back together on the server.

Also, it clocks up your memory with all the chunks right away: FileReader works async, so it'll continue looping over the file while FileReader isn't done yet. That way a lot of FileReader (and thus XHR and Blob) instances are created, eating memory like it's ice cream on a summer day. Your users won't like that.

Gears works great though. Try this: http://code.google.com/p/google-ajax-ex … l.js?r=183
Features chunks, retries, pausing, etc.

Re: Retry if error

Thanks for example. Silverlight is also a good candidate for such usage scenario. Flash is closer to HTML5 in that it does load the whole thing in memory, chunking or not. They do not support reading from the file stream. Or actually they do, but only for AIR.

If you want to see your issue fixed, do not report it here, do it on - GitHub.

Re: Retry if error

Spocke wrote:

The current implementation doesn't have support for retrying the upload of chunks. We added chunk support so that the server doesn't time out or block to large uploads mainly PHP. But I guess it could be used as a reliability feature as well.

Has this changed, or is there a way to retry a whole file on error? I noticed that the logic I'm using includes files with errors. How would I make plupload try them again?

uploader.bind('StateChanged', function() {
     if (uploader.files.length === (uploader.total.uploaded + uploader.total.failed)) {
         $('#plupload_form')[0].submit();
     }
});

Last edited by 7liter (2012-06-01 19:55:32)

Re: Retry if error

Spocke wrote:

Has this changed, or is there a way to retry a whole file on error? I noticed that the logic I'm using includes files with errors. How would I make plupload try them again?

+1

Re: Retry if error

ecentinela wrote:
Spocke wrote:

Has this changed, or is there a way to retry a whole file on error? I noticed that the logic I'm using includes files with errors. How would I make plupload try them again?

+1

wondering the same thing

Re: Retry if error

This is going to change with Plupload 2. In previous version it wasn't possible due to variety of reasons. So, hold on smile

If you want to see your issue fixed, do not report it here, do it on - GitHub.

Re: Retry if error

That's good news! Any details on how this would be solved using Plupload 2? Is there an example somewhere?

Is it enough the change the file status to "plupload.QUEUED" and restart the queue?

Last edited by Thomas Jaggi (2013-01-03 22:26:09)

Re: Retry if error

Thomas Jaggi wrote:

Is it enough the change the file status to "plupload.QUEUED" and restart the queue?

In Plupload 2 - yes. That's the way currently.

If you want to see your issue fixed, do not report it here, do it on - GitHub.

Re: Retry if error

davit wrote:

In Plupload 2 - yes. That's the way currently.

I'm actually trying to implement a retry-mechanism in plupload 2 based on your recommendation, but sadly I can't find any way to get this to work:

uploader.bind("FileUploaded", function(up, file, result){
    // parse result and check for error, if so, requeue file
    up.stop();
    file.status = plupload.QUEUED;
    up.start();
});

If I bind my FileUploaded function prior to the uploader.init() method, the file.status will be overwritten in pluploader.js own binding afterwards to file.status = plupload.DONE and the uploader continues to upload the next file.

In case I bind my FileUploaded function after the initialization method, I have no chance in preventing the uploader from continuing with the next file.

I'm currently using JSON responses to return the upload response from the server to the client which does not trigger an HTTP Error on the client side. I think these are intended for "real" HTTP errors.

Or should the file be cloned somehow in case of an error and appended to the queue? But I couldn't find any code which could confirm this assumption.

It would be great if you could point me into the right direction. - Thanks!

Re: Retry if error

@Marc Fritsche, if you need to restart upload with from specific file, rather them simply re-queueing failed on, try something like this:

uploader.bind("FileUploaded", function(up, file, result){
    // parse result and check for error, if so, requeue file
    up.stop();
    file.status = plupload.UPLOADING;
    up.state = plupload.STARTED;
    up.trigger("StateChanged");
    up.trigger('UploadFile', file);
});

This should work in both versions, btw. Bit clunky, I got to give it some good thinking, taking into account new possibilities.

If you want to see your issue fixed, do not report it here, do it on - GitHub.

Re: Retry if error

Thanks for your help. I finally managed to get it up and running. However there was an additional change I had to made in order to get it properly running. I introduced a property "retry" on the file to prevent the plupload.js binding from messing with the retry mechanism handled within my code.

plupload.js (version 1.5.5, around line 1175)

self.bind("FileUploaded", function(up, file) {
    if (!file.retry) {
        file.status = plupload.DONE;
        file.loaded = file.size;
        up.trigger('UploadProgress', file);

        // Upload next file but detach it from the error event
        // since other custom listeners might want to stop the queue
        delay(function() {
            uploadNext.call(self);
        }, 1);
    }
});

And here is my solution for the retry mechanism:

uploader.bind('FileUploaded', function(up, file, response) {
    var error = false; // parse response and check for error, if so, requeue file
    if (error) {
        // we try to upload the file three times
        if (file.attempt >= 3) {
            // too many attempts, stop uploader, cancel this file and display an error
            up.stop();
            file.status = plupload.FAILED;
            file.retry = false;
            
            // display error for file…

            // ... start the uploader again to process the next files
            up.state = plupload.STARTED;
            up.trigger("StateChanged");

        } else {
            // retry to upload file
            up.stop();
            file.status = plupload.UPLOADING;
            file.attempt++;
            file.retry = true;

            // ... start the uploader to process the current file again
            up.state = plupload.STARTED;
            up.trigger("StateChanged");
            up.trigger('UploadFile', file);
        }
    } else {
        file.retry = false;
        // show uploaded file
    }
});

Re: Retry if error

The traditional approach is to use return codes. The open method returns some specific value to say it failed. This value is then propagated back through the layers of calling routines until someone wants to take responsibility for it.