Skip to content

Figure out which file uploads are in an error state and need to be restarted. #1

@crspybits

Description

@crspybits

I'll focus this on uploads for now, but eventually need to consider (a) upload deletions, and (b) downloads.

Currently UploadFileTracker's can have three states:

    enum Status : String {
        case notStarted
        case uploading
        
        // This is for both successfully uploaded files and files that cannot be uploaded due to a gone response. For vN files this just means the first stage of the upload has completed. The full deferred upload hasn't necessarily completed yet.
        case uploaded
    }

In general, multiple files can be uploaded together (see UploadObjectTracker), so I need to consider the possible values for the Status for each of these uploads:

  1. all notStarted
    Current TriggerUploads will handle

  2. all uploading
    Expected normal state. No action needed.

  3. all uploaded
    Should be transient and only happen just before removing all trackers.

  4. some notStarted, some uploading
    Error state. Current TriggerUploads will not handle. Deal with here. Perhaps wait until .uploading state changes?

  5. some notStarted, some uploaded
    Error state. Current TriggerUploads will not handle. Deal with here.

  6. some uploading, some uploaded
    Expected normal state. No action needed.

  7. some notStarted, some uploading, some uploaded
    Error state. Current TriggerUploads will not handle. Deal with here. Perhaps wait until .uploading state changes?

Do we need to keep track of the amount of time a file is in an .uploading state? If that is too long, we may need to deal with it as an error case.

Possible algorithm:

  1. Get the fileGroupUUID for any UploadObjectTracker with a uploading state file.
  2. We can re-trigger the upload for any upload in an error state where it's object is not one of those objects currently uploading.
  3. It seems like this is the time we want to consider the time of times an upload has been tried/retried. We should ony try a specific number of times before giving up.

How to test? We need some way to reset the state of an upload to .notStarted. Might be able to do that in a test just after learning its upload was successful.

What happens if the same upload is carried out more than once? That is: How does the server respond? Let's analyze the cases, below.

1 out of 1 cases

1.) 1 out of 1 upload for v0, no change resolver

1.1) Server never received upload (due to error):
RESULT: Re-attempted upload works.

1.2) Server received upload: (*)
RESULT: [FAIL]
This will be received as a second upload for the same file. It will fail.
How to handle? Server could possibly report some error code analogous to "gone" e.g., "duplicate". But, how to know error vs. duplicate?

2.) 1 out of 1 upload for v0, with change resolver

2.1) Server never received upload (due to error):
RESULT: Re-attempted upload works.

2.2) Server received upload:
RESULT: [FAIL]
The v0 upload will contain v0 only data, such as change resolver, mime type etc. It will fail. This case is really the same as (*) above.

3.) 1 out of 1 upload for vN

3.1) Server never received upload (due to error):
RESULT: Re-attempted upload works.

3.2) Server received upload:
This is handled normally, and applied as a change to the file.
Since we require re-applying of the same change to be handled without error, this is OK.

N out of M cases

1.) N out of M uploads for v0-- only one file is .notStarted on client; the rest were successfully uploaded.

1.1) Server never received upload (due to error):
RESULT: Re-attempted upload works.

1.2) Server received upload:
RESULT: The uploads would have been processed and (possibly) flushed from the Upload table.
NEEDS TESTING:
What happens if the deferred uploading hasn't finished yet and the re-upload arrives before the Uploads flushed?
If the uploads have been flushed, then this is a case where the upload would just linger. We would never get the deferred upload id in response to a "last upload.

Possibilities:
A) Server responds with deferred id after each upload in batch.
That id could be used to query.
B) Client queries state of uploads.
C) Client queries to get deferred upload id's. These are specific to sharing group and file group, and user. We could also tag them with some user supplied info such as a client side date.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions