SFTPFailure : Failure when opening files on remote and when transferring

[lborcard]

I am getting SFTPFailure error when using sftp.open() (used to do to checksums) and sftp.get(). I simply use this to transfer data and sftp.get() worked before but is now throwing an error. I do not know what I can provide you as infos since there is nothing

Do you know what could be causing and if it would be possible to get a more exhaustive error?

here is a snapshot of the last debug message before the SFTPFailure during sftp.read():

2025-10-27 13:35:15,356 [DEBUG] [conn=0, chan=0] Sending fstat for handle 000003fa
2025-10-27 13:35:15,356 [DEBUG] [conn=0, chan=0] Received handle 000003fb
2025-10-27 13:35:15,356 [DEBUG] Calculating MD5 for remote file: file_244.fastq.gz
2025-10-27 13:35:15,356 [DEBUG] [conn=0, chan=0] Sending fstat for handle 000003fb
2025-10-27 13:35:15,357 [DEBUG] [conn=0, chan=0] Received handle 000003fc
2025-10-27 13:35:15,357 [DEBUG] Calculating MD5 for remote file: file.gz
2025-10-27 13:35:15,357 [DEBUG] [conn=0, chan=0] Sending fstat for handle 000003fc

best,

Loïc

[lborcard]

and when using sftp.get() i am getting this log in debug mode:

2025-10-27 13:44:30,274 [DEBUG] [conn=0, chan=0] Received type: directory, size: 4096, uid: 1000, gid: 1000, permissions: 40775, atime: Mon Oct 27 09:08:16 2025, mtime: Wed Oct 22 12:05:50 2025
2025-10-27 13:44:30,275 [INFO] [conn=0, chan=0]   Starting copy of directory /home/dir2/
2025-10-27 13:44:30,275 [DEBUG] [conn=0, chan=0] Sending opendir for /home/dir2
 Failure to transfer dir2 to /home/local/dir2 : Failure

[ronf]

Can you provide some example code where you are seeing this problem, particularly related to the MD5 calculation?

In general, I would expect your application to either call sftp.open() followed by a loop calling sftp.read() or for your application to call sftp.get() to download the file and then call open() and read() on the resulting local file.

Also, the pathnames don't look right to me here. It looks like you are trying to copy starting from the path /home/dir2, without any kind of username on the remote file path. Also, does /home/local exist locally, and would the application have write access to that?

Finally, since you are trying to copy a directory, are you setting recurse to True?

[lborcard]

Yes recurse is set to true and the directories paths were changed for privacy reasons.
this is the part doing the md5 checksums. The list of files is given and are processed with a create_task and gather(*tasks).

  
                try:
                    remote_file = await sftp.open(file_path, "rb")
                    logging.debug(f"Calculating MD5 for remote file: {file_path}")
                    md5_hash = hashlib.md5()
                    try:

                        byte_block = await remote_file.read()
                        while byte_block:
                            md5_hash.update(byte_block)
                            try:
                                byte_block = await remote_file.read()
                            except ValueError as e:
                                logging.error(f'Error while reading bytes blocks: {e}')
                    except ValueError as e:
                        logging.error(f'Error while reading bytes blocks:{e}')
                        raise
                    
                    return md5_hash.hexdigest(), Path(file_path).relative_to("/data/")
                except SFTPFailure as r:
                    logging.error(f'Error during (SFTP) md5 checksums on remote: {r}')

                    return None,None
                except Exception as e:
                    logging.error(f'Unexpected error during remote MD5 calculation for {file_path}: {e}')
                    return None, None

[lborcard]

It seems that the SFTPFailure returning upon failure caused it to stop the process . I initially wanted to stop when it fails but I did not expect to have SFTPFailure most of the time I do the checksums. In fact, this only happens with certain folder and not all of them which is also very confusing about this. My main issue now is that the sftp.get() is not working since it's never showed any issues before.

[ronf]

Is it possible you are running into permission errors, where some of the directories aren't readable to the user you are logging in as on the SSH target host? That could cause SFTPFailure to come back from the SFTP server. The same would go for there being files within the directories which the user didn't have read access to. I would expect errors like that to return SFTPPermissionDenied, though, and not SFTPFailure. That said, I've seen a lot of SFTP servers that don't do a very good job returning errors, and the protocol itself doesn't really let you provide any kind of descriptive text in the error when the error occurs on the server side. You'd have to try and look at logs on the SFTP server to get more details.

Is there any chance you are passing a directory into your function to calculate MD5 checksums? Trying to call sftp.open() on a directory will work, but if you try to call read() on that, you'll get back an SFTPFailure.

As an aside, calling read() with no length around is going to load the entire file into memory at once on the client. Since you have a loop calling read(), you might want to add a byte count argument to that, so it fetches the data a block at a time, if you have any really large files. It's faster to let SFTP download all the data at once, but you need to make sure none of the files are larger than available memory on the client if you do that. A good compromise would be to limit it to reads which are something like a few megabytes in size. This is large enough to get the benefit of the SFTP parallel fetching, while minimizing memory use.

Also, I'm not sure the ValueError exception handlers are quite right. That would only ever be returned by hashlib and not by the SFTP read, so you might want to limit the scope of the try..except to the top-level try block, or something more specific ahead of that when you instantiate the md5 object.

[lborcard]

Thank you for the details answer, I have explored the permission error route but did not find any differences between the files. Thanks for pointing out the value error exception handler, and I was indeed attempting to read dirs.

I am still getting the Failure during transfer with sftp.get() and it seems that if I get it once during the transfer then all operations are getting Failure too (e.g. sftp.listdir() ).

[lborcard]

Something else that I noticed is that it usually fails when there are many files to read or transfer ,<1000 is ok but ~20000 is usually causing problems.

[ronf]

Are you currently opening only a single SFTP session on each SSH connection and then using that SFTP session for all your file transfers to/from a given target host, or are there cases where you are opening multiple SFTP sessions on a connection? If you are opening more than one session, keep in mind that most SSH servers limit the max sessions allowed on a connection. I think the default is around 100. You shouldn't need more than one SFTP session per connection even if you want to do multiple transfers in parallel, but I thought I'd mention this just in case.

Can you provide a debug trace showing one of the sftp.get() failures, and what messages came before it? Also, are you able to get debug logs off of the server? That might provide more useful information than just the SFTPFailure result.

[ronf]

One more thought: If you are opening multiple remote files at once on either the server or the client side, you could be running into a limit on max open file descriptors in the OS. On some systems, this can be as small at 256 if you don't adjust it, and I think may Linux distros limit you to something like 1024 by default. So, that would align with your "<1000 is ok". You can raise this limit with ulimit, but you'd probably need to do this on both the client side and the server side (meaning the SSHD process) for it to help.

[lborcard]

Here is what I see before the failure of sftp.get()
await sftp.get(remote_path,local_path,recurse=True,block_size=32*1024*1024,max_requests=30,progress_handler=self.progress_handler)
it seems that get with recursive=True is more problematic, at least with the non recursive version I can limit the requests with a semaphore. I changed max_requests to 30 but it did not help.

/
2025-11-04 09:48:47,739 [INFO] [conn=0, chan=0] Starting SFTP get of /data//to /storage/tmp/ES/LOIC_TMP_NOBACKUP/my_dir
2025-11-04 09:48:47,740 [DEBUG] [conn=0, chan=0] Sending lstat for /data/my_dir
2025-11-04 09:48:47,744 [DEBUG] [conn=0, chan=0] Received type: directory, size: 4096, uid: 1000, gid: 1000, permissions: 40775, atime: Tue Nov  4 09:22:39 2025, mtime: Thu Oct 30 09:15:33 2025
2025-11-04 09:48:47,746 [INFO] [conn=0, chan=0]   Starting copy of directory /data/my_dir to /storage/tmp/ES/LOIC_TMP_NOBACKUP/my_dir/my_dir
2025-11-04 09:48:47,749 [DEBUG] [conn=0, chan=0] Sending opendir for /data/my_dir
2025-11-04 09:50:48,957 [ERROR] Failure to transfer /data/my_dir to /storage/tmp/ES/LOIC_TMP_NOBACKUP/my_dir : Failure`

[ronf]

Is there any chance that there might be a "loop" in your directory structure due to a symlink? I'm not sure I have any protection against that when follow_symlinks is set (which it is by default). It's possible that's causing the sftp.get() to infinitely recurse, eventually exhausting all the file descriptors on the server.

Other than that, the code currently copies a single file at a time, with the only parallelism being when copying different ranges of the file (if the file is big enough). It would also keep open handles for all parent directories, but unless you have a very deep nested directory structure that generally shouldn't cause a problem. It even uses scandir() rather than listdir() to more efficiently handle directories with a very large number of files, avoiding the need to read the entire directory contents in memory.

You could try running with follow_symlinks=False to see if that makes any difference.

If that's not it, I still think you should look for ways in which the server might be exhausting all its available file descriptors. One way to do that would be to manually start the SSH daemon on the server from a shell with a higher ulimit on file descriptors. You can run sshd with the "-D" option to keep it in the foreground, and maybe add "-ddd" as well to crank up its logging. You'll need to temporarily shut off the daemon version of sshd first, or alternatively you could set up a custom config file for it to run on an alternate port and then temporarily change your client to match.