libssh async SFTP project created as a part of GSoC 2023

My role : CONTRIBUTOR

• Project Name : Async SFTP Client
• Mentors :
  • Jakub Jelen : (GitLab Profile Link )
  • Norbert Pocs : (GitLab Profile Link )
  • Sahana Prasad : (GitLab Profile Link )
• Contributor : Eshan Kelkar (GitLab Profile Link )

1. Creating a low level asynchronous API for performing remote reads and writes.
2. Creating a high level file transfer API that uses the low level async API internally.

For the created APIs, the following also had to be written :

• Tests
• Benchmarks (to analyse performance)
• Documentation

In the GSOC community bonding period, some issues were identified in the then existing libssh async SFTP API for reading. Those issues were :

• Incorrect file offset handling in certain scenarios by the async API.
• API user needed to perform cumbersome typecasting of the request ids used by the API.

On discussion with the mentors, it was decided to fix these issues in the existing API before developing the high level file transfer API.

Fixing these issues required changing function return types and parameter types. Since the API was already public, that couldn't have been done without breaking the existing libssh user code.

Hence, it was decided that a new async API had to be created. In future, the users would use the new async API and libssh would deprecate the old async API.

This new async API was named as SFTP AIO (async i/o) and it :

• Solves the above described issues in the old async API.
• Provides the user ability to perform async read/write operations on remote files.

After creating the SFTP AIO API, the SFTP FT (File transfer) API was created. Features of this API :

• Provides functions to perform 4 kinds of transfers :
  • Local to Remote (Upload)
  • Remote to Local (Download)
  • Remote to Remote (Remote copy)
  • Local to Local (Local copy)
• Uses the low level async SFTP AIO API internally to get faster transfers as compared to transfers performed using synchronous SFTP read/write API.
• Makes transfers efficient by minimizing the number of data copies between buffers as much as possible.
• Provides a feature to resume interrupted transfers.
• Provides a progress callback feature to monitor the progress of the transfer. This could be useful for applications that want to show some progress related UI.

Merge requests created during the GSoC period :

1. SFTP AIO API : MR
2. SFTP FT API : MR
3. Windows compatibility layer for file i/o : MR
4. Adding helpers ssh_readn(), ssh_writen() : MR

Merge requests created before the GSOC period (These aren't directly related to the async SFTP project) :

1. Adding support for posix-rename@openssh.com extension : MR
2. Adding support for hardlink@openssh.com extension : MR

What's done ?

• Implementation
• Function documentation
• Tests
• Benchmark code
• Tutorial for the SFTP AIO API

What's left ?

• Adding support for the non-blocking mode in the API.

What's done ?

• Code to perform a local to remote transfer.
• Code to perform a remote to local transfer.
• Adding support for the SFTP "copy-data" extension by introducing sftp_copy_file_range()
• Tests for sftp_copy_file_range()
• Code to perform a remote to remote transfer.
• Public API code
• Function documentation
• Tests
• Benchmark code
• Tutorial for the SFTP FT API

What's left ?

• Code to perform a local to local transfer

• I thank Google for conducting GSoC.
• I thank libssh mentors and community for :
  • Providing well written code, documentation and tutorials.
  • Being responsive and supportive before and during the GSoC period.
  • Accepting my GSoC proposal.
  • Patiently answering my questions.
  • Suggesting useful features that could be provided by the created APIs.
  • Providing suggestions related to API design.
• I thank God.
• I thank my parents, relatives and friends.