ipfs::unixfs::cat prefetching or prefetching in general

[koivunej]

A larger issue: when for example cat'ing an UnixFs file the slowest possible way is the one we are currently doing:

1. fetch a block
2. yield file contents
3. inspect it's links
4. select next applicable link, goto 1

A faster way would be to go ahead of time fetch more blocks but of course the issue is not so clear. It might be a good idea to study how go-ipfs and js-ipfs have solved this issue.

WIP code in https://github.com/koivunej/rust-ipfs/tree/refs_to_ipfs_streaming_input. In the branch I made Ipfs::refs take in a Stream of IpfsPath, which I then used to start refs for the pending links in different ways. Going totally unbounded traversals was of course fastest, probably limited by the capacity of how much tokio's stdout adapter could push. Did not immediatedly figure out any wiser strategy, even though I played around with starting to prefetch only when the list of links started contracting, or doing it in reverse so to download the beginning and the end at the same time.

[koivunej]

Related and possible cause:

rust-ipfs/src/p2p/behaviour.rs

Lines 456 to 462 in 347bf8a

	// FIXME: it would be best if get_providers is called only in case the already connected
	// peers don't have it
	pub fn want_block(&mut self, cid: Cid) {
	let key = cid.to_bytes();
	self.kademlia.get_providers(key.into());
	self.bitswap.want_block(cid, 1);
	}

Testing the prefetching without starting any queries (just commenting the self.kademlia.get_providers line out) could be a proper next step.