fix: Use target tar toolchain for cross-platform execution

[sfc-gh-dbieliaiev]

Hi @thesayyn, @alexeagle.

I'd like to help solve the cross-platform remote execution issue described in bazel-contrib/rules_oci#769, which significantly affects us. I see this is connected to the fundamental Bazel limitation described in bazelbuild/bazel#19645.

I've noticed that a similar issue was addressed for crane and yq toolchains in bazel-contrib/rules_oci#590, but the same transition-based solution for tar toolchain (bazel-contrib/rules_oci#762) got stuck because it doesn't work well in other cases.

Proposal

I'm suggesting to use target_toolchain which was recently implemented instead of toolchain to determine it correctly. As a prerequisite we need to migrate to tar.bzl.

Changes summary

1. tar.bzl was used instead of aspect_bazel_lib/tar.bzl
2. target toolchain was used instead of toolchainl.

[aspect-workflows]

Test

All tests were cache hits

49 tests (100.0%) were fully cached saving 31s.

[thesayyn]

I'd like to fix these issues as well, i just don't see a good way going forward. I thought toolchain attribute (exec groups) was going to solve some of these issues and i'd like to give that a try.

[sfc-gh-dbieliaiev]

@thesayyn Do you have objections against using target toolchain and more in favor of going with toolchain exec groups attribute? I'm not exactly sure how's it going to work(don't have an experience with execution groups) so it might be more elegant solution but if I understand correctly nobody is sure that it's going to work.

On the other side target toolchain is a proven solution(although is not fancy), that might be a good idea to implement as a short term mitigation of this issue.

Also I encourage @alexeagle and @fmeum to join discussion here as there were comments in related PR with opinions that this solution makes sense.

[fmeum]

oci_load generates a script that should run on the target platform and needs a tar binary that it can execute at runtime. That is exactly what a toolchain with target_compatible_with constraints and no exec_compatible_with constraints provides. This needs a separate toolchain type because it has legitimately different resolution behavior than a toolchain used to run tar during rule execution.

There are no correct fancy alternatives. If we started again from scratch, a simpler alternative would have been to only offer toolchains with target_compatible_with constraints as those can be reused by build rules via the resolved alias pattern, but this doesn't work the other way round.

[sfc-gh-dbieliaiev]

@fmeum @thesayyn @alexeagle I'm not sure where the migration to tar.bzl is being tracked, but I did some work related to this in scope of this PR.
I expect some tests to be fixed after rebase from #842 but some tests are related to tar.bzl/bazel 6 support and I'm not sure what way did you decide to go, to support bazel 6 there or to drop support in some major version of rules_oci.