[Core] Add default Ray Node labels at Node init

[ryanaoleary]

Why are these changes needed?

This PR adds support for populating several default Ray node labels (described here) in the Ray runtime environment when a node is initialized. This change will help support autoscaling with the Label Selector API. This PR is related to ray-project/kuberay#3699 which passes several environment variables from the K8s stack which are used to set ray.io/ labels. I'll leave a comment on this PR with manual tests showing the ray.io/accelerator-type and ray.io/availability-zone labels getting set.

Related issue number

#51564

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ryanaoleary]

cc: @MengjinYan

[MengjinYan]

The logic change looks good to me! Just one comment about testing.

[MengjinYan]

@edoakes @jjyao Wondering if you can help take a look and merge the change to get default set the default labels for ray nodes? Thanks!

[edoakes]

CI tests failures @ryanaoleary

[ryanaoleary]

CI tests failures @ryanaoleary

@edoakes Think I fixed the CI failure with bd51036. test_exit_logging was failing because the calls to AcceleratorManager in Node init would add the Traceback to the driver log, when none was expected. I updated the test to patch calls to _get_default_ray_node_labels so that these logs could be ignored for the test. An alternative would be to update calls like this in each AcceleratorManager definition to log but not add to the traceback. I also added a try: except block when we make calls to AcceleratorManager in the new code to ensure Node init doesn't fail for whatever reason when adding the default labels.

[edoakes]

The solution looks worrisome/hacky. What causes the accelerator manager calls to fail on startup? We might be breaking some kind of assumption there.

@jjyao PTAL at the usage of accelerator manager here

[ryanaoleary]

The solution looks worrisome/hacky. What causes the accelerator manager calls to fail on startup? We might be breaking some kind of assumption there.

@jjyao PTAL at the usage of accelerator manager here

It'd raise an exception here due to missing dependencies such as ModuleNotFoundError: No module named 'acl' for NPU. Detecting the accelerator type would fail, but node startup would still succeed as expected. I added the try: except just to be safe since the code is on a critical path, but maybe it's not needed, since it's not used in other places AcceleratorManager is used. Maybe a better solution is just to logger.debug when we see a ModuleNotFoundError rather than raise an exception.

[ryanaoleary]

@MengjinYan rebased and re-pushed since there were some changes to resource_spec.py in master which gets deleted in this PR. I made sure the same change is included in this PR (changed import for from ray._common.usage import usage_lib in node.py to match).