Skip to content

Conversation

@gulsumgudukbay
Copy link
Collaborator

@gulsumgudukbay gulsumgudukbay commented Oct 14, 2025

Description

Enable running the test suite completely offline (no gs://, no gcloud, no Vertex AI) when the environment variable: DECOUPLE_GCLOUD=TRUE

Key Environment Variables
DECOUPLE_GCLOUD=TRUE: Primary toggle for offline/"decouple from gcloud" mode.
LOCAL_BASE_OUTPUT: Optional override for local run/output directory (defaulted to decoupled_datasets/gcloud_decoupled_test_logs in most tests).
LOCAL_GCLOUD_PROJECT: Placeholder to satisfy code paths expecting a project (set in CI).

A minimal in-repo dataset directory (i.e. decoupled_datasets) is used instead of:
gs://maxtext-dataset (TFDS / ArrayRecord)
gs://maxtext-dataset/hf/... (Parquet)

In decoupled mode:

  • All tests and helper scripts redirect base_output_directory to decoupled_datasets/gcloud_decoupled_test_logs and modify input dataset accesses to decoupled_datasets if DECOUPLE_GCLOUD=TRUE.
  • vertex_tensorboard no-op in decoupled mode.
  • max_utils.get_project() returns local placeholder without gcloud invocation when decoupled.
  • Decoupling script added (decouple.py) with stubs for JetStream, Tunix, and diagnostics, so that if we are in decoupled mode, these stubs are used and GCE-dependent imports are cancelled earlier.
  • Serving/JetStream/vLLM entrypoints fail-fast when decoupled so that training continues with safe fallbacks.
  • Pytest markers created (external_serving, external_training, tpu_only). If a test has any of these markers, it is marked as decoupled, so that the end user can run the GCE-independent tests using -m decoupled marker.

Tests

In progress...

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed.

@gulsumgudukbay gulsumgudukbay deleted the decouple_gcloud branch October 27, 2025 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants