How can I disable FlashAttention (to avoid GLIBC_2.32 requirement)?

[nariaki3551]

I’m training with VERL (agentlightning.verl) and hit an ImportError because the prebuilt FlashAttention wheel requires GLIBC_2.32, while my host glibc is older. I don’t need FlashAttention for now — I just want to run with it disabled.

• verl : 0.4.1
• flash-attn : 2.8.3

  File "/home/tateiwa/webserf-playground/.venv/lib/python3.10/site-packages/agentlightning/verl/entrypoint.py", line 12, in main
    run_ppo(config)
  File "/home/tateiwa/webserf-playground/.venv/lib/python3.10/site-packages/agentlightning/verl/entrypoint.py", line 26, in run_ppo
    ray.get(runner.run.remote(config))
  File "/home/tateiwa/webserf-playground/.venv/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/home/tateiwa/webserf-playground/.venv/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
  File "/home/tateiwa/webserf-playground/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 2882, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/home/tateiwa/webserf-playground/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 968, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ImportError): ray::TaskRunner.run() (pid=2732978, ip=10.130.255.217, actor_id=e7ef749a1d20c651cdda278c01000000, repr=<agentlightning.verl.entrypoint.TaskRunner object at 0x7f2ab8ae7790>)
  File "/home/tateiwa/webserf-playground/.venv/lib/python3.10/site-packages/agentlightning/verl/entrypoint.py", line 151, in run
    trainer.init_workers()
  File "/tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/verl/trainer/ppo/ray_trainer.py", line 793, in init_workers
    self.actor_rollout_wg.init_model()
  File "/tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/verl/single_controller/ray/base.py", line 51, in __call__
    output = ray.get(output)
ray.exceptions.RayTaskError(ImportError): ray::WorkerDict.actor_rollout_init_model() (pid=2734353, ip=10.130.255.217, actor_id=192201e60d5fbcd080f77fce01000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7fe5bd70fb50>)
  File "/home/tateiwa/.pyenv/versions/3.10.8/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/tateiwa/.pyenv/versions/3.10.8/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/verl/single_controller/ray/base.py", line 663, in func
    return getattr(self.worker_dict[key], name)(*args, **kwargs)
  File "/tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/verl/single_controller/base/decorator.py", line 540, in inner
    return func(*args, **kwargs)
  File "/tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/verl/workers/fsdp_workers.py", line 498, in init_model
    from verl.workers.actor import DataParallelPPOActor
  File "/tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/verl/workers/actor/__init__.py", line 16, in <module>
    from .dp_actor import DataParallelPPOActor
  File "/tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/verl/workers/actor/dp_actor.py", line 42, in <module>
    from flash_attn.bert_padding import index_first_axis, pad_input, rearrange, unpad_input
  File "/tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/flash_attn/__init__.py", line 3, in <module>
    from flash_attn.flash_attn_interface import (
  File "/tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 15, in <module>
    import flash_attn_2_cuda as flash_attn_gpu
ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/ray/session_2025-10-14_12-27-54_779556_2728634/runtime_resources/working_dir_files/_ray_pkg_0cde68e4ed4ead2f/.venv/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so)

[Hans-M-Yin]

hi. there are 2 solutions.
(1) downgrade flash-attn to 2.7.4.post1 or 2.7.3. (i tests with GLIBC 2.31)
(2) Using polyfill-glibc to help old-version device install flash-attn 2.8.x. Following guide in Dao-AILab/flash-attention#1708 (comment).