ray-project/ray

Description With Kubernetes token auth (RAY_ENABLE_K8S_TOKEN_AUTH=true, which KubeRay sets for K8s auth mode), workers started through the image_uri or container runtime env probably can't authenticate. KubeRay mounts a projected service account token at /var/run/secrets/ray.io/serviceaccount/token in the Ray container, and the token loader reads it from there. The runtime env plugin in python/ray/_private/runtime_env/image_uri.py forwards RAY_AUTH_MODE and RAY_ENABLE_K8S_TOKEN_AUTH into the podman worker container, but only mounts the Ray temp dir, so the worker has no token. There may be other files need to be mounted, so contributor should mount those files too. This comes from reading the code and hasn't been reproduced. Contributors should verify it on a KubeRay cluster with K8s token auth enabled first, then decide on a fix, for example mounting the token directory into the container. The kubelet rotates projected tokens, so a fix should keep working after rotation. Related: #664

What happened + What you expected to happen We hit this while starting Ray workers in 1-CPU containers. A .pth hook imported torch unconditionally in every Python process, delaying dashboard/metrics agent startup. The agents did not write their port files within the fixed 15-second WaitForPersistedPort timeout. The raylet then exited through RCHECK in NodeManager::WaitForDashboardAgentPorts, before registering with GCS: (raylet) node_manager.cc:3557: An unexpected system state has occurred. ... Check failed: ... Timed out waiting for file /tmp/ray/session_<session>/metrics_agent_port_<node_id> ray::raylet::NodeManager::WaitForDashboardAgentPorts() ray start did not include this error in its output. In our instrumented production attempt, it failed about 12.5 seconds later when the separate 30-second get_node wait in node.py expired: Exception: The current node timed out during startup. This could happen because some of the raylet failed to startup or the GCS has become overloaded.

Description register_nixl_memory's cache is keyed on the storage's data_ptr(). If a tensor's storage is freed and a later allocation reuses the same address but with a larger size, the cache sees a hit on the pointer and skips re-registration. NIXL still only has the old, smaller extent [P, P+S1) registered. A subsequent transfer of that tensor will fail with NIXL_ERR_NOT_FOUND, since NIXL requires the full transfer range to be covered by a single registered region. Use case On register_nixl_memory(tensor), when the cache has a hit on data_ptr(), also compare the previously-registered size against the current storage's nbytes(): Sizes match → current behavior (cache hit, skip registration). Sizes differ → deregister the old registration and register the new extent as if fresh. Add a docstring/comment on register_nixl_memory making explicit that resizing a preregistered buffer in place is inefficient and user should prefer allocating a big buffer size.

What happened + What you expected to happen Description Ray 2.55.1 passes the keyword argument storage_options_provider to PyLance write APIs, but PyLance removed that argument lance-format/lance#6186 and replaced it with namespace_client / table_id. As a result Dataset.write_lance() fails with newer versions of lance: TypeError: write_fragments() got an unexpected keyword argument 'storage_options_provider' Reproduction import ray ds = ray.data.from_items([{"id": 1, "text": "a"}, {"id": 2, "text": "b"}]) ds.write_lance("/tmp/ray-lance-test") Actual: ray.exceptions.RayTaskError(TypeError): ray::Write() TypeError: write_fragments() got an unexpected keyword argument 'storage_options_provider' Expected: the dataset is written, or Ray fails fast with a clear version-compatibility error. Root cause In /ray/data/_internal/datasource/lance_datasink.py the argument is passed unconditionally, even when no namespace is configured and the provider is None (lines 130-150): storage_options_prov

Currently operations on (reads) a Databricks Unity Catalog on GCP doesn't work with the Catalog abstraction for Delta tables. The reason for this is because internally we use the delta-rs library to write to delta tables, and delta-rs relies on an older version of object_store which doesm't support google oauth bearer tokens (the latest version does). We should upgrade our delta-rs dependency when they release a new version and update code as necessary. Issue on delta-rs: delta-io/delta-rs#4566