flyteorg/flyte

Why InternalRunService.UpdateActionStatus reports failures in the response body, not as a transport error. The handler always returns a nil error and puts the failure in Status: // runs/service/internal_run_service.go:204 func (s *RunService) UpdateActionStatus(...) (*connect.Response[workflow.UpdateActionStatusResponse], error) { resp := s.updateActionStatus(ctx, req.Msg) // errors become resp.Status via statusFromError return connect.NewResponse(resp), nil // <- always nil } Its only caller ignores that field and keys off the transport error alone: // actions/k8s/client.go:840 if _, err := c.runClient.UpdateActionStatus(ctx, connect.NewRequest(statusReq)); err != nil { logger.Warnf(ctx, "Failed to update action status in run service for %s: %v", update.ActionID.Name, err) } else if isTerminalPhase(update.Phase) && !update.IsDeleted { c.markTerminalStatusRecorded(ctx, taskAction) // runs even when the run service rejected the update } So when the run service fails to re

Issue When injecting a secret into task pod like this: env = flyte.TaskEnvironment( name="hello_world", resources=flyte.Resources(cpu=1, memory="1Gi"), image=image, secrets=[ flyte.Secret(key="my_secret", as_env_var="MY_SECRET_KEY"), ] ) When getting pod spec with kubectl describe command, we will directly see the secret value in ENVVAR, which is not secure: Suggestion We use webhook to inject secret into task pod. We should insert EnvVarSource instead of EnvVar value directly

Why A PostgreSQL NOTIFY in the runs-service is a wakeup hint — the payload carries only an action identifier, and every listener re-reads state from the database when it wakes. Losing one is recoverable. But today, sending one can block a write RPC indefinitely. notifyActionUpdate does a blocking channel send: select { case r.actionNotifyCh <- payload: case <-ctx.Done(): logger.Errorf(ctx, "Action NOTIFY send cancelled for %s: %v", payload, ctx.Err()) } The channel is buffered at 256 and drained by a single goroutine that round-trips to Postgres once per payload — a ceiling of roughly 240 notifications/second (measured: 4.1 ms per pg_notify round trip from the same VPC, versus 32 µs for the pg_notify call itself). When demand exceeds that, the buffer fills and the send blocks — inside the RPC handler, because UpdateActionPhase calls it right after the row is written. What is not at risk: the database write Worth stating up front, because it is the first question everyone ask

Why The runs-service sends one PostgreSQL NOTIFY per action update, and each one costs a full network round trip on a single dedicated connection. That serial round trip — not Postgres, not CPU — is what limits action-update throughput today. Measured on a dev cluster during a 100,000-action benchmark: RPC rate p50 p95 mean InternalRunService.UpdateActionStatus 101.8/s 394 ms 3085 ms 780 ms EventsProxyService.Record 25.6/s 653 ms 2418 ms 801 ms InternalRunService.RecordActionEvents 25.6/s 652 ms 2418 ms 800 ms ActionsService.Enqueue 57.5/s — 1351 ms 375 ms InternalRunService.RecordAction 52.9/s 326 ms 588 ms 288 ms The mean is 780 ms, so this is the whole distribution being slow, not a tail. And it is not the usual suspects: Not Postgres — RDS WriteLatency 0.9 ms, CPU 58–72%, and only 14 database connections open against a pool max of 100. Every connection sat in ClientRead, i.e. the database was waiting on the application. Not CPU — the pod used 1.34 of

Summary There's a typo in one of our protobuf fields: funtion_name should be function_name (missing the c). It lives on the ActionMetadata message and flows through the generated code in every language, the Go backend, and the published flyteidl2 package that the Python SDK depends on. This spans two repos: most of the work is in flyteorg/flyte (Parts 1–3), plus a small follow-up in flyteorg/flyte-sdk (Part 4). It's totally fine to do Part 4 as a second PR — see the note there. Background The field is defined here: flyteidl2/workflow/run_definition.proto (around line 188) message ActionMetadata { ... // Function name string funtion_name = 13; // 👈 typo: "funtion" → "function" ... } The = 13 is the field's wire number — we keep it as 13 so this change is fully backward-compatible on the wire. We're only fixing the human-readable name. What needs to change 1. Fix the proto (the only file you edit by hand for the IDL) — repo: flyteorg/flyte In flyteidl2/workflow/r