grafana/alloy

Component(s) loki.source.docker What's wrong? Stopping a container deletes its read position, so the next time that container starts, Alloy re-reads its log from the beginning and re-sends the whole retained history to Loki. The container does not need to be removed — an ordinary docker stop is enough. When a container leaves discovery while the component is still running, tailer.stop() removes its cursor: // internal/component/loki/source/docker/tailer.go:176-179 (v1.16.1) // If the component is not stopping, then it means that the target for this component is // gone and that we should clear the entry from the positions file. if !t.componentStopping() { t.positions.Remove(positions.CursorKey(t.containerID), t.labelsStr) } componentStopping is func() bool { return c.exited.Load() } (docker.go:241) — it reports whether Alloy's component is shutting down, and nothing else. It cannot distinguish "the container was removed", where discarding the cursor is right, from "the container wa

What's missing The chart's templates/service.yaml renders internalTrafficPolicy from .Values.service.internalTrafficPolicy, but there is no way to set spec.trafficDistribution on the Service. For a DaemonSet-mode Alloy acting as a node-local OTLP receiver, that leaves two unsatisfactory options: internalTrafficPolicy: Cluster (the default) — every export is load-balanced across all DaemonSet pods, so telemetry usually leaves the node that produced it. internalTrafficPolicy: Local — node-local, but with no fallback. While a node's Alloy pod is unready — notably during any DaemonSet rollout — that node's telemetry is dropped rather than routed elsewhere. What would fix it spec.trafficDistribution: PreferSameNode prefers an endpoint on the caller's own node and falls back when there isn't one, which is exactly the semantics a node-local collector wants. It is GA as of Kubernetes 1.35 (alpha in 1.33, beta and enabled by default in 1.34). See KEP-3015, kubernetes/enhancements#3015

Component(s) loki.enrich Request prometheus.enrich now supports multi-label matching with #5822 using the target_to_metric_match parameter. Could this feature be added to loki.enrich using a target_to_log_match parameter as well? Use case Would align these two components on feature set and configuration parameters. AI disclosure This issue was substantially generated with AI assistance. Tip No response

Summary Every component that uses the shared positions package (loki.source.kubernetes, loki.source.file, loki.source.docker, loki.source.podlogs, …) rewrites its positions file every 10 seconds through renameio.WriteFile. renameio always suffixes its temp file with a fresh random number, so each write creates a filename that has never been seen before. Under cgroup v2 the kernel memory for those dentries is charged to the container and counts toward memory.max. The result is a linear, unbounded growth of slab_reclaimable while the Go heap and RSS stay completely flat — so it is invisible to kubectl top, to RSS-based dashboards, and to Alloy's own go_memstats_* metrics. In our production EKS cluster this reached 90% of the container memory limit in 52 days and fired our OOM-risk alert. Evidence Alloy v1.17.0, DaemonSet, EKS / Amazon Linux 2023, kernel 6.12.x, cgroup v2, loki.source.kubernetes. The Go heap is identical between a growing pod and a flat pod — only kernel memory diff

Component(s) prometheus.operator.servicemonitors Request I'd like to request support for Prometheus operator scrape classes. https://prometheus-operator.dev/docs/developer/scrapeclass/ https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/proposals/implemented/202305-scrapeclasses.md prometheus-operator/prometheus-operator#6199 This is the suggested alternative for setting bearerTokenFile on ServiceMonitor resources. Use case From the ScrapeClass documentation: ScrapeClass is a feature that allows you to define common configuration settings to be applied across all scrape resources( PodMonitor, ServiceMonitor, ScrapeConfig and Probe ). This feature is similar to StorageClass in Kubernetes and is very useful when it comes to standardising configurations such as common relabelling rules, TLS certificates and authentication. In Alloy this is needed to support the scrapeClass parameter of collected PodMonitor, ServiceMonitor, ScrapeConfig and Probe resource