CVE-2023-47108: DoS vulnerability in otelgrpc (uncontrolled resource consumption) due to unbound cardinality metrics
Summary
The grpc Unary Server Interceptor opentelemetry-go-contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go
// UnaryServerInterceptor returns a grpc.UnaryServerInterceptor suitable // for use in a grpc.NewServer call. func UnaryServerInterceptor(opts ...Option) grpc.UnaryServerInterceptor { out of the box adds labels
- net.peer.sock.addr - net.peer.sock.port
that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent.
Details
An attacker can easily flood the peer address and port for requests.
PoC
Apply the attached patch to the example and run the client multiple times. Observe how each request will create a unique histogram and how the memory consumption increases during it. Impact
In order to be affected, the program has to configure a metrics pipeline, use UnaryServerInterceptor, and does not filter any client IP address and ports via middleware or proxies, etc.
Others
It is similar to already reported vulnerabilities.
GHSA-rcjv-mgp8-qvmr (open-telemetry/opentelemetry-go-contrib) - GHSA-5r5m-65gx-7vrh (open-telemetry/opentelemetry-go-contrib) - GHSA-cg3q-j54f-5p7p (prometheus/clientgolang)
Workaround for affected versions
As a workaround to stop being affected, a view removing the attributes can be used.
The other possibility is to disable grpc metrics instrumentation by passing otelgrpc.WithMeterProvider option with noop.NewMeterProvider.
Solution provided by upgrading
In PR #4322, to be released with v0.46.0, the attributes were removed.
References
- #4322
Other sources
OpenTelemetry-Go Contrib is a collection of third-party packages for OpenTelemetry-Go. Prior to version 0.46.0, the grpc Unary Server Interceptor out of the box adds labels net.peer.sock.addr and net.peer.sock.port that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent. An attacker can easily flood the peer address and port for requests. Version 0.46.0 contains a fix for this issue. As a workaround to stop being affected, a view removing the attributes can be used. The other possibility is to disable grpc metrics instrumentation by passing otelgrpc.WithMeterProvider option with noop.NewMeterProvider.
OpenTelemetry-Go Contrib is a collection of third-party packages for OpenTelemetry-Go. Starting in version 0.37.0 and prior to version 0.46.0, the grpc Unary Server Interceptor out of the box adds labels net.peer.sock.addr and net.peer.sock.port that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent. An attacker can easily flood the peer address and port for requests. Version 0.46.0 contains a fix for this issue. As a workaround to stop being affected, a view removing the attributes can be used. The other possibility is to disable grpc metrics instrumentation by passing otelgrpc.WithMeterProvider option with noop.NewMeterProvider.
— MITRE
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
go/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpcto a version that resolves this vulnerability.Fixed in 0.46.0 - Upgrade
Upgrade
redhat/opentelemetry-go-contribto a version that resolves this vulnerability.Fixed in 0.46.0 - Configuration
Configure a metrics view to remove/drop the net.peer.sock.addr and net.peer.sock.port attributes so they are not recorded (use a view that excludes these attributes).
otelgrpc metrics instrumentation metrics_view_drop_attributes = net.peer.sock.addr, net.peer.sock.port - Configuration
Disable grpc metrics instrumentation by passing otelgrpc.WithMeterProvider(noop.NewMeterProvider()) when creating the interceptor.
otelgrpc MeterProvider = noop.NewMeterProvider - Compensating control
Filter or normalize client IP addresses and ports at middleware or proxies (e.g., drop/aggregate or restrict peer address/port) to prevent unbounded cardinality of metrics labels.
Event History
Frequently Asked Questions
What is CVE-2023-47108?
CVE-2023-47108 is a Denial-of-Service (DoS) vulnerability in otelgrpc due to uncontrolled resource consumption.
How severe is CVE-2023-47108?
CVE-2023-47108 has a severity rating of high with a CVSS score of 7.5.
What is the affected software?
The affected software is go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc version up to exclusive 0.46.0.
How do I fix CVE-2023-47108?
To fix CVE-2023-47108, upgrade the otelgrpc package to version 0.46.0 or later.
What is the Common Weakness Enumeration (CWE) for CVE-2023-47108?
The CWE for CVE-2023-47108 is CWE-770 (Allocation of Resources Without Limits or Throttling).