Skip to content

Fix span log oom #773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 6, 2025
Merged

Fix span log oom #773

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
22f14e8
fix:oom due to span log too many
Sep 4, 2025
5e9e53f
fix:add fix log to CHANGES.md
Sep 5, 2025
a17eb60
code style and CHANGES.md fixed
Sep 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Release Notes.
* Bump up cli to the 0.15.0-dev.latest(77b4c49e89c9c000278f44e62729d534f2ec842e) in e2e.
* Bump up apache parent pom to v35.
* Update Maven to 3.6.3 in mvnw.
* Fix OOM due to too many span logs.

All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/242?closed=1)

Expand Down
5 changes: 5 additions & 0 deletions ...niffer/apm-agent-core/src/main/java/org/apache/skywalking/apm/agent/core/conf/Config.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,11 @@ public static class Agent {
*/
public static int TRACE_SEGMENT_REF_LIMIT_PER_SPAN = 500;

/**
* The max number of logs in a single span to keep memory cost estimatable.
*/
public static int LOG_LIMIT_PER_SPAN = 300;

/**
* The max number of spans in a single segment. Through this config item, SkyWalking keep your application
* memory cost estimated.
Expand Down
6 changes: 6 additions & 0 deletions ...src/main/java/org/apache/skywalking/apm/agent/core/context/trace/AbstractTracingSpan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,9 @@ public AbstractTracingSpan log(Throwable t) {
if (!errorOccurred && ServiceManager.INSTANCE.findService(StatusCheckService.class).isError(t)) {
errorOccurred();
}
if (logs.size() >= Config.Agent.LOG_LIMIT_PER_SPAN) {
return this;
}
logs.add(new LogDataEntity.Builder().add(new KeyValuePair("event", "error"))
.add(new KeyValuePair("error.kind", t.getClass().getName()))
.add(new KeyValuePair("message", t.getMessage()))
Expand All @@ -196,6 +199,9 @@ public AbstractTracingSpan log(long timestampMicroseconds, Map<String, ?> fields
if (logs == null) {
logs = new LinkedList<>();
}
if (logs.size() >= Config.Agent.LOG_LIMIT_PER_SPAN) {
return this;
}
LogDataEntity.Builder builder = new LogDataEntity.Builder();
for (Map.Entry<String, ?> entry : fields.entrySet()) {
builder.add(new KeyValuePair(entry.getKey(), entry.getValue().toString()));
Expand Down
3 changes: 3 additions & 0 deletions apm-sniffer/config/agent.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ agent.authentication=${SW_AGENT_AUTHENTICATION:}
# The max number of TraceSegmentRef in a single span to keep memory cost estimatable.
agent.trace_segment_ref_limit_per_span=${SW_TRACE_SEGMENT_LIMIT:500}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No point in adding this extra line.

# The max number of logs in a single span to keep memory cost estimatable.
agent.log_limit_per_span=${SW_LOG_LIMIT_PER_SPAN:500}

# The max amount of spans in a single segment.
# Through this config item, SkyWalking keep your application memory cost estimated.
agent.span_limit_per_segment=${SW_AGENT_SPAN_LIMIT:300}
Expand Down
Loading