Looking for a suggested "default" backend when using OTLP exporters

[svrnm]

More and more languages provide support for metrics & logs, our current documentations for exporters recommend jaeger to verify OTLP for traces, but right now there is no immediate option for logs & metrics. We could suggest that people setup a collector and look at their telemetry with a logging exporter, but it would be nice to have something visual like open-telemetry/oteps#230 suggests.

for reference: #3144 (comment)

[cijothomas]

Our demo uses opensearch for logs. We can use that?
https://github.com/open-telemetry/opentelemetry-demo/tree/main/src/opensearch

And prom/grafana for metrics (demo already use that).

[svrnm]

yeah, we can update for OpenSearch, I think when I created the discussion there was no OTLP yet for OpenSearch?

[cijothomas]

to verify OTLP

if the goal is to just verify that OTLPExporter is setup correctly and it works, then collector with stdout/debug exporter is sufficient IMHO!
If the intent is more than that, there is no single tool today, so a combination of jaeger/opensearch/prom+grafana might be the best bet and is followed in demo already.

[svrnm]

Yes, for pure verficiation it's fine, but what many end-users want is something visual. We anticipate that in our docs already, e.g. here for JS:

https://opentelemetry.io/docs/languages/js/exporters/

By adding OpenSearch we have at least one OTLP endpoint per signal.

[cartermp]

I would say prometheus for metrics, for logs I'm unsure (and unopinionated right now). OpenSearch isn't bad as per @cijothomas's suggestion?

[svrnm]

So, I looked into the OpenSearch documentation and the basic setup is already ... complicated:

https://opensearch.org/docs/latest/quickstart/

If we compare that to jaeger & prometheus where one simply starts an "all-in-one" docker container, this is a lot of effort for quickly taking a look at your metrics.

I have no idea about OpenSearch but maybe there is an equivalent where I can get a single container up and running?

[kamphaus]

There are several Clickhouse based projects that provide combined support for logs+metrics+traces:

• https://github.com/coroot/coroot
• https://github.com/SigNoz/signoz
• https://qryn.metrico.in/

Maybe one of these is a good fit?

[svrnm]

thanks for sharing. All three of them are commercial as well as open source, so this would not work easily because of our promise to be vendor neutral. We had a related discussion around this topic in #5040 as well.

[cartermp]

Yeah, if we were to suggest Clickhouse, it would just be Clickhouse. But that also requires some setup!

[kamphaus]

I’ve learned about Apache SkyWalking in the Linux Foundation Istio course LFS144. It supports Otlp ingestion for metrics, logs and traces.

[svrnm]

thanks for sharing, I was not aware that skywalking supports all signals as of today. Wonder how good that support is? Also, can it be run easily?

[kamphaus]

From its docs, SkyWalking supports tracing, metrics and logging over the OTLP protocol.
I haven't tried any of the projects I listed though (it's on my todo list).

It looks like SkyWalking has docker-compose and helm support so it should be easy to get started.
How well it handles under production load I don't know.

[kamphaus]

Another ClickHouse based observability stack: hyperdx
Since ClickHouse acquired HyperDX they are rebranding it into ClickStack.
https://clickhouse.com/blog/clickstack-a-high-performance-oss-observability-stack-on-clickhouse

[svrnm]

thanks for sharing this @kamphaus !

As you can see this discussion is stalled for a while. Happy to make some progress, if you have ideas on how to do it.

[kamphaus]

I've tried both ClickStack and and Qryn for a personal project.

ClickStack works fine and is easier to setup.
Their all-in-one image might be fine for displaying standalone telemetry: https://clickhouse.com/docs/use-cases/observability/clickstack/deployment/all-in-one
However their docs are lacking when it comes to a production-like setup like how to connect to a clickhouse cluster managed by the clickhouse operator.
Their UI is less familiar for people coming from a LGTM or ELK stack background.
Since it's a newer project there is no library of dashboards or alerts that could be used with it.

Qryn allows easy integration into Grafana for display.
It's more difficult to setup because the qryn helm chart does not include the clickhosue persistence backend.
While the qryn otel-collector could export the otlp stream from the otel-demo application to Clickhouse fine and it was displayable in Grafana. Using other otlp could have issues: Fluentd logs could not be directly imported for display in Grafana: I needed to add a transform in the qryn otel collector of logs to flatten the attributes and dedot the attribute keys so that they were queryable via LogQL.
Other issues I found with Qryn was high latencies in the queries for logs volume and logs themselves. This resulted in flaky results in the Logs Explorer needing to refresh multiple times.

As a conclusion I would not use either for a production setup unless you have a competent DevOps team to handle the drawbacks.

You could checkout the hyperdx all-in-one image to see if it fits the purpose for visualizing some otlp signals in a dev environment.

I won't look into the other projects I've listed above.
If someone else has done so feel free to share your point of view here.

[svrnm]

thanks for taking a closer look into these @kamphaus !