Self-hosted observability stack for Ruby on Rails apps

This blog post introduces the opentelemetry-rails-example repository - a demo of fully instrumented Rails app with a self-hosted observability stack.

OpenTelemetry has done remarkable work on providing specification of integrated telemetry data as well as on delivering the tooling - instrumentation SDKs for various languages and the OpenTelemetry Collector for data ingestion. However, by design, it doesn’t specify how the data is persisted and accessed. This area is mainly owned by vendors - cloud observability providers who compete in making use of the gathered data by providing all sorts of visualizations and alerting.

Yet, application owners may consider an entirely in-house approach. There are many reasons why one may want to self-host their observability stack. The motivation could be that they want to avoid a $65B bill from the observability tooling provider, or they want to have full control over how and what data is collected, or simply their policy disallows sending the data to a third-party provider.

Regardless of the motivation, I believe it’s worthwhile to know what can be achieved using generally available software that can be self-hosted, as opposed to leveraging observability cloud providers.

I was fortunate to work on implementing observability in numerous Ruby (and Rails) apps from scratch. It involves connecting quite a few moving parts - starting from the application instrumentation, through data ingestion and storage to building up dashboards and alerts - it may be a struggle. As an afterthought, I distilled a minimum stack that gives very powerful insights into the system’s performance.

For details, check out this repository: -> opentelemetry-rails-example <-

The example stack covers three main layers of the system observability:

• instrumentation of the app (using the OpenTelemetry SDK, but not only);
• ingestion and storage of the observability data (featuring the OpenTelemetry Collector, StatsD, Prometheus, Promtail and Loki);
• accessing the observability data - seamless navigation between logs, metrics and traces with Grafana

Of course, it all comes with costs. Production deployments may need tuning and sampling to limit the infrastructure burden. Business-wise, with an in-house stack, the luxury of logs, metrics, and traces doesn’t have to connote an enormous bill from a third-party observability provider.

By adapting the in-house approach development teams may benefit from an integrated instrumention while having a better understanding of data and a better control over the costs. I believe it’s worth the effort - debugging a system with traversable logs, metrics, and traces is mentally easier and more efficient, especially when the time is critical.