Observability

Observability Stack Overview

Our observability infrastructure comprises various tools designed to capture and analyze telemetry data, ensuring the health and performance of our systems. The stack is structured around three primary data types: traces, logs, and metrics.

Metrics

Metrics collection utilizes the Otel Collector along with a suite of exporters included within our Platform Data Collection (PDC) framework. This includes tools like the Node Exporter, MongoDB Exporter, and cAdvisor, among others. Together, they provide a comprehensive view of our system's performance and usage statistics.

Traces

For tracing, we employ the OpenTelemetry (Otel) Collector, which facilitates capturing and managing trace data across our distributed systems. This component is key for understanding request lifecycles and inter-service dependencies.

Logs

Log aggregation and management are handled by Fluent Bit. This lightweight data processor is part of our future work plans to enhance log analysis and storage capabilities. Stay tuned for updates in this area.

cAdvisor (short for container Advisor) analyzes and exposes resource usage and performance data from running containers. cAdvisor exposes Prometheus metrics out of the box.

cAdvisor is not enabled by default.

To enable cAdvisor:

Navigate to deployment folder:

cd
cd /opt/pentaho/pdc-docker-deployment/vendor
sudo nano docker-compose.mom.yml

Scroll down to the mon-cadvisor: section
Uncomment all the lines:

Save.

CTRL + o
ENTER
CTRL + x

Note the profile: mon_enhanced

Enable the OTEL collector to scrap the logs.

Navigate to deployment folder:

cd
cd /opt/pentaho/pdc-docker-deployment/vendor/mon/otel_col
sudo nano otelcol-config.yml

Uncomment cadvisor.

Save.

CTRL + o
ENTER
CTRL + x

COMPOSE_PROFILES

Ensure the profile has been enabled

Navigate to deployment folder:

cd
cd /opt/pentaho/pdc-docker-deployment/vendor
sudo nano .env.default

Add mon_enhanced

Save.

CTRL + o
ENTER
CTRL + x

You will need to restart PDC to deploy the cAdvisor container.

cd
cd /opt/pentaho/pdc-docker-deployment
./pdc.sh restart

Check cAdvisor container is up and running.

docker ps -n 1

OpenObserve is a comprehensive observability platform designed to provide insights into the health and performance of IT systems. At its core, OpenObserve integrates seamlessly with a suite of tools for collecting metrics, traces, and logs— the three primary pillars of observability.

This section is for reference only.

OpenObserve has been enabled by default.

Edit the .env.default

cd
cd /opt/pentaho/pdc-docker-deployment/vendor
sudo nano .env.default

Check the following parameter has been added.

COMPOSE_PROFILES=mongodb,collab,pdso,mon_enhanced

Save.

CTRL + o
ENTER
CTRL + x

You will need to restart PDC to deploy the container.

cd
cd /opt/pentaho/pdc-docker-deployment
./pdc.sh up

OpenObserve

Open your web browser and navigate to the following URL: http://localhost/internal/openobserve/web
Log in using the credentials provided below:

Username

[email protected]

Password

Complexpass123

Ensure that the organisation on the top right of the home page has been set to: pdc

Logs

OpenObserve provides a centralized log management interface that allows users to easily search, filter, and analyze log data from various sources. This aids in troubleshooting issues and understanding the system's behavior over time.

Select: Logs from the left-hand menu.

You can filter logs by specific fields or keywords and visualize the log events in chronological order. This enables quick identification of patterns and anomalies in the log data.

As you type the SELECT statement, you will be prompted ..!

Ensure your in SQL mode and type in the following Query.

SELECT * FROM "default" WHERE container_name='/pdc-mongodb-1' AND attr_principalname='root'

A simplier method is to select the field/value from the list

Delete the query and select the required fields.

This will also give us an idea of the number of records.

Metrics

Each exporter utilizes a specific naming convention for its metrics, facilitating the identification of their sources.

For instance, metrics from the node exporter, responsible for server-specific data, start with node_*, while metrics collected by cAdvisor, which targets metrics from all Docker containers, begin with container_*

Select: Metrics from the left-hand menu.

In this example: node_memory_MemAvailable_bytes

The screenshot displays the total memory (bytes) available for the last 4 days.

Choose any metric and confirm that:

A time series graph can be produced for any metric stored in Prometheus
Metric values are being recorded in real time
The PromQL editor can be used to drill down on metrics values with a specific parameter.

Traces are one of its key components of OpenObserve.

Traces help you understand the flow of requests across multiple services in a distributed system. This is crucial for identifying bottlenecks and optimizing performance.

By linking errors to specific traces, you can quickly identify the root cause of issues and the context in which they occurred.

Traces reveal how different services interact, helping you understand and manage dependencies in your system.

With trace data, you can identify areas for potential optimization, such as reducing unnecessary API calls or improving database queries.

Enable Tracing

Log into OpenObserve and select Ingestion -> Traces (Open Telementry)

Make a note of the OLTP gRPC settings (you will have a different Auth token)

endpoint: localhost
  Authorization: "Basic cm9vdEBleGFtcGxlLmNvbTpKeFJHYTh6UEFSZUZUSUlt"
  organization: pdc
  stream-name: default
tls:
  insecure: false

Edit the following OTEL configuration file.

cd
sudo nano /opt/pentaho/pdc-docker-deployment/vendor/mon/otelcol.config.yml

Restart PDC

cd
cd /opt/pentaho/pdc-docker-deployment
./pdc.sh restart

Searching for a Specific Trace

To locate a specific trace by its traceID, you can refine your search by editing the query in the query editor. Use the field name trace_id to direct your search to a particular trace.

On the left side of the page, you'll find a list of field names that assist in filtering traces. For instance, to explore traces originating from the front end of the PDC, you can input the following query into the editor:

str_match(service_name, 'pdc-web-client')

This query retrieves all traces associated with the pdc-web-client service, allowing for a focused analysis of front-end activities.

Query Functions

OpenObserve supports a variety of functions to manipulate and analyze data effectively. These functions can be used within queries to perform operations like aggregations, calculations, and transformations on collected metrics, traces, and logs.

Aggregation Functions: Functions such as SUM(), AVG(), and COUNT() allow for the aggregation of data points over a specified interval.

Transformation Functions: Functions like TOPK(), PERCENTILE(), and RATE() help in transforming raw data into useful insights.

Math Functions: Basic arithmetic functions (+, -, *, /) can be applied to metrics for custom calculations.

String Functions: Functions such as str_match() and str_replace() aid in manipulating text-based log and trace data.

Aggregation Functions: Functions such as SUM(), AVG(), and COUNT() allow for the aggregation of data points over a specified interval.

Transformation Functions: Functions like TOPK(), PERCENTILE(), and RATE() help in transforming raw data into useful insights.

Math Functions: Basic arithmetic functions (+, -, *, /) can be applied to metrics for custom calculations.

String Functions: Functions such as str_match() and str_replace() aid in manipulating text-based log and trace data.

Combining these functions within queries can help in deriving meaningful and actionable insights from your telemetry data.

VRL Functions

OpenObserve integrates VRL (Vector Remap Language) for complex data transformations and log manipulations. VRL functions provide a powerful way to convert, enrich, and process logs, metrics, and traces with ease.

Basic Functions: Use functions like parse_json(), to_string(), and to_int() for basic data type conversions.

Conditional Functions: Implement conditional logic with if, else, and case statements.

String Functions: Manipulate text data using upcase(), downcase(), trim(), and substring() functions.

Log Enrichment: Enrich logs with metadata or additional context using functions like add_field() and merge().

As a simple example we're going to add a timestamp to the logs.

Let's take a look at the different fields in the default log stream.

Click on a row and select table to view the fields.

We're going to add a formatted_date field to the logs to help with selecting range - dates in our dashboard.

The following VRL function will add a field: formatted_date to the logs.

# Create a timestamp (current time)
current_timestamp = now()

# Format the current timestamp into a standard date format
.formatted_date, err = format_timestamp(current_timestamp, "%Y-%m-%d %H:%M:%S")

# Output the formatted date
.

Copy and paste the following into the VRL Function Editor.

current_timestamp = now()
.formatted_date, err = format_timestamp(current_timestamp, "%Y-%m-%d %H:%M:%S")
.

Execute the VRL function & check the log file.

Save the function so that you can apply the function to all incoming pdc logs.

Click on the function option in the sidebar & select: Stream Association.
Highlight the default logs stream & associate the formatted_date function.

The function will now be applied to all incoming pdc logs.

PreviousWorker Server

Last updated 9 months ago