Sending Traces

This guide describes how to send OpenTelemetry traces to the Unomiq OTel Gateway API. It covers two approaches:

Collector Sidecar — an OTel Agent runs alongside your app, handles OAuth2 automatically, and forwards traces to the gateway.
Direct from Application — your application code manages OAuth2 tokens and sends traces to the gateway without a sidecar.

Both approaches use the OAuth2 client credentials grant type.

Prerequisites

API Credentials

Create API credentials from the Unomiq Management API or the Unomiq Dashboard. The credentials must have the write:traces permission. This will give you an API key (Client ID) and secret (Client Secret).

Configuration

Set the following environment variables:

Variable	Description
`OAUTH_CLIENT_ID`	API Key from the Unomiq Management API / dashboard
`OAUTH_CLIENT_SECRET`	API Secret from the Unomiq Management API / dashboard
`OAUTH_TOKEN_URL`	`https://oauth-api.unomiq.com/token`
`OAUTH_SCOPES`	Set to `write:traces`
`OTEL_EXPORTER_OTLP_ENDPOINT`	`https://gateway-api.unomiq.com`

Approach 1: OTel Collector Sidecar

In this approach, your application sends telemetry to a local OTel Collector over an unauthenticated connection. The collector handles OAuth2 token acquisition and renewal, then forwards the authenticated telemetry to the gateway.

Your App ──(no auth)──▶ OTel Collector Sidecar ──(OAuth2 Bearer token)──▶ Gateway

This is the recommended approach for containerized environments (Docker, Kubernetes) because it decouples authentication from application code entirely.

Collector Configuration

The collector uses the oauth2clientauthextension from the OpenTelemetry Collector Contrib distribution. Create an otel-collector-config.yaml:

extensions:
  oauth2client:
    client_id: ${env:OAUTH_CLIENT_ID}
    client_secret: ${env:OAUTH_CLIENT_SECRET}
    token_url: ${env:OAUTH_TOKEN_URL}
    scopes:
      - ${env:OAUTH_SCOPES}
    timeout: 10s

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 100

exporters:
  otlphttp:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    auth:
      authenticator: oauth2client

service:
  extensions: [oauth2client]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Key points:

The oauth2client extension acquires a token using the client credentials grant and automatically refreshes it before expiry.
The otlphttp exporter references the extension via auth.authenticator, so every outgoing request includes the Authorization: Bearer <token> header.
The receiver listens on standard OTLP ports (4317 for gRPC, 4318 for HTTP) without requiring any authentication from your application.

Running the Collector

Use the contrib distribution of the collector, which includes the oauth2clientauthextension. The core distribution does not include it. Docker Compose example:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
    environment:
      - OAUTH_CLIENT_ID
      - OAUTH_CLIENT_SECRET
      - OAUTH_TOKEN_URL
      - OAUTH_SCOPES
      - OTEL_EXPORTER_OTLP_ENDPOINT
    ports:
      - "4317:4317"
      - "4318:4318"

Configuring Your Application

Point your application’s OTLP exporter at the local collector. No authentication configuration is needed in the application itself. Python (automatic instrumentation):

export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
opentelemetry-instrument python app.py

Python (SDK):

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(endpoint="http://otel-collector:4318/v1/traces")

Any language: Set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to http://localhost:4318 (or the collector’s hostname in Docker/Kubernetes).

Approach 2: Direct Authentication from Application (Python)

In this approach, your application manages OAuth2 tokens and sends authenticated traces directly to the gateway. No sidecar is needed.

Your App ──(OAuth2 Bearer token)──▶ Gateway

This is useful for serverless environments (Cloud Run, Lambda) or when you want fewer infrastructure components.

Step 1: OAuth2 Token Manager

Create a token manager that handles the client credentials flow and automatic refresh:

"""OAuth2 token manager for OTLP exporters."""

import threading
import logging
from typing import Optional
from datetime import datetime, timedelta

import requests

logger = logging.getLogger(__name__)


class OAuthTokenManager:
    """Manages OAuth2 token retrieval and automatic refresh."""

    def __init__(
        self,
        client_id: str,
        client_secret: str,
        token_url: str,
        scopes: str,
        refresh_buffer: int = 300,  # Refresh 5 minutes before expiry
    ):
        self.client_id = client_id
        self.client_secret = client_secret
        self.token_url = token_url
        self.scopes = scopes
        self.refresh_buffer = refresh_buffer

        self._token: Optional[str] = None
        self._token_expiry: Optional[datetime] = None
        self._lock = threading.Lock()
        self._stop_refresh = threading.Event()

        # Get initial token
        self._fetch_token()

        # Start background refresh
        self._refresh_thread = threading.Thread(
            target=self._refresh_loop, daemon=True, name="oauth-token-refresh"
        )
        self._refresh_thread.start()

    def _fetch_token(self) -> None:
        """Fetch a new access token from the OAuth2 token endpoint."""
        response = requests.post(
            self.token_url,
            data={
                "grant_type": "client_credentials",
                "client_id": self.client_id,
                "client_secret": self.client_secret,
                "scope": self.scopes,
            },
            headers={"Content-Type": "application/x-www-form-urlencoded"},
            timeout=10,
        )
        response.raise_for_status()

        token_data = response.json()
        self._token = token_data["access_token"]
        expires_in = token_data.get("expires_in", 3600)
        self._token_expiry = datetime.now() + timedelta(seconds=expires_in)
        logger.info("OAuth token fetched, expires in %d seconds", expires_in)

    def _should_refresh(self) -> bool:
        if self._token is None or self._token_expiry is None:
            return True
        return (self._token_expiry - datetime.now()).total_seconds() <= self.refresh_buffer

    def _refresh_loop(self) -> None:
        while not self._stop_refresh.is_set():
            try:
                if self._should_refresh():
                    with self._lock:
                        if self._should_refresh():
                            self._fetch_token()
            except Exception:
                logger.exception("Error refreshing OAuth token")
            self._stop_refresh.wait(60)

    def get_headers(self) -> dict:
        """Return authorization headers for the OTLP exporter."""
        with self._lock:
            if self._should_refresh():
                self._fetch_token()
            return {"Authorization": f"Bearer {self._token}"}

    def stop(self) -> None:
        """Stop the background refresh thread."""
        self._stop_refresh.set()

Step 2: Configure the Tracer

Wire the token manager into the OTLP exporter:

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

from oauth_token_manager import OAuthTokenManager


def configure_tracing(service_name: str, service_version: str) -> trace.Tracer:
    """Set up OpenTelemetry tracing with OAuth2-authenticated export."""

    # Create the token manager
    token_manager = OAuthTokenManager(
        client_id=os.environ["OAUTH_CLIENT_ID"],
        client_secret=os.environ["OAUTH_CLIENT_SECRET"],
        token_url=os.environ["OAUTH_TOKEN_URL"],
        scopes=os.environ.get("OAUTH_SCOPES", ""),
    )

    # Build the OTLP exporter with auth headers
    endpoint = os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"].rstrip("/") + "/v1/traces"
    exporter = OTLPSpanExporter(
        endpoint=endpoint,
        headers=token_manager.get_headers(),
    )

    # Assemble the tracer provider
    resource = Resource.create({
        SERVICE_NAME: service_name,
        SERVICE_VERSION: service_version,
    })
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

    return trace.get_tracer(service_name, service_version)

Step 3: Use the Tracer

tracer = configure_tracing("my-service", "1.0.0")

with tracer.start_as_current_span("my-operation") as span:
    span.set_attribute("key", "value")
    # ... your code here

For frameworks like Flask or Django, you can also use the OpenTelemetry instrumentor libraries (e.g., opentelemetry-instrumentation-flask) after calling configure_tracing().

Required Dependencies

opentelemetry-api
opentelemetry-sdk
opentelemetry-exporter-otlp-proto-http
requests

Choosing Between the Two Approaches

	Collector Sidecar	Direct from Application
Auth handling	Collector manages tokens	Application manages tokens
Code changes	None (env vars only)	Token manager + exporter setup
Infrastructure	Requires running a collector	No extra infrastructure
Language support	Any (language-agnostic)	Requires per-language implementation
Best for	Docker, Kubernetes	Serverless (Cloud Run, Lambda)
Token refresh	Handled by collector	Handled by background thread

Use the sidecar when you’re in a containerized environment and want to keep authentication out of your application code. It also makes it easy to add processing (batching, filtering, sampling) without changing your app. Use direct export when running in serverless or lightweight environments where an extra sidecar is impractical, or when you need precise control over the export pipeline.

Monitoring Sent Traces

Once your traces are flowing to the gateway, you can monitor them in two ways:

API — Use the Get Live Traces endpoint to programmatically retrieve and inspect incoming traces in real time.
Dashboard — View and explore traces visually from the Unomiq Dashboard.

Troubleshooting

Common Issues

401 Unauthorized from the gateway

Verify that OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, and OAUTH_TOKEN_URL are correct.
Check that the scopes match what the gateway expects.
Ensure the token endpoint is reachable from your environment.

No traces appearing at the gateway

Confirm OTEL_EXPORTER_OTLP_ENDPOINT is set to the correct gateway URL.
For the sidecar approach, check collector logs for export errors.
For direct export, enable debug logging: logging.getLogger('opentelemetry').setLevel(logging.DEBUG).

Token refresh failures

The sidecar collector and the Python token manager both refresh tokens automatically before expiry. Check logs for errors from the token endpoint.
Ensure your OAuth2 client has not been revoked or rate-limited.

Viewing Collector Logs (Sidecar Approach)

Add the debug exporter to your collector pipeline for verbose output:

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      exporters: [otlphttp, debug]

Getting started

Sending Data

Cost Attribution

Sending Traces

Prerequisites

API Credentials

Configuration

Approach 1: OTel Collector Sidecar

Collector Configuration

Running the Collector

Configuring Your Application

Approach 2: Direct Authentication from Application (Python)

Step 1: OAuth2 Token Manager

Step 2: Configure the Tracer

Step 3: Use the Tracer

Required Dependencies

Choosing Between the Two Approaches

Monitoring Sent Traces

Troubleshooting

Common Issues

Viewing Collector Logs (Sidecar Approach)

Getting started

Sending Data

Cost Attribution

​Prerequisites

​API Credentials

​Configuration

​Approach 1: OTel Collector Sidecar

​Collector Configuration

​Running the Collector

​Configuring Your Application

​Approach 2: Direct Authentication from Application (Python)

​Step 1: OAuth2 Token Manager

​Step 2: Configure the Tracer

​Step 3: Use the Tracer

​Required Dependencies

​Choosing Between the Two Approaches

​Monitoring Sent Traces

​Troubleshooting

​Common Issues

​Viewing Collector Logs (Sidecar Approach)

Prerequisites

API Credentials

Configuration

Approach 1: OTel Collector Sidecar

Collector Configuration

Running the Collector

Configuring Your Application

Approach 2: Direct Authentication from Application (Python)

Step 1: OAuth2 Token Manager

Step 2: Configure the Tracer

Step 3: Use the Tracer

Required Dependencies

Choosing Between the Two Approaches

Monitoring Sent Traces

Troubleshooting

Common Issues

Viewing Collector Logs (Sidecar Approach)