Go Custom Instrumentation

This guide explains how to implement Custom Instrumentation in Go applications using the Motadata Custom Instrumentation package.

Custom instrumentation enables you to add validated, auto-prefixed business attributes to your traces when using Motadata APM with eBPF zero-code auto-instrumentation. The package ensures attribute validation, consistent namespacing, and safe runtime behavior — without affecting the existing auto-instrumentation flow.

Prerequisites

• The application must already be instrumented using Motadata Auto Instrumentation so that span context is available.
• Go 1.22 or higher
• Motadata Agent 8.2.0 or higher
• The application must already be generating traces in APM

Configuration Steps

Step 1: Install the Go Custom Instrumentation Package

Run the following command in your project directory:

go get github.com/motadata2025/motadata-apm-custom-instrumentation-go@latest

note

Do not call otel.SetTracerProvider(...) anywhere in your application. This package uses autosdk.TracerProvider() directly — the eBPF agent hooks into this automatically. Initializing a global TracerProvider manually will conflict with the Auto SDK and break span correlation.

Step 2: Import the Package

import motadata "github.com/motadata2025/motadata-apm-custom-instrumentation-go"

Step 3: Create a Span and Add Custom Attributes

Use motadata.StartSpan to create a child span linked to the eBPF parent trace, then use the typed attribute setters to attach business context.

Basic Example

import motadata "github.com/motadata2025/motadata-apm-custom-instrumentation-go"

func (h *UserHandler) CreateUser(w http.ResponseWriter, r *http.Request) {
    ctx, span, _ := motadata.StartSpan(r.Context(), "CreateUser")
    defer span.End()

    _ = span.SetString("user.username", req.Username)
    _ = span.SetInt("user.age", int64(req.Age))
    _ = span.SetBool("user.verified", true)

    result, err := h.repo.CreateUser(ctx, req)
    if err != nil {
        span.RecordError(err)
        return
    }
    // ...
}

Keys are automatically prefixed with apm. if the prefix is not provided. Attribute errors are non-fatal — the span continues normally and only that one attribute is skipped.

Recommended Production Pattern

if err := span.SetString("user.email", email); err != nil {
    log.Printf("custom attribute skipped: %v", err)
}

Supported Methods

Span Lifecycle

Method	Description
motadata.StartSpan(ctx, spanName)	Creates a child span linked to the eBPF parent trace. Returns (context.Context, *Span, error). Returns ErrEmptySpanName if spanName is empty or whitespace-only. On error, a no-op span is returned — defer span.End() is always safe to call.
span.End()	Finalizes and exports the span. Always call immediately after StartSpan using defer.
span.RecordError(err)	Records err as a span event with stack trace and sets the span status to Error. No-op if err is nil.

Scalar Attribute Setters

Method	Type
span.SetString(key, value)	string
span.SetInt(key, value)	int64
span.SetFloat(key, value)	float64 — returns ErrInvalidFloat if value is NaN or ±Inf
span.SetBool(key, value)	bool

Slice / Array Attribute Setters

Method	Type
span.SetStringSlice(key, values)	[]string
span.SetIntSlice(key, values)	[]int64
span.SetFloatSlice(key, values)	[]float64 — returns ErrInvalidFloat if any element is NaN or ±Inf
span.SetBoolSlice(key, values)	[]bool

Attribute Behavior and Validation Rules

The Go Custom Instrumentation package validates both keys and values before attaching attributes to the span.

Key Handling

• Keys are trimmed and normalized to lowercase.
• Keys are automatically prefixed with apm. when the prefix is missing. Double-prefixing is prevented.
• Keys allow only alphanumeric characters and dots (a-z, 0-9, .).
• Keys must not be empty or whitespace-only.

Value Handling

• Float values must be finite. NaN and ±Inf are rejected.
• Slice methods reject any element that is NaN or ±Inf (floats).

Runtime Behavior

• Attribute errors are non-fatal. The span continues normally; only the invalid attribute is skipped.
• A no-op span is returned on StartSpan error — defer span.End() is always safe to call.
• Designed to work safely alongside existing eBPF auto-instrumentation.

Example Scenarios

HTTP Handler

func (h *UserHandler) CreateUser(w http.ResponseWriter, r *http.Request) {
    ctx, span, _ := motadata.StartSpan(r.Context(), "CreateUser")
    defer span.End()

    _ = span.SetString("http.method", r.Method)
    _ = span.SetString("http.url", r.URL.Path)
    _ = span.SetString("operation", "create_user")
    _ = span.SetString("user.username", req.Username)

    result, err := h.repo.CreateUser(ctx, req)
    if err != nil {
        span.RecordError(err)
        http.Error(w, "internal error", http.StatusInternalServerError)
        return
    }
    // ...
}

Repository / Data Access

func (r *UserRepo) CreateUser(ctx context.Context, req CreateUserRequest) (*User, error) {
    _, span, _ := motadata.StartSpan(ctx, "db:CreateUser")
    defer span.End()

    _ = span.SetString("db.operation", "INSERT")
    _ = span.SetString("db.table", "users")
    _ = span.SetString("db.param.username", req.Username)

    err := r.db.QueryRowContext(ctx, query, req.Username).Scan(&user.ID)
    if err != nil {
        span.RecordError(err)
        return nil, err
    }
    return &user, nil
}

Passing ctx from the handler into StartSpan here automatically creates a parent-child span relationship. The child span is linked to the handler span without any manual wiring.

Service / Business Logic

func ProcessOrder(ctx context.Context, orderID string) error {
    ctx, span, _ := motadata.StartSpan(ctx, "ProcessOrder")
    defer span.End()

    _ = span.SetString("order.id", orderID)
    _ = span.SetBool("order.express", true)
    _ = span.SetInt("order.items", 5)
    _ = span.SetFloat("order.weight.kg", 2.5)
    _ = span.SetStringSlice("order.tags", []string{"urgent", "vip"})

    return nil
}

Key Validation Rules

Rule	Example (input → stored key)
Automatically prefixed with apm.	"user.name" → apm.user.name
Leading/trailing whitespace trimmed	" user.name " → apm.user.name
Converted to lowercase	"User.Name" → apm.user.name
Only a-z, 0-9, . allowed	"user-name" → ErrInvalidKey
Empty key rejected	"" → ErrEmptyKey
Double prefix prevented	"apm.user.name" → apm.user.name
Float NaN / Inf rejected	math.NaN() → ErrInvalidFloat

Error Reference

Error	Cause
motadata.ErrEmptyKey	Attribute key is empty or whitespace-only
motadata.ErrInvalidKey	Attribute key contains characters other than a-z, 0-9, .
motadata.ErrInvalidFloat	Float value is NaN or ±Inf
motadata.ErrEmptySpanName	spanName passed to StartSpan is empty or whitespace-only

Best Practices

• Use hierarchical dot-separated keys such as "order.customer.tier" and "payment.method" for organized grouping in the APM UI.
• Always pass ctx downstream — the child span chain only works if ctx flows through every function call.
• Always defer span.End() — a span that is never ended is never exported.
• Never let attribute errors break business logic — always use _ = or check and log; attribute failures are non-fatal by design.
• Be consistent with key naming across services so attributes are filterable across traces in the APM UI.

What to Avoid

• Do not inject sensitive data such as passwords, tokens, or secrets.
• Do not call otel.SetTracerProvider(...) — it will conflict with the Auto SDK and break span correlation.
• Do not use inconsistent key naming across services.
• Do not pass invalid keys with unsupported characters.
• Do not share span objects across goroutines — use span.getContext() and create a new child span in the target goroutine.

Verify the Attributes

After implementing custom instrumentation:

1. Go to APM Explorer.
2. Open the required trace.
3. Select the relevant span.
4. Confirm custom attributes are visible under span attributes.

Summary

The Motadata Go Custom Instrumentation package provides a lightweight and enterprise-ready mechanism to inject validated, namespaced, business-specific attributes into active spans. It enhances trace visibility while maintaining full compatibility with eBPF-based auto-instrumentation — no manual TracerProvider setup required.