Implementing Distributed Tracing in a Golang software

This tutorial was initially posted on SigNoz Weblog and is written by Naman Jain

On this article, we’ll implement distributed tracing for a Golang software with three microservices. To implement distributed tracing, we shall be utilizing open-source options – SigNoz and OpenTelemetry, so you may simply observe the tutorial.

What’s distributed tracing?

Fashionable software structure utilizing cloud-native, containerization, and microservices is a really complicated distributed system. A typical web-search instance will illustrate a number of the challenges such a system wants to deal with.

A front-end service might distribute an online question to many lots of of question servers. The question may additionally be despatched to quite a few different sub-systems that will course of ads or search for specialised outcomes like photographs, information, and many others. This would possibly contain database entry, cache lookup, community name, and many others. In whole, 1000’s of machines and many various companies may be wanted to course of one search question.

Furthermore, web-search customers are delicate to delays, which might be brought on by poor efficiency in any sub-system. An engineer trying solely on the general latency might know there’s a downside however might not have the ability to guess which service is at fault nor why it’s behaving poorly. And such companies are additionally not written and managed by a single workforce. Additionally, daily, new elements would possibly get added to the system. Distributed tracing gives insights into the internal workings of such a fancy system. Tracing such complicated techniques allows engineering groups to arrange an observability framework.

Distributed tracing provides insights into how a selected service is performing as a part of the entire in a distributed software program system. It includes passing a hint context with every person request which is then handed throughout hosts, companies, and protocols to trace the person request.

On this article, we’ll use OpenTelemetry and SigNoz to allow distributed tracing in a pattern Golang software with microservices. However earlier than we deep dive into the implementation steps, allow us to provide you with a short context on OpenTelemetry and SigNoz.

OpenTelemetry and SigNoz

OpenTelemetry is a vendor-agnostic set of instruments, APIs, and SDKs used to instrument purposes to create and handle telemetry information(logs, metrics, and traces). It goals to make telemetry information a built-in function of cloud-native software program purposes.

OpenTelemetry gives the instrumentation layer to generate and export your telemetry information to a backend. Then, it is advisable to select a backend device that can present the info storage and visualization in your telemetry information. That’s the place SigNoz comes into the image.

SigNoz is a full-stack open-source APM device that gives metrics monitoring and distributed tracing.

OpenTelemetry is the way in which ahead for cloud-native software house owners who need to arrange a sturdy observability framework. It additionally gives you the liberty to decide on any backend evaluation device. SigNoz is constructed to help OpenTelemetry natively, thus making an important combo.

Distributed Tracing in a Golang software

We are going to display implementing distributed tracing in a Golang software within the following sections:

• Instrumenting the Golang app with OpenTelemetry
• Operating the pattern Golang software
• Visualizing traces information with SigNoz dashboards

Stipulations

Putting in SigNoz

First, it is advisable to set up SigNoz in order that OpenTelemetry can ship the info to it.

SigNoz might be put in on macOS or Linux computer systems in simply three steps by utilizing a easy set up script.

The set up script routinely installs Docker Engine on Linux. Nonetheless, on macOS, you will need to manually set up Docker Engine earlier than operating the set up script.

git clone -b important https://github.com/SigNoz/signoz.git
cd signoz/deploy/
./set up.sh

You possibly can go to our documentation for directions on the right way to set up SigNoz utilizing Docker Swarm and Helm Charts.

When you’re executed putting in SigNoz, you may entry the UI at http://localhost:3301.

Instrumenting the Golang app with OpenTelemetry

Now we have constructed a pattern Golang software for the aim of this tutorial. It has 3 companies:

• user-service
• payment-service, and
• order-service

These companies are instrumented with OpenTelemetry libraries, and once they work together with one another, OpenTelemetry emits the telemetry information to OTel collector which comes bundled with SigNoz.

Step 1: Clone pattern Golang app repository and go to the basis folder
We shall be utilizing a pattern Golang app at this GitHub repo.

git clone https://github.com/SigNoz/distributed-tracing-golang-sample.git
cd distributed-tracing-golang-pattern

Step 2: Set up the required dependencies

Verify the checklist of all of the required modules from go.mod. For OpenTelemetry, we want:

go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux v0.32.0
go.opentelemetry.io/contrib/instrumentation/web/http/otelhttp v0.32.0
go.opentelemetry.io/otel v1.7.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.7.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.7.0
go.opentelemetry.io/otel/sdk v1.7.0
go.opentelemetry.io/otel/hint v1.7.0
github.com/XSAM/otelsql v0.14.1

All of the dependencies might be put in utilizing:

go mod tidy
go mod vendor

Step 3: Configure the OpenTelemetry collector

Ideally, you must begin OpenTelemetry at first of important, earlier than another companies begin operating. When your program exits, name Shutdown on the SDK to make sure the final little bit of telemetry is flushed earlier than this system exits.

tp := config.Init(serviceName)
defer func() {
    if err := tp.Shutdown(context.Background()); err != nil {
        log.Printf("Error shutting down tracer supplier: %v", err)
    }
}()
// tracer is later used to create spans
tracer = otel.Tracer(serviceName)

We additionally initialized the tracer which is later used to create customized spans.

Let’s now perceive what does Init perform in config/config.go does.

1. Initialize exporter:

   The exporter in SDK is accountable for exporting the telemetry sign (hint) out of the appliance to a distant backend, logging to a file, and many others.
   On this demo, we’re making a gRPC exporter to ship out traces to an OpenTelemetry Collector backend operating at collectorURL (SigNoz). It additionally helps TLS and software auth utilizing headers.

   secureOption := otlptracegrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, "")) // config might be handed to configure TLS
   if len(insecure) > 0 {
       secureOption = otlptracegrpc.WithInsecure()
   }
   exporter, err := otlptrace.New(
       context.Background(),
       otlptracegrpc.NewClient(
           secureOption,
           otlptracegrpc.WithEndpoint(collectorURL),
           otlptracegrpc.WithHeaders(headers),
       ),
   )
   

2. Assemble hint supplier:

   TracerProvider gives entry to instrumentation Tracers. We configure it to pattern all of the traces and ship the traces in batches to the collector. The useful resource describes the article that generated the telemetry indicators. Primarily, it should be the title of the service or software. We set it to serviceName:

   traceProvider := sdktrace.NewTracerProvider(
       sdktrace.WithSampler(sdktrace.AlwaysSample()),
       sdktrace.WithSpanProcessor(sdktrace.NewBatchSpanProcessor(exporter)),
       sdktrace.WithResource(useful resource.NewWithAttributes(semconv.SchemaURL, semconv.ServiceNameKey.String(serviceName))),
   )
   

Now, we’re able to configure varied elements in our software.

Step 4: Instrument HTTP handler with OpenTelemetry

We’re utilizing gorilla/mux for the HTTP router. It may be instrumented with OpenTelemetry utilizing otelmux:

router.Use(otelmux.Middleware(serviceName))

Now, all of the HTTP calls cross by the OpenTelemetry middleware.

Our companies talk with one another utilizing HTTP APIs. We have to configure our consumer to cross on the tracing metadata. We will do this utilizing:

func SendRequest(ctx context.Context, methodology string, url string, information []byte) (*http.Response, error) {
    request, err := http.NewRequestWithContext(ctx, methodology, url, bytes.NewBuffer(information))
    if err != nil {
        return nil, fmt.Errorf("create request error: %w", err)
    }

    consumer := http.Consumer{
        // Wrap the Transport with one which begins a span and injects the span context
        // into the outbound request headers.
        Transport: otelhttp.NewTransport(http.DefaultTransport),
        Timeout:   10 * time.Second,
    }

    return consumer.Do(request)
}

Observe that the ctx parameter accommodates the tracing metadata of the mother or father span. So now the consumer sends the metadata and the server can extract this and join the tracing info of assorted companies.

Aside from the instrumentation already supplied by otelhttp library, we’d need to have customized spans for varied functions (e.g. hint database name, or hint perform execution). We will do this utilizing:

ctx, span := tracer.Begin(r.Context(), "replace person quantity")
defer span.Finish()

We will additionally connect attributes, occasions, and many others. to this span. Please check with the documentation for that.

Step 5: Instrument MySQL with OpenTelemetry

Database lies within the scorching path for many of the purposes and any insights into its efficiency are priceless. We instrument it with the assistance of github.com/XSAM/otelsql. And whereas making any DB name, we cross on the context.

db, err = otelsql.Open("mysql", datasourceName(username, password, host, dbName))
....
....
res, err := stmt.ExecContext(ctx, p.Vars...)

Observe: http://github.com/XSAM/otelsql just isn’t but formally supported by OpenTelemetry.

Operating the pattern Golang software

Step 1: Configuration

To arrange OpenTelemetry to gather and export telemetry information, it is advisable to specify OTLP (OpenTelemetry Protocol) endpoint. It consists of the IP of the machine the place SigNoz is put in and the port quantity at which SigNoz listens.

OTLP endpoint for SigNoz – <IP of the machine>:4317

When you’ve got put in SigNoz in your native machine, then your endpoint is 127.0.0.1:4317.

When you’ve got put in SigNoz on some area, then your endpoint is http://check.com:4317

Configuration for the next might be arrange in .env

# service config
USER_URL=localhost:8080
PAYMENT_URL=localhost:8081
ORDER_URL=localhost:8082

# database config
SQL_USER=root
SQL_PASSWORD=password
SQL_HOST=localhost:3306
SQL_DB=signoz

# telemetry config
OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
INSECURE_MODE=true

Step 2: Run the microservices

As we have now already cloned the repo within the above part, from the basis folder, run these instructions, every in a separate terminal:

go run ./customers
go run ./fee
go run ./order

Step 3: Affirm desk creation:

After operating the companies, verify if the tables ORDERS and USERS are created utilizing the instructions under:

mysql> use signoz;
mysql> present tables;

Visualizing Distributed Tracing information with Signoz

To visualise the hint information with SigNoz, we first have to generate some person information by interacting with the frontend.

Producing person information by interacting with the pattern app:

You’ll want to generate some person information to see the way it seems within the SigNoz dashboard. The pattern software comes with a UI to work together with the app. Use the under command within the root folder to launch the UI:

serve -l 5000 frontend

Now go to the app frontend operating at localhost:5000. Carry out the under given steps 4-5 occasions to generate some information.

1. Create a person:
   Click on Create Consumer button to create a brand new person within the MySQL db.

1. Switch the fund:
   Switch some quantity by clicking Switch Fund button.

1. Place an order:
   Place an order by choosing a product from the dropdown.

Now go to the SigNoz dashboard (operating on http://localhost:3301/ by default), look forward to a while, and refresh the dashboard. You’ll discover the checklist of service names that we configured:

• user-service
• order-service
• payment-service

Analyze traces and metrics utilizing the Signoz dashboard

Within the metrics tab, you may see Software Metrics, Exterior Calls, and Database Calls:

Software metrics:

Right here, we will see the appliance latency, requests per second(rps), error share, and the endpoints that had been hit for a given service.

Exterior Calls:

Right here, you may see the metrics in regards to the calls made to exterior companies. In our case, we’re operating the companies on localhost; therefore we see a single line. Metrics like exterior name period (by deal with) give a fast glimpse of the community connectivity with the exterior service. This may be helpful to detect a community subject.

For extra options on metrics, please learn the documentation.

Determine latency points with Flamegraphs and Gantt charts

You possibly can examine every occasion within the spans desk with Flamegraphs and Gantt charts to see an entire breakdown of the request. Establishing a sequential circulation of the person request together with data on time taken taken by every a part of the request might help establish latency points shortly. Let’s see the way it works within the case of our pattern Go app.

Go to the service title filter on the left and choose order-service. Now choose any span:

Right here, increasing on insert order you will note the time utilised in varied SQL DB calls.

Additionally, word that we even have extra details about the question that was run within the tags panel on the appropriate.

Conclusion

Distributed tracing is a strong and important toolkit for builders creating purposes primarily based on microservices structure. For Golang purposes utilizing microservices structure, distributed tracing can allow a central overview of how requests are performing throughout microservices.

This lets software house owners reconstruct the entire path of the request and see how particular person elements carried out as a part of your entire person request.

OpenTelemetry and SigNoz present an important open-source answer to implement distributed tracing in your purposes. You possibly can check out SigNoz by visiting its GitHub repo 👇

When you’ve got any questions or want any assist in setting issues up, be part of our slack group and ping us in #help channel

If you wish to know extra about distributed tracing or SigNoz, be at liberty to observe these hyperlinks:

SigNoz – an open-source different to DataDog

Information on Distributed Tracing