Skip to main content

Producing structured logs

Overview​

Structured logs can be written from within an IPC using the RPCs provided by the DataLogger service, which are accessible through the structured logging clients in the SDK.

This guide covers the various RPCs available for fetching logs using the C++ StructuredLoggingClient.

INTR_ASSIGN_OR_RETURN(
    StructuredLoggingClient client,
    StructuredLoggingClient::Create("logger.app-intrinsic-base.svc.cluster.local:8080", absl::Now() + absl::Seconds(5)));

LogItems and event sources​

Structured logging data must be represented as a LogItem proto, existing in streams of data that we call "event sources".

You must designate which event source the log should be sent to by specifying specify the name of the event source at write time.

important

Every log that is logged to the same event source must be of the same schema.

Event source names​

The following are valid characters for event source names:

  • alphanumeric characters (a-z, A-Z, 0-9)
  • some special characters, with caveat
    • periods (.)
    • hyphens (-)
    • underscores (_)
    • forward slashes (/)

It is recommended to keep event source names under 60 characters in length.

important

It’s important to prevent collisions in event source names, as they can cause unrelated logs to become interleaved—or worse, lead the logging infrastructure into an invalid state by mixing logs with incompatible schemas.

Furthermore, as a caveat due to backend constraint reasons, all valid special characters are treated as the same character. Take care to avoid collisions under that caveat.

For example, the following event source names would result in a collision, as the special characters resolve to the same character in the backend:

  • my_prefix.event_source_name
  • my_prefix-event_source_name
  • my_prefix_event_source_name
  • my_prefix/event_source_name
tip

One way to avoid collisions is to prefix your event source names (e.g., my_org_prefix/event_source_name). Or another way is to list what event sources currently exist on an IPC, to see whether a name is already in use.

Basic write​

You can write a structured log using the Log RPC.

MyProto custom_proto;
intrinsic_proto::data_logger::LogItem log_item;

log_item.mutable_metadata()->set_event_source("event_source_name");
log_item.mutable_payload()->mutable_any()->PackFrom(custom_proto);

INTR_RETURN_IF_ERROR(client.Log(log_item));

Blobs​

You can also attach a byte blob to a structured log by populating its blob_payload field.

note

Blob IDs should be unique. Otherwise, subsequent writes to the same blob ID will overwrite the blob.

log_item.mutable_blob_payload()->set_blob_id("my_blob_id");
log_item.mutable_blob_payload()->set_data("blob_data");

INTR_RETURN_IF_ERROR(client.Log(log_item));

Labels​

You may attach optional labels to a structured log by populating the context.labels field. This will enable the log to be filtered using label-based filtering

LogItem item;

// Attach key-value labels.
for (int j = 0; j < i % 4; ++j) {
  item.mutable_context()->mutable_labels()->insert(
      {absl::StrFormat("key_%d", j), absl::StrFormat("val_%d", j)});
}


// Set other log fields... The log it.

INTR_RETURN_IF_ERROR(client.Log(log_item));