Skip to content

Commit 35cfdb0

Browse files
committed
Restructure log data model to be more like metrics
1 parent ce16301 commit 35cfdb0

4 files changed

Lines changed: 131 additions & 95 deletions

File tree

specification/logs/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,7 +196,7 @@ Logs can be correlated with the rest of observability data in a few dimensions:
196196
- By the **origin of the telemetry**, also known as the Resource context.
197197
OpenTelemetry traces and metrics contain information about the Resource they
198198
come from. We extend this practice to logs by including the
199-
[Resource](data-model.md#field-resource) in LogRecords.
199+
[Resource](../resource/README.md) in Log data streams.
200200

201201
These 3 correlations can be the foundation of powerful navigational, filtering,
202202
querying and analytical capabilities. OpenTelemetry aims to record and collects

specification/logs/api.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,10 +67,10 @@ The `LoggerProvider` MUST provide the following functions:
6767

6868
#### Get a Logger
6969

70-
This API MUST accept the following [instrumentation scope](data-model.md#field-instrumentationscope)
70+
This API MUST accept the following [instrumentation scope](../common/instrumentation-scope.md)
7171
parameters:
7272

73-
* `name`: Specifies the name of the [instrumentation scope](../common/instrumentation-scope.md),
73+
* `name`: Specifies the name of the instrumentation scope,
7474
such as the [instrumentation library](../glossary.md#instrumentation-library)
7575
(e.g. `io.opentelemetry.contrib.mongodb`), package, module or class name.
7676
If an application or library has built-in OpenTelemetry instrumentation, both

specification/logs/data-model.md

Lines changed: 125 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -12,32 +12,36 @@ weight: 2
1212

1313
<!-- toc -->
1414

15+
- [Overview](#overview)
1516
- [Design Notes](#design-notes)
1617
* [Requirements](#requirements)
17-
* [Events](#events)
1818
* [Definitions Used in this Document](#definitions-used-in-this-document)
1919
+ [Type `any`](#type-any)
2020
+ [Type `map`](#type-mapstring-any)
2121
* [Field Kinds](#field-kinds)
22-
- [Log and Event Record Definition](#log-and-event-record-definition)
23-
* [Field: `Timestamp`](#field-timestamp)
24-
* [Field: `ObservedTimestamp`](#field-observedtimestamp)
25-
* [Trace Context Fields](#trace-context-fields)
22+
- [Model Details](#model-details)
23+
* [Open Telemetry Protocol data model](#open-telemetry-protocol-data-model)
24+
+ [Field: `Timestamp`](#field-timestamp)
25+
+ [Field: `ObservedTimestamp`](#field-observedtimestamp)
2626
+ [Field: `TraceId`](#field-traceid)
2727
+ [Field: `SpanId`](#field-spanid)
2828
+ [Field: `TraceFlags`](#field-traceflags)
29-
* [Severity Fields](#severity-fields)
3029
+ [Field: `SeverityText`](#field-severitytext)
3130
+ [Field: `SeverityNumber`](#field-severitynumber)
32-
+ [Mapping of `SeverityNumber`](#mapping-of-severitynumber)
33-
+ [Reverse Mapping](#reverse-mapping)
34-
+ [Error Semantics](#error-semantics)
35-
+ [Displaying Severity](#displaying-severity)
36-
+ [Comparing Severity](#comparing-severity)
37-
* [Field: `Body`](#field-body)
38-
* [Field: `Attributes`](#field-attributes)
39-
+ [Errors and Exceptions](#errors-and-exceptions)
40-
* [Field: `EventName`](#field-eventname)
31+
+ [Field: `Body`](#field-body)
32+
+ [Field: `Attributes`](#field-attributes)
33+
- [Errors and Exceptions](#errors-and-exceptions)
34+
+ [Field: `EventName`](#field-eventname)
35+
- [Log Records](#log-records)
36+
* [Logs](#logs)
37+
* [Events](#events)
38+
- [Severity](#severity)
39+
* [Mapping of `Severity`](#mapping-of-severity)
40+
+ [Mapping to `SeverityNumber`](#mapping-to-severitynumber)
41+
+ [Mapping from `SeverityNumber`](#mapping-from-severitynumber)
42+
* [Error Semantics](#error-semantics)
43+
* [Displaying Severity](#displaying-severity)
44+
* [Comparing Severity](#comparing-severity)
4145
- [Example Log Records](#example-log-records)
4246
- [Example Mappings](#example-mappings)
4347
- [References](#references)
@@ -46,6 +50,8 @@ weight: 2
4650

4751
</details>
4852

53+
## Overview
54+
4955
This is a data model and semantic conventions that allow to represent logs from
5056
various sources: application log files, machine generated events, system logs,
5157
etc. Existing log formats can be unambiguously mapped to this data model.
@@ -102,14 +108,6 @@ The Data Model aims to successfully represent 3 sorts of logs and events:
102108
we include in the logs. We can likely modify the source code of the
103109
application if needed.
104110

105-
### Events
106-
107-
Events are OpenTelemetry's standardized format for LogRecords. All semantic
108-
conventions defined for logs SHOULD be formatted as Events. Requirements and details for the Event format can be found in the [semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/events.md).
109-
110-
Events are intended to be used by OpenTelemetry instrumentation. It is not a
111-
requirement that all LogRecords are formatted as Events.
112-
113111
### Definitions Used in this Document
114112

115113
In this document we refer to types `any` and `map<string, any>`, defined as
@@ -189,38 +187,48 @@ decision about when to use a top-level named field:
189187
Both of the above conditions were required to give the field a place in the
190188
top-level structure of the record.
191189

192-
## Log and Event Record Definition
190+
## Model Details
191+
192+
### Open Telemetry Protocol data model
193+
194+
The OpenTelemetry protocol (OTLP) data model is composed of Log data streams. These streams are in turn composed of log records.
193195

194-
[Appendix A](./data-model-appendix.md#appendix-a-example-mappings) contains many examples that show how
195-
existing log formats map to the fields defined below. If there are questions
196-
about the meaning of the field reviewing the examples may be helpful.
196+
Log streams are grouped into individual Log objects, identified by:
197197

198-
Here is the list of fields in a log record:
198+
- The originating Resource attributes
199+
- The instrumentation Scope (e.g., instrumentation library name, version)
199200

200-
| Field Name | Description |
201-
| ---------- | ----------- |
202-
| Timestamp | Time when the event occurred. |
203-
| ObservedTimestamp | Time when the event was observed. |
204-
| TraceId | Request trace id. |
205-
| SpanId | Request span id. |
206-
| TraceFlags | W3C trace flag. |
207-
| SeverityText | The severity text (also known as log level). |
208-
| SeverityNumber | Numerical value of the severity. |
209-
| Body | The body of the log record. |
210-
| Attributes | Additional information about the event. |
211-
| EventName | Name that identifies the class / type of event. |
201+
The `Log` record is defined by the following properties:
202+
203+
| Field Name | Category | Description |
204+
| ---------- | -------- | ----------- |
205+
| Timestamp | | Time when the event occurred. |
206+
| ObservedTimestamp | | Time when the event was observed. |
207+
| TraceId | Trace Context | Request trace id. |
208+
| SpanId | Trace Context | Request span id. |
209+
| TraceFlags Trace Context | Trace Context | W3C trace flag. |
210+
| SeverityText | Severity | The severity text (also known as log level). |
211+
| SeverityNumber | Severity | Numerical value of the severity. |
212+
| Body | | The body of the log record. |
213+
| Attributes | | Additional information about the event. |
214+
| EventName | | Name that identifies the class / type of event. |
215+
216+
> [!NOTE]
217+
> [Appendix A](./data-model-appendix.md#appendix-a-example-mappings) contains many examples that show how
218+
> existing log formats map to the fields defined above. If there are questions
219+
> about the meaning of the field reviewing the examples may be helpful.
212220
213221
Below is the detailed description of each field.
214222

215-
### Field: `Timestamp`
223+
#### Field: `Timestamp`
216224

217225
Type: Timestamp, uint64 nanoseconds since Unix epoch.
218226

219227
Description: Time when the event occurred measured by the origin clock, i.e. the
220228
time at the source. This field is optional, it may be missing if the source
221229
timestamp is unknown.
222230

223-
### Field: `ObservedTimestamp`
231+
#### Field: `ObservedTimestamp`
224232

225233
Type: Timestamp, uint64 nanoseconds since Unix epoch.
226234

@@ -238,8 +246,6 @@ timestamp internally the following logic is recommended:
238246

239247
- Use `Timestamp` if it is present, otherwise use `ObservedTimestamp`.
240248

241-
### Trace Context Fields
242-
243249
#### Field: `TraceId`
244250

245251
Type: byte sequence.
@@ -266,8 +272,6 @@ Description: Trace flag as defined in
266272
specification. At the time of writing the specification defines one flag - the
267273
SAMPLED flag. This field is optional.
268274

269-
### Severity Fields
270-
271275
#### Field: `SeverityText`
272276

273277
Type: string.
@@ -305,7 +309,77 @@ The following table defines the meaning of `SeverityNumber` value:
305309

306310
`SeverityNumber=0` MAY be used to represent an unspecified value.
307311

308-
#### Mapping of `SeverityNumber`
312+
#### Field: `Body`
313+
314+
Type: [`any`](#type-any) or [AnyValue](../common/README.md#anyvalue).
315+
316+
Description: A value containing the body of the log record. Can be for example
317+
a human-readable string message (including multi-line) describing the event in
318+
a free form or it can be a structured data composed of arrays and maps of other
319+
values. Body MUST support [AnyValue](../common/README.md#anyvalue)
320+
to preserve the semantics of structured logs emitted by the applications.
321+
Can vary for each occurrence of the event coming from the same source.
322+
This field is optional.
323+
324+
#### Field: `Attributes`
325+
326+
Type: [`map<string, any>`](#type-mapstring-any) or [Attribute Collection](../common/README.md#attribute-collections).
327+
328+
Description: Additional information about the specific event occurrence. Unlike
329+
the [`Resource`](../resource/sdk.md), which is fixed for a particular source, `Attributes` can
330+
vary for each occurrence of the event coming from the same source. Can contain
331+
information about the request context.
332+
This field is optional.
333+
334+
##### Errors and Exceptions
335+
336+
Additional information about errors and/or exceptions that are associated with
337+
a log record MAY be included in the structured data in the `Attributes` section
338+
of the record.
339+
If included, they MUST follow the OpenTelemetry
340+
[semantic conventions for exception-related attributes](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/exceptions/exceptions-logs.md).
341+
342+
#### Field: `EventName`
343+
344+
Type: string.
345+
346+
Description: Name that identifies the class / type of the [Event](#events).
347+
This name SHOULD uniquely identify the event structure (both attributes and body).
348+
A log record with a non-empty event name is an [Event](#events).
349+
350+
## Log Records
351+
352+
It is not a requirement that all LogRecords are formatted as Events hence the 2 formats described below.
353+
354+
> [!NOTE]
355+
> Events are intended to be used by OpenTelemetry instrumentation.
356+
357+
### Logs
358+
359+
The requirements of logs are:
360+
361+
- A log MUST NOT have an event name.
362+
- Attributes SHOULD be used to represent details and provide additional context about the event.
363+
- Body may be used.
364+
- Events SHOULD specify a severity number.
365+
366+
Logs are not defined in semantic conventions.
367+
368+
### Events
369+
370+
Events are OpenTelemetry's standardized format for LogRecords. The requirements of events are:
371+
372+
- An event MUST have an event name that uniquely identifies the event structure.
373+
- Attributes SHOULD be used to represent details and provide additional context about the event.
374+
- Events SHOULD specify a severity number.
375+
376+
All semantic conventions defined for logs SHOULD be formatted as Events with the requirements and details for the Event format can be found in the [semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/events.md).
377+
378+
## Severity
379+
380+
### Mapping of `Severity`
381+
382+
#### Mapping to `SeverityNumber`
309383

310384
Mappings from existing logging systems and formats (or **source format** for
311385
short) must define how severity (or log level) of that particular format
@@ -336,7 +410,7 @@ records with missing severity information distinctly or may interpret log
336410
records with missing `SeverityNumber` and `SeverityText` fields as if the
337411
`SeverityNumber` was set equal to INFO (numeric value of 9).
338412

339-
#### Reverse Mapping
413+
#### Mapping from `SeverityNumber`
340414

341415
When performing a reverse mapping from `SeverityNumber` to a specific format
342416
and the `SeverityNumber` has no corresponding mapping entry for that format
@@ -347,7 +421,7 @@ For example Zap has only one severity in the INFO range, called "Info". When
347421
doing reverse mapping all `SeverityNumber` values in INFO range (numeric 9-12)
348422
will be mapped to Zap’s "Info" level.
349423

350-
#### Error Semantics
424+
### Error Semantics
351425

352426
If `SeverityNumber` is present and has a value of ERROR (numeric 17) or higher
353427
then it is an indication that the log record represents an erroneous situation.
@@ -364,7 +438,7 @@ recommended value in this case is INFO (numeric 9). See
364438
[Appendix B](./data-model-appendix.md#appendix-b-severitynumber-example-mappings) for more mapping
365439
examples.
366440

367-
#### Displaying Severity
441+
### Displaying Severity
368442

369443
The following table defines the recommended short name for each
370444
`SeverityNumber` value. The short name can be used for example for representing
@@ -417,51 +491,13 @@ dropdown list, which lists all distinct `SeverityText` values that are known to
417491
the system (which can be a large number of elements, often differing only in
418492
capitalization or abbreviated, e.g. "Info" vs "Information").
419493

420-
#### Comparing Severity
494+
### Comparing Severity
421495

422496
In the contexts where severity participates in less-than / greater-than
423497
comparisons `SeverityNumber` field should be used.
424498
Special handling MAY be given to `SeverityNumber=0`
425499
when it is used to represent an unspecified severity.
426500

427-
### Field: `Body`
428-
429-
Type: [`any`](#type-any) or [AnyValue](../common/README.md#anyvalue).
430-
431-
Description: A value containing the body of the log record. Can be for example
432-
a human-readable string message (including multi-line) describing the event in
433-
a free form or it can be a structured data composed of arrays and maps of other
434-
values. Body MUST support [AnyValue](../common/README.md#anyvalue)
435-
to preserve the semantics of structured logs emitted by the applications.
436-
Can vary for each occurrence of the event coming from the same source.
437-
This field is optional.
438-
439-
### Field: `Attributes`
440-
441-
Type: [`map<string, any>`](#type-mapstring-any) or [Attribute Collection](../common/README.md#attribute-collections).
442-
443-
Description: Additional information about the specific event occurrence. Unlike
444-
the [`Resource`](../resource/sdk.md), which is fixed for a particular source, `Attributes` can
445-
vary for each occurrence of the event coming from the same source. Can contain
446-
information about the request context (other than [Trace Context Fields](#trace-context-fields)).
447-
This field is optional.
448-
449-
#### Errors and Exceptions
450-
451-
Additional information about errors and/or exceptions that are associated with
452-
a log record MAY be included in the structured data in the `Attributes` section
453-
of the record.
454-
If included, they MUST follow the OpenTelemetry
455-
[semantic conventions for exception-related attributes](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/exceptions/exceptions-logs.md).
456-
457-
### Field: `EventName`
458-
459-
Type: string.
460-
461-
Description: Name that identifies the class / type of the [Event](#events).
462-
This name SHOULD uniquely identify the event structure (both attributes and body).
463-
A log record with a non-empty event name is an [Event](#events).
464-
465501
## Example Log Records
466502

467503
For example log records see

specification/logs/sdk.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -266,8 +266,8 @@ In this document we refer to `ReadableLogRecord` and `ReadWriteLogRecord`, defin
266266

267267
A function receiving this as an argument MUST be able to access all the
268268
information added to the [LogRecord](data-model.md#log-and-event-record-definition). It MUST also be able to
269-
access the [Instrumentation Scope](./data-model.md#field-instrumentationscope)
270-
and [Resource](./data-model.md#field-resource) information (implicitly)
269+
access the [Instrumentation Scope](../common/instrumentation-scope.md)
270+
and [Resource](../resource/README.md) information (implicitly)
271271
associated with the `LogRecord`.
272272

273273
The [trace context fields](./data-model.md#trace-context-fields) MUST be populated from
@@ -412,7 +412,7 @@ in order to support filtering via [`Logger.Enabled`](api.md#enabled).
412412

413413
* [Context](../context/README.md) explicitly passed by the caller or the current
414414
Context
415-
* [Instrumentation Scope](./data-model.md#field-instrumentationscope) associated
415+
* [Instrumentation Scope](../common/instrumentation-scope.md) associated
416416
with the `Logger`
417417
* [Severity Number](./data-model.md#field-severitynumber) passed by the caller
418418
* [Event Name](./data-model.md#field-eventname) passed by the caller

0 commit comments

Comments
 (0)