CTF2-PROP-2.0: Proposal for a major revision of the Common Trace Format, version 1.8

This document is an informal proposal for the next major revision of the Common Trace Format (CTF) version 2 (hereafter named CTF 2).

This is not a formal reference. Some parts of this document, however, may be formal enough to be elligible for a specification document.

RFC 2119

The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, MAY, and OPTIONAL in this document, when emphasized, are to be interpreted as described in RFC 2119.

1. Revision history

Significant changes since CTF2-PROP-1.0:

Written in Asciidoctor's AsciiDoc syntax.

The original AsciiDoc project is pretty much deprecated.

Asciidoctor is also what GitHub, GitLab, and more use to render AsciiDoc documents.
Terminology update.

See the CTF 2 terminology update.
Mostly written from the consumer’s perspective, especially the data stream decoding procedure.
Metadata types are removed: the document uses JSON directly now.

This makes the specification easier to read, removing one nonessential abstraction layer.
Constant integer JSON object is removed: we can use plain JSON integers.
Splitting the specification into three layers is removed: the document specifies the whole format monolithically.

This makes the specification easier to read.
Format simplifications:
- Trace class's default/native byte order property is removed: a fixed-length bit array field class MUST have an explicit byte order property.
- Not all field classes have an alignment property: only a fixed-length bit array field class, and all the field classes which inherit it, need one.
  
  To remain backward compatible with CTF 1, a structure field class still needs a minimum alignment property.
- Relative field location (field path in CTF2-PROP-1.0) is removed.
  
  You can always use absolute field locations.
- Field class aliases are removed.
- “Tags” replaced with roles attached to field classes.
  
  It makes sense for a field class to have a role now that each field class is unique (no field class aliases).
- A data stream MAY have a single default clock. Therefore, a data stream class MAY have a single, default clock class.
  
  This is enough to satisfy the current use cases.
- Null, variable-length boolean, and union field classes are removed: we don’t have use cases for them.
Optional field class added.

This is why CTF2-PROP-1.0 introduced the null field class (removed in CTF2-PROP-2.0): in the end, a variant field class with a null field class option and another option really is an optional field class.
BLOB field classes added.

A BLOB field contains a BLOB.

Having dedicated static- and dynamic-length BLOB field classes instead of using static- and dynamic-length array field classes makes it possible for consumer APIs to provide the values of such fields as byte buffers.
An integer field class MAY have a preferred display base, even if it’s insignificant to decode its fields, because this property exists in CTF 1 and makes the transition easier.
An event record class MAY have a name and other properties which exist in CTF 1, but which are insignificant to decode its instances, because it makes the transition easier.
Extension feature added.

This is how we plan to add core features and modify existing ones in the future without having to rely on a minor version.
Auxiliary stream feature added.

2. CTF publication authority

The DiaMon Workgroup, a Linux Foundation workgroup which creates de facto open standards and tools for tracing, monitoring, and diagnostics, oversees the Common Trace Format design and the publication of official CTF documents.

The DiaMon Workgroup is also responsible for providing a platform for interested parties to comment on CTF-related proposals.

3. CTF 2 document identification

We suggest that all official CTF 2 documents bear a unique document identifier (ID) having the following format:

CTF2-SHORTNAME-MAJOR.MINOR[rREV]

Table 1. Descriptions and roles of CTF 2 document ID parts
Part	Description	Bump MAY introduce new concepts, procedures, and formats?	Bump MAY remove or change existing concepts, procedures, and formats?
`SHORTNAME`	The capitalized short name of the document, unique amongst all the official CTF 2 documents.	N/A	N/A
`MAJOR`	The major version number of the document.	Yes	Yes
`MINOR`	The minor version number of the document.	Yes	No
`REV`	The revision letter of the document (from `A` to `Z`, then `ZA` to `ZZ`, and so on). We’ll use document revisions to add examples, clarify existing concepts, or fix grammar/content mistakes, for example.	No	No

As an example, the short name of this document is PROP, for proposal, and its full document ID is CTF2-PROP-2.0. The next revision would be CTF2-PROP-2.0rA, and the following would be CTF2-PROP-2.0rB.

A CTF 2 document MUST refer to another CTF 2 document using only its ID, for example:

This concept is further explained in CTF2-SOMEID-1.2.

There’s no need to refer to a specific revision: a reference always targets the latest document’s revision.

We suggest the following IDs for the initial documents:

CTF 2 document ID	Description
CTF2-DOCID-1.0	CTF 2 document identifier format.
CTF2-FS-2.0	Layout of a CTF 2 trace when stored on a file system.
CTF2-PMETA-2.0	CTF 2 metadata stream packet format.
CTF2-SPEC-2.0	Common Trace Format (CTF), version 2. We don’t plan to ever bump this document’s minor version as we can use the proposed extension mechanism to add or modify core features.

CTF 2 document ID

Description

CTF2-DOCID-1.0

CTF 2 document identifier format.

CTF2-FS-2.0

Layout of a CTF 2 trace when stored on a file system.

CTF2-PMETA-2.0

CTF 2 metadata stream packet format.

CTF2-SPEC-2.0

Common Trace Format (CTF), version 2.

We don’t plan to ever bump this document’s minor version as we can use the proposed extension mechanism to add or modify core features.

4. Why CTF 2?

Why do we need a major version bump of the CTF specification?

A major version bump is never an easy choice when it comes to revisiting a software library, a communication protocol, a file format, or anything else that serves as a contract between a producer and a consumer.

When such a decision is taken, it must be justified by solid arguments, since it makes it impossible for old consumers to consume the product of new producers.

In this proposal, for instance, CTF 2 traces are not backward compatible with CTF 1 traces. Although the data stream binary format is compatible, the metadata stream is not: it’s written in a different language (JSON instead of TSDL).

CTF 1 has been used and tested for over ten years now, by different producers and consumers. Over that time, we have noted a few gaps in the trace format, gaps that make it hard to extend CTF 1 as much as we’d like, amongst other things.

This specification proposition designs CTF 2 to overcome those gaps, to the best of our knowledge, and to be flexible enough to gracefully accept future additions while avoiding another major version bump in the upcoming years.

4.1. Design goals

The design goals of CTF 2 are as follows, in order, beginning with the most important:

CTF 2 data streams MUST be backward compatible with CTF 1 data streams.

Many applications are already written to output valid CTF 1 packets. Modifying the code of those applications to produce different binary packets can be cumbersome, and sometimes impossible if the application passed acceptance tests, for example.

Making sure that applications producing CTF 1 traces can also produce CTF 2 traces only by changing the metadata stream is therefore a hard requirement.
The CTF 2 data streams MUST be as efficient as possible to produce by a tracer.

This design goal was also one of the major ones which drove the design of CTF 1.

In other words, a small embedded system MUST be able to produce CTF 2 data streams natively. Moreover, the tracer MUST be able to copy binary data to the packet buffer of a data stream without altering it.
A CTF 2 metadata stream MUST be extensible by users (including the DiaMon Workgroup) of the specification.

CTF 1’s TSDL grammar is pretty restrictive when it comes to customizing existing blocks with user-defined attributes.

Many protocols and declarative languages support custom user data in their payload. For example, HTML5 allows any element to have user attributes with the data- prefix.

A CTF 2 producer MUST be able to add custom attributes to almost any specified metadata object. This makes it possible for standard consumers to read any CTF 2 trace and ignore unknown user attributes, providing a “bland”, yet complete view of the trace fields, while you can write a special consumer (or extend an existing one) to interpret specific user attributes and use them to render a meaningful visualization.

This design goal also means that an “old” CTF 2 consumer MUST be able to either:
- Decode a “new” CTF 2 trace completely.
- Indicate that it cannot decode a “new” CTF 2 trace completely.
CTF 2’s specification MUST focus on use cases known on its publication date.

The previous design goal indicates that a CTF 2 metadata stream MUST be extensible by users. Knowing this, the CTF 2 specification itself MUST focus on currently known use cases, leaving anything else to future extensions to be described in other CTF 2 documents.

As per design goal 1, those known use cases include everything available in CTF 1.
CTF 2’s specification MUST NOT specify how to transport or store a trace.

In the CTF 2 specification, a CTF 2 trace MUST be defined as a set of streams, without specifying how you transport or store said streams.

Other official CTF 2 documents published by the DiaMon Workgroup can define standard ways to transport and store CTF 2 streams for targeted use cases. Trace producers and consumers can choose to implement one or more transport/storage strategies following the other documents.
A CTF 2 trace SHOULD be as easy as possible to consume.

CTF 1 focuses on being easy to produce, which is a good idea since producers are often tracers in this context, and as per design goal 2, a minimal tracer MUST be able to produce a correct CTF trace with minimal code.

However, because a CTF 1 metadata stream is written in TSDL, a custom, declarative, C-like DSL designed for CTF, writing a minimal consumer of CTF 1 is not an easy task. TSDL is an intricate language, with many special cases and features, many of which are borrowed from the C language, which you can’t ignore when writing a consumer supporting all its features. TSDL was developed to ease the manual (human) production of metadata streams.

Over time, we realized that, while producing traces is important, consuming them to solve problems with event record analysis is just as important, if not more.

This is why CTF 2 SHOULD encourage the development of CTF consumers in any programming language by reducing the number of special cases, as well as by using a very simple, yet well-known descriptive language for the metadata stream.

CTF 1 tries to accomodate other trace formats, which can be converted to CTF without changing the data streams by writing the matching metadata stream. This is also a source of special cases. CTF 2 SHOULD build on binary trace conversion (from another, non-CTF trace format to CTF 2) rather than trying to accomodate other formats.
CTF 2’s model SHOULD be similar to CTF 1’s.

Some APIs are already written to deal with CTF 1 “objects”, or concepts (event record classes, event records, field classes, and clock classes, for example).

The model of CTF 2 SHOULD be, as much as possible, compatible with the model of CTF 1, so that those existing APIs can operate on CTF 2 objects too without requiring significant upgrades.

5. Changes since CTF 1

Here is a brief summary of the changes, from CTF 1 to CTF 2, as introduced by this proposal.

The terminology of the specification and the binary layouts of the data streams are completely detached from the C language and from the behaviour of any C compiler.

CTF 2 is a programming language-agnostic trace format.

Terminology update:

CTF 1 term	Equivalent CTF 2 term
Absolute (clock class property)	Origin is Unix epoch
Array (field class)	Static-length array
Base (integer field class property)	Preferred display base
Binary stream	Data stream
Clock (when it names a block of metadata which describes actual clocks)	Clock class
Content size	Packet’s content size
Declaration	Field class
Enumeration (field class)	Fixed-length enumeration
Event (when it names a block of metadata which describes event records)	Event record class
Event (when it names an actual recorded event contained in a packet)	Event record
Event context	Event record specific context
Event fields	Event record payload
Event ID	Event record class ID
Events discarded	Discarded event record counter
Field (structure field class)	Structure field member class
Field (structure field)	Structure field member
Floating point (field class)	Fixed-length floating point number
Integer (field class)	Fixed-length integer
Packet size	Packet’s total size
Sequence (field class)	Dynamic-length array
Size (integer field class property)	Length
Stream (when it names a block of metadata which describes data streams)	Data stream class
Stream event context	Event record common context
Stream event header	Event record header
Stream ID	Data stream class ID
Stream instance ID	Data stream ID
Stream packet context	Packet context
String (field class)	Null-terminated string
Tag (variant field class)	Selector
Trace (when it names a block of metadata which describes traces)	Trace class
Trace packet header	Packet header

CTF 1 term

Equivalent CTF 2 term

Absolute (clock class property)

Origin is Unix epoch

Array (field class)

Static-length array

Base (integer field class property)

Preferred display base

Binary stream

Data stream

Clock (when it names a block of metadata which describes actual clocks)

Clock class

Content size

Packet’s content size

Declaration

Field class

Enumeration (field class)

Fixed-length enumeration

Event (when it names a block of metadata which describes event records)

Event record class

Event (when it names an actual recorded event contained in a packet)

Event record

Event context

Event record specific context

Event fields

Event record payload

Event ID

Event record class ID

Events discarded

Discarded event record counter

Field (structure field class)

Structure field member class

Field (structure field)

Structure field member

Floating point (field class)

Fixed-length floating point number

Integer (field class)

Fixed-length integer

Packet size

Packet’s total size

Sequence (field class)

Dynamic-length array

Size (integer field class property)

Length

Stream (when it names a block of metadata which describes data streams)

Data stream class

Stream event context

Event record common context

Stream event header

Event record header

Stream ID

Data stream class ID

Stream instance ID

Data stream ID

Stream packet context

Packet context

String (field class)

Null-terminated string

Tag (variant field class)

Selector

Trace (when it names a block of metadata which describes traces)

Trace class

Trace packet header

Packet header

The metadata stream is written in JSON, as specified by ECMA-404.
The CTF 2 specification doesn’t specify a “packetized” metadata stream format.

This is a back-end-specific way of wrapping a metadata stream as defined in this document.

Another document specifies the packetized metadata stream format, which trace storage and transport documents can use if needed.
Most objects of a CTF 2 metadata stream MAY contain custom user attributes.
Field class aliases are removed.

This removes indirection, simplifying the format.
New field classes:
- A fixed-length bit array field class describes the most primitive fields (fixed-length bit array fields) of a data stream which actually have values.
  
  A fixed-length integer field class is now specified as a fixed-length bit array field class with an OPTIONAL preferred display base property and integer type semantics.
  
  A fixed-length enumeration field class is now specified as a fixed-length integer field class with mappings.
  
  A fixed-length floating point number field class is now specified as a fixed-length bit array field class: its length property is enough to determine its encoding under IEEE 754-2008's binary interchange format.
- A fixed-length boolean field class, which is a fixed-length bit array field class with a special meaning, describes fixed-length boolean fields.
  
  When all the bits of a fixed-length boolean field are cleared (zero), the field’s value is said to be false. Otherwise, the field’s value is said to be true.
- A variable-length bit array field class describes variable-length bit array fields.
  
  Each byte of a variable-length bit array field has its most significant bit set if one more byte is part of the field.
  
  The seven low-order bits of each byte are concatenated in a specific way to form a final, equivalent fixed-length bit array field.
- A variable-length integer field class, which is to a variable-length bit array field class what a fixed-length integer field class is to a fixed-length bit array field class, describes variable-length integer fields.
- A variable-length enumeration field class, which is to a variable-length integer field class what a fixed-length enumeration field class is to a fixed-length integer field class, describes variable-length enumeration fields.
- A static-length string field class and a dynamic-length string field class describe static- and dynamic-length string fields, which MAY not be null-terminated.
  
  Such string fields hold possibly null-terminated UTF-8 string values (they MAY also hold an exact number of UTF-8 bytes without a terminating null byte).
- An optional field class describes an optional field.
  
  It is similar to a variant field class with two options: the optional field class and an “empty” field class.
- A static-length BLOB field class and a dynamic-length BLOB field class describe static- and dynamic-length sequences of contiguous bytes to represent BLOBs.
Modified field classes:
- The default alignment property of a fixed-length bit array field (and therefore also of fixed-length boolean, integer, enumeration, and floating point number fields) is 1.
  
  In CTF 1, it’s 8 if the field’s class’s size property is a multiple of 8, and 1 otherwise.
- The encoding property is removed from the fixed-length integer field class.
  
  The encoding property in CTF 1 indicates that a byte is a UTF-8 character, for example, but since there are UTF-8 characters which are encoded on more than one byte, this property actually makes no sense at the fixed-length integer field class level.
  
  You can use the new static-length string field class and dynamic-length string field class to achieve the same result.
- The exponent and mantissa size properties are removed from the fixed-length floating point number field class.
  
  A fixed-length floating point number field class is now defined as a fixed-length bit array field class, which has a single length property to indicate its total, fixed length, in bits.
  
  A CTF 2 consumer can decode a fixed-length floating point number field encoded following IEEE 754-2008's binary interchange format knowing only this parameter—the storage width of the bit array—from which it can deduce other parameters according to the standard.
  
  In other words, in CTF 1, only specific pairs of exponent and mantissa size properties are valid: 8 and 24, 11 and 53, 15 and 113, and so on.
- The encoding property is removed from the null-terminated string field class.
  
  CTF 2 null-terminated string fields always contain a sequence of bytes which encode a null-terminated string with UTF-8. If a null-terminated string field is encoded with the ASCII encoding in a CTF 1 data stream, it’s still valid in CTF 2 since an ASCII string is a UTF-8 string.
Relative field locations (variant field class’s tag/selector and dynamic-length field class’s length) are removed.

In CTF 2, you can specify a location to any valid field with an absolute field location.

This simplifies the format.
“Special” structure field member classes, such as the packet magic number member class, the data stream class ID member class, and the packet’s total size member class, can have any name: a field class MAY have one or more roles instead to indicate the purpose of its instances.

This means a CTF 2 consumer doesn’t need to rely on reserved names like CTF 1’s magic, stream_id, and packet_size.

This new approach also makes it possible for a given field class to have multiple roles.
The native byte order property is removed from the trace class.

In CTF 2, each fixed-length bit array field class MUST explicitly indicate the byte order of its instances.

This removes indirection, simplifying the format.
The EMF URI property is removed from the event record class.

This Eclipse Modeling Framework property is not common enough for the Common Trace Format.

A producer can still write such an URI as an event record class’s user attribute.
The log level property is removed from the event record class.

Log level integral values are tracer-specific.

A producer can still write such a log level value as an event record class’s user attribute.
A CTF 2 trace MAY have zero or more auxiliary streams.

An auxiliary stream contains structured information related to a specific trace which doesn’t fit the data stream model, whereas a CTF 2 metadata stream now describes a class of traces (multiple traces can share the same metadata stream).
The “call site” block feature is removed.

We don’t know any producer using this.

6. Common definitions

Common definitions for this specification:

Byte

A group of eight bits operated on as a unit.

Class

A set of values (instances) which share common properties.

For example, a fixed-length unsigned integer field class with an 8-bit length is the set of the all the fixed-length unsigned integer fields from 00000000 to 11111111 (values 0 to 255). .

This document often states that some class describes instances. For example, an event record class describes event records.

Consumer

A software or hardware system which consumes (reads) the streams of a trace.

A trace consumer is often a trace viewer or a trace analyzer.

Namespace

A string of which the purpose is to avoid naming conflicts.

This document doesn’t specify the format of a namespace. It is recommended to use a URI, or at least to include a domain name owned by the organization defining the objects under a namespace.

The std namespace is reserved for the CTF 2 specification.

Producer

A software or hardware system which produces (writes) the streams of a trace.

A trace producer is often a tracer.

Sequence

A set of related items that follow each other in a particular order.

Stream

A sequence of bytes.

7. Trace composition

A trace is:

One metadata stream.
One or more data streams.
Zero or more auxiliary streams.

As a reminder, a stream is defined as a sequence of bytes.

This document doesn’t specify how to transport or store CTF 2 streams. A producer could serialize all streams as a single file on the file system, or it could send the streams over the network using TCP, to name a few examples.

7.1. Metadata stream (overview)

A metadata stream describes trace data streams with JSON objects.

A metadata stream describes things such as:

The class of the data stream default clocks.
The names of event record classes.
The classes of event record fields.

Multiple traces MAY share the same metadata stream: a given trace MAY contain specific information in its own auxiliary streams.

See Metadata stream for the full metadata stream specification.

7.2. Data stream

A data stream is a sequence of one or more data packets:

In the metadata stream, a data stream class describes data streams.

A packet MUST contain one or more bytes of data.

Although a packet MAY contain padding at the end itself, from the data stream’s point of view, there’s no padding between packets. In other words, the byte following the last byte of a packet is the first byte of the next packet.

A data stream MAY have, conceptually:

One default, monotonic clock: Described by a clock class in the metadata stream.

Packets and event records MAY contain snapshots, named timestamps, of their data stream’s default clock.
One counter of discarded event records: Indicates the number of event records which the producer needed to discard for different reasons.

For example, a tracer could discard an event record when it doesn’t fit some buffer and there’s no other available buffer.

A packet MAY contain a snapshot of this counter.

See Data stream decoding procedure to learn how to decode a CTF 2 data stream.

7.2.1. Packet

A packet is a part of a data stream.

A packet contains a sequence of data fields or padding (garbage data). In the metadata stream, field classes describe data fields.

A packet P, contained in a data stream S, contains, in this order:

OPTIONAL: A header structure field, described at the trace class level in the metadata stream, which contains:
1. OPTIONAL: A packet magic number field (0xc1fc1fc1, or 3254525889).
2. Order not significant:
  - OPTIONAL: A trace class UUID field.
  - OPTIONAL: One or more fields which contain the numeric ID of the class of S.
  - OPTIONAL: One or more fields which contain the numeric ID of S.
OPTIONAL: A context structure field, described at the data stream class level in the metadata stream, which contains (order not significant):
- OPTIONAL: A field which contains the total size of P, in bits (always a multiple of 8).
- OPTIONAL: A field which contains the content size of P, in bits.
- OPTIONAL: A field which contains the beginning timestamp of P.
- OPTIONAL: A field which contains the end timestamp of P.
- OPTIONAL: A field which contains a snapshot of the discarded event record counter of S at the end of P.
- OPTIONAL: A field which contains the sequence number of P within S.
- OPTIONAL: User fields.
Zero or more event records.

A packet MUST contain one or more bytes of data.

A packet MAY have padding (garbage data) after its last event record. The size of this padding is the difference between its total size and its content size (as found in its context structure field).

Packets are independent of each other: if you remove a packet from a data stream, a consumer can still decode the whole data stream. This is why:

Packets MAY contain snapshots of their data stream’s discarded event record counter.
Packets and event records MAY contain timestamps which are snapshots of their data stream’s default clock.

If the packet context fields of a data stream’s packets contain a packet sequence number field, a consumer can recognize missing packets.

See Packet decoding procedure to learn how to decode a CTF 2 packet.

7.2.2. Event record

An event record is the result of a producer writing a record with OPTIONAL user data when an event occurs during its execution.

A packet contains zero or more event records.

An event record class describes the specific parts of event records.

An event record E, contained in a data stream S, contains, in this order:

OPTIONAL: A header structure field, described at the data stream class level in the metadata stream, which contains (order not significant):
- OPTIONAL: One or more fields which contain the numeric ID of the class of E which has the class of S as its parent.
- OPTIONAL: One or more fields which contain a timestamp or a partial timestamp.
OPTIONAL: A common context structure field, described at the data stream class level in the metadata stream, which contains user fields.
OPTIONAL: A specific context structure field, described at the event record class level in the metadata stream, which contains user fields.
OPTIONAL: A payload structure field, described at the event record class level in the metadata stream, which contains user fields.

An event record MUST contain one or more bits of data.

The default clock timestamp of an event record, that is, the value of its data stream's default clock after its header field, if any, is encoded/decoded MUST be greater than or equal to the default clock timestamp of the previous event record, if any, within the same data stream.

See Event record decoding procedure to learn how to decode a CTF 2 event record.

7.3. Auxiliary stream

An auxiliary stream is a JSON object, as specified by ECMA-404, which contains extra, structured information about the trace which doesn’t fit the data stream model.

An auxiliary stream has a single property:

Name	Auxiliary stream’s namespace.
Value	A JSON value.

Two auxiliary streams of a given trace MUST NOT have the same namespace.

Example 1. Auxiliary stream with the my.tracer namespace.

{
  "my.tracer": {
    "version": [1, 3, 2],
    "session-name": "hanon"
  }
}

Example 2. Auxiliary stream of which the value is just 42.

{
  "328c7a2d-a959-4f60-bd22-cca74359326f": 42
}

7.3.1. Trace environment

To remain backward compatible with CTF 1, a trace MAY contain an auxiliary stream having the std namespace which contains trace environment variables under the environment property.

The trace environment variables are a single JSON object where each property is:

Name	Trace environment variable name.
Value	Trace environment variable value (any JSON value).

This document doesn’t specify trace environment variable names.

Example 3. std auxiliary stream with trace environment variables.

{
  "std": {
    "environment": {
      "hostname": "hanon",
      "domain": "kernel",
      "sysname": "Linux",
      "kernel_release": "4.12.12-1-ARCH",
      "kernel_version": "#1 SMP PREEMPT Sun Sep 10 09:41:14 CEST 2017",
      "tracer_name": "lttng-modules",
      "tracer_major": 2,
      "tracer_minor": 10,
      "tracer_patchlevel": 0
    }
  }
}

8. Metadata stream

A metadata stream is a JSON array, as specified by ECMA-404, of fragments.

Together, the fragments of a metadata stream contain all the information about the data streams of one or more traces.

A fragment is a JSON object; its allowed properties depend on its type property.

Table 2. Common properties of a fragment F.
Name	Type	Description	Required?	Default
`type`	JSON string	Type of F. The value of this property MUST be one of: `"preamble"` F is a preamble fragment. `"trace-class"` F is a trace class fragment. `"clock-class"` F is a clock class fragment. `"data-stream-class"` F is a data stream class fragment. `"event-record-class"` F is a event record class fragment.	Yes
`user-attributes`	User attributes	User attributes of F.	No	`{}`
`extensions`	Extensions	Extensions of F. For any fragment except a preamble fragment, any extension which exists under this property must also be declared in the metadata stream’s preamble fragment.	No	`{}`

The metadata stream is designed as a flat JSON array of fragments instead of a single JSON object containing nested objects to enable real-time, or “live”, tracing: a consumer can always decode event records having known event record classes while a producer can always add new event record classes to a data stream class by appending additional fragments to the metadata stream. Once a producer appends a fragment to a metadata stream, the fragment is considered “frozen”, in that it never needs to change.

A metadata stream:

MUST start with a preamble fragment.
MUST contain exactly one preamble fragment.
MAY contain one trace class fragment.
MUST contain one or more data stream class fragments which MUST follow the trace class fragment, if any.
MAY contain one or more event record class fragments which MUST follow their parent data stream class, if any.

Example 4. Partial metadata stream.

[
  {
    "type": "preamble",
    "version": 2
  },
  …
]

This section doesn’t specify how a metadata stream translates into data stream encoding and decoding rules; it only describes objects and their properties.

See Data stream decoding procedure to learn how to decode a data stream.

8.1. UUID

Both a trace class fragment and a clock class fragment MAY have a UUID property.

Within a metadata stream, a UUID is a JSON string of which the value is the canonical textual representation of the UUID.

Example 5. UUID.

"e53e0ab8-50a1-4f0a-b710-b5f0bba9c4ac"

8.2. Extensions

A producer MAY add extensions to many metadata stream JSON objects.

The purpose of an extension is to add core features to CTF 2 or to modify existing core features, as specified by this document. In other words, an extension MAY alter the format itself. This document doesn’t specify what an extension exactly is.

The metadata stream’s preamble fragment contains extension declarations:

Any extension in metadata stream objects MUST be declared, by namespace and name, in the preamble fragment.

Declaring an extension is said to enable it.
If a consumer doesn’t support any declared extension, it MUST NOT consume the trace's data streams.

The consumer SHOULD report unsupported extensions as an error.

Extensions are a single JSON object, where each property is:

Name	A namespace
Value	A namespaced extensions object

A namespaced extensions object is a JSON object, where each property is:

Name	An extension name
Value	A JSON value

The metadata stream JSON objects which MAY contain extensions as their extensions property are:

Any fragment.

An extension in the preamble fragment also makes it declared.
Any field class.
A structure field member class.
A variant field class option.

Example 6. Three extensions under two namespaces.

{
  "my.tracer": {
    "piano": {
      "keys": 88,
      "temperament": "equal"
    },
    "ramen": 23
  },
  "abc/xyz": {
    "sax": {
      "variant": "alto"
    }
  }
}

8.3. User attributes

A producer MAY add custom user attributes to many metadata stream JSON objects.

This document doesn’t specify what a user attribute exactly is. Unlike extensions, a consumer MUST NOT consider user attributes to decode data streams.

User attributes are a single JSON object, where each property is:

Name	A namespace
Value	A JSON value

The metadata stream JSON objects which MAY contain user attributes as their user-attributes property are:

Any fragment.
Any field class.
A structure field member class.
A variant field class option.

Example 7. User attributes under two namespaces.

{
  "my.tracer": {
    "max-count": 45,
    "module": "sys"
  },
  "abc/xyz": true
}

8.4. Field classes

A field class describes fields, that is, sequences of bits as found in a data stream.

A field class contains all the properties a consumer needs to decode a given field.

A field is a field class instance.

This document specifies the following types of field classes:

Abstract field classes

You cannot use the following field classes directly: they are bases for other, concrete field classes:

Abstract integer field class
Abstract enumeration field class
Abstract array field class
Abstract BLOB field class

Fixed/static-length field classes

Fixed-length bit array field class
Fixed-length boolean field class
Fixed-length integer field class
Fixed-length enumeration field class
Fixed-length floating point number field class
Static-length string field class
Static-length BLOB field class

Variable/dynamic-length field classes

Variable-length bit array field class
Variable-length integer field class
Variable-length enumeration field class
Null-terminated string field class
Dynamic-length string field class
Dynamic-length BLOB field class

Compound field classes

The following field classes contain one or more field classes.

Structure field class
Static-length array field class
Dynamic-length array field class
Optional field class
Variant field class

A field class is a JSON object; its properties depend on its type property.

Table 3. Common properties of a field class F.
Name	Type	Description	Required?	Default
`type`	JSON string	Type of F. The value of this property MUST be one of: `"fixed-length-bit-array"` F is a fixed-length bit array field class. `"fixed-length-boolean"` F is a fixed-length boolean field class. `"fixed-length-unsigned-integer"` `"fixed-length-signed-integer"` F is a fixed-length integer field class. `"fixed-length-unsigned-enumeration"` `"fixed-length-signed-enumeration"` F is a fixed-length enumeration field class. `"fixed-length-floating-point-number"` F is a fixed-length floating point number field class. `"variable-length-bit-array"` F is a variable-length bit array field class. `"variable-length-unsigned-integer"` `"variable-length-signed-integer"` F is a variable-length integer field class. `"variable-length-unsigned-enumeration"` `"variable-length-signed-enumeration"` F is a variable-length enumeration field class. `"null-terminated-string"` F is a null-terminated string field class. `"static-length-string"` F is a static-length string field class. `"static-length-blob"` F is a static-length BLOB field class. `"dynamic-length-string"` F is a dynamic-length string field class. `"dynamic-length-blob"` F is a dynamic-length BLOB field class. `"structure"` F is a structure field class. `"static-length-array"` F is a static-length array field class. `"dynamic-length-array"` F is a dynamic-length array field class. `"optional"` F is a optional field class. `"variant"` F is a variant field class.	Yes
`roles`	Roles	Roles of an instance of F. See Trace class fragment and Data stream class fragment which indicate accepted roles for their root field classes.	No	`[]`
`user-attributes`	User attributes	User attributes of F.	No	`{}`
`extensions`	Extensions	Extensions of F. Any extension which exists under this property must also be declared in the metadata stream’s preamble fragment.	No	`{}`

The following fragment properties require a structure field class as their value:

Trace class fragment

packet-header-field-class

Data stream class fragment

packet-context-field-class
event-record-header-field-class
event-record-common-context-field-class

Event record class fragment

specific-context-field-class
payload-field-class

8.4.1. Field location

A field location is a means for a consumer to find a field which it needs to decode another field.

A consumer needs to find another field to decode instances of the following classes:

Dynamic-length array field class
Dynamic-length string field class
Dynamic-length BLOB field class: Needs a fixed-length unsigned integer or variable-length unsigned integer length field.
Optional field class: Needs a fixed-length boolean, fixed-length integer, or variable-length integer selector field.
Variant field class: Needs a fixed-length integer or variable-length integer selector field.

A field location is a JSON array, where, in this order:

The first element is the name (JSON string) of a root field from where to start the lookup, amongst:

`"packet-header"`	Packet header
`"packet-context"`	Packet context
`"event-record-header"`	Event record header
`"event-record-common-context"`	Event record common context
`"event-record-specific-context"`	Event record specific context
`"event-record-payload"`	Event record payload

The following elements are structure field member names (JSON strings) to follow to find the target field.

The length of a field location MUST be greater than or equal to two.

Let T be a field which a consumer needs to decode another field S:

If S is in a packet header, then T MUST be in the same packet header.
If S is in a packet context, then T MUST be in one of:
- The packet header the same packet.
- The same packet context.
If S is in an event record header, then T MUST be in one of:
- The packet header of the same packet.
- The packet context of the same packet.
- The same event record header.
If S is in an event record common context, then T MUST be in one of:
- The packet header of the same packet.
- The packet context of the same packet.
- The event record header of the same event record.
- The same event record common context.
If S is in an event record specific context, then T MUST be in one of:
- The packet header of the same packet.
- The packet context of the same packet.
- The event record header of the same event record.
- The event record common context of the same event record.
- The same event record specific context.
If S is in an event record payload, then T MUST be in one of:
- The packet header of the same packet.
- The packet context of the same packet.
- The event record header of the same event record.
- The event record common context of the same event record.
- The event record common specific of the same event record.
- The same event record payload.
If S and T are not in the same root field, then T MUST NOT be in any array or optional field.
If S and T are in the same root field, then:
- If S is in an array field, then T must be in the same array field element.
- If S is in an optional field, then T must be in the same optional field.

If any structure member name N of a field location L names a variant field, then:

If N is not the last element of L: The variant field MUST select a structure field, from which the lookup process can continue, recursively.
If N is the last element of L: The variant field MUST select the target field (fixed-length integer, variable-length integer, or fixed-length boolean), recursively.

In both cases, all the options of the variant field class MUST make it possible for the lookup process to continue.

Example 8. Dynamic-length array field and its length field in the same root field.