CTF2-PROP-1.0: Proposal for a major revision of the Common Trace Format, version 1.8

This document is an informal proposal for the next major revision of the Common Trace Format (CTF): version 2.0 (hereafter named CTF 2).

This is not a formal reference. Some parts of this document, however, may be formal enough to be elligible for a specification document.

Note	RFC 2119 The key words must, must not, required, shall, shall not, should, should not, recommended, may, and optional in this document, when emphasized, are to be interpreted as described in RFC 2119.

A new workgroup responsible for the publication of CTF documents

The DiaMon Workgroup, a Linux Foundation workgroup which creates de-facto standards and tools for tracing, monitoring, and diagnostics, is now responsible for the publication of the official documents about CTF.

The DiaMon Workgroup is also responsible for making available a platform where interested parties can comment on the proposals related to CTF.

A new method for identifying CTF 2 documents

We suggest that all documents related to CTF 2 bear a unique document identifier (ID) having the following format:

CTF2-<short name>-<major>.<minor>[r<revision>]

Table 1. Descriptions and roles of CTF 2 document ID parts
Part	Description	Bump may introduce new concepts, procedures, and formats?	Bump may remove or change existing concepts, procedures, and formats?
`<short name>`	The capitalized short name of the document, unique amongst all the CTF 2 documents.	N/A	N/A
`<major>`	The major version number of the document.	Yes	Yes
`<minor>`	The minor version number of the document.	Yes	No
`<revision>`	The revision letter of the document (from `A` to `Z`). Document revisions are used to add examples, clarify existing concepts, fix grammar or content mistakes, or reword existing parts, for example.	No	No

For example, the short name of this document is PROP, for proposal, and its full document ID is CTF2-PROP-1.0. The next revision would be CTF2-PROP-1.0rA, and the following would be CTF2-PROP-1.0rB.

In any CTF 2 document, another CTF 2 document must be referred to by using only its ID. For example: This concept is further explained in CTF2-SOMEID-1.2. There is no need to refer to a specific revision: the reference always targets the latest document’s revision.

We suggest the following IDs for the initial documents:

CTF2-DOCID-1.0: CTF 2 document identifier format
CTF2-SPEC-2.0: The Common Trace Format (CTF), version 2.0
CTF2-BASICATTRS-1.0: Basic CTF 2 user attributes
CTF2-FS-1.0: Layout of a CTF 2 trace stored on a file system
CTF2-PMETA-1.0: CTF 2 metadata stream packet format

Why CTF 2?

Why do we need a major version bump of the CTF specification?

A major version bump is never an easy choice when it comes to revisiting a software library, a communication protocol, a file format, or anything that serves as a contract between a producer and a consumer. When such a decision is taken, it must be justified by solid arguments, since it makes it impossible for old consumers to consume the product of new producers.

In this proposal, for instance, CTF 2 traces are not backward compatible with CTF 1 traces. Although the binary format is not changed, the metadata stream is written in a different language.

CTF 1 has been used and tested for many years now, by different producers and consumers. Over that time, we have noted a few gaps in the trace format, gaps that prevent us from extending CTF 1 as much as we would like, amongst other things. CTF 2 is designed to overcome those gaps, as far as we know them, and to be flexible enough to gracefully accept future additions while avoiding another major version bump in the following years.

Design goals

The design goals of CTF 2 are as follows, in order, beginning with the most important:

CTF 2 data streams must be backward compatible with CTF 1 data streams.

Many applications are already programmed to write valid CTF 1 packets. Modifying the code of those applications to produce different binary packets can be cumbersome, and sometimes impossible if the application passed acceptance tests, for example.

Making sure that applications producing CTF 1 traces can also produce CTF 2 traces only by changing the metadata stream is an absolute necessity.
The CTF 2 streams must be as efficient as possible to produce by a tracer.

This design goal was also one of the major ones which motivated the design of CTF 1.

In other words, a small embedded system must be able to produce CTF 2 streams natively. Moreover, the tracer must be able to copy binary data to the packet buffer of a data stream without altering it.
CTF 2’s model should be backward compatible with CTF 1’s.

Some APIs are already written to deal with CTF 1 “objects”, or concepts (event record classes, event records, field types, and data stream clock classes, to name a few). The model of CTF 2 should be compatible with the model of CTF 1, so that those existing APIs can operate on CTF 2 objects too without requiring huge refactorings.
The size of any CTF 2 static field, that is, any field with a non-dynamic type, must always be the same in data streams to speed up the validation in some situations.

In other words, a field described by a fixed-size type, or by a compound type containing only fixed-size types, should always have the same size, no matter its offset in the data stream.

This guarantee allows, in certain situations, to greatly speed up the validation process by having a great part of it done at the metadata stream level.
A CTF 2 trace should be as easy as possible to consume.

CTF 1 focuses on being easy to be produced, which is a good idea since producers are tracers in this context, and a minimal tracer should be able to produce a correct CTF trace with minimal code (design goal 2).

However, because CTF 1’s metadata stream is written in TSDL, a custom, declarative, C-like DSL designed for CTF, writing a minimal consumer of CTF 1 is not an easy task. TSDL is an intricate language, with many special cases, many of which are borrowed from the C language, which cannot be ignored when writing a consumer supporting all its features. TSDL was developed to ease the manual (human) production of metadata streams.

Over time, we realized that, while producing traces is important, consuming them to solve problems by analysing the event records is just as important, if not more.

This is why CTF 2 should encourage the development of CTF consumers in any programming language by reducing the number of special cases, as well as by using a very simple, yet well-known grammar for the metadata stream.

CTF 1 tries to accomodate other trace formats, which can be converted to CTF without changing the data streams by writing the matching metadata stream. This is also a source of special cases. CTF 2 should build on binary trace conversion (from another, non-CTF trace format to CTF 2) rather than trying to accomodate other formats.
A CTF 2 metadata stream must be extensible by users and by future minor revisions of the specification (forward compatibility).

CTF 1’s TSDL grammar is pretty restrictive when it comes to customizing existing blocks with user-defined attributes.

Many protocols and declarative languages support custom user data in their payload. For example, HTML5 allows any element to have user attributes by prefixing their names with data-.

A producer of CTF 2 should be able to add custom attributes to almost any object defined by the specification. This allows standard consumers to read any CTF 2 trace and ignore unknown user attributes, providing a “bland”, yet complete view of the trace fields, while special consumers can be written (or existing consumers can be extended) to interpret specific user attributes and use them to present a meaningful visualization.

It is also possible for the DiaMon Workgroup to publish a proposal for new properties for a next minor revision of the specification, but test them as temporary user attributes for some time in order to collect comments before updating the specification itself.

This design goal also means that an “old” CTF 2 consumer should be able to decode a “new” CTF 2 trace, possibly with missing field semantics if field types are added in minor revisions.
CTF 2’s specification must focus on how to encode and decode data streams.

CTF 1 has a few base properties in its metadata stream that are not strictly needed to encode or decode data streams. For example, the base property of the integer field type is only useful to visualize the decoded integer fields: the decoding process does not depend on a preferred radix. Also, the name of an event record class is not needed to decode the event records it describes: only its numeric ID is needed to select the appropriate context and payload field types to use for the encoding/decoding process.

All the object properties defined by the CTF 2 specification document should exist only because they have a role in the encoding/decoding process. Everything else should be delegated to other documents which define optional extension layers using the mechanisms designed as a response to design goal 6.
CTF 2’s specification must not specify how to transport a trace, nor how a trace should be stored.

In the CTF 2 specification, a CTF 2 trace should be defined as a set of bit streams, without specifying how those streams are transported or stored. Other official documents published by the DiaMon Workgroup can define standard ways to support and transport CTF 2 streams for specific use cases. Trace producers and consumers can choose to implement one or more transport/storage mechanisms by following the other documents.

Changes since CTF 1

Here is a brief summary of the changes, from CTF 1 to CTF 2, introduced by this proposal.

The terminology of the specification and the binary layouts of the data streams are completely detached from the C language and from the behaviour of any C compiler.

CTF 2 is a programming language-agnostic trace format.
Terminology update:
- Binary stream → data stream.
- Trace → trace class, when it names a block of metadata which describes traces.
- Stream → data stream class, when it names a block of metadata which describes data streams.
- Stream ID → data stream class ID.
- Stream instance ID → data stream ID.
- Event → event record class, when it names a block of metadata which describes event records.
- Event ID → event record class ID.
- Event → event record, when it names an actual recorded event contained in a packet.
- Declaration → field type.
- Type alias → field type alias.
- Clock → data stream clock class, when it names a block of metadata which describes actual data stream clocks.
- Clock → data stream clock, when it names a per-data stream instance of a specific data stream clock class.
- Native byte order → default byte order.
- Stream packet context → data stream packet context.
- Stream event header → data stream event record header.
- Stream event context → data stream event record context.
- Event context → event record context.
- Event fields → event record payload.
- Events discarded → discarded event record count.
- Packet size → packet’s total size.
- Content size → packet’s content size.
The metadata stream is written in JSON. JSON values are used to represent metadata types, the internal type system of the CTF 2 metadata.
The CTF 2 specification does not specify a “packetized” metadata stream format. This is a back-end-specific way of wrapping a metadata stream as defined in this document. Another document specifies the packetized metadata stream format, which can be used by standard storage and transport documents if needed.
Most objects (called m‑map values) of the CTF 2 metadata model may contain custom user attributes.
Properties that are not strictly needed to encode or decode the data streams are defined as user attributes in other documents. This includes, but is not limited to, the trace’s environment, the event record class’s name and log level, and the integer field type’s preferred display base (CTF 1’s base property).
New field types:
- A bit array field type describes the most primitive fields of a data stream actually holding values.
  
  An integer field type is now defined as a bit array field type with a signedness property.
  
  A floating point number field type is now defined as a bit array field type: its size property is enough to determine its encoding under IEEE 754-2008’s binary interchange format.
- A null field type, which represents nonexistent, or missing fields. The intended use of this field type is as one of the possible types of a variant field type, especially when representing the types of a dynamically-typed language using a variant field type.
  
  This field type should be used to represent Python’s None, Java’s and JavaScript’s null, and Ruby’s nil, for example.
- A boolean field type, which is a bit array field type with a special meaning. When all the bits of a boolean field are cleared, the field’s value is said to be false. Otherwise, the field’s value is said to be true.
- A variable-length bit array field type. Each byte of a field having this type has its most significant bit set if one more byte must be encoded. The 7 low-order bits of each byte are concatenated in a specific way to form the final, equivalent bit array.
- A variable-length boolean field type, a variable-length integer field type, and a variable-length enumeration field type, which use the encoding mechanism of the variable-length bit array field type.
- A text array field type and a text sequence field type, which are specialized versions of the array field type and sequence field type with a byte as the element field type. Text array and sequence fields hold possibly null-terminated UTF-8 string values.
- A union field type, which provides a list of one or more field types, each of which is an alternative representation of the same binary field.
  
  This mechanism may be used to add field types in minor revisions of CTF 2, while still ensuring the forward compatibility of “old” consumers.
Modified field types:
- All CTF 2 field types have an alignment property. Some field types, however, impose alignment constraints to match the constraints of CTF 1’s field types and to make consumers easier to develop.
- The default alignment of an integer field type is 1. It used to be 8 if the size is a multiple of 8, and 1 otherwise, in CTF 1.
- The base and encoding properties are removed from the integer field type. The encoding property is used in CTF 1 to indicate that a byte is a UTF-8 character, for example, but since some UTF-8 characters are encoded on more than one byte, this property makes no sense here. The new text array field type and the text sequence field type may be used to achieve the same result instead.
- The exp and mant properties are removed from the floating point number field type. As mentioned above, a floating point number field type is now defined as a bit array field type, which has a size property to indicate its total, fixed size, in bits. A floating point number field encoded following IEEE 754-2008’s binary interchange format can be decoded knowing only this parameter, the storage width of the bit array, from which other parameters can be deduced according to the standard.
  
  In other words, as far as CTF 1’s floating point number field type is following IEEE 754-2008’s binary interchange format for encoding its fields, only specific pairs of exp and mant properties are valid: 8 and 24, 11 and 53, 15 and 113, 20 and 237, and so on.
- The encoding property of the string field type is removed. CTF 2 string fields always contain a sequence of bytes which encode a string with UTF-8. If a string is encoded with the ascii encoding in a CTF 1 data stream, it’s still valid in CTF 2 since an ASCII string is a UTF-8 string.
The mechanism by which a relative variant field type’s tag or sequence field type’s length is searched for in preceding scopes (trace packet size, data stream packet context, event record context, event record payload, etc.) when it’s not found in the current scope is removed. It has not be shown to be useful and it adds complexity to consumers. It is always possible to convert a CTF 1 metadata stream using this behaviour to a CTF 2 metadata stream not using it.
“Special” fields, such as the magic number field, the data stream class ID field, and the packet’s total size field, can have any name: they are tagged with specified tag names by the producer for the consumer to know that they have a special meaning without relying on reserved names like CTF 1’s magic, stream_id, and packet_size.

With this mechanism, it’s possible for a field to be tagged multiple times, with different meanings.
The frequency of a data stream clock class no longer defaults to the arbitrary 1 GHz (it’s a mandatory property).
There is no more lexical scopes to limit the scope of field type aliases and other definitions. Field type aliases must have unique names within the whole metadata stream.

CTF 2 actors

There are two main actors when it comes to tracing:

Producer: A software or hardware system which produces (writes) the streams of a trace.

A trace producer is often called a tracer.

A producer is only concerned with how to write the metadata stream and how to encode supported values as CTF 2 data fields and serialize them to one or more data streams.
Consumer: A software or hardware system which consumes (reads) the streams of a trace.

A trace consumer is often called a trace viewer or a trace analyzer.

A consumer is only concerned with how to read and interpret the metadata stream, and how to deserialize CTF 2 data fields from data streams and decode them to retrieve the values they represent.

Note that a piece of software can be both a consumer and a producer. This is the case of a trace converter, for example.

What is a CTF 2 trace?

A CTF 2 trace is a set of zero or more data streams and exactly one metadata stream.

The data streams contain actual packets of event records, while the metadata stream contains information on how to interpret the data streams of the same trace.

The support or transport of data streams and the metadata stream is not specified. A stream may be serialized as a single file on a file system, or it may be sent over the network using TCP, to name a few examples. The mechanism to identify the metadata stream amongst the streams of a trace is also not specified.

Metadata types

Many of the following sections of this document describe the required and optional properties of metadata m‑map values. All the metadata m‑map values can be represented using a defined set of types. The values allowed by those types have no specific textual or binary representation.

To avoid any confusion with field types and JSON types, the m- prefix is used before the names of the metadata types. An m-value is any value allowed by any of the following types.

Note

Not to be confused with field types!

The metadata types define the possible values that can be used to define the metadata m‑map values, for example:

Integer field type m‑map
Structure field type m‑map
Event record class m‑map
Data stream class m‑map

As an example, here’s a JSON representation of a possible integer field type m‑map:

{
  "field-type": "int",
  "size": 23,
  "alignment": 16
}

Here:

The whole JSON object represents an m‑map value.
"field-type" represents an m‑string value, which is an m‑map key here.
23 represents an m‑number value.

The whole m‑map value represents an integer field type. This integer field type can be used to encode and decode integer values to and from binary data fields, depending on where exactly this m‑map is placed within the whole metadata m‑array.

The metadata types are:

m‑null

Nullable type: the only possible value of this type is the null value.

m‑bool

Boolean type, that is, the following set of values:

True
False

m‑int

Integer type, that is, the set of all the negative and positive integer values.

m‑number

Number type, that is, the set of all the rational numbers that can be represented with the decimal representation.

m‑string

String type, that is, the set of all the possible finite sequences of Unicode characters, including the zero-length sequence.

m‑array

Array type, that is, the set of all the possible finite sequences of any m-values, including the zero-length sequence.

m‑map

Unordered map type, that is, the set of all the possible sets of (m‑string value, m-value) pairs, including the zero-length set. The m‑string value in a given pair is called the key of the association. An association may also be called a property.

Note	For reasons of brevity and style, the word value after a metadata type name is sometimes discarded in this text. For example, you can use an `m‑int` to… means you can use an `m‑int` value to….

Bidirectional association between metadata types and JSON values

The metadata types can be bidirectionally mapped to JSON values as follows:

Table 2. Bidirectional association between metadata types and JSON values
Metadata type	JSON values
`m‑null`	The null m-value is mapped to the JSON `null` value.
`m‑bool`	The true and false m-values are mapped to the `true` and `false` JSON values respectively.
`m‑int`	Any allowed m-value is mapped to a JSON number without a fractional part or to a constant integer JSON object.
`m‑number`	Any allowed m-value is mapped to a JSON number or to a constant integer JSON object.
`m‑string`	Any allowed value is mapped to a JSON string.
`m‑array`	Any finite sequence is mapped to a JSON array, where each m-value is mapped to a JSON value using this table.
`m‑map`	Any set of associations is mapped to a JSON object, where each pair’s `m‑string` value is mapped to a key (JSON string, using this table) in this JSON object, and its associated m-value is mapped to a JSON value using this table and associated to this key.

All the examples of m-values in this document use this mapping to show textual representations.

Constant integer JSON object

Unfortunately, JSON does not support binary, octal, or hexadecimal constant integers. Also, it is known that some JSON parsers have a limited support for big integers (generally, integer values which do not fit a 64-bit representation). A constant integer JSON object can represent an m‑int value or an m‑number value without a fractional part.

It is recommended to use a constant integer JSON object instead of a JSON number when the m‑int or m‑number value to represent is lesser than -9223372036854775808 or greater than 9223372036854775807 (signed 64-bit range).

Table 3. Constant integer JSON object properties
Name	Allowed JSON values	Description	Required?	Default value
`base`	`2`, `8`, `10`, and `16`	Radix of the number.	Optional	`10`
`value`	JSON string	Integer’s value using the digits allowed by the base, without the typical `0b`/`0`/`0x` prefix. The string may be prefixed with `-` (U+002D) if the value is negative.	Required

Example 1. Constant integer JSON object: positive decimal integer

Equivalent to 2876321721982327:

{"value": "2876321721982327"}

Example 2. Constant integer JSON object: negative binary integer

Equivalent to -253339:

{"base": 2, "value": "-111101110110011011"}

Example 3. Constant integer JSON object: positive octal integer

Equivalent to 420:

{"base": 8, "value": "644"}

Example 4. Constant integer JSON object: positive hexadecimal integer

Equivalent to 3735928559:

{"base": 16, "value": "deadbeef"}

Example 5. Constant integer JSON object: negative decimal integer

Equivalent to -2317:

{"value": "-2317"}

Metadata `m‑array`

The metadata m‑array is an m‑array of fragments which contains all the metadata information of a given trace.

A fragment is an m-value.

A fragment is either:

The version fragment, that is, an m‑string which is always CTF 2.
One of the other allowed fragments, which are described in the upper layers of CTF 2.

The first fragment in the metadata m‑array is always the version fragment. It is followed by one or more fragments, as described by the upper layers of CTF 2.

Metadata stream

A metadata stream is the JSON representation of a metadata m‑array, that is, a UTF-8 JSON array which is written by the producer to describe the data streams of the same trace.

The rationale for choosing JSON over another representation, for example TSDL (CTF 1’s metadata language), is as follows:

JSON can represent all the possible m-values using the m-value to JSON value assocation table.
JSON is a simple language to consume. A very basic JSON parser can be written in a few hundred lines of C code. Moreover, tested and documented JSON parsers exist for all the major programming languages.
JSON is a very simple language to produce.
JSON strings support Unicode.

One of the design goals of CTF 2 is to make consumption as easy as possible. Relieving the burden of implementing a custom TSDL parser is a substantial part of how this goal is achieved.

Keep in mind that this JSON metadata is expected to be generated by machines, thus shortcuts that would save time to human beings are avoided in favor of easier consumption, but without compromising easy and fast machine generation.

Data streams

A CTF 2 data stream is a sequence of packets.

Each packet starts with an optional header field followed by an optional context field, after which is a sequence of event records.

An event record starts with an optional header, followed by an optional context field defined at the data stream class level, followed by an optional context field defined at the event record class level, followed by an optional payload field. An event record’s total binary size must be greater than 0.

Summary

A CTF 2 trace is a set of:

One metadata stream, which is the UTF-8 JSON representation of a metadata m‑array. A metadata m‑array is an m‑array containing fragments. Fragments are m-values which describe properties of the trace and the binary layouts of its various parts.
Zero or more data streams. A data stream contains packets. A packet contains event records. The layout of packet headers and contexts, and of event record headers, contexts, and payloads, are described by the metadata stream of the same trace.

Structure of the CTF 2 specification

We suggest that the concepts of CTF 2 be presented in the specification document as three layers:

The first layer, named the CTF 2 field encoding rules (CTFFER), shows how to encode common programming language values as binary data fields according to the their descriptions, field types. This is a serialization protocol. This is the foundation of CTF 2, in that the other layers need data fields to have any meaning. This layer is independent of the tracing domain, in that it can be used to encode any self-described bit stream for any application.
The second layer adds the concept of packets and event records, and how different layouts of packet header and context fields, and of event record header, context, and payload fields, may exist within different data streams and event records of the same trace thanks to data stream class fragments and event record class fragments in the metadata stream.

This layer also introduces the field type alias fragment and the trace class fragment.
The third layer adds the concept of time (clocks). Clocks are essential data stream variables in a CTF 2 trace because they associate event records and packets with one or more points in time. Data stream clocks are sampled by the producer when writing specific, regular data fields. They are updated by the consumer when reading the corresponding fields. Data stream clocks are described by data stream clock class fragments in the metadata stream.

CTF 2 is designed so that each layer may be implemented in its own software package. This structure separates the concepts of CTF 2 into different sections of the text, making it more easy to read. This structure should also make testing more easy.

Each layer depends on the previous one.

Compliance

A CTF 2 producer, either a piece of software or a machine, must implement the first two layers of CTF2-SPEC-2.0.

A CTF 2 consumer, either a piece of software or a machine, must implement all three layers of CTF2-SPEC-2.0.

Layer 1: CTF 2 field encoding rules (CTFFER)

The CTF 2 field encoding rules, or CTFFER, dictate how to serialize a value to a binary data field by using the properties of a field type, that is, an m‑map which describes a set of possible binary data field values. The field type can later be used to deserialize a data field back to a value.

The representation of a value depends on the programming language in which the CTF 2 producer or consumer is written here. For example, a producer written in C may serialize an int variable as a CTF 2 integer field described by an integer field type having the appropriate size, alignment, byte order, and signedness properties to accommodate any value that an int variable could hold for a specific architecture (32-bit for IA-32 and 16-bit for AVR, for example). However, since a Python 3 int object can hold any integer value, a better choice for a Python 3 producer would be to serialize such an object as a CTF 2 variable-length integer field described by a variable-length integer field type.

The following subsections only describe how to encode values as binary fields by using field types. The specification does not suggest specific field type configurations. It is up to the producer side to choose appropriate field type properties depending on its environment. The procedures to encode values presented in the following subsections are very generic: they take no account of optimizations that would be possible in specific situations. For example, it is often possible, in programming languages which are aware of their memory layout, to encode a whole complex structure of values by a simple memory copy to the data stream, as far as appropriate field types are used to describe this exact memory layout for further correct decoding. It is in order to satisfy those situations, which support faster producers, that field types are flexible: a produder can encode some value in various ways by choosing the resulting field’s alignment, byte order, size, and the rest within the data stream destination.

The values which can be encoded are:

Null values: A value with no size, which usually represents missing or unknown data.
Bit array values: A finite sequence of contiguous bits without a specific meaning. The required size, in bits, to represent a bit array value can be known statically or dynamically.
Boolean values: A bit array value which is either false (all bits are cleared) or true (anything else).
Integer values: A bit array value which represents an integer (signed or not).
Number values: A real number which can be represented with IEEE 754-2008’s binary interchange format.
Enumeration values: An integer value with an associated label (known by its type).
String values: A finite sequence, with a length known dynamically, of Unicode characters.
Structure values: A finite sequence of values of different types.
Array values: A finite sequence, with a length known statically, of values sharing the same type.
Sequence values: A finite sequence, with a length known dynamically, of values sharing the same type.

The CTFFER also support variant and union fields. A variant field is an encoded value of a given type amongst many possible types. This type is dynamically chosen by a tag (a previously encoded enumeration value). A union field is a data field which represents, at the same time, different values of different types.

Byte order `m‑string`

A byte order m‑string is one of the following values:

default: Use default byte order.
be: Big-endian.
le: Little-endian.

CTF 2 does not support middle or mixed endianness.

User attributes `m‑map`

Many metadata m‑map values described in this document may have a user-attrs property, which must be set to an m‑map, if set at all.

Each key of the user attributes m‑map is a namespace. The value of a given key is the custom user attribute within this namespace (any m-value is valid).

The format of a namespace is not specified. It is recommended to use a URI, or at least to include a domain name owned by the organization defining the attributes nested under this namespace. A UUID is also a rational option.

What to do with those user attributes from a consumer’s standpoint is not specified by this document. The values of those attributes are not needed to decode the CTF data streams, and may be safely ignored by any CTF 2 consumer.

It is expected that “industrial standards” defining sets of useful attributes within given namespaces will emerge naturally over time. Producers and consumers supporting the same attributes can enhance the experience of the whole tracing ecosystem.

It is expected that user attributes usually fall into one of the following categories:

Model: Information about the application data model of an object.
Textual style: Style attributes/hints that could be applied to a textual rendering of the object (color, font attributes, print format, etc.).
Graphical style: Style attributes/hints that could be applied to a graphical output of the object.

Example 6. User attributes m‑map with namespace diamon.org/ctf/ns/std

JSON representation:

{
  "diamon.org/ctf/ns/std": {
    "base": 16
  }
}

Example 7. User attributes m‑map with different namespaces

JSON representation:

{
  "diamon.org/ctf/ns/std": {
    "name": "sched_switch",
    "ns": "lttng.org/ctf-ns/modules/2.9"
  },
  "lttng.org/ctf-ns": {
    "tmp-event": true,
    "ignore-ip": true
  }
}

Example 8. User attributes m‑map with namespace mytracer.org/ctf-ns/hints

JSON representation:

{
  "mytracer.org/ctf-ns/hints": {
    "format-string": "{src} sent {size} bytes to {dst} at {addr}"
  }
}

Although not recommended, an empty m‑string is a valid namespace:

Example 9. User attributes m‑map with empty namespace

JSON representation:

{
  "": {
    "my-option": 23,
    "include": ["this", "and", "that"]
  }
}

The value of user attributes for a given namespace need not be an m‑map:

Example 10. User attributes m‑map with an m‑number attribute

JSON representation:

{
  "my namespace": -17.22
}

Scope

A scope is a specific field within a data stream. The exact location of a scope within a data stream depends on the current encoding context. The upper layer defines which scopes are available in its context and how to find them by name (m‑string).

Field path m-value

A field path m-value, used by sequence and variant field types, is a path leading to a previously encoded data field by “digging” into structure and union fields. It can be either relative (starting from a known field), or absolute (starting from a user-specified scope field).

Absolute field path `m‑map`

An absolute field path m‑map defines a field path from a specific scope.

Table 4. Absolute field path `m‑map` properties
Name	Type	Description	Required?	Default m-value
`scope`	`m‑string`	Name of the scope.	Required
`path`	`m‑array` of `m‑string` values	Field names, from the scope’s root field, to follow to reach the desired field (last element of this path).	Required

The path property may be empty: this targets the scope field itself.

Example 11. Absolute field path m‑map

JSON representation:

{
  "scope": "data-stream-packet-context",
  "path": ["path", "to", "cpu_id"]
}

Relative field path `m‑array`

A relative field path m‑array is an m‑array of field names (m‑string values) to follow, starting from the sequence/variant field using the field path.

Example 12. Relative field path array

JSON representation:

["path", "to", "cpu_id"]

Field lookup mechanism

Field path elements are names of structure or union fields. If one of those fields is a variant field, then the lookup must recursively find the variant’s current field.

For example, let’s say we have the following scope named my-scope (FT means field type):

a: int FT
b: struct FT
  v: variant FT
    choice1: int FT
    choice2: int FT
    choice3: struct FT
      a: int FT
      b: float FT
    choice4: enum FT
  i: int FT

All the following field path m-values are valid sequence field type lengths here:

Trivial:

{
  "scope": "my-scope",
  "path": ["a"]
}

and

{
  "scope": "my-scope",
  "path": ["b", "i"]
}

If choice1, choice2, or choice4 is selected in v when performing the lookup:
{ "scope": "my-scope", "path": ["b", "v"] }

If choice3 is selected in v when performing the lookup:

{
  "scope": "my-scope",
  "path": ["b", "v", "a"]
}

Relative field paths are looked up by going back into the current structure/union field, and then back into the current structure/union field’s parent structure/union field, and so on. For example:

z: int FT <------------------------------------.
y: struct FT                                   |
  a: int FT                                    |
  b: struct FT                                 |
    c: int FT                                  |
    d: string FT                               |
    e: struct FT                               |
      f: int FT <----------------------------. |
      g: int FT                              | |
  h: int FT                                  | |
  i: struct FT                               | |
    j: int FT                                | |
    k: sequence FT, length: ["b", "e", "f"] -' |
x: sequence FT, length: ["z"] -----------------'

Encoding context

We define a current encoding head, an integer variable which is initialized by the upper layer before encoding a value to a data field. This variable is the current position of the writing “head”.

When a bit array is written, the current encoding head is updated by adding the written size to it.

Before encoding a value as a data field, the current encoding head must be aligned to respect the alignment requirements of said field (given by its field type). The following operation can be used to update the current encoding head p to the beginning of a field with an effective alignment of a (bits):

p = (p + a - 1) & -a

For example, if the current encoding head is 37 and the alignment of the next field to write is 8, then the current encoding head must be updated to 40 before writing the field. If the current encoding head is 48 and the alignment of the next field to write is 8, then the current head is already aligned.

Field type m-value

A field type m-value can be either an m‑string value or an m‑map value:

If it’s an m‑string value, it is the name of a field type alias. The upper layer has a mapping of field type alias names to complete field types (m‑map values).
Example 13. Field type alias
JSON representation:

"my alias"

If it’s an m‑map value, it has the following base properties:

Table 5. Field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Field type’s class.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1

The following table summarizes the available field types:

Table 6. Available field types
`field-type` property (`m‑string`)	Name	Inherits properties from
`null`	Null field type.	Field type m-map.
`bitarray`	Bit array field type.	Field type m-map.
`bool`	Boolean field type.	Field type m-map.
`int`	Integer field type.	Bit array field type.
`float`	Floating point number field type.	Bit array field type.
`enum`	Enumeration field type.	Integer field type.
`varbitarray`	Variable-length bit array field type.	Field type m-map.
`varbool`	Variable-length boolean field type.	Variable-length bit array field type.
`varint`	Variable-length integer field type.	Variable-length bit array field type.
`varenum`	Variable-length enumeration field type.	Variable-length integer field type.
`string`	Null-terminated UTF-8 string field type.	Field type m-map.
`struct`	Structure field type.	Field type m-map.
`array`	Array field type.	Field type m-map.
`textarray`	Text array field type.	Field type m-map.
`sequence`	Sequence field type.	Field type m-map.
`textsequence`	Text sequence field type.	Field type m-map.
`variant`	Variant field type.	Field type m-map.
`union`	Union field type.	Field type m-map.

Note that, on the decoding side, any unknown field type must be ignored, and any unknown field type m‑map property must also be ignored.

Null field type

A null field type describes null fields.

A null value usually represents missing or unknown data.

Table 7. Null field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `null`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1

Example 14. Null field type

JSON representation:

{
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  },
  "field-type": "null",
  "alignment": 4
}

Example 15. Minimal null field type

JSON representation:

{"field-type": "null"}

Encode a null value as a null field

To encode a null value using a null field type:

Align the current encoding head using the alignment property of the null field type.

Bit array field type

A bit array field type describes bit array fields.

A bit array value is a simple array of bits. It is not an integer value (it has no signedness).

Table 8. Bit array field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `bitarray`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`size`	`m‑int`	Field’s size, in bits (must be greater than 0).	Required
`byte-order`	Byte order `m‑string`	Field’s byte order.	Optional	`default`

Example 16. Bit array field type

JSON representation:

{
  "field-type": "bitarray",
  "alignment": 16,
  "size": 5,
  "byte-order": "le",
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 17. Minimal bit array field type

JSON representation:

{
  "field-type": "bitarray",
  "size": 32
}

Encode a bit array value as a bit array field

To encode a bit array value using a bit array field type:

Align the current encoding head using the alignment property of the bit array field type.
Follow the rules of Common Trace Format v1.8.2, section 4.1.5, to encode the bit array value according to its byte-order and size properties.
Add the value of the size property to the current encoding head.

Boolean field type

A boolean field type describes boolean fields.

A boolean value is a bit array value which, when all its bits are cleared, is said to be false, and otherwise is said to be true.

Table 9. Boolean field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `bool`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`size`	`m‑int`	Field’s size, in bits (must be greater than 0).	Required
`byte-order`	Byte order `m‑string`	Field’s byte order.	Optional	`default`

Example 18. Boolean field type

JSON representation:

{
  "field-type": "bool",
  "size": 8,
  "byte-order": "be",
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 19. Minimal boolean field type

JSON representation:

{
  "field-type": "bool",
  "size": 8
}

Encode a boolean value as a boolean field

To encode a boolean value using a boolean field type:

Encode the boolean value as a bit array value. This process is platform-dependent.
Follow the rules of how to encode a bit array value as a bit array field.

Integer field type

An integer field type describes integer fields.

Table 10. Integer field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `int`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`size`	`m‑int`	Field’s size, in bits (must be greater than 0).	Required
`byte-order`	Byte order `m‑string`	Field’s byte order.	Optional	`default`
`signed`	`m‑bool`	True if the integer field represents a signed integer value, in which case the field has the two’s complement format.	Optional	False

Example 20. Integer field type

JSON representation:

{
  "field-type": "int",
  "alignment": 16,
  "size": 5,
  "byte-order": "le",
  "signed": true,
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 21. Minimal integer field type

JSON representation:

{
  "field-type": "int",
  "size": 32
}

Encode an integer value as an integer field

To encode an integer value using an integer field type:

Encode the integer value as a bit array value. Follow the two’s complement representation if the integer value is signed. This process is platform-dependent.
Follow the rules of how to encode a bit array value as a bit array field.

Floating point number field type

A floating point number field type describes floating point number fields encoded with IEEE 754-2008’s binary interchange format.

A number value (real number) can be encoded as a floating point number field provided it is representable with one of the versions of IEEE 754-2008’s binary interchange format.

Table 11. Floating point number field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `float`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`size`	`m‑int`	Field’s size, in bits (must be greater than 0).	Required
`byte-order`	Byte order `m‑string`	Field’s byte order.	Optional	`default`

The value of the size property corresponds to the value of the parameter k (storage width in bits) in Table 3.5, Binary interchange format parameters, of the IEEE Std 754-2008 document. All the other parameters of the format needed to encode and decode the floating point number value can be deduced from the value of k.

Example 22. Floating point number field type describing the basic binary64 format

This floating point number field type describes fields encoded with the parameters of the basic binary64 format, which is the encoding used by the “double” type of most programming languages.

JSON representation:

{
  "field-type": "float",
  "alignment": 8,
  "size": 64,
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 23. Minimal floating point number field type

JSON representation:

{
  "field-type": "float",
  "size": 32
}

Encode a number value as a floating point number field

To encode a number value using a floating point number field type:

Encode the number value as a bit array value following IEEE 754-2008’s binary interchange format. This process is platform-dependent.
Follow the rules of how to encode a bit array value as a bit array field.

Enumeration field type member m-value

An enumeration field type member m-value represents the range of values mapped to the label of an enumeration field type member.

An enumeration field type member m-value is either:

An m‑int value, in which case the member’s label is mapped to this value. For example (JSON representation):
```
28
```
An m‑map with the lower and upper properties (m‑int values) which indicate the lower (inclusive) and upper (inclusive) limits of a range to which the member’s label is mapped. For example (JSON representation):
```
{"lower": -3, "upper": 17}
```

Enumeration field type

An enumeration field type describes enumeration fields.

An enumeration value is an integer value mapped to a label.

Table 12. Enumeration field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `enum`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`size`	`m‑int`	Field’s size, in bits (must be greater than 0).	Required
`byte-order`	Byte order `m‑string`	Field’s byte order.	Optional	`default`
`signed`	`m‑bool`	True if the integer field is signed, in which case the field has the two’s complement format.	Optional	False
`members`	`m‑map` which maps label names (`m‑string`) to `m‑array` values of enumeration field type member m-values.	Enumeration field type’s members.	Required

Example 24. Enumeration field type

JSON representation:

{
  "field-type": "enum",
  "alignment": 16,
  "size": 32,
  "signed": true,
  "members": {
    "NEW": [0],
    "TERMINATED": [-1],
    "READY": [2, 17],
    "RUNNING": [-3],
    "WAITING": [
        {"lower": 19, "upper": 199},
        1000
    ],
    "RESTARTING": [
        {"base": 8, "value": "126674015"},
        {"lower": -155, "upper": -98}
    ]
  },
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

With this enumeration field type, the following enumeration fields would have the following associated labels:

-1: TERMINATED
17: READY
-101: RESTARTING
1000: WAITING
22771725: RESTARTING
2: READY
50: WAITING

Example 25. Minimal enumeration field type

JSON representation:

{
  "field-type": "enum",
  "members": {
    "": [0]
  }
}

Encode an enumeration value as an enumeration field

To encode an enumeration value (which is an integer value) using an enumeration field type:

Follow the rules of how to encode an integer value as an integer field.

String field type

A string field type describes string fields.

A string value is a finite sequence of Unicode characters.

Table 13. String field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`field-type`	`m‑string`	Must be set to `string`.	Required
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than or equal to 8).	Optional	8

Example 26. String field type

JSON representation:

{
  "field-type": "string",
  "alignment": 16,
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 27. Minimal string field type

JSON representation:

{"field-type": "string"}

Encode a string value as a string field

To encode a string value using a string field type:

Align the current encoding head using the alignment property of the string field type.
For each Unicode character of the string value:
1. Encode this Unicode character as a sequence of 8-bit bit arrays (bytes) following UTF-8. This process is platform-dependent.
2. For each resulting UTF-8 byte of this character:
  - Follow the rules of how to encode a bit array value as a bit array field.
Follow the rules of how to encode an integer value as an integer field to encode the UTF-8 null character (U+0000).

Variable-length bit array field type

A variable-length bit array field type describes variable-length bit array fields.

A bit array value of any size that is a multiple of 7 bits, and at least 7 bits, can be dynamically encoded as a variable-length bit array field.

Table 14. Variable-length bit array field type `m‑map`
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `varbitarray`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than or equal to 8).	Optional	8

Example 28. Variable-length bit array field type

JSON representation:

{
  "field-type": "varbitarray",
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 29. Minimal variable-length bit array field type

JSON representation:

{"field-type": "varbitarray"}

Encode a bit array value as a variable-length bit array field

To encode a bit array value using a variable-length bit array field type:

Align the current encoding head using the alignment property of the variable-length bit array field type.
Encode the bit array value as a bit array field following the unsigned LEB128 format.
Add the encoded variable-length bit array field’s size (not the original bit array value’s size) to the current encoding head.

Variable-length boolean field type

A variable-length boolean field type describes variable-length boolean fields.

A boolean value is a bit array value which, when all its bits are cleared, is said to be false, and otherwise is said to be true.

A boolean value of any size that is a multiple of 7 bits, and at least 7 bits, can be dynamically encoded as a variable-length bit array field.

Table 15. Variable-length boolean field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `varbool`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than or equal to 8).	Optional	8

Example 30. Variable-length boolean field type

JSON representation:

{
  "field-type": "varbool",
  "alignment": 32,
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 31. Minimal variable-length boolean field type

JSON representation:

{"field-type": "varbool"}

Encode a boolean value as a variable-length boolean field

To encode a boolean value using a variable-length boolean field type:

Encode the boolean value as a bit array value. This process is platform-dependent.
Follow the rules of how to encode a bit array value as a variable-length bit array field.

Variable-length integer field type

A variable-length integer field type describes variable-length integer fields.

An integer value of any size can be encoded dynamically as a variable-length integer field.

Table 16. Variable-length integer field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `varint`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than or equal to 8).	Optional	8
`signed`	`m‑bool`	True if the integer field is signed, in which case the field has the two’s complement format.	Optional	False

Example 32. Variable-length integer field type

JSON representation:

{
  "field-type": "varint",
  "alignment": 16,
  "signed": true,
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 33. Minimal variable-length integer field type

JSON representation:

{"field-type": "varint"}

Encode an integer value as a variable-length integer field

To encode an integer value using a variable-length integer field type:

Encode the integer value as a bit array value. Follow the two’s complement representation if the integer value is signed. Sign-extend the bit array to the next multiple of 7 bits. This process is platform-dependent.
Follow the rules of how to encode a bit array value as a variable-length bit array field.

Variable-length enumeration field type

A variable-length enumeration field type describes variable-length enumeration fields.

An enumeration value is an integer value mapped to a label.

An enumeration value of any size can be encoded dynamically as a variable-length enumeration field.

Table 17. Variable-length enumeration field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `varint`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than or equal to 8).	Optional	8
`signed`	`m‑bool`	True if the integer field is signed, in which case the field has the two’s complement format.	Optional	False
`members`	`m‑map` which maps label names (`m‑string`) to `m‑array` values of enumeration field type member m-values.	Enumeration field type’s members.	Required

Example 34. Variable-length enumeration field type

JSON representation:

{
  "field-type": "varenum",
  "signed": true,
  "members": {
    "NEW": [0],
    "TERMINATED": [-1],
    "READY": [2, 17],
    "RUNNING": [-3],
    "WAITING": [
      {"lower": 19, "upper": 199},
      1000
    ],
    "RESTARTING": [
      {"base": 8, "value": "126674015"},
      {"lower": -155, "upper": -98}
    ]
  },
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

With this enumeration field type, the following enumeration fields would have the following associated labels:

-1: TERMINATED
17: READY
-101: RESTARTING
1000: WAITING
22771725: RESTARTING
2: READY
50: WAITING

Example 35. Minimal variable-length enumeration field type

JSON representation:

{
  "field-type": "varenum",
  "members": {
    "": [0]
  }
}

Encode an enumeration value as a variable-length integer field

To encode an enumeration value (which is an integer value) using a variable-length enumeration field type:

Encode the integer value as a bit array value. Follow the two’s complement representation if the integer value is signed. Sign-extend the bit array to the next multiple of 7 bits. This process is platform-dependent.
Follow the rules of how to encode a bit array value as a variable-length bit array field.

Structure/union/variant field type field `m‑map`

A structure/union/variant field type field m‑map represents one field of a structure field type or of a union field type, or one choice of a variant field type.

Table 18. Structure/union/variant field type field `m‑map` properties
Name	Type	Description	Required?	Default m-value
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`name`	`m‑string`	Structure/union/variant field type field’s name.	Required
`field-type`	Field type m-value	Structure/union/variant field type field’s type.	Required

Example 36. Structure/union/variant field type field m‑map

JSON representation:

{
  "name": "src_addr",
  "field-type": {
    "field-type": "array",
    "length": 4,
    "element-field-type": "uint8"
  },
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Structure field type

A structure field type describes structure fields.

A structure value is a finite sequence of values of different types. This is sometimes also called a record.

Table 19. Structure field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `struct`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s minimal alignment (must be a power of two, greater than 0).	Optional	1
`fields`	`m‑array` of structure/union/variant field type field `m‑map` values	Fields of the structure field type.	Optional	Empty `m‑array` value.

The alignment property indicates the minimal alignment of the structure fields encoded with this field type. The automatic alignment of the structure field type is the greatest value amongst all the effective alignments of the field type’s fields. The effective alignment of the structure field type is the greatest value amongst the field type’s minimal and automatic alignments.

Example 37. Structure field type

JSON representation:

{
  "field-type": "struct",
  "alignment": 8,
  "fields": [
    {
      "name": "timestamp_begin",
      "field-type": "uint64"
    },
    {
      "name": "timestamp_end",
      "field-type": "uint64"
    },
    {
      "name": "packet_size",
      "field-type": "uint32"
    },
    {
      "name": "content_size",
      "field-type": "uint32"
    },
    {
      "name": "core location",
      "field-type": {
        "field-type": "struct",
        "fields": [
          {
            "name": "x",
            "field-type": {
              "field-type": "int",
              "size": 3
            }
          },
          {
            "name": "y",
            "field-type": {
              "field-type": "int",
              "size": 5
            }
          }
        ]
      }
    }
  ],
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

In this example, assuming that the effective alignment of the timestamp_end field is 64, and that this is the greatest alignment amongst all the alignments of the structure field type’s fields, then the structure field type’s effective alignment is also 64.

Example 38. Minimal (empty) structure field type

JSON representation:

{"field-type": "struct"}

Encode a structure value as a structure field

To encode a structure value using a structure field type:

Align the current encoding head using the effective alignment of the structure field type.
For each field of the structure value:
- Encode the field’s value using the field type of the structure/union/variant field type field m‑map at the corresponding position in the fields m‑array of the structure field type.

Array field type

An array field type describes array fields.

An array value is a finite sequence of values.

The length of all the possible array fields represented by a given array field type is known statically (when producing the array field type).

Table 20. Array field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `array`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`length`	`m‑int`	Number of elements contained in array fields described by this field type.	Required
`element-field-type`	Field type m-value	Field type of the elements contained in array fields described by this field type.	Required

Example 39. Array field type

{
  "field-type": "array",
  "alignment": 64,
  "length": 72,
  "element-field-type": {
    "field-type": "float",
    "size": 64,
    "byte-order": "be"
  },
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 40. Array field type describing UUID fields

JSON representation:

{
  "field-type": "array",
  "alignment": 8,
  "length": 16,
  "element-field-type": {
    "field-type": "int",
    "size": 8
  },
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 41. Minimal array field type

JSON representation:

{
  "field-type": "array",
  "length": 32,
  "element-field-type": "elem-ft"
}

Encode an array value as an array field

To encode an array value using an array field type:

Align the current encoding head using the alignment property of the array field type.
For each element of the array value:
- Encode the value using the element-field-type property of the array field type.

Text array field type

A text array field type describes text array fields. It is a specialized version of the array field type.

A text array value is a finite sequence of bytes which form a UTF-8 string. The length of all the possible text array fields represented by a given text array field type is known statically (when producing the text array field type). The text array value’s length may be greater than the number of effective UTF-8 bytes, as long as the string is null-terminated. In this case, the padding bytes after the UTF-8 null character can have any value.

Table 21. Text array field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `textarray`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`length`	`m‑int`	Number of bytes contained in text array fields described by this field type.	Required

Example 42. Text array field type

JSON representation:

{
  "field-type": "textarray",
  "alignment": 32,
  "length": 16,
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 43. Minimal text array field type

JSON representation:

{
  "field-type": "textarray",
  "length": 32
}

Encode a text array value as a text array field

A text array value is an array value with bytes as elements.

To encode a text array value using a text array field type:

Follow the rules of how to encode an array value as an array field.

Sequence field type

A sequence field type describes sequence fields.

A sequence value is a finite sequence of values. Its length is known dynamically.

Table 22. Sequence field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `sequence`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`length`	Field path m-value	Path to a previously encoded integer, enumeration, variable-length integer, or variable-length enumeration field which indicates the number of elements contained in the sequence field.	Required
`element-field-type`	Field type m-value	Field type of the elements contained in sequence fields described by this field type.	Required

Example 44. Sequence field type

JSON representation:

{
  "field-type": "sequence",
  "alignment": 32,
  "length": ["msg", "info", "count"],
  "element-field-type": {
    "field-type": "string"
  },
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 45. Minimal sequence field type

JSON representation:

{
  "field-type": "sequence",
  "length": ["len"],
  "element-field-type": "elem-ft"
}

Encode a sequence value as a sequence field

To encode a sequence value using a sequence field type:

Align the current encoding head using the alignment property of the sequence field type.
For each element of the sequence value (the count given by a previously encoded integer or enumeration field, located thanks to the length property):
- Encode the value using the element-field-type property of the sequence field type.

Text sequence field type

A text sequence field type describes text sequence fields. It is a specialized version of the sequence field type.

A text sequence value is a finite sequence of bytes which form a UTF-8 string. Its length is known dynamically. The text sequence value’s length may be greater than the number of effective UTF-8 bytes, as long as the string is null-terminated. In this case the padding bytes after the UTF-8 null character can have any value.

Table 23. Text sequence field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `textsequence`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`length`	Field path m-value	Path to a previously encoded integer, enumeration, variable-length integer, or variable-length enumeration field which indicates the number of bytes contained in the text sequence field.	Required

Example 46. Text sequence field type

JSON representation:

{
  "field-type": "textsequence",
  "alignment": 32,
  "length": {
    "scope": "my-scope",
    "path": "cmd-len"
  },
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 47. Minimal text sequence field type

JSON representation:

{
  "field-type": "textsequence",
  "length": ["len"]
}

Encode a text sequence value as a text sequence field

A text sequence value is a sequence value with bytes as elements.

To encode a text sequence value using a text sequence field type:

Follow the rules of how to encode a sequence value as a sequence field.

Variant field type

A variant field type describes variant fields.

A variant value is a value of some type amongst many possible types. The exact type of the value is indicated dynamically by a previously encoded tag field (an enumeration or variable-length enumeration field).

Table 24. Variant field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `textsequence`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s alignment (must be a power of two, greater than 0).	Optional	1
`tag`	Field path m-value	Path to a previously encoded enumeration or variable-length enumeration field which indicates the type, by label name to choice name association, of the variant field.	Required
`choices`	`m‑array` of structure/union/variant field type field `m‑map` values	Choices of the variant field type.	Required

The name property of the all the structure/union/variant field type field m‑map values listed in the choices property must exist as a member’s label name in the tag’s enumeration field type.

Example 48. Variant field type

JSON representation:

{
  "field-type": "variant",
  "alignment": 16,
  "tag": ["path", "to", "tag"],
  "choices": [
    {
      "user-attrs": {
        "ns": {
          "split": 5
        }
      },
      "name": "ID",
      "field-type": {
        "field-type": "int",
        "size": 35,
        "signed": true,
        "alignment": 32
      }
    },
    {
      "name": "NAME",
      "field-type": {
        "field-type": "string"
      }
    }
  ],
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 49. Minimal variant field type

JSON representation:

{
  "field-type": "variant",
  "tag": ["tag"],
  "choices": [
    {
      "name": "",
      "field-type": "some-ft"
    }
  ]
}

Note that such a variant field type, with a single choice, is useless because this choice could be used directly instead.

Encode a variant value as a variant field

To encode a variant value using a variant field type:

Align the current encoding head using the alignment property of the variant field type.
Follow the encoding rules of the field type, amongst the field types of the choices listed in the choices property, currently selected by the previously encoded tag field.

Union field type

A union field type describes union fields.

A union field is a binary field which, once encoded using some field type, can be decoded using other field types.

Table 25. Union field type `m‑map` properties
Name	Type	Description	Required?	Default m-value
`field-type`	`m‑string`	Must be set to `struct`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`alignment`	`m‑int`	Field’s minimal alignment (must be a power of two, greater than 0).	Optional	1
`fields`	`m‑array` of at least one structure/union/variant field type field `m‑map` value	Fields of the union field type.	Required

Example 50. Union field type

In this example, the union fields represented by this field type can always be decoded as either string fields, or as unsigned 64-bit integer fields. This means that the producer ensures that, when writing a string field, its length is always 64 bits (including the terminating null character).

JSON representation:

{
  "field-type": "union",
  "alignment": 8,
  "fields": [
    {
      "name": "as string",
      "field-type": {
        "field-type": "string"
      }
    },
    {
      "name": "as int",
      "field-type": {
        "field-type": "int",
        "size": 64
      }
    }
  ],
  "user-attrs": {
    "my-namespace": {
      "my-attr": "desc"
    }
  }
}

Example 51. Minimal union field type

JSON representation:

{
  "field-type": "union",
  "length": 32,
  "fields": [
    {
      "name": "",
      "field-type": "some-ft"
    }
  ]
}

Note that such a union field type, with a single field, is useless because this field could be used directly instead.

Encode a value as a union field

To encode a value using a union field type:

Align the current encoding head using the alignment property of the union field type.
Follow the encoding rules of the chosen field type, amongst the field types of the fields listed in the fields property, corresponding to the type of the value to encode.

An important condition when encoding a union field is that, whichever field type is chosen to decode the field afterwards, the current decoding head must always be the same after the process.

Layer 2: Data streams, packets, and event records

This layer adds the following concepts to the CTF 2 specification:

Field type alias fragment.
Field tag m‑map.
A trace contains zero or more data streams. A data stream is a sequence of zero or more packets. A packet contains zero or more event records.
The concepts and layouts of data streams, packets, and event records.
Trace class, data stream class, and event record class fragments.

Data stream

A CTF 2 data stream is defined as a sequence of packets.

[packet]
[packet]
[packet]
[packet]
[packet]
...

A data stream class fragment describes, in the metadata m‑array, a class of data streams. For a single data stream class, there can be zero or more data streams in the trace.

There are two ways to distinguish individual data streams:

Rely on the storage or transport back-end to separate individual data streams.

For example, a trace stored on a file system could contain one data stream per file. If a trace is sent over the network, the wrapping network protocol could assign a unique ID to each data stream.
Tag a field of the trace packet header field with the data-stream-id field tag.

This is the recommended way to isolate a data stream: it is more robust and more portable. With this method, each packet holds the unique ID of the data stream to which it logically belongs.

Packet

A packet contains two optional, contiguous scope fields, named the trace packet header and the data stream packet context fields, followed by zero or more event records, and finally by optional padding bits to honor its total size:

[trace packet header field]
[data stream packet context field]
[event record]
[event record]
[event record]
...
[padding]

Before the first bit of a packet is written, the current encoding head of the CTFFER encoding context is set to 0. This current encoding head keeps on incrementing as fields are encoded following the CTFFER until the beginning of the next packet in the same data stream, where it is reset to 0 again. This means that the reference to align any binary field within a packet is the beginning of a packet, not the beginning of a data stream.

To simplify matters for CTF 2 consumers, the size of a packet, in bits, must be greater than 8, and must be a multiple of 8.

The first packet’s padding bit is reached when the current encoding head is equal to the value of the packet’s content size field found in the data stream packet context field (if any).

If there is more than one data stream class in the trace, the trace packet header field contains a data stream class ID field which indicates which data stream class describes the data stream packet context field and the event records.

Event record

An event record contains four optional, contiguous scope fields, named the data stream event record header, the data stream event record context, the event record context, and the event record payload fields.

[data stream event record header field]
[data stream event record context field]
[event record context]
[event record payload]

If there is more than one event record class in a given data stream class, the data stream event record header field contains an event record class ID field which indicates which event record class describes the event record context and event record payload fields.

The encoded size of any event record must be at least 1 bit.

Field type alias fragment

A field type alias fragment is a fragment which associates a name (m‑string) to a field type m-value (complete field type m‑map, or previously defined field type alias’s name).

The name of a field type alias can be used by field types which are written after the field type alias fragment in the metadata m‑array.

Table 26. Field type alias fragment
Name	Type	Description	Required?	Default m-value
`fragment`	`m‑string`	Must be set to `field-type-alias`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`name`	`m‑string`	Name of the field type alias.	Required
`field-type`	Field type m-value	Field type of the field type alias.	Required

Within a given metadata m‑array, two field type alias fragments cannot have the same name property.

Example 52. Field type alias fragment

JSON representation:

{
  "fragment": "field-type-alias",
  "name": "uint8",
  "field-type": {
    "field-type": "integer",
    "size": 8,
    "alignment": 8
  }
}

Example 53. Field type alias fragment giving another name to a previously defined alias

JSON representation:

{
  "fragment": "field-type-alias",
  "name": "uint8_t",
  "field-type": "uint8"
}

Field tag `m‑map`

A field tag m‑map “tags” a scope’s field with a special meaning.

The following sections define specific field tag m‑map values which can be used in specific contexts.

Table 27. Field tag `m‑map` base properties
Name	Type	Description	Required?	Default m-value
`tag`	`m‑string`	Name of the tag (fragment-dependent).	Required
`path`	Absolute field path `m‑map`	Absolute path to the tagged field.	Required

Discarded event record count field tag `m‑map`

A discarded event record count field tag m‑map is a field tag which tags a field as being a counter of discarded event records.

Event records can be discarded for multiple reasons from the producer’s perspective. This document specifies the available reasons, when event records are lost for those reasons, and how to compute the total number of discarded event records so far for a given data stream.

Table 28. Discarded event record count field tag `m‑map` properties
Name	Type	Description	Required?
`tag`	`m‑string`	Must be set to `discarded-event-record-count`.	Required
`path`	Absolute field path `m‑map`	Absolute path to the tagged field.	Required
`reason`	`m‑string`	Must be set to `legacy` as of this version.	Required

A field tagged with the legacy reason has the same behaviour as the events_discarded field of the stream packet context of CTF v1.8.2.

The type of the field located using the path property must be an unsigned integer field type, an unsigned enumeration field type, an unsigned variable-length integer field type, or an unsigned variable-length enumeration field type.

Example 54. Discarded event record count field tag

JSON representation:

{
  "tag": "discarded-event-record-count",
  "path": {
    "scope": "data-stream-packet-context",
    "path": ["number of discarded events"]
  },
  "reason": "legacy"
}

Trace class fragment

A trace class fragment is a fragment which defines properties that are common to the whole trace, that is, to all the data streams.

Exactly one trace class fragment must exist in a given metadata m‑array, and it must precede any data stream class fragment.

Table 29. Trace class fragment `m‑map` properties
Name	Type	Description	Required?	Default m-value
`fragment`	`m‑string`	Must be set to `trace-class`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`default-byte-order`	Byte order `m‑string` (excluding `default`)	Trace’s default byte order.	Optional	No trace’s default byte order. Note that, if this property is missing, no field types in the whole metadata `m‑array` can have a `byte-order` property set to `default`.
`uuid`	`m‑string`	UUID (canonical form) of the trace.	Optional	No trace UUID.
`packet-header-field-type`	Field type m-value	Field type of the trace packet header field. The name of this scope is `trace-packet-header`, to locate it with an absolute field path `m‑map`.	Optional	1-bit aligned null field type.
`tags`	`m‑array` of field tag `m‑map` values.	Field tags of this trace class. See the allowed field tags below. Upper layers may also define specific field tags that are allowed here.	Optional	Empty `m‑array` value.

Example 55. Trace class fragment

JSON representation:

{
  "fragment": "trace-class",
  "default-byte-order": "le",
  "uuid": "1255cddc-5afe-4b4a-92b2-17aa5dde2ea6",
  "packet-header-field-type": {
    "field-type": "struct",
    "fields": [
      {
        "name": "the magic",
        "field-type": "uint32"
      },
      {
        "name": "the uuid",
        "field-type": {
          "field-type": "array",
          "element-field-type": "uint8"
        }
      },
      {
        "name": "the data stream class ID",
        "field-type": "uint8"
      }
    ]
  },
  "tags": [
    {
      "tag": "magic",
      "path": {
        "scope": "trace-packet-header",
        "path": ["the magic"]
      }
    },
    {
      "tag": "uuid",
      "path": {
        "scope": "trace-packet-header",
        "path": ["the uuid"]
      }
    },
    {
      "tag": "data-stream-class-id",
      "path": {
        "scope": "trace-packet-header",
        "path": ["the data stream class ID"]
      }
    }
  ],
  "user-attrs": {
    "my ns": "yes"
  }
}

Allowed field tags targetting trace packet header fields

In addition to the tags below, discarded event record count field tags are allowed, and any field tag defined by the upper layers of CTF 2.

Table 30. Trace class fragment’s allowed tags for the trace packet header fields
Tag name	Meaning	Tagged field constraints
`magic`	Magic number field. This field indicates the CTF 2 data stream magic number.	This field must be the first field of the trace packet header field type. This field’s type must be a 32-bit unsigned integer field type. The value of this field in all the data streams must be 3254525889 (0xc1fc1fc1).
`uuid`	Trace’s UUID field. This field indicates the UUID of the trace.	This field’s type must be an array field type of length 16, with an 8-bit aligned, 8-bit integer field type as its element. The value of this field in all the data streams must be equal to the binary equivalent of the trace class fragment’s `uuid` property.
`data-stream-class-id`	Data stream class ID field. This field indicates the numeric ID of the data stream class used to encode the rest of the packet. If this tag is not specified, the ID of the data stream class used to encode the rest of the packet is implicitly 0. If more than one field are tagged with this tag, the last one to be encoded in the entire trace packet header field is the effective ID of the data stream class used to encode the rest of the packet.	This field’s type must be an unsigned integer field type, an unsigned enumeration field type, an unsigned variable-length integer field type, or an unsigned variable-length enumeration field type.
`data-stream-id`	Data stream ID field. This field indicates the unique numeric ID of the data stream to which the packet belongs. If this tag is not specified, the data stream to which the packet belongs is identified using the storage or transport back-end of the trace. If more than one field are tagged with this tag, the last one to be encoded in the entire trace packet header field is the effective unique data stream ID.	This field’s type must be an unsigned integer field type, an unsigned enumeration field type, an unsigned variable-length integer field type, or an unsigned variable-length enumeration field type.

Data stream class fragment

A data stream class fragment is a fragment which defines properties that are common to one or more data streams.

More than one data stream class fragment may exist in a given metadata m‑array, but they must come after the trace class fragment.

Any event record class fragment which is a child of a given data stream class fragment must come after the latter in the metadata m‑array.

Table 31. Data stream class fragment `m‑map` properties
Name	Type	Description	Required?	Default m-value
`fragment`	`m‑string`	Must be set to `data-stream-class`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`id`	`m‑int`	Data stream class’s numeric ID.	Optional	0
`packet-context-field-type`	Field type m-value	Field type of the data stream packet context field. The name of this scope is `data-stream-packet-context`, to locate it with an absolute field path `m‑map`.	Optional	1-bit aligned null field type.
`event-record-context-field-type`	Field type m-value	Field type of the data stream event record context field. The name of this scope is `data-stream-event-record-context`, to refer to it with an absolute field path `m‑map`.	Optional	1-bit aligned null field type.
`event-record-header-field-type`	Field type m-value	Field type of the data stream event record header field. The name of this scope is `data-stream-event-record-header`, to refer to it with an absolute field path `m‑map`.	Optional	1-bit aligned null field type.
`tags`	`m‑array` of field tag `m‑map` values.	Field tags of this data stream class. See the allowed field tags below. Upper layers may also define specific field tags that are allowed here.	Optional	Empty `m‑array` value.

Two data stream class fragments with the same id property cannot exist in the same metadata m‑array.

Example 56. Data stream class fragment

JSON representation:

{
  "fragment": "data-stream-class",
  "id": 3,
  "packet-context-field-type": {
    "field-type": "struct",
    "fields": [
      {
        "name": "packet size",
        "field-type": "uint32"
      },
      {
        "name": "content size",
        "field-type": "uint32"
      },
      {
        "name": "discarded events",
        "field-type": "uint32"
      },
      {
        "name": "timestamp begin",
        "field-type": "uint64"
      },
      {
        "name": "timestamp end",
        "field-type": "uint64"
      },
      {
        "name": "cpu ID",
        "field-type": "uint8"
      }
    ]
  },
  "event-record-header-field-type": {
    "field-type": "struct",
    "fields": [
      {
        "name": "ID",
        "field-type": {
          "field-type": "enum",
          "size": 5,
          "members": {
            "compact": [{"lower": 0, "upper": 30}],
            "extended": [31]
          }
        }
      },
      {
        "name": "compact or extended",
        "field-type": {
          "field-type": "variant",
          "tag": ["ID"],
          "choices": [
            {
              "name": "compact",
              "field-type": {
                "field-type": "struct",
                "fields": [
                  {
                    "name": "timestamp",
                    "field-type": "uint27"
                  }
                ]
              }
            },
            {
              "name": "extended",
              "field-type": {
                "field-type": "struct",
                "fields": [
                  {
                    "name": "ID",
                    "field-type": "uint32"
                  },
                  {
                    "name": "timestamp",
                    "field-type": "uint64"
                  }
                ]
              }
            }
          ]
        }
      }
    ]
  },
  "tags": [
    {
      "tag": "packet-total-size",
      "path": {
        "scope": "data-stream-packet-context",
        "path": ["packet size"]
      }
    },
    {
      "tag": "packet-content-size",
      "path": {
        "scope": "data-stream-packet-context",
        "path": ["content size"]
      }
    },
    {
      "tag": "discarded-event-record-count",
      "path": {
        "scope": "data-stream-packet-context",
        "path": ["discarded events"]
      },
      "reason": "legacy"
    },
    {
      "tag": "event-record-class-id",
      "path": {
        "scope": "data-stream-event-record-header",
        "path": ["ID"]
      }
    },
    {
      "tag": "event-record-class-id",
      "path": {
        "scope": "data-stream-event-record-header",
        "path": ["compact or extended", "ID"]
      }
    }
  ],
  "user-attrs": {
    "my ns": "yes"
  }
}

Allowed field tags targetting data stream packet context fields

In addition to the tags below, discarded event record count field tags are allowed, and any field tag defined by the upper layers of CTF 2.

Table 32. Data stream class fragment’s allowed tags for the data stream packet context fields
Tag name	Meaning	Tagged field constraints
`packet-total-size`	Packet’s total size field. This field indicates the size, in bits, of the whole packet in which this data stream packet context field is encoded. This size includes the padding bits after the packet’s content, if any. If this tag is not specified, this is the only packet of its data stream, that is, it ends where the data stream ends.	This field’s type must be an unsigned integer field type, an unsigned enumeration field type, an unsigned variable-length integer field type, or an unsigned variable-length enumeration field type.
`packet-content-size`	Packet’s content size field. This field indicates the content size, in bits, of the packet in which this data stream packet context field is encoded. The packet’s content size is the number of bits from the packet’s first bit to the last bit of the last event record (included). The difference between the packet’s total size and the packet’s content size is the padding size. If this tag is not specified, the packet has no padding bits, thus its content size is the same as its total size.	This field’s type must be an unsigned integer field type, an unsigned enumeration field type, an unsigned variable-length integer field type, or an unsigned variable-length enumeration field type.
`packet-sequence-number`	Packet’s sequence number field. This field indicates the sequence number of the packet in which this data stream packet context field is encoded. This is the zero-based index of the packet within its data stream.	This field’s type must be an unsigned integer field type, an unsigned enumeration field type, an unsigned variable-length integer field type, or an unsigned variable-length enumeration field type.

Allowed field tags targetting data stream event record header fields

In addition to the tags below, any field tag defined by the upper layers of CTF 2 is allowed.

Table 33. Data stream class fragment’s allowed tags for the data stream event record header fields
Tag name	Meaning	Tagged field constraints
`event-record-class-id`	Event record class ID field. This field indicates the numeric ID of the event record class used to encode the rest of the event record. If this tag is not specified, the ID of the event record class used to encode the rest of the event record is implicitly 0. If more than one field are tagged with this tag, the last one to be encoded in the entire data stream event record header field is the effective ID of the event record class used to encode the rest of the event record.	This field’s type must be an unsigned integer field type, an unsigned enumeration field type, an unsigned variable-length integer field type, or an unsigned variable-length enumeration field type.

Event record class fragment

An event record class fragment is a fragment which defines properties that are common to one or more event records.

More than one event record class fragment may exist in a given metadata m‑array, but they must come after their parent data stream class fragment.

Table 34. Event record class fragment `m‑map` properties
Name	Type	Description	Required?	Default m-value
`fragment`	`m‑string`	Must be set to `event-record-class`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`id`	`m‑int`	Event record class’s numeric ID.	Optional	0
`parent-data-stream-class-id`	`m‑int`	Numeric ID of parent data stream class.	Optional	0
`context-field-type`	Field type m-value	Field type of the event record context field. The name of this scope is `event-record-context`, to locate it with an absolute field path `m‑map`.	Optional	1-bit aligned null field type.
`payload-field-type`	Field type m-value	Field type of the event record payload field. The name of this scope is `event-record-payload`, to locate it with an absolute field path `m‑map`.	Optional	1-bit aligned null field type.
`tags`	`m‑array` of field tag `m‑map` values.	Field tags of this event record class. Upper layers may also define specific field tags that are allowed here.	Optional	Empty `m‑array` value.

Two event record class fragments with the same id property, and with the same parent-data-stream-class-id property, cannot exist in the same metadata m‑array.

Example 57. Event record class fragment

JSON representation:

{
  "fragment": "event-record-class",
  "id": 45,
  "parent-data-stream-class-id": 3,
  "payload-field-type": {
    "field-type": "struct",
    "fields": [
      {
        "name": "prev_comm",
        "field-type": {
          "field-type": "textarray",
          "length": 16
        }
      },
      {"name": "prev_tid", "field-type": "uint32"},
      {"name": "prev_prio", "field-type": "uint32"},
      {"name": "prev_state", "field-type": "uint64"},
      {
        "name": "next_comm",
        "field-type": {
          "field-type": "textarray",
          "length": 16
        }
      },
      {"name": "next_tid", "field-type": "uint32"},
      {"name": "next_prio", "field-type": "uint32"}
    ]
  },
  "user-attrs": {
    "my ns": "yes"
  }
}

Layer 3: Timekeeping

This layer adds concepts of time to the CTF 2 specification. With this layer, timestamps may be associated to packet and event record fields.

Data stream clock class fragment

A data stream clock class fragment is a fragment which defines properties that are common to one or more data stream clocks.

Table 35. Data stream clock class fragment `m‑map` properties
Name	Type	Description	Required?	Default m-value
`fragment`	`m‑string`	Must be set to `data-stream-clock-class`.	Required
`user-attrs`	User attributes `m‑map`	User attributes.	Optional	Empty `m‑map` value.
`name`	`m‑string`	Name of this data stream clock class.	Required
`freq`	`m‑int`	Frequency (Hz).	Required
`uuid`	`m‑string`	UUID (canonical form) of this data stream clock class.	Optional	No UUID.
`error-cycles`	`m‑int`	Error in cycles.	Optional	0
`offset-seconds`	`m‑int`	Offset in seconds.	Optional	0
`offset-cycles`	`m‑int`	Offset in cycles.	Optional	0
`is-absolute`	`m‑bool`	True if this data stream clock class defines absolute data stream clocks, that is, clocks that are considered to be global references, even across different UUIDs.	Optional	False

Two data stream clock class fragments with the same name property cannot exist in the same metadata m‑array.

See Common Trace Format v1.8.2, section 8, for more information about data stream clock classes (equivalent to TSDL’s clock block).

Example 58. Data stream clock class fragment

JSON representation:

{
  "fragment": "data-stream-clock-class",
  "name": "monotonic",
  "uuid": "96a25753-91af-4602-a71b-d53b7c3dde45",
  "freq": 1000000000,
  "error-cycles": 1000,
  "offset-seconds": 1326476837,
  "offset-cycles": 897235420,
  "is-absolute": true,
  "user-attrs": {
    "my ns": "yes"
  }
}

Data stream clock

A data stream clock is an instance of a data stream clock class.

Each data stream has one such clock instance for each known data stream clock class.

A data stream clock is an integer variable initialized to 0 before decoding the data stream to which it is attached. This variable holds the current value, in cycles, for this specific data stream, of its data stream clock class.

When any field tagged with one of the data stream clock update tags is decoded, its associated data stream clock can be updated.

Update data stream clock now field tag `m‑map`

An update data stream clock now field tag m‑map is a field tag which tags a field to update a data stream clock with its value when it is decoded.

Table 36. Update data stream clock now field tag `m‑map` properties
Name	Type	Description	Required?
`path`	Absolute field path `m‑map`	Absolute path to the tagged field.	Required
`tag`	`m‑string`	Must be set to `update-data-stream-clock-now`.	Required
`data-stream-clock-class-name`	`m‑string`	Name of the data stream clock class of the data stream clock to update.	Required

This tag may be used in the tags property of a trace class fragment m‑map, a data stream class fragment m‑map, or an event record class fragment m‑map.

If this tag is used, a matching data stream clock class fragment must exist in the metadata m‑array before the fragment in which it is used.

The tagged field’s type must be an unsigned integer field type, an unsigned enumeration field type, an unsigned variable-length integer field type, or an unsigned variable-length enumeration field type.

Example 59. Data stream class fragment using this field tag

JSON representation:

{
  "fragment": "data-stream-class",
  "id": 2,
  "event-record-header-field-type": {
    "field-type": "struct",
    "fields": [
      {
        "name": "ts",
        "field-type": "uint32"
      }
    ]
  },
  "tags": [
    {
      "tag": "update-data-stream-clock-now",
      "data-stream-clock-class-name": "monotonic",
      "path": {
        "scope": "data-stream-event-record-header",
        "path": ["ts"]
      }
    }
  ]
}

Update data stream clock after packet field tag `m‑map`

An update data stream clock after packet field tag m‑map is a field tag which tags a field to update a data stream clock with its value after the whole packet is decoded (delayed update).

Table 37. Update data stream clock after packet field tag `m‑map` properties
Name	Type	Description	Required?
`path`	Absolute field path `m‑map`	Absolute path to the tagged field.	Required
`tag`	`m‑string`	Must be set to `update-data-stream-clock-after-packet`.	Required
`data-stream-clock-class-name`	`m‑string`	Name of the data stream clock class of the data stream clock to update.	Required

This tag is only allowed in the tags property of a data stream class fragment m‑map. It must target the data-stream-packet-context scope.

If this tag is used, a matching data stream clock class fragment must exist in the metadata m‑array before the fragment in which it is used.

Complete metadata stream example

Here’s an example of a complete, valid metadata stream. It contains one data stream class fragment with two event record class fragments as its children.

[
  "CTF 2",
  {
    "fragment": "field-type-alias",
    "name": "uint8",
    "field-type": {
      "field-type": "int",
      "size": 8,
      "align": 8
    }
  },
  {
    "fragment": "field-type-alias",
    "name": "uint16",
    "field-type": {
      "field-type": "int",
      "size": 16,
      "align": 8
    }
  },
  {
    "fragment": "field-type-alias",
    "name": "uint32",
    "field-type": {
      "field-type": "int",
      "size": 32,
      "align": 8
    }
  },
  {
    "fragment": "field-type-alias",
    "name": "uint64",
    "field-type": {
      "field-type": "int",
      "size": 64,
      "align": 8
    }
  },
  {
    "fragment": "field-type-alias",
    "name": "int64",
    "field-type": {
      "field-type": "int",
      "size": 64,
      "align": 8,
      "signed": true
    }
  },
  {
    "fragment": "field-type-alias",
    "name": "uint27",
    "field-type": {
      "field-type": "int",
      "size": 27
    }
  },
  {
    "fragment": "field-type-alias",
    "name": "uuid",
    "field-type": {
      "field-type": "array",
      "alignment": 8,
      "length": 16,
      "field-type": "uint8"
    }
  },
  {
    "fragment": "trace-class",
    "default-byte-order": "le",
    "uuid": "908d7a0d-2bbc-4584-ba3b-a9c73a62f52e",
    "packet-header-field-type": {
      "field-type": "struct",
      "fields": [
        {
          "name": "the magic",
          "field-type": "uint32"
        },
        {
          "name": "the UUID",
          "field-type": "uuid"
        },
        {
          "name": "ids",
          "field-type": {
            "field-type": "struct",
            "fields": [
              {
                "name": "data stream class",
                "field-type": "uint8"
              },
              {
                "name": "data stream",
                "field-type": "uint8"
              }
            ]
          }
        }
      ]
    },
    "tags": [
      {
        "tag": "magic",
        "path": {
          "scope": "trace-packet-header",
          "path": ["the magic"]
        }
      },
      {
        "tag": "uuid",
        "path": {
          "scope": "trace-packet-header",
          "path": ["the UUID"]
        }
      },
      {
        "tag": "data-stream-class-id",
        "path": {
          "scope": "trace-packet-header",
          "path": ["ids", "data stream class"]
        }
      },
      {
        "tag": "data-stream-id",
        "path": {
          "scope": "trace-packet-header",
          "path": ["ids", "data stream"]
        }
      }
    ]
  },
  {
    "fragment": "data-stream-clock-class",
    "name": "clock src",
    "uuid": "96a25753-91af-4602-a71b-d53b7c3dde45",
    "freq": 1000000000,
    "error-cycles": 1000,
    "offset-seconds": 1326476837,
    "offset-cycles": 897235420,
    "is-absolute": true,
  },
  {
    "fragment": "data-stream-class",
    "packet-context-field-type": {
      "field-type": "struct",
      "fields": [
        {
          "name": "packet sizes",
          "field-type": {
            "field-type": "struct",
            "fields": [
              {
                "name": "total",
                "field-type": "uint32"
              },
              {
                "name": "content",
                "field-type": "uint32"
              }
            ]
          }
        },
        {
          "name": "discarded events",
          "field-type": "uint32"
        },
        {
          "name": "sequence number",
          "field-type": "uint64"
        },
        {
          "name": "packet begin TS",
          "field-type": "uint64"
        },
        {
          "name": "packet end TS",
          "field-type": "uint64"
        },
        {
          "name": "cpu ID",
          "field-type": "uint8"
        },
      ]
    },
    "event-record-header-field-type": {
      "field-type": "struct",
      "fields": [
        {
          "name": "ID",
          "field-type": {
            "field-type": "enum",
            "size": 5,
            "members": {
              "compact": [{"lower": 0, "upper": 30}],
              "extended": [31]
            }
          }
        },
        {
          "name": "compact or extended",
          "field-type": {
            "field-type": "variant",
            "tag": ["ID"],
            "choices": [
              {
                "name": "compact",
                "field-type": {
                  "field-type": "struct",
                  "fields": [
                    {
                      "name": "timestamp",
                      "field-type": "uint27"
                    }
                  ]
                }
              },
              {
                "name": "extended",
                "field-type": {
                  "field-type": "struct",
                  "fields": [
                    {
                      "name": "ID",
                      "field-type": "uint32"
                    },
                    {
                      "name": "timestamp",
                      "field-type": "uint64"
                    }
                  ]
                }
              }
            ]
          }
        }
      ]
    },
    "tags": [
      {
        "tag": "packet-total-size",
        "path": {
          "scope": "data-stream-packet-context",
          "path": ["packet sizes", "total"]
        }
      },
      {
        "tag": "packet-content-size",
        "path": {
          "scope": "data-stream-packet-context",
          "path": ["packet sizes", "content"]
        }
      },
      {
        "tag": "discarded-event-record-count",
        "path": {
          "scope": "data-stream-packet-context",
          "path": ["discarded events"]
        }
      },
      {
        "tag": "packet-sequence-number",
        "path": {
          "scope": "data-stream-packet-context",
          "path": ["sequence number"]
        }
      },
      {
        "tag": "update-data-stream-clock-now",
        "data-stream-clock-class-name": "clock src",
        "path": {
          "scope": "data-stream-packet-context",
          "path": ["packet begin TS"]
        }
      },
      {
        "tag": "update-data-stream-clock-after-packet",
        "data-stream-clock-class-name": "clock src",
        "path": {
          "scope": "data-stream-packet-context",
          "path": ["packet end TS"]
        }
      },
      {
        "tag": "event-record-class-id",
        "path": {
          "scope": "data-stream-event-record-header",
          "path": ["ID"]
        }
      },
      {
        "tag": "event-record-class-id",
        "path": {
          "scope": "data-stream-event-record-header",
          "path": ["compact or extended", "ID"]
        }
      },
      {
        "tag": "update-data-stream-clock-now",
        "data-stream-clock-class-name": "clock src",
        "path": {
          "scope": "data-stream-event-record-header",
          "path": ["compact or extended", "timestamp"]
        }
      }
    ]
  },
  {
    "fragment": "field-type-alias",
    "name": "comm",
    "field-type": {
      "field-type": "textarray",
      "length": 16
    }
  },
  {
    "fragment": "event-record-class",
    "id": 0,
    "payload-field-type": {
      "field-type": "struct",
      "fields": [
        {"name": "prev_comm", "field-type": "comm"},
        {"name": "prev_tid", "field-type": "uint32"},
        {"name": "prev_prio", "field-type": "uint32"},
        {"name": "prev_state", "field-type": "uint64"},
        {"name": "next_comm", "field-type": "comm"},
        {"name": "next_tid", "field-type": "uint32"},
        {"name": "next_prio", "field-type": "uint32"}
      ]
    }
  },
  {
    "fragment": "event-record-class",
    "id": 1,
    "payload-field-type": {
      "field-type": "struct",
      "fields": [
        {"name": "ret", "field-type": "int64"},
        {"name": "addr", "field-type": "uint64"},
        {"name": "len", "field-type": "uint64"},
        {"name": "prot", "field-type": "uint64"},
        {"name": "flags", "field-type": "uint64"},
        {"name": "fd", "field-type": "uint64"},
        {"name": "pgoff", "field-type": "uint64"}
      ]
    }
  }
]

CTF2-PROP-1.0: Proposal for a major revision of the Common Trace Format, version 1.8

A new workgroup responsible for the publication of CTF documents

A new method for identifying CTF 2 documents

Why CTF 2?

Design goals

Changes since CTF 1

CTF 2 actors

What is a CTF 2 trace?

Metadata types

Bidirectional association between metadata types and JSON values

Constant integer JSON object

Metadata m‑array

Metadata stream

Data streams

Summary

Structure of the CTF 2 specification

Compliance

Layer 1: CTF 2 field encoding rules (CTFFER)

Byte order m‑string

User attributes m‑map

Scope

Field path m-value

Absolute field path m‑map

Relative field path m‑array

Field lookup mechanism

Encoding context

Field type m-value

Null field type

Encode a null value as a null field

Bit array field type

Encode a bit array value as a bit array field

Boolean field type

Encode a boolean value as a boolean field

Integer field type

Encode an integer value as an integer field

Floating point number field type

Encode a number value as a floating point number field

Enumeration field type member m-value

Enumeration field type

Encode an enumeration value as an enumeration field

String field type

Encode a string value as a string field

Variable-length bit array field type

Encode a bit array value as a variable-length bit array field

Variable-length boolean field type

Encode a boolean value as a variable-length boolean field

Variable-length integer field type

Encode an integer value as a variable-length integer field

Variable-length enumeration field type

Encode an enumeration value as a variable-length integer field

Structure/union/variant field type field m‑map

Structure field type

Encode a structure value as a structure field

Array field type

Encode an array value as an array field

Text array field type

Encode a text array value as a text array field

Sequence field type

Encode a sequence value as a sequence field

Text sequence field type

Encode a text sequence value as a text sequence field

Variant field type

Encode a variant value as a variant field

Union field type

Encode a value as a union field

Layer 2: Data streams, packets, and event records

Data stream

Packet

Event record

Field type alias fragment

Field tag m‑map

Discarded event record count field tag m‑map

Trace class fragment

Allowed field tags targetting trace packet header fields

Data stream class fragment

Allowed field tags targetting data stream packet context fields

Allowed field tags targetting data stream event record header fields

Event record class fragment

Layer 3: Timekeeping

Data stream clock class fragment

Metadata `m‑array`

Byte order `m‑string`

User attributes `m‑map`

Absolute field path `m‑map`

Relative field path `m‑array`

Structure/union/variant field type field `m‑map`

Field tag `m‑map`

Discarded event record count field tag `m‑map`

Update data stream clock now field tag `m‑map`

Update data stream clock after packet field tag `m‑map`