Hi everyone! We’re very excited to announce the first Babeltrace 2.0 release candidate! 😃

What’s new since Babeltrace 2.0.0-pre5?

Babeltrace 2.0.0-pre5 was released 3 May 2019.

We brought many improvements to the CLI, the project plugins, and the library and its Python bindings.

We also updated all the Babeltrace 2 manual pages.

The build-time and run-time requirements for Babeltrace 2.0.0-rc1 are updated:

  • Babeltrace 2.0.0-rc1 does not need the popt library anymore to parse command-line arguments.

    We wrote our own command-line argument parser to remove this obsolete requirement and to support advanced, position-dependent parsing.

  • Babeltrace 2.0.0-rc1 requires GLib ≥ 2.28 at build time and run time instead of the previous ≥ 2.22 requirement.

Command-line interface

Automatic source component discovery

With the convert command’s new automatic source component discovery feature, specify a path, URI, or arbitrary string to let babeltrace2 find the most appropriate components to use:

$ babeltrace2 /path/to/ctf/trace
$ babeltrace2 net://localhost/host/myhost/my-session

This means you don’t need to create an explicit component with the --component option most of the time.

Behind the scenes, this feature queries the babeltrace.support-info object from all known component classes: a component class can reply with a confidence score to indicate its ability to handle a given path or custom string.

For example, a source.ctf.fs (CTF trace directory source) component class replies to this query whether or not the given path looks like a CTF trace directory.

Note

Source component classes do not have to implement the babeltrace.support-info query.

If a component class does not implement this query, the automatic source component discovery algorithm won’t consider it. In that case, you must instantiate such a component class explicitly with the --component option.

Plugins

New sink.text.details component class

This new component pretty-prints all the details of the messages it consumes.

The output is optimized to be human-readable, yet it is deterministic for testing and debugging purposes.

You can use a sink.text.details component to easily inspect messages flowing at any point of a graph and make it easy to test and troubleshoot other component classes.

The Babeltrace 2 project uses this component class for many tests.

source.ctf.fs

A source.ctf.fs component can now open and read malformed CTF traces produced by tracers having known bugs and corner cases (LTTng-UST, LTTng-modules, and barectf).

For example, a source.ctf.fs component can now read data streams which the lttng-crash tool generates, even if the last packets can be malformed.

See the Trace quirks section of the babeltrace2-source.ctf.fs(7) manual page for more details.

Library and Python bindings

Library-specific: simple sink component

Add a sink component to a graph easily using the new simple sink component C API.

A simple sink component is a sink component with a single input port and a single message iterator.

This interface reduces the amount of code you need to write to create a basic Babeltrace 2 message consumer.

Just provide a consumption user function which receives the component’s message iterator to consume upstream messages.

You can also implement other user functions:

Initialization function

Called once during the initialization phase of the graph.

Finalization function

Called once during the finalization phase of the graph.

Python-specific: automatic source component discovery using bt2.TraceCollectionMessageIterator

Integrate the same automatic source component discovery mechanism which the babeltrace2 CLI’s convert command uses to a Python application using a bt2.TraceCollectionMessageIterator object.

With a bt2.TraceCollectionMessageIterator object, your application can access additional component classes installed on the system from the get go.

New field class types

New field classes are available in libbabeltrace2 and its Python bindings to support eventual, new CTF 2 field classes.

The new fields are:

Boolean

Contains a boolean value.

Bit array

Contains a fixed-size array of bits.

A bit array field can represent a CPU status or flag register value, for example.

Unlike integer fields, a bit array field is not a numerical field.

Option

Contains an optional field.

An option field class is conceptually equivalent to a variant field class with two options: no field class and the optional field class.

User attributes

Attach custom user attributes to any of the following trace IR objects:

  • Clock class

  • Event class

  • Field class

  • Stream

  • Stream class

  • Trace

  • Trace class

This new property exists because we expect the CTF 2 metadata objects to contain user attributes too.

Error reporting

Report rich error causes from user functions with the new thread-safe error reporting API.

The error reporting API makes it possible for the various actors interacting through libbabeltrace2 to precisely describe the chain of events that lead to a given error. As a library user, you can access this list of causes and reduce the time needed to troubleshoot problems.

This feature is similar to the stack trace which exception objects contain in many programming languages.

When an error occurs, the babeltrace2 CLI uses this error reporting API to show the top-level error itself, but also a detailed list of its causes, for example:

ERROR:    [Babeltrace CLI] (babeltrace2.c:2546)
  Graph failed to complete successfully
CAUSED BY [Babeltrace library] (graph.c:604)
  Component's "consume" method failed: ..., comp-name="sink.text.details", ...
CAUSED BY [Babeltrace library] (iterator.c:889)
  Component input port message iterator's "next" method failed: ..., iter-upstream-comp-class-name="debug-info", ...
CAUSED BY [Babeltrace library] (iterator.c:889)
  Component input port message iterator's "next" method failed: ..., iter-upstream-comp-name="muxer", ...
CAUSED BY [Babeltrace library] (iterator.c:889)
  Component input port message iterator's "next" method failed: ..., iter-upstream-comp-class-name="MyFirstSource", ...
CAUSED BY [src.demo.MyFirstSource (some-name): 'source.demo.MyFirstSource'] (bt2/native_bt_log_and_append_error.h:100)
  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/dist-packages/bt2/message_iterator.py", line 151, in _bt_next_from_native
      msg = next(self)
    File "./bt_plugin_foo.py", line 33, in __next__
      raise Exception
  Exception

In the Python bindings, raise any exception object as usual from a user function: the bt2 package converts it to a libbabeltrace2 error cause for other parties.

Coinstallation with Babeltrace 1

To make the transition from Babeltrace 1 to Babeltrace 2 easier, we decided to make both projects coinstallable thanks to those changes:

  • The Babeltrace 2 CLI tool is named babeltrace2.

  • The Babeltrace 2 library’s include directory is named babeltrace2.

  • The Babeltrace 2 library is named libbabeltrace2.

  • The Babeltrace 2 manual page names start with babeltrace2-.

Distribution packages will be available for both Babeltrace 1 and Babeltrace 2.

Message Interchange Protocol versioning

Future-proof component classes using Message Interchange Protocol versioning.

As Babeltrace 2 evolves, we expect to introduce new concepts and objects to libbabeltrace2 such as field class types, message types, methods, and message ordering requirements. To ensure forward and backward compatibility can be maintained across future releases of Babeltrace 2, Babeltrace 2.0.0-rc1 adds the concept of Message Interchange Protocol (MIP).

Should we introduce a breaking change to the protocol which trace processing graph components must honor, we’ll bump the MIP version to announce the modified interface.

You can consider the MIP version as an implicit precondition variable for almost all the libbabeltrace2 functions.

Given the expected initialization parameters, a component class can report a range of supported MIP versions. You can then build a graph which operates with a specific MIP version to ensure all its components exchange messages using the same protocol.

const correctness

The Babeltrace 2 C API uses the const qualifier liberally to catch many programming errors at compile time.

As the Python language does not have a const feature to express immutability constraints, the Python classes are now split into constant and mutable variants. Mutable variants of the classes have methods to modify and read object properties whereas their constant counterpart only have reading methods.

All constant classes end with Const.

If you try to modify a constant object, Python raises an attribute error.

Optional packet support

The Babeltrace 2 packet concept exists because CTF needs it. However, many trace formats don’t.

With Babeltrace 2.0.0-rc1, you don’t need to create packet objects to conceptually contain event objects. As such, you don’t need to emit packet beginning and end messages.

By default, packets are not supported for a given stream class: you need to explicitly enable their support.

TRACE logging level

The VERBOSE logging level is now the TRACE logging level.

It’s not obvious whether the VERBOSE level is more or less verbose than the DEBUG level. With TRACE instead, it becomes evident.

What’s new since Babeltrace 1.5?

This section presents a very high level view of the changes from Babeltrace 1.5 to Babeltrace 2.

Read babeltrace2-intro(7) to learn more about the Babeltrace 2 project and its core concepts.

Trace processing graph

Connect components within a trace processing graph and run it to execute trace manipulation or conversion tasks.

Components are instances of component classes. Babeltrace 2 plugins (shared libraries or Python files) are distributable packages of component classes.

The Babeltrace 2 project’s component classes are:

CTF file system source

Read CTF traces from the file system.

LTTng live source

Connect to an LTTng relay daemon and receive CTF streams.

CTF file system sink

Write CTF traces to the file system.

Debugging information filter for LTTng traces

Augment compatible events with debugging information.

Linux kernel ring buffer source

Read Linux ring buffer lines (dmesg(1) output) from a file or from standard input.

Detailed sink

Print messages with details.

Pretty-printing sink

Pretty-print messages (text format of Babeltrace 1).

Message muxer filter

Sort messages from multiple input ports to a single output port by time.

Message trimmer filter

Discard messages that occur outside a specific time range.

Message counter sink

Count messages and print the statistics.

Dummy sink

Consume messages and discard them.

Read multiple CTF traces sharing the same UUID

Merge multiple physical traces sharing the same UUID to a single logical trace.

This is especially useful to read traces produced by the tracing session rotation feature introduced in LTTng 2.11.

Developer mode

Verify your custom component classes by building and running Babeltrace 2 in developer mode.

This special operation mode enables a large number of libbabeltrace2 function precondition and postcondition checks. This is at the cost of making the library less efficient.

With the developer mode, you can develop and test new plugins while ensuring that they honour the contract of the various libbabeltrace2 functions you use.

Since we assume that component classes have been tested in developer mode by their authors, the release, or production build configuration of Babeltrace 2 does not check fast path preconditions and postconditions at run time, resulting in improved performance.

To build Babeltrace 2 in developer mode and test your own plugins, set the BABELTRACE_DEV_MODE environment variable to 1 at configuration time:

$ BABELTRACE_DEV_MODE=1 ./configure

When a function precondition or postcondition is not satisfied, libbabeltrace2 prints why and details to the standard error and calls abort(), just like assert(), so that you can inspect the programming error’s context with a debugger.

For example:

10-06 09:12:20.228 62362 62362 F LIB/VALUE bt_value_array_get_length@value.c:887 Babeltrace 2 library precondition not satisfied; error is:
10-06 09:12:20.228 62362 62362 F LIB/VALUE bt_value_array_get_length@value.c:887 Value object is NULL:
10-06 09:12:20.228 62362 62362 F LIB/VALUE bt_value_array_get_length@value.c:887 Aborting...

Colors

The Babeltrace CLI and some components now use colors when the connected terminal supports it.

LTTng live source’s subscribe mode

Subscribe to a tracing session served by an LTTng relay daemon with a source.ctf.lttng-live component in subscribe mode.

To subscribe to a session, set the source.ctf.lttng-live component’s session-not-found-action initialization parameter to continue. In subscribe mode, the source attempts to consume an LTTng relay daemon’s tracing session and attempts to reconnect periodically if it does not find the target session.

By default, the CLI’s convert command uses the end session-not-found action: babeltrace2 exits with success, just like Babeltrace 1’s babeltrace program.

To use the continue action explicitly, do:

$ babeltrace2 net://relayhost/host/tgthost/my-session \
              --params='session-not-found-action="continue"'

API changes

The libbabeltrace2 API shares nothing with Babeltrace 1’s libbabeltrace API.

The libbabeltrace2 API is trace-format agnostic and aims at making it easy to correctly manipulate traces and logs produced in a variety of formats.

Babeltrace 2 still provides Python bindings through the bt2 package, exposing all the features of the C API. The bt2 package is not a drop-in replacement of the Babeltrace 1’s babeltrace package.

Upcoming

As we enter the release candidate phase of Babeltrace 2’s development cycle, we are hard at work putting the finishing touches on our way to the final 2.0.0 release.

We invite you to try this release candidate and report any problems you may encounter to the lttng-dev@lists.lttng.org mailing list or through the Babeltrace bug tracker.

Documentation

We are currently documenting the entire Babeltrace 2 C API to make the development of new component classes as easy as possible.

We’ll also work on the Python bindings documentation.

The documentation of the API will be made available on the official Babeltrace website.

Other tasks

  • Improve test coverage.

  • Improve resilience to corrupted/malformed CTF traces.

  • Minor internal cleanups and bug fixes.