Trace Compass

July 27, 2014

This proposal is in the Project Proposal Phase (as defined in the Eclipse Development Process) and is written to declare its intent and scope. We solicit additional participation and input from the community.

Background

The Tracing and Monitoring Framework (which we will refer to as TMF) originated from the LTTng component in the Eclipse Linux Tools project. At first, the aim of this component was to provide a control interface for the LTTng tracer, as well as views to graphically display traces that were taken with this tool. TMF was present from the beginning, as a generic framework for reading, parsing and analyzing traces independently from their type, and LTTng was to be the reference implementation.

Over time, support for other trace types was added. Nowadays, LTTng is but one of the many trace types supported by the framework. The developers and users mostly refer to the project as “TMF”, even though it is still technically “The LTTng component of the Linux Tools project”.

It is becoming increasingly confusing when presenting the project to new people, like at conferences, to explain how we build a generic framework for building trace viewers, but that is is located under Linux Tools – LTTng. Especially since the project is specific to neither LTTng nor Linux anymore! Since the project has grown outside of its original scope, and its own community is building around it, we feel like it would make sense for it to become its own project, separate from Linux Tools.

We were also looking at re-branding the project while doing so (as there is already a TMF project at Eclipse in the Textual Modeling Framework). The name that was suggested for the new project is “Trace Compass”.

Scope

Trace Compass is a tool for viewing and analyzing both logs and traces. It provides views, graphs, metrics, etc. to help extract useful information from logs or traces, in a way that is user-friendly and informative. The project provides:

Both a stand-alone RCP application and a standard Eclipse plugin.
A core framework written in Java that exposes a generic interface for integration of logs or trace data input.
Parsers to ingest input logs or traces and convert to the frameworks internal data representation.
Support for the following trace formats natively (no third-party libraries needed):
Linux LTTng kernel traces
Linux LTTng-UST userspace traces
Other traces in the Common Trace Format (CTF), like bare metal and HW traces (e.g. IEEE Nexus 5001 conversion). See also this link.
GDB traces for debugging
The Best Trace Format (BTF) for OSEK
The libpcap (PAcket CAPture) format, for network traces
Custom text or XML parsers that can be added right from the graphical interface by the user
Can be (and has been) extended to support various proprietary log or trace files.
Support for live trace reading and monitoring.
Tracer control (currently available for LTTng traces)
Configurable data-driven views and analysis.
A trace synchronization infrastructure, which allow matching traces taken from different hosts at the same time.
A repository of application-specific or problem-specific modules of all known trace type integration plugins.

The following items would be considered outside of the scope of this project:

Trace viewing and analyzing facilities that do not integrate with the base Trace Compass framework and “do their own thing”. To prevent project bloat, reduce the maintenance burden, and encourage reusability, all views and analyses integrated in the project should use the Trace Compass framework.
Parsers/readers depending on native utilities or libraries. Parsers should be implemented in the framework, in Java, as much as possible. Keeping the native third-party libraries to a minimum makes distribution easier.

Description

Trace Compass is a Java tool for viewing and analyzing any type of logs or traces. Its goal is to provide views, graphs, metrics, etc. to help extract useful information from traces, in a way that is more user-friendly and informative than huge text dumps.

Trace Compass currently supports many trace formats natively (no third-party libraries needed), such as:

Linux LTTng kernel traces
Linux LTTng-UST userspace traces
Other traces in the Common Trace Format (CTF), like bare metal and HW traces (e.g. IEEE Nexus 5001 conversion). See also this link.
GDB traces for debugging
The Best Trace Format (BTF) for OSEK
The libpcap (PAcket CAPture) format, for network traces
Custom text or XML parsers that can be added right from the graphical interface by the user
Can be (and has been) extended to support various proprietary log or trace files.

It is available both as a stand-alone RCP application, or as standard Eclipse plugins. It supports Windows, Mac and Linux. End users can download a pre-compiled package and start using it right away. Developers can create plugins to the framework to add support for new trace types, or new analysis to existing trace types.

They can also create their own re-spin (like their own RCP) that contains the base framework with a different set of plugins included. You’d need a specific parser for each log type, for example, to tell the framework which part of the line corresponds to the timestamp, which part to the source component, which part is the payload, etc. This is better described here. As additional background on a comparison of logs vs. traces; please see this page.

Why Here?

There are many players, both in the Eclipse community and outside, that are using what is today the TMF framework. By remaining within Eclipse, we maintain the guarantees that are offered by the Eclipse Foundation with regards to code provenance, IP due diligence, etc. This means those users will not have to reassess their use of this tool; for them this move is not much more than a simple rename. Since the project is already hosted at Eclipse, and is quite mature and active, we feel that a standard Eclipse project is a natural fit, and that going through Eclipse Labs is not needed.

Licenses

Eclipse Public License 1.0

Legal Issues

Since the code base is already part of an Eclipse project and is under the EPL, there should not be any issues regarding the licensing.

While we haven’t done an exhaustive trademark search for the “Trace Compass” name, we did make sure that the name would not clash with another existing project. Searching for “Trace Compass” in Google returns nothing similar.

Project Scheduling

Since we already have an existing code base and Eclipse-ready build system, it should not take long to get the project going. Once the new project gets approved, we could upload the initial contribution in the following days, and get a build ready within a week.

Future Work

An important point we want to improve once the project is set up is the community/open source aspect. Some of the planned tasks, in no particular order:

Basic marketing, in presentations and in our usual mailing lists, to get the new (and clearer) “Trace Compass” branding known.
Involve the community a bit more. This can simply mean talking about big framework changes and planned features on our mailing list before implementing them. Even if nobody responds at first, it keeps a “paper trail”, and makes the project seem more reliable if breakages are announced in advance.
Go over and cleanup the core framework’s public API. Many parts are convoluted and/or unintuitive. There is a lot of legacy dating back to the LTTng 0.x days that is not needed anymore.

As for the development, we are planning to work on the following main features:

Support for the new features in LTTng 2.5+, including live trace control and reading
Adding more types of data-driven views, improving the existing ones
Automatic X-axis alignment of views, as seen in most performance monitoring tools
Infrastructure for performance testing, à la arewefastyet.com
Trace control and analysis at the model level (integration with Papyrus?)