Real-time monitoring statistics from your system at runtime
Filter and group statistics across your software stack - user IDs, source application, end-host, and many more
Define your own monitoring, at runtime, without re-deploying your system
Pivot Tracing is a monitoring framework for distributed systems that can seamlessly correlate statistics across applications, components, and machines at runtime, without needing to change or redeploy system code. Users can define and install monitoring queries on-the-fly, to collect arbitrary statistics from one point in the system while being able to select, filter, and group by events meaningful at other points in the system.
Pivot Tracing does not correlate cross-component events using expensive global aggregations, nor does it perform offline analysis. Instead, Pivot Tracing directly correlates events as they happen, by piggybacking metadata alongside requests as they execute – even across component and machine boundaries. This gives Pivot Tracing a very low runtime overhead – less than 1% for many cross-component monitoring queries.
Read more about Pivot Tracing in our research paper (Best Paper Award in SOSP 2015)
To get started with Pivot Tracing, check out the code from the Brown University Systems Research GitHub
Pivot Tracing currently supports Apache Hadoop (YARN, MapReduce, HDFS), HBase, and ZooKeeper.
Support for Apache Spark, Cassandra, and others are on the way!
This page is a work in progress - watch this space!
Jonathan Mace is a PhD student in the Brown University systems group. Jonathan's research focuses on end-to-end tracing, monitoring, and resource management in distributed systems.
Rodrigo Fonseca is a professor in the department of computer science at Brown University. He leads the department's research group on Networking and Distributed Systems.