Using the debug tree#

Warning

This debug functionality is highly experimental. Some of the information it exposes may be considered “internal” state of VW and is therefore not guaranteed to be stable between releases.

VW is composed of a stack of reductions that implement individual parts of the computation. Problems are solved by transforming the problem into something that has an existing reduction that can solve it. This is done by chaining reductions together. The debug tree is a tool that can be used to visualize the reduction stack and the data that is flowing through it.

How to enable the debug tree#

When the Workspace is constructed the enable_debug_tree argument must be set to True. This will cause any calls to predict_one(), learn_one(), or predict_then_learn_one() to also return a DebugNode object that represents the root of the tree of computation.

The DebugNode essentially represents a snapshot of state at that place in the reduction stack. There are many properties that can be inspected, and can be seen here: DebugNode.

Let’s say we just wanted to see which reductions were called for a simple default VW model. We can do this by traversing the returned tree and reading the properties of each node as we go.

def print_tree(node, depth=0):
    print(f"{'  '* depth} {node.name}({node.function}) pred:{node.output_prediction}")
    for child in node.children:
        print_tree(child, depth + 1)

import vowpal_wabbit_next as vw

workspace = vw.Workspace(enable_debug_tree=True)
parser = vw.TextFormatParser(workspace)

workspace.learn_one(parser.parse_line("0 | price:.23 sqft:.25 age:.05 2006"))
workspace.learn_one(
    parser.parse_line("1 2 'second_house | price:.18 sqft:.15 age:.35 1976")
)
workspace.learn_one(
    parser.parse_line("0 1 0.5 'third_house | price:.053 sqft:.32 age:.87 1924")
)

prediction, dbg_node = workspace.predict_one(
    parser.parse_line("| price:0.25 sqft:0.8 age:0.1")
)
print_tree(dbg_node)

 count_label(predict) pred:0.3052094280719757
   scorer-identity(predict) pred:0.3052094280719757
     gd(predict) pred:0.3052094280719757

VW uses a variety of different reductions, some of which are for collecting stats or transforming the state of the example as it flows through. In the above example count_label is used for reporting and therefore is not going to have a visible effect on the state of the example.

In this example we can see that the prediction doesn’t change as it propagates back up the tree. However, if we did something like change the link function used, then we could see the effect of that more clearly.

import vowpal_wabbit_next as vw

workspace = vw.Workspace(["--link=logistic"], enable_debug_tree=True)
parser = vw.TextFormatParser(workspace)

workspace.learn_one(parser.parse_line("0 | price:.23 sqft:.25 age:.05 2006"))
workspace.learn_one(
    parser.parse_line("1 2 'second_house | price:.18 sqft:.15 age:.35 1976")
)
workspace.learn_one(
    parser.parse_line("0 1 0.5 'third_house | price:.053 sqft:.32 age:.87 1924")
)

prediction, dbg_node = workspace.predict_one(
    parser.parse_line("| price:0.25 sqft:0.8 age:0.1")
)
print_tree(dbg_node)
print(prediction)

 count_label(predict) pred:0.5757155418395996
   scorer-logistic(predict) pred:0.5757155418395996
     gd(predict) pred:0.3052094280719757
0.5757155418395996

Fetching the scores using the debug tree#

When using cb_explore_adf it can be helpful to inspect the predicted scores were used to generate the exploration distribution. Using the debug tree, you can gain access to those scores.

from typing import Optional
import vowpal_wabbit_next as vw


workspace = vw.Workspace(["--cb_explore_adf"], enable_debug_tree=True)
parser = vw.TextFormatParser(workspace)

ex = [
    parser.parse_line("shared | s_1"),
    parser.parse_line("0:0.1:0.25 | a:0.5 b:1"),
    parser.parse_line("| a:-1 b:-0.5"),
    parser.parse_line("| a:-2 b:-1"),
]

workspace.learn_one(ex)
prediction, debug_node = workspace.predict_one(ex)


def find_cb_adf_node(node) -> Optional[vw.DebugNode]:
    if node.name == "cb_adf":
        return node

    for child in node.children:
        found = find_cb_adf_node(child)
        if found:
            return found

    return None


cb_adf_node = find_cb_adf_node(debug_node)

print(f"Action probabilities: {prediction}")
print(f"Action scores: {cb_adf_node.output_prediction}")

Action probabilities: [(2, 0.9666666388511658), (1, 0.01666666753590107), (0, 0.01666666753590107)]
Action scores: [(2, -0.07499927282333374), (1, -0.01249987818300724), (0, 0.09999903291463852)]

We can also use this to understand the entire process of how the scores are produced.

import vowpal_wabbit_next as vw


def print_tree(node, depth=0):
    pred = (
        f"pred:{node.output_prediction}"
        if isinstance(node.output_prediction, list)
        else ""
    )
    partial_pred = (
        f"partial_pred:{node.partial_prediction}"
        if not isinstance(node.partial_prediction, list)
        else ""
    )
    print(f"{'  '* depth} {node.name}({node.function}) {pred}{partial_pred}")
    for child in node.children:
        print_tree(child, depth + 1)


workspace = vw.Workspace(["--cb_explore_adf"], enable_debug_tree=True)
parser = vw.TextFormatParser(workspace)

ex = [
    parser.parse_line("shared | s_1"),
    parser.parse_line("0:0.1:0.25 | a:0.5 b:1"),
    parser.parse_line("| a:-1 b:-0.5"),
    parser.parse_line("| a:-2 b:-1"),
]

workspace.learn_one(ex)
prediction, debug_node = workspace.predict_one(ex)

print_tree(debug_node)

 shared_feature_merger(predict) pred:[(2, 0.9666666388511658), (1, 0.01666666753590107), (0, 0.01666666753590107)]
   cb_explore_adf_greedy(predict) pred:[(2, 0.9666666388511658), (1, 0.01666666753590107), (0, 0.01666666753590107)]
     cb_adf(predict) pred:[(2, -0.07499927282333374), (1, -0.01249987818300724), (0, 0.09999903291463852)]
       csoaa_ldf-rank(predict) pred:[(2, -0.07499927282333374), (1, -0.01249987818300724), (0, 0.09999903291463852)]
         scorer-identity(predict) partial_pred:0.09999903291463852
           gd(predict) partial_pred:0.09999903291463852
         scorer-identity(predict) partial_pred:-0.01249987818300724
           gd(predict) partial_pred:-0.01249987818300724
         scorer-identity(predict) partial_pred:-0.07499927282333374
           gd(predict) partial_pred:-0.07499927282333374

Visualizing runtimes with a flamechart#

Runtimes of the reductions can be measured and then visualized as a flamechart.

import vowpal_wabbit_next as vw


def print_node(node, stack_so_far=[]):
    extra_colon = ";" if len(stack_so_far) != 0 else ""
    print(
        f"{';'.join(stack_so_far)}{extra_colon}{node.name}({node.function}) {node.self_duration_ns}"
    )
    for child in node.children:
        print_node(child, stack_so_far + [f"{node.name}({node.function})"])
    if len(stack_so_far) != 0:
        print(f"{';'.join(stack_so_far)}{extra_colon} 0")


workspace = vw.Workspace(["--cb_explore_adf", "--cb_type=ips"], enable_debug_tree=True)
parser = vw.TextFormatParser(workspace)

ex = [
    parser.parse_line("shared | s_1"),
    parser.parse_line("0:0.1:0.25 | a:0.5 b:1"),
    parser.parse_line("| a:-1 b:-0.5"),
    parser.parse_line("| a:-2 b:-1"),
]

dbg_node = workspace.learn_one(ex)
print_node(dbg_node[0])

shared_feature_merger(learn) 4010
shared_feature_merger(learn);cb_explore_adf_greedy(learn) 1854
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn) 4297
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict) 4680
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict);scorer-identity(predict) 1407
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict);scorer-identity(predict);gd(predict) 9713
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict);scorer-identity(predict); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict);scorer-identity(predict) 1137
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict);scorer-identity(predict);gd(predict) 658
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict);scorer-identity(predict); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict);scorer-identity(predict) 1131
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict);scorer-identity(predict);gd(predict) 632
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict);scorer-identity(predict); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(predict); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn) 3130
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn);scorer-identity(learn) 1562
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn);scorer-identity(learn);gd(learn) 13844
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn);scorer-identity(learn); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn);scorer-identity(learn) 1183
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn);scorer-identity(learn);gd(learn) 661
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn);scorer-identity(learn); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn);scorer-identity(learn) 1158
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn);scorer-identity(learn);gd(learn) 647
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn);scorer-identity(learn); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn);csoaa_ldf-rank(learn); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn);cb_adf(learn); 0
shared_feature_merger(learn);cb_explore_adf_greedy(learn); 0
shared_feature_merger(learn); 0

This output can then be processed using this tool. For example:

perl flamegraph.pl --flamechart stacktrace.txt > stacktrace.svg

To get a something like:

flamechart