Hackthology

Taming the Variants: Multi-Architecture Continuous Testing at Google

2026-03-19T00:00:00-04:00

Tim A. D. Henderson, Sushmita Azad, Chandrakanth Chittappa, Ali Esmaeeli, Laura Macaddino, Sam Manfreda, David Margolin, Dharma Naidu, Sabuj Pattanayek, Sachin Sable, Ruslan Sakevych, Dushyant Acharya, Adrian Berding, Kevin Crossan, Wolff Dobson, Avi Kondareddy, and Abhayendra Singh. Taming the Variants: Multi-Architecture Continuous Testing at Google. ICST 2026.
DOI. PDF. WEB.

Note

This is a conversion from a latex paper I wrote. If you want all formatting correct you should read the pdf version.

Abstract

Enterprises are increasingly adopting multiple general-purpose computer architectures in the data center. This leads to new testing challenges as it creates demand to qualify the software for the additional architectures. Naively double-testing all software for both architectures is costly and unnecessary. Further, reconfiguring CI/CD to take advantage of the new architecture can be non-trivial at scale. This paper introduces CI/CD variants and an optimized testing cycle to solve these twin challenges. We empirically evaluate our solution's impact on human and machine expenses using 44k projects at Google on real production data. First, we estimate saving ~25% of machine expenses at the negligible cost of a few delayed breakage detections per day. Second, we estimate a 90+% reduction in human cost for migrating the configuration. All features described in this paper are now Generally Available at Google and we report this as an empirical case study in scaling CI/CD to new architectures.

I. Introduction

Over the last decade, hyperscalers have introduced multiple general-purpose architectures into the data center [1]. At Google, this creates challenges for our very large scale continuous integration and delivery platforms (CI/CD) [2]. We aim to enable adoption by ensuring the software is appropriately built and tested. We face two challenges.

First, a high performance Continuous Integration [3] system identifies novel build and test failures the moment they are introduced to the source repository. This enables developers and automation to find and fix these failures as soon as possible. The sooner a failure is found, the cheaper it is to fix. For instance, a bug-introducing change can be automatically reverted without review, provided there haven't been further changes that would cause merge conflicts [4, 5]. Finding failures as quickly as possible is in tension with running tests as cheaply as possible (achieved by adjusting testing frequency). Can we add the new architecture without constantly retesting on both platforms?

Second, when a new compute architecture is introduced, all projects need to determine:

Is the project's code build-compatible?
Do the project's tests pass on the new architecture?
Is the project's runtime performance acceptable?

Answering these questions requires modifying the CI/CD configuration to additionally build, test, and deploy the code to the new architecture(s). This mechanical, unrewarding task can be a costly undertaking for an organization. Prior to the start of this project we estimate that it took an expert from the central Arm migration team (whose primary job was to perform this qualification) hours of hands-on time just to make the required configuration changes to a single project. Google operates a monorepo with billions of lines of source code and over 100,000 binaries. Making manual changes to every project was impractical [1].

The two challenges are intertwined. We neither want to double test nor do we want our users to manually edit configuration files. Further, manual changes create bespoke additional CI/CD configurations requiring future maintenance. While guides exist, users don't always follow instructions. This means that the CI/CD systems would need complex logic to link functionally-equivalent x86 and Arm configurations.

Without reliably linking configurations, the system cannot optimize the testing process. We solved both problems by introducing variants: automatically reconfigured user projects. A variant adapts the original configuration for each targeted platform. This flexible infrastructure also supports variants for compiler optimization level, address and memory sanitization, and other build modifications.

Because the variants are automatically configured, they are also structured. This allows our systems to make consistent choices on how building and testing should occur for each variant. Our primary CI system, the Test Automation Platform (TAP) [2, 6, 4, 5, 7], introduced a new testing mode: Target Comprehensive Cycles, optimizing the cost of testing across multiple architectures. Our CD system also adopted Target Comprehensive cycles to realize the cost savings. However, to prevent bugs escaping to production, it introduced Late Stage Testing. This phase executes the remaining tests prior to production, but at a lower frequency.

A. Contributions

We detail our solution, Variants, and empirically analyze its efficacy and performance. We measure the realized human migration savings from variants and the machine costs avoided by Target Comprehensive (TC) testing. Finally, because optimizations involve a trade-off, we examine the productivity cost of delaying the detection of bugs by skipping tests.

Variants are generally available and used by 44k projects. We present a large-scale empirical case study on the efficacy of our methods using production data. We estimated both the human and machine savings and the productivity cost of delayed breakage detection.
A new Target Comprehensive (TC) scheduling mode prevents doubled testing costs. We measured savings of approximately ~25% over the naive double testing strategy.
Late-Stage Testing (LST) shifts test load into later, less frequent, parts of the CI/CD pipeline. This executes test configurations skipped by TC prior to user release. Late-stage testing failures occur less than 0.1% of the time.
Automated configuration for building and testing for alternative architectures reduces migration effort by approximately 90%.

II. Background

At Google, developers use multiple tools as part of their CI/CD system. Testing is typically handled by TAP (the Test Automation Platform), which performs testing both prior to code being submitted (presubmit testing) and after code is submitted to the mainline branch of the mono-repository. TAP only handles tests that fit on a single machine and are version hermetic (i.e. only use code from a single version and do not make network calls). Some teams need larger integration tests that need multiple machines or need to compare behavior between the test environment or a production environment. These teams additionally use an Integration Testing platform. While our work on variants is also used by the Integration Testing platform, we constrain the focus of this paper to TAP.

TAP runs testing cycles, where all projects and their Bazel targets are batched together at the same version [4, 7]. Once all the targets for a project have executed, TAP computes a Project Status (ex. PASSING, FAILING, ...). A PASSING status enables the next service in the CI/CD system to take over: Rapid.

Rapid is a generalized CI/CD workflow execution engine. It is similar in capability to commercial and open source systems such as GitHub Actions or Jenkins. It allows developers to orchestrate a complex series of processes to build production packages, trigger integration tests, and trigger production deployments. While Rapid's capabilities are very general, all Rapid "releases" start with a Candidate Creation phase, during which testing results from TAP and other systems are verified before the production binary package is created. A developer may configure some tests to only trigger in Rapid instead of the more common and recommended method of using TAP.

Due to the general complexity of software development at Google, we have a managed platform that is built on top of all of this called Urfin. It provides a streamlined intent-based mechanism for developers to configure their projects. Urfin standardizes the workflows that Rapid executes and the production environments. When a project is configured by Urfin (instead of "Rapid Native"), we are able to automatically determine what workflows modify production environments (versus those used for additional verification or validation with non-production data or users). This structural distinction is relevant for our Late-Stage Testing strategy and empirical evaluation, motivating our introduction of these terms.

A. Glossary of Google Terminology

Piper – The version control system used at Google [8].
Changelist (CL) – A commit in Piper. CLs have integer numbers and submitted CLs have monotonically increasing numbers.
Bazel – The build tool used at Google: https://bazel.build/ [9].
Target – The compilation unit in Bazel. TAP doesn't run individual test methods or even test classes / suites. Instead, it runs Bazel test targets that may contain many test methods and classes.
Forge – The large remote build execution cluster used by Bazel [9].
Build System – A collection of services for batching and running builds in production [9, 10, 11].
TAP – The primary continuous integration platform [12, 13, 14, 15, 16, 2, 17, 6, 4, 5, 18, 7].
Rapid – The primary workflow automation tool use to orchestrate release processes [9].
Urfin – A managed platform built on top of tools like TAP and Rapid that dramatically simplifies configuration and usage of release automation.
Rapid Native – Our term for release projects that don't use the managed platform (Urfin). These projects have non-standard workflows and configurations.
Blueprint File – The common configuration file format used by TAP, Rapid, and many other tools. It is defined in a bespoke configuration language (NCL).

B. Configuring TAP and Rapid

TAP, Rapid, and other tools are configured in Blueprint files. Blueprints are written in a bespoke, proprietary configuration language (NCL). NCL is a high-level functional language for describing and manipulating protocol buffers. It supports complex data types, functions, and libraries. The power of the language has led to proliferation of complex Blueprints that are difficult to mechanically modify for Large Scale Changes [9], even with the support of modern LLM tooling [1].

Users specify in the Blueprints what TAP should run by defining TAP Projects. A TAP Project has patterns (that match Bazel targets), Bazel flags (which control compilation modes such as building for Arm), a maximum execution frequency, and notification routing. Rapid Projects are also defined in Blueprints. These projects can be extremely complex. We will focus just on how users configure testing. A Rapid Project can be gated on one or more TAP Projects, which must all pass. When they all pass it is called a Shared Green or a Golden.

Urfin, the managed platform, generates Blueprints (and other configuration) from a higher level "intent"-based configuration format. Urfin's configuration is less flexible than Blueprints and large scale changes are easier to mechanize.

C. TAP Overview

The continuous integration system, TAP, has two phases: TAP Presubmit and TAP Postsubmit. Presubmit runs as part of the developer's "code review loop", executing when they send out changes for review and when they attempt to submit the code to the repository. Because of the massive scale of the repository, TAP Presubmit does not conduct exhaustive testing. Instead, it operates in a "best-effort" mode that attempts to prevent most breakages from escaping into submitted code. TAP Presubmit uses static build dependence-based test selection [12, 16], machine learning driven selection [17], and machine learning driven flakiness mitigation [18]. In this paper, we are focused on TAP Postsubmit's behavior and look forward to analyzing changes to Presubmit in a future manuscript.

TAP Postsubmit runs tests after changes have been submitted. While it originally ran the affected tests (based on the build graph) at every single change [13], this quickly proved too expensive. Today, Postsubmit runs periodically and avoids running if its Forge is too full. This strategy effectively batches many changes together, leading to a search problem when breakages occur [4, 5]. We call these "periodic runs" Fully Comprehensive (FC) testing cycles since TAP runs all tests in all projects with all Bazel flag configurations that have been affected since the last cycle. Because FC cycles currently run at a maximum rate of once per hour, in 2025 we introduced a faster ML driven cycle to find breakages faster called Speculative cycles [7]. Speculative cycles only run targets whose executions are likely to reveal novel breakages.

D. Rapid Overview

Rapid runs user-defined "workflows" that are generally triggered off of a schedule. The first workflow, Candidate Creation, always creates a release "candidate", which includes verifying test results and building the binary packages. When Rapid is configured via Urfin, the managed platform, the workflows are standardized and structured such that we know which workflows can change production environments. Rapid's schedule for Candidate Creation defaults to once every 2 hours in Urfin, and production pushes happen once per day on business days. Many Rapid Native projects are configured at a slower cadence.

When verifying test results, Rapid can execute any tests not previously run. Because it evolved independently from TAP, Rapid has always maintained its own execution logic, which historically led to redundant and wasteful testing effort. As part of this project, Rapid now recognizes which tests TAP has already completed; by eliminating these redundant executions, we have also reduced the risk of a release being blocked by flaky test failures.

III. Configuring CI/CD for the new Architecture

In order for a project to take advantage of the new compute platform, it needs to be built and tested for the new platform. Practically, this requires adjusting its Blueprint (the configuration, Section II-B) to additionally instruct the CI/CD systems to build and test for the new platform.

Listing 1 provides an example of the work it takes to manually create new CI/CD projects to build and test a binary for Arm. As shown in the listing, multiple sections need to be modified and these modifications cannot always be done atomically. Depending on the complexity of a project's configuration, it may require refactoring of the functions or libraries used to generate the various components (such as buildable units) to generate both Arm and x86 versions of those components. Approximately 25% of active Rapid projects have complex Blueprints requiring manual modifications.

Listing 1: From a developers perspective, lines of code needed to manually configure arm-based testing and release automation.

 // I need to manually create Arm Buildable Units
local arm_buildable_unit=::blueprint::BuildableUnit(
  name="arm_ex",
  build_flags=["--compilation_mode=opt", "--cpu=arm"],
  test_patterns=[ "//ex/pattern/...", "-//ex/pattern/x86tests/..."]);

// And associate them with a new Arm TAP Project
blueprint_file=::blueprint::BlueprintFile(
  continuous_tests=[
    ::blueprint::ContinuousTest(name="ex", buildable_unit_name="ex"),
    ::blueprint::ContinuousTest(name="arm_ex", buildable_unit_name="arm_ex"),
  ],
  buildable_units=[buildable_unit, arm_buildable_unit]);

continuous_tests=[
    ContinuousTest(name="ex", buildable_unit_name="ex"),
    ContinuousTest(name="ex", buildable_unit_name="ex_config"),
    ContinuousTest(name="ex.arm", buildable_unit_name="ex.arm"),
    ContinuousTest(name="ex.arm", buildable_unit_name="ex_config.arm"),
]
// which I add to my Release project's Golden Info.
golden_changelist=[
    GoldenInfo(tap_build="ex"), GoldenInfo(tap_build="ex_config"),
    GoldenInfo(tap_build="ex.arm"), GoldenInfo(tap_build="ex_config.arm")
]
// create duplicate release with Arm config.
ReleasableUnit(
  name="ex_multiarch_release",
  buildable_unit_names=[ "ex", "ex_config", "ex.arm", "ex_config.arm"])

// and I add --fat_package_cpu=k8,arm to all Release BuildableUnits
BuildableUnit(
  name="ex_package",
  mpm_patterns=["//ex/pattern/..."],
  test_patterns=[ "//ex/pattern/...", "-//ex/pattern/x86tests/..."],
  build_flags=["--fat_mpm_cpu=k8, arm"],
  ...
)

Additionally, as the central team gained experience migrating our largest binaries to Arm they discovered a few best practices. For instance, it was better to already have continuous testing results before going to the product team with requested changes. This allowed the product team to better assess the changes. However, setting up a normal CI pipeline for running the Arm tests before notifying the product team is a recipe for friction and pushback, as there are likely to be real and noisy failures. Thus, the central porting team needed a way to run the tests in the same way the product team would.

To solve this problem, we came up with the idea of having "shadow CI projects" that would be duplicates of the production CI projects, but turn off notifications, switch the cost accounting, and add the necessary flags to run the project on Arm. To create these "shadow CI projects", the central team had to create Blueprint files in their own directory that imported and modified the target project's configuration. While the central team had a process that let them drive migrations to increase the migration rate, we needed a more automated solution.

A. Variants

The experiences of the central migration team led us to create Variant projects. These are automatically generated CI Projects that take the user-supplied configuration and adjust it to use different Bazel flags. For example, adding the flag --cpu=arm will compile and test for Arm. Variants are general and allow us, as the CI system owner, to dynamically define new variants as needed.

Figure 1

Figure 1 shows the structure of how Rapid and TAP interact with the new ~arm variant. Each TAP project has a copy made (potentially removing or adding targets as necessary), and Rapid utilizes the shared green between the original projects and the variant projects when creating release candidates. The new TAP projects are largely identical to the original project, but we have the ability to globally modify any setting in the variant projects we choose. We used this to implement multiple "modes" for a variant: DISABLED, SHADOW, and RELEASE. We were then able to launch variants in SHADOW mode globally with no user involvement or toil from the central migration team. This allowed collection of repository wide testing results to guide porting effort. When a project is ready to adopt Arm, a simple top-level field lets them switch to RELEASE mode.

Variant projects were designed to be largely identical to regular user configured projects. This meant most portions of TAP, downstream systems, and API clients initially worked unmodified with the variants. However, we eventually introduced differentiated behavior. For instance, we changed the web user interface to explicitly show the user the relationships between the projects when a variant is enabled and has valid targets.

B. Porting Process with Variants

By default, all projects run their Arm variant in shadow mode. Once a project's tests all pass on Arm it becomes eligible for enabling the Arm release (but not necessarily running in production [1]). To enable the Arm release, we added a single top level field to the Blueprint (plus service level opt-outs) that enables Arm for all CI and Release projects defined in the Blueprint. This field arm_variant_mode is shown in Listing 2.

Listing 2: Simplified variant configuration

blueprint_file = ::blueprint::BlueprintFile(
  arm_variant_mode = ::devtools_blueprint:: VariantMode::VARIANT_MODE_RELEASE;
  continuous_tests = [
    ::blueprint::ContinuousTest(name = "example", buildable_unit_name = "example"),
  ],

  buildable_units = [buildable_unit]);

This new field drastically simplifies the porting process. It can usually be mechanically added (although not always). In Section VI-E, we estimate the total savings of human labor from this simplification. With the automated construction of shadow projects and the ability to mechanically drive changes to Blueprints, we were able to fully automate portions of the porting process.

C. Complexity of Implementation

In Section VI-E, we analyze the Return on Investment of this project and find that it was positive. However, the actual implementation of variants was complicated for many reasons that are outside of the scope of this paper to fully describe. For instance, TAP manages a set of projects and their targets at the scale of Google's multi-billion line mono-repository. Those projects and targets form a bipartite graph where a project contains multiple targets but a target may be inside multiple projects. Adding a single variant doubles the size of that graph. There are technical challenges involved with optimizing the variant implementation for computing, storing, and serving this graph. Note, the graph is dynamic as it changes at practically every submitted change.

Other, more prosaic challenges emerged, such as finding the correct policy for detecting incompatible projects, redesigning data storage layers for project status, and renegotiating the contract between Rapid and TAP, among many other challenges. We regret that we do not provide the full saga of the development.

IV. Target Comprehensive Testing

One benefit of variants we have not yet discussed is their structured nature. When developers manually create new CI and CD projects as shown in Listing 1, there are no constraints on what they might change. We cannot enforce standard naming conventions automatically, nor can we even ensure they create them in the same Blueprint configuration file. Developers may add or subtract additional flags in non-standard ways. They may add or remove sections of their code from the patterns to be built and tested. Because of this, the central services (TAP and Rapid) would need to make conservative choices when qualifying releases.

However, variants have none of the above problems. Their definitions (what flags are added, what kind of targets are excluded, etc...) are centrally defined and audited. This enables direct monitoring (as shown in Section VI-C) of behavior divergence between x86 and Arm. It also enables us to make changes to how we schedule tests in TAP and Rapid.

As discussed in Section II, TAP has traditionally run all tests in all configurations. We call this testing mode Fully Comprehensive (FC). In order to find breakages faster, we then also ran tests in a Speculative mode [7]. Now, we introduce a third mode enabled by variants: Target Comprehensive (TC) testing. Under TC, we guarantee that targets are run at least once in some configuration (x86, arm, or other variant). Because of the automatically generated nature of variants and our central monitoring, results in one configuration are generally substitutable for another configuration (see Section VI-C).

A. Integration with Release

A TAP project is eligible for Target Comprehensive testing when all release projects that reference it are enabled for Arm (see Listing 2). Under TC, TAP only guarantees to run a target on a single architecture (but it can choose to run it on more). However, before a release goes to production (and potentially executes on an architecture different than the one TAP tested on), we want to qualify it in all configurations. To do so, we run the remainder of the tests that TAP skipped during the release process. Figure 2 visualizes how this all connect together.

Figure 2

In release, there are two types of projects: Rapid Native (unmanaged) projects and Urfin (managed) projects (Section II). By default, Urfin projects "Create Candidates" every two hours and push production releases once per business day (excluding Fridays). Because of the structured nature of Urfin, we know centrally when an Urfin project pushes to production. This allows us to delay the finalization of testing to immediately before the production push. We call this Late Stage Testing.

In Rapid Native, we do not know when in the release process a developer may be modifying production. Therefore, we need to continue to complete testing in the traditional Create Candidate phase. Luckily, many Rapid Native projects tend to be configured to Create Candidates once per day.

B. Cost Savings

Target Comprehensive testing shifts the load right in the Software Development Life Cycle (closer to the user and further away from the developer). This is expected to save machine cost by reducing the frequency at which tests are executed on both architectures. We expect significant savings, since production pushes usually only occur approximately once per day and TAP testing cycles occur once per hour. In Section VI-A, we empirically evaluate our realized savings. While lower than the theoretical maximum for a variety of reasons, the results are significant and financially meaningful to us.

V. Empirical Evaluation

A. Research Questions

RQ1: How does the Target Comprehensive scheduling strategy, which executes a union of test targets once across architectures ($T_{\text{x86}} \cup T_{\text{Arm}}$), compare to a Fully Comprehensive baseline ($T_{\text{x86}} \times T_{\text{Arm}}$) in terms of compute resource consumption and project status latency?
RQ2: Does the Target Comprehensive approach maintain release success rate compared to Fully Comprehensive testing? Specifically, what is the impact on breakage detection latency and rate of late failure detections?
RQ3: How does an intent-based configuration abstraction, arm_variant_mode, compare to manual Blueprint configuration in terms of upfront engineering cost (measured as saved migration effort)?

B. Note on Statistical Testing

Throughout our analysis below, our primary method for assessing validity and effect size is Bootstrap resampling (with 10,000 samples) to construct confidence intervals at the 95% confidence level using the percentile method [19]. While our sample sizes are large enough to generally "ignore" normality violations, the violations are at times quite severe in our datasets. To build confidence in the conclusions we usually apply a secondary parametric analysis (such as an appropriate t-test or regression model). The results of the parametric tests confirmed the non-parametric Bootstrap method. Finally, the Bootstrap method returns larger confidence intervals than the corresponding parametric method employed. We present these more conservative intervals in the text of the paper.

C. Analyzing the Effect on Machine Costs

The motivating reason for Target Comprehensive testing cycles was to save machine costs as double testing was deemed unsustainable. So, did the cost savings materialize? Forge provides highly-detailed per action logs of executor cost and time. An action is a single build action that Bazel invokes. For instance, a compiler or linker command. Test actions (when a test is executed) are trivially differentiable from the build actions.

We analyzed Forge usage data from TAP Postsubmit (CI runs occurring after code submission) for 44,243 projects that had Arm testing enabled. We collected data over several weeks, joining Bazel targets with Forge actions to estimate the amortized build cost, and collecting the direct execution cost for each test action. For each testing cycle run by TAP Postsubmit, we summed the amortized build costs and test costs for all targets in the project to create a total cost per cycle. Target costs were normalized when shared by multiple projects in the same testing cycle.

We performed a paired comparison (by project) of the Target Comprehensive (TC) and Fully Comprehensive (FC) costs. We evaluated performance using two distinct metrics: per-project savings and global system savings. In both cases, savings are defined as $1 - \text{TC} / \text{FC}$.

To estimate the expected "per project" savings, we used the Geometric Mean (GM) of the ratio for all projects: $1 - \text{GM}(\text{TC} / \text{FC})$. To estimate the expected system (or total) savings, we estimated the ratio of the sum of costs across all projects in a scheduling cycle: $1 - \sum \text{TC} / \sum \text{FC}$.

For both estimates, we used the non-parametric method Bootstrap re-sampling [19], which does not require an assumption about normality of the data. As can be seen in Figure 3, the data has heavy tails and multiple peaks violating this assumption. Using Bootstrap, we resampled (for each estimate) 10,000 times and computed the expected savings (the median) and 95% confidence interval using the 2.5th - 97.5th percentiles. Finally, we cross checked the per-project estimate using a paired t-test.

D. Analyzing the Effects on Project Status Latencies

A project status includes the results of all targets and indicates if the project is passing, failing, broken, etc. Project status latency is the time elapsed between successive calculations of project statuses at each cycle. To account for distribution variability, we utilize a Poisson Bootstrap method with 10,000 resamplings to analyze latency across 2.75 million FC statuses (before launch) and 2.94 million TC statuses (after launch). The cohort consists of 31,996 TAP projects that were Arm-enabled just prior to launch of TC.

E. Analyzing Breakage Detection Latency

Breakage detection latency is defined as the amount of time between when a developer submits a bug that breaks a test and the breakage being detected through a CI execution. We use Bootstrap analysis to compute confidence intervals to compare this breakage detection latency before and after the launch of Target Comprehensive testing. Further parametric analysis was completed that agrees with the non-parametric analysis, which we omit for brevity.

F. Analyzing the Effects on Release Failure Rate

Our second question, RQ2, asks whether the reduction in coverage from Target Comprehensive testing caused additional failures in release. As a reminder, for binaries that will actually be released, we run the remaining tests that were not run during Target Comprehensive. Thus, every test is run on both platforms if it runs in production. Running on both platforms happens much less frequently and enables the savings shown in RQ1.

We gathered data on 91,815 Release Projects, including both projects with Arm enabled and not enabled. Initial normality checks revealed the failure data to be extremely heavy-tailed and zero-inflated: the vast majority of projects never fail, while a small minority fail constantly. We filtered out approximately 130 projects that failed 100% of the time in both periods, as manual inspection confirmed these were abandoned and would otherwise skew volume-weighted metrics.

We analyzed the two project types: "Rapid Native" (flexible workflows) and "Urfin" (managed/structured workflows). For Urfin, we distinguish between the standard "Candidate Creation" phase and the additional "Late-stage" testing performed immediately before production push.

We compare across these 3 groups the change in test failure rates. We look at the comparison in two ways:

Paired Analysis (System Burden): We calculated the mean arithmetic difference in failure rate for each project before and after Target Comprehensive was enabled. This measures the expected change in the total failure rate on the system.
Geometric Analysis (Typical Experience): We analyzed the change in the geometric mean of project failure rates. This metric controls for volume bias (projects with high execution counts) and outliers, answering the question: "Did the failure rate of a typical project change?"

Given the non-normal shape of the data, we utilized Bootstrap re-sampling (10k iterations) to estimate 95% confidence intervals as our primary method. We cross-checked the paired analysis using a Wilcoxon Signed-Rank Test to detect non-random drifts in rank order, and cross-checked the geometric analysis using a Robust Linear Model (RLM) to provide parametric validation robust to outliers.

G. Estimating Engineering Cost Savings

We want to compare the engineering costs of migrating projects using the new variants approach versus a more complex legacy method. We measured effort using the following normalized units that represent the time that a single software engineer spends working.

Definition 1 (SWEy). A SWE Year is a normalized unit that represents a software engineer working for a year.

SWEy are treated as interchangeable regardless of which person or how many persons work on the migration. Note that coordination costs between multiple people working on a migration are also neglected. Due to confidentiality constraints, we are unable to report our internal SWEy numbers that we have computed. Instead, total effort saved will be reported, which provides a percentage of working time avoided due to the improved configuration mechanism.

To estimate the total effort, we began by estimating the average or expected effort to migrate a single configuration file (i.e., Blueprint). This was estimated using data from the team conducting central high-priority migrations. This team spent approximately 80% of their total time on conducting manual migrations before variants were introduced. Based on their migration output over the course of an entire quarter, we derived a specific (but confidential) estimate of the average time spent per Blueprint.

Definition 2 (BPe). BPe = "Blueprint Effort", or the time spent to migrate a single Blueprint without variants. (Actual number is confidential).

Note, the above estimation was computed from expert engineers who were tasked only with conducting migrations. This central team only migrated a small fraction of the total projects. Engineers who don't work on migrations every day are less expert in conducting the migration and they take longer. We estimate that the migration would take twice as long for a non-expert engineer. We use the range [BPe, 2 ⋅ BPe] to estimate a normal distribution of effort.

In the same way, we can estimate Ve:

Definition 3. Ve = "Variant Effort" of the time spent to migrate a single Blueprint using variants. (Actual number is confidential).

Using these estimates, we can estimate total migration costs:

$$\text{baseline-total}{\text{(SWEy)}} = \text{project count} \times 2 \times \text{BPe}$$}

$$\text{variant-total}{\text{(SWEy)}} = \text{project count} \times \text{Ve}$$}

To compute return on investment, we account for realized machine savings (see Section VI-A) and the development cost (number of SWEy spent on development).

$$\text{ROI} = 1 - \frac{\text{variant-total} + \text{development} - \text{machine savings}}{\text{baseline total}}$$

To estimate the return on investment when many of the parameters for the computation are estimates themselves, we used a Monte Carlo simulation parameterized by our ranges of values for quantities such as BPe, VPe, and the machine cost savings. We assumed these values could be drawn from a roughly normal distribution and took 100k samples of each for the MC simulation.

H. Threats to Validity

Developer behavior and resource usage predictably varies throughout the year, with large observed dips during holiday periods. Our comparison accounts for this effect by excluding holidays. Additionally, for our most business critical question (machine cost savings), we were fortunate to be able to conduct a paired analysis by executing a combination of both cycle types after launch. This provided us a robust data set of FC cycles before and after launch and TC cycles after launch, which helps us account for any time based factor. Second, our return on investment computation relies on estimation of effort by experts and may not be perfectly reflective of actual effort. While we attempt to account for this effect, we acknowledge the limitation of both our attempt, and the Monte Carlo simulation used to provide the confidence interval. Further, factors outside the scope of this paper, such as any differences in actual machine cost or other unaccounted costs (either human or machine) could reduce (or strengthen) the actual return.

Finally, this is a case study of one organization conducting a single project to scale their environment to another machine platform. Our results may not generalize. We observe in our own data that the existence of platform specific test behavior is highly dependent on the context of the specific project. We expect other organizations may have a different mixture of code, and if most code is sensitive to the runtime platform then the savings realized here will be unrealizable for that organization.

VI. Results

A. RQ1: Compute Resource Consumption

Figure 3

Figure 3 shows a breakdown of machine costs for running projects under Target Comprehensive and Fully Comprehensive testing cycles. As seen in the figure, there is a high variance in project costs. The plots indicate that x86 Test costs were generally lower under Target Comprehensive (as expected), while Arm Test costs were generally higher (as expected). Table 1 summarizes the results of our statistical analysis. We observed total savings of ~25%; statistical analysis confirmed the trends evident in the plots, with the most savings derived from avoiding x86 Testing. However, we also saw a substantial 14.8% decrease in build costs attributable to a decrease in double compilation under TC testing.

Table 1: Build and test cost savings of Target Comprehensive scheduling.

Category	Typical Project Savings (Geometric Mean)	Global System Savings (Expected Total Savings)
Total	20.6% (20.3% - 21.0%)	24.9% (19.6% - 29.6%)
x86 Build	16.0% (15.6% - 16.3%)	14.9% (7.7% - 21.4%)
x86 Test	56.9% (56.4% - 57.5%)	66.7% (58.0% - 72.8%)
Arm Test	-25.5% (-26.8% - -24.3%)	-18.4% (-28.7% - -9.3%)

We validated the per-project savings using a paired t-test on the per-project mean of log-transformed costs. The result was highly significant ($t(44242) = -101.3, p < 0.001$), and the derived confidence interval aligned closely with the non-parametric Bootstrap estimates reported in Table 1. This convergence reinforces the statistical validity of our findings. Diagnostic checks (Q-Q plots) confirmed that the data is left-skewed (skewness: -1.46) and leptokurtic (kurtosis: 19.8), exhibiting heavy tails. Despite these deviations from normality, the large sample size supports the use of the parametric t-test, while the Bootstrap analysis provides a robust, assumption-free confirmation.

Target Comprehensive testing has an expected system wide savings of ~25% versus Fully Comprehensive.

B. RQ1: Comparison of Project Status Latencies

Table 2 presents the project status latencies and 95% CI for FC and TC, along with the percentage difference between the two periods. Significant architectural changes and varying project definitions occurred between the FC and TC periods. However, our analysis deliberately focuses on the holistic, end-to-end latency to capture the net impact on system performance, regardless of these confounding factors.

Table 2: Project Status Latency in minutes (p50 and p90).

	p50 [95% CI]	p90 [95% CI]
FC	88.50 [88.43 - 88.58] min	200.78 [200.43 - 201.12] min
TC	75.08 [75.03 - 75.13] min	176.58 [176.25 - 176.90] min
% Diff	-15.16%	-12.05%

Target Comprehensive testing has a median latency savings of 15% and p90 savings of 12%.

C. RQ2: Breakage Detection Latency

Figure 4

Figure 4 summarizes the results of our analysis. We compared the latency of detecting newly-introduced breakages (from submission of the culprit change to the first failure being detected) before and after launch of Target Comprehensive. While the median breakage detection latency was unchanged, the mean and 90th percentile showed significant changes. To understand why, we plotted a breakdown of the number of breakages occurring in ranges of latencies (ex. 17 breakages in 90 minutes - 105 minutes) before and after. We then used Bootstrap re-sampling to estimate the change in the number of breakages likely to occur in each bucket. As visualized in Figure 4, the results indicate that we can expect approximately 4 additional breakages per day that take more than 12 hours to detect. This is a regression and a cost of Target Comprehensive testing as implemented today. We look forward to further optimizations to address this in the future.

We expect 4 additional breakages per day that take more than 12 hours to detect.

D. RQ2: Late-Stage Defect Detection Rates

Figure 5

Figure 5 illustrates the distribution of release-time testing failures for projects with Arm enabled versus disabled, distinguishing rates before and after the rollout of Target Comprehensive testing. In general, the Urfin failure rate is approximately 1/10th that of Rapid Native. Notably, Arm-disabled Rapid Native projects (which were unaffected by the rollout) also saw a regression during this period, suggesting a potential global regression in failure rates unrelated to the launch.

Tables 3 and 4 summarize the results of our statistical analysis. We highlight two key findings. First, the paired bootstrap analysis for Rapid Native indicates a worst-case regression of 0.27%, but the confidence interval also spans zero, allowing for a possible 0.1% improvement. However, the Wilcoxon Signed-Rank Test reports a statistically significant difference in rank order ($p = 0.003$). Given this conflict between magnitude (CI includes 0) and rank drift (Significant P-value), we classify this as a potential regression.

Second, both analytical methods confirm an unambiguous improvement in the failure rate for Urfin Late-Stage testing. Prior to this rollout, Urfin projects executed Arm tests exclusively in the Late-stage phase (a MVP optimization while we waited for Target Comprehensive implementation to complete). The introduction of Target Comprehensive testing effectively shifted left this coverage. TC can identify Arm and x86 failures earlier in the TAP Postsubmit CI phase. This operational change explains the significant decrease in Late-Stage failures ($p < 0.001$). This result justifies the strategy of probabilistic early testing (TC) over exclusive late-stage verification (MVP optimization).

Table 3: Paired Analysis of Change in Test Failure Rate (Arithmetic Mean).

	δ Failure Rate	Wilcoxon Pval	Verdict
Rapid Native	0.09% (-0.10, 0.27)	p=0.003	-
Urfin - CC	0.01% (-0.00, 0.03)	p<0.001	-
Urfin - LS	-0.08% (-0.11, -0.06)	p<0.001	+++

Verdicts: (+++) Improvement, (-) Potential Regression (CI spans 0).

Table 4: Analysis of Change in Typical Test Failure Rate (Geometric Mean).

	δ Failure Rate	RLM Pval	Verdict
Rapid Native	0.008% (0.004, 0.012)	p<0.001	--
Urfin - CC	0.001% (0.000, 0.003)	p=0.159	~-
Urfin - LS	-0.003% (-0.003, -0.002)	p<0.001	++

Verdicts: (++) Small Improvement, (--) Small Regression, (~-) Conflict between Robust Linear Model (RLM) and Bootstrap, Prefer Bootstrap Prediction of Small Regression.

Rapid Native projects show a potential 0.1% regression in failure rates. Urfin projects show a clear 0.08% improvement (decrease) in late-stage failures.

E. RQ3: Engineering Savings

As shown in Listings 1 and 2, the amount of code that must be added to enable a project to take advantage of Arm is much less with variants. Typically, projects need to change only a single line. We estimate that for the typical Blueprint, variants require only 1/10th of the lines of code to maintain compared to the legacy system. Not all Blueprints needed humans to migrate them.

Many Blueprints could be handled by tooling (either traditional tooling or LLMs); however, there was a large number of Blueprints that did need human effort. We restrict our ROI computation below to those requiring human effort.

The new configuration syntax significantly reduces the effort to change a complicated Blueprint that cannot be automatically changed. We quantified this as percent change from legacy configuration to the modern configuration. With the modern syntax, 55% fewer Blueprints needed to be changed by hand.

Based on the Monte Carlo simulation, we believe our return on investment is approximately 40% and between (10%, 56%) when all these different costs are accounted for. We also estimated the savings in just migration cost. The MC simulation estimates a migration savings of 95%, with an interval of (91%, 98%). As more projects adopt Arm over the coming year, our realized return on investment for this capability will continue to grow, both in terms of saved migration costs and increased savings of machine costs as more projects adopt Target Comprehensive testing.

Our estimated ROI for this project is 40%.

In this paper, we presented an empirical case study examining the cost and benefit of the Target Comprehensive (TC) testing strategy. TC runs each test on either Arm or x86, but not both, for high frequency CI executions. This can be viewed as a trivial case of Combinatorial Interaction Testing (CIT) [20, 21] where we only have 1 parameter. CIT has long been a standard tool for testing the t-way interactions between various options in software testing. The most direct analogy for this paper comes from the world of mobile software testing. With the proliferation of devices, operating system variants, and manufacturer customizations, mobile application developers have long had to test their software across many devices. This has led to the proliferation of commercial services to facilitate such testing (ex. Firebase Test Lab, AWS Device Farm, Sauce Labs, etc...).

Vilkomir explicitly studied the number of (random) devices one should test on to identify all bugs [22, 23]. They found that 13 devices achieved 100% coverage in their study and 5 devices provided 80%+ confidence that all bugs had been identified. They also examined if there were smarter ways to select the devices such that fewer devices would be required to achieve the same degree of confidence. In 2018, Kowalczyk et al. [24] also studied interactions between different properties of devices and test failures. Kowalczyk also found that the applications studied showed a variance in both failure rate and program coverage that depended on the configuration.

As reported by Lin et al. in 2023 [25], the difficulties in managing the extensive (and expensive) on device testing strategy led ByteDance to create a "virtual device farm" [25]. Here, we find a direct analogy to Target Comprehensive testing. The Virtual Device Farm mimicked the real devices with emulators, though the fidelity was imperfect. To accommodate this, they would run the majority of their testing effort on the virtual farm. However, final validation tests ran on the physical farm: "Before the app's public releases, we conduct testing on physical devices to uncover the remaining bugs, which are mostly related to vendor-specific system services or drivers (§5.2). Since our continuous testing has captured most app-level bugs (which account for 93% of all bugs), tests on physical devices are much less frequently interrupted by test failures." [25].

In addition to the work in mobile testing, there is a large body of work assessing the feasibility of porting x86 code bases to Arm [1, 26]. We do not attempt to assess Google's overall migration effort here (see Christopher et al. paper [1]), but note that there are significant architectural differences between x86 and Arm that lead to correctness violations. These violations require precise changes to rectify, as they often involve synchronization between threads. We note that before a program is (initially) eligible to run on the Arm architecture it goes through additional qualification beyond its test suite [1]. One example of research on automatic migration is Beck et al.'s work [26]. They use static analysis to automatically translate synchronization patterns to safe and performant Arm equivalents during compilation.

Finally, our empirical assessment on changes to Breakage Detection Latency relied on our past work on TAP [4, 5, 7]. In 2023, we introduced a highly accurate culprit finder, and more importantly, a methodology for automatically assessing its accuracy [4]. Culprit finding simply identifies which changes introduced a test breakage into the repository. This is made challenging by flaky and non-hermetic tests. We conduct extensive reruns (as documented in [4]) to automatically verify the output of the culprit finder. In Kondareddy et al.'s 2025 paper, we hand re-verified a sample of the output and determined our verification strategy is 99.2 ± 3.3% accurate. This verified dataset provides our definitive source of truth (± 3%) on breakages and enables us to accurately report Breakage Detection Latency while accounting for flakiness to the best of our ability.

VIII. Conclusion

Automated configuration, variants, and Target Comprehensive (TC) testing cycles were introduced to enable large scale adoption of the Arm architecture at Google. A naive approach of double testing would have been unsustainable, doubling compute costs and potentially increasing friction from additional flaky failures. Further, requiring all projects to modify their individual configuration files was infeasible.

Our solution enabled the adoption of Arm throughout the organization by controlling machine costs and minimizing human toil. We saved 90%+ human migration effort. We saved ~25% of the machine costs. Even though we don't execute all the tests all the time, the delays in breakage detection were marginal (4 per day) over the total number of breakages across the repository. Finally, our final safety net (Late Stage Testing) only detects a failure in fewer than 0.1% of production promotion attempts, a negligible amount and dwarfed by other reasons a production promotion may be aborted. Collectively, this work demonstrates the feasibility of large scale adoption of heterogeneous computing platforms without doubling the cost of testing.

Contributions and Acknowledgments

Primary Author: Tim A. D. Henderson, Secondary Author: Sushmita Azad

Additional authors, alphabetic:

Data Analysis for Empirical Evaluation: Chandrakanth Chittappa, Ali Esmaeeli, Laura Macaddino, Sam Manfreda, David Margolin, Dharma Naidu, Sabuj Pattanayek, Sachin Sable, Ruslan Sakevych
Drafting and Editing: Dushyant Acharya, Adrian Berding, Kevin Crossnan, Wolff Dobson, Avi Kondareddy, Abhayendra Singh

Additionally, we would also like to thank the following people for their technical contributions towards the success of this effort. In alphabetic order: Michael Adelemoni, David Alfonso, Sterling Augustine, Mark Balch, Leeann Bent, Paul Bethe, Lexi Bromfield, Eric Burnett, Saahithi Chillara, Brian Chiu, Dan Cho, Eric Christopher, Eduardo Colaco, Nicolo Davis, Rumeet Dhindsa, Paul Diebold, Bobby Dorward, Pat Doyle, Ákos Frohner, Charlie Fu, Russell Gottfried, Mark Halberstadt, Minh Hoang, Jung Woo Hong, JaJan Hsu, Jin Huang, Xin Huang, Kevin Jiang, Pranav Kant, Danial Klimkin, Gregory Kwok, Leon Lee, Chris Lewis, Drew Lewis, Jordan Lin, Kevin Liston, Sean Luchen, David Michaelson, Dominic Mitchell, Morgan Mitchell, Maksym Motornyy, Katherine Nadell, Danny Nguyen, Denis Nikitin, John Penix, Evelyn Price, Daniel Rall, Emma Rapati, Ashley Richter, Alberto Rojas, Daniel Romero Sanchez, Carly Schaeffer, Aaron Schooley, Alexander Schrepfer, Will Schwarz, Markus Schwenk, Manish Shah, Santanu Sinha, Peter Spragins, Arvind Sundararajan, Xenia Tay, Greg Tener, Marjorie Thibodaux, Dima Tsumarau, Shanid Seerae Valappil, Sri Raga Velagapudi, Andrew Vorwald, Shu-Chun Weng, Michael Wexler, DJ Whang, Jeff Xia, Anthony Xiang, and Jun Xiang Yak.

References

[1] E. Christopher, K. Crossan, W. Dobson, C. Kennelly, D. Lewis, K. Lin, M. Maas, P. Ranganathan, E. Rapati, and B. Yang, "Instruction Set Migration at Warehouse Scale," Oct. 2025. [Online]. Available: http://arxiv.org/abs/2510.14928

[2] A. Memon, Zebao Gao, Bao Nguyen, S. Dhanda, E. Nickell, R. Siemborski, and J. Micco, "Taming Google-scale continuous testing," in 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). Piscataway, NJ, USA: IEEE, May 2017, pp. 233-242.

[3] M. Fowler, "Continuous Integration," 2006. [Online]. Available: https://martinfowler.com/articles/continuousIntegration.html

[4] T. A. D. Henderson, B. Dorward, Eric Nickell, C. Johnston, and Avi Kondareddy. "Flake Aware Culprit Finding," in 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, Apr. 2023.

[5] T. A. D. Henderson, Avi Kondareddy, Sushmita Azad, and Eric Nickell. "SafeRevert: When Can Breaking Changes be Automatically Reverted?" in 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). Toronto, ON, Canada: IEEE, May 2024, pp. 395-406.

[6] C. Leong, Abhayendra Singh, Mike Papadakis, Yves Le Traon, and J. Micco, "Assessing Transition-Based Test Selection Algorithms at Google," in 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, May 2019, pp. 101-110.

[7] Avi Kondareddy, Sushmita Azad, Abhayendra Singh, and T. A. D. Henderson, "Speculative Testing at Google with Transition Prediction," in 2025 IEEE Conference on Software Testing, Verification and Validation (ICST). Napoli, Italy: IEEE, Mar. 2025, pp. 510-521.

[8] R. Potvin and J. Levenberg, "Why google stores billions of lines of code in a single repository," Communications of the ACM, vol. 59, no. 7, pp. 78-87, 2016.

[9] T. Winters, T. Manshreck, and H. Wright, Software Engineering at Google: Lessons Learned from Programming over Time. Beijing: O'Reilly, 2020.

[10] K. Wang, G. Tener, V. Gullapalli, X. Huang, Ahmed Gad, and Daniel Rall. "Scalable build service system with smart scheduling service," in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. Virtual Event USA: ACM, Jul. 2020, pp. 452-462.

[11] K. Wang, Daniel Rall, Greg Tener, V. Gullapalli, X. Huang, and Ahmed Gad. "Smart Build Targets Batching Service at Google," in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). Madrid, ES: IEEE, May 2021, pp. 160-169.

[12] P. Gupta, M. Ivey, and J. Penix, "Testing at the speed and scale of Google," 2011. [Online]. Available: https://google-engtools.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html

[13] M. Bland, "The Chris/Jay Continuous Build," Jun. 2012. [Online]. Available: https://mike-bland.com/2012/06/21/chris-jay-continuous-build.html

[14] J. Micco, "Tools for Continuous Integration at Google Scale," Google NYC, Jun. 2012. [Online]. Available: https://youtu.be/KH2_sB1A6lA

[15] J. Micco, "Continuous Integration at Google Scale," EclipseCon 2013, Mar. 2013. [Online]. Available: https://web.archive.org/web/20140705215747/https://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/2013-03-24%20Continuous%20Integration%20at%20Google%20Scale.pdf

[16] S. Elbaum, Gregg Rothermel, and J. Penix, "Techniques for improving regression testing in continuous integration development environments," in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering - FSE 2014. New York, New York, USA: ACM Press, 2014, pp. 235-245.

[17] P. Spragins, "Efficacy Presubmit," Sep. 2018. [Online]. Available: https://testing.googleblog.com/2018/09/efficacy-presubmit.html

[18] Minh Hoang and Adrian Berding, "Presubmit Rescue: Automatically Ignoring FlakyTest Executions," in Proceedings of the 1st International Workshop on Flaky Tests. Lisbon Portugal: ACM, Apr. 2024, pp. 1-2.

[19] T. Hesterberg, David S. Moore, S. Monaghan, A. Clipson, and R. Epstein, "Bootstrap Methods and Permutation Tests," in Introduction to the Practice of Statistics, 7th ed. New York: W.H. Freeman, 2012.

[20] D. R. Kuhn, R. N. Kacker, and Y. Lei, Introduction to Combinatorial Testing, ser. Chapman & Hall / CRC Innovations in Software Engineering and Software Development. Boca Raton, FL: CRC Press, 2013.

[21] D. R. Kuhn, Rick Bryce, F. Duan, L. S. Ghandehari, Y. Lei, and R. N. Kacker, "Chapter one - combinatorial testing: Theory and practice," ser. Advances in Computers, A. Memon, Ed. Elsevier, 2015, vol. 99, pp. 1-66.

[22] S. Vilkomir, K. Marszalkowski, C. Perry, and S. Mahendrakar, "Effectiveness of Multi-device Testing Mobile Applications," in 2015 2nd ACM International Conference on Mobile Software Engineering and Systems. Florence, Italy: IEEE, May 2015, pp. 44-47.

[23] S. Vilkomir, "Multi-device coverage testing of mobile applications," Software Quality Journal, vol. 26, no. 2, pp. 197-215, Jun. 2018.

[24] E. Kowalczyk, Myra B. Cohen, and Atif M. Memon, "Configurations in Android testing: They matter," in Proceedings of the 1st International Workshop on Advances in Mobile App Analysis. Montpellier France: ACM, Sep. 2018, pp. 1-6.

[25] Jordan Lin, J. Qiu, H. Wang, Z. Li, L. Gong, D. Gao, Y. Liu, F. Qian, Z. Zhang, P. Yang, and T. Xu, "Virtual Device Farms for Mobile App Testing at Scale: A Pursuit for Fidelity, Efficiency, and Accessibility," in Proceedings of the 29th Annual International Conference on Mobile Computing and Networking. Madrid Spain: ACM, Oct. 2023, pp. 1-17.

[26] Manuel Beck, K. Bhat, L. Stričević, G. Chen, D. Behrens, M. Fu, V. Vafeiadis, H. Chen, and H. Härtig, "AtoMig: Automatically Migrating Millions Lines of Code from TSO to WMM," in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. Vancouver BC Canada: ACM, Jan. 2023, pp. 61-73.

Speculative Testing at Google with Transition Prediction

2025-04-02T00:00:00-04:00

Avi Kondareddy, Sushmita Azad, Abhayendra Singh, and Tim A. D. Henderson. Speculative Testing at Google with Transition Prediction. ICST Industry Track 2025.
DOI. PDF. WEB.

Note

This is a conversion from a latex paper I wrote. If you want all formatting correct you should read the pdf version.

Abstract

Google's approach to testing includes both testing prior to code submission (for fast validation) and after code submission (for comprehensive validation). However, Google's ever growing testing demand has lead to increased continuous integration cycle latency and machine costs. When the post code submission continuous integration cycles get longer, it delays detecting breakages in the main repository which increases developer friction and lowers productivity. To mitigate this without increasing resource demand, Google is implementing Postsubmit Speculative Cycles in their Test Automation Platform (TAP). Speculative Cycles prioritize finding novel breakages faster. In this paper we present our new test scheduling architecture and the machine learning system (Transition Prediction) driving it. Both the ML system and the end-to-end test scheduling system are empirically evaluated on 3-months of our production data (120 billion test$\times$cycle pairs, 7.7 million breaking targets, with $\sim$20 thousand unique breakages). Using Speculative Cycles we observed a median (p50) reduction of approximately 65\% (from 107 to 37 minutes) in the time taken to detect novel breaking targets.

Introduction

Continuous integration (CI) coordinates the development activity of large numbers of developers (Fowler 2006). When two developers are working in the same portion of the code base, continuous integration ensures that conflicting changes combine into a conflict-free version before reaching the end user. In general, adopters of CI execute builds and tests to ensure that the final release artifact (server binary, mobile application, etc...) passes all relevant tests. These build and tests are executed frequently to reduce the amount of time conflicts are present in the code base. Continuous integration is both a system and a practice of automatically merging changes into a source of truth for an organization's source code and related artifacts.

For small software repositories and organizations, the implementation of continuous integration is well supported by off-the-shelf software such as Github Actions, CircleCI, Jenkins, and numerous other tools. However, as the repository and organization scales up challenges emerge.

As an organization adds projects and engineers, there are two distinct paths that emerge: the many-small-repository path and the single-mono-repository path. Both paths have distinct challenges and advantages. Neither path will enable organizations to use vanilla off-the-shelf solutions for their development environment. This paper is focused on a particular challenge faced in a mono-repository environment at Google, but similar problems arise in organizations with many small repositories that are strongly coupled together.

Google, an early adopter of mono-repositories and continuous integration (Potvin and Levenberg 2016), faces an enormous ever expanding code base that is centrally tested by the Test Automation Platform (TAP) (Micco 2012). As the code base and company grows, so does the demand for compute resources for continuous testing. Left unchecked, we have observed a double digit percentage organic demand growth rate year-over-year (compounding). Such a growth rate is untenable even for a large company like Google. To prevent this unchecked growth in resource demand, TAP has long had features that reduce demand by skipping some tests (before submission) and batching many versions together (after submission) to amortize the cost (Gupta, Ivey, and Penix 2011; Bland 2012; Micco 2012, 2013; Memon et al. 2017; Leong et al. 2019; K. Wang et al. 2020, 2021; Henderson et al. 2023, 2024; Hoang and Berding 2024).

In this paper, we are primarily concerned with testing that occurs after the code has been submitted. We examine how to improve the developer experience by finding novel build/test breakages faster while continuing to control our resource footprint growth rate.

The Google Development Environment

Google's development environment uses a centralized repository (Potvin and Levenberg 2016) where code changes are managed with Blaze/Bazel¹ and automatically tested, primarily by TAP. Developers write code, build and test locally, then submit it for review via Critique (Sadowski et al. 2018), triggering Presubmit checks. After review and approval, changes undergo further checks before merging. Due to high submission rates, Presubmit testing is limited to avoid excessive resource consumption and delays.

TAP Postsubmit

When a developer's change gets submitted it may still have a bug that breaks an existing test or causes a compile breakage in another part of the repository. To find these bugs that have slipped through the pre-submission testing and validation process, TAP also has a "post-submission" mode (TAP Postsubmit).

In Postsubmit, TAP periodically (subject to compute resource availability in the Build system) schedules all tests that have been affected (based on build dependencies) since the last run. We refer this execution cycle as Comprehensive Testing Cycle. Previously, this cycle has also been referred as Milestones (Memon et al. 2017). There are two primary objectives for the Comprehensive Cycle:

Uncover all test failures at the version comprehensive testing was conducted at.
Provide project health signals for downstream services to trigger more expensive testing and/or start production rollout by triggering release automation

As of today's writing, it takes ∼1-2 hours for a test broken by a developer's change to start failing in TAP Postsubmit. Once the failure is detected automatic culprit finders (Henderson et al. 2023) and rollback system (Henderson et al. 2024) will spring into action to help keep the repository healthy and our developers productive. Automatically rolling back the change (after the first breakage was detected) might take as little as 30 minutes or as long as several hours depending on a number of factors. Thus, it could be 4 hours or more before our developer learns they have broken a test and either been notified or automatically had their change rolled back.

Cost of Breakages

The longer it takes to identify and fix a code breakage, the more challenging and expensive the remediation process becomes. This delay can lead to a loss of context for the original developer, potentially requiring teammates to resolve the issue. Additionally, a bad change can affect other developers, especially if it occurs in widely used parts of the code base. When interrupted by such a failure programmers need to distinguish between the change they are making and existing fault in the repository. Ultimately, prolonged breakages can disrupt release automation and even delay production releases.

Our Contributions and Findings

In this paper we are looking to increase developer productivity by reducing their friction with respect to broken code in the main code base. Specifically we are solving the following problem.

How can CI minimize the time between the submission of a faulty change and its identification within a continuous integration (CI) system to accelerate the detection and mitigation of bugs?

To solve the problem, we propose a new scheduling mode to our CI system, Speculative Testing Cycles, that opportunistically runs a small subset of tests to uncover novel failures and reduce the mean time to detection (MTTD) for new failures. Speculative Cycle prioritizes finding novel breakages by identifying tests that are very likely to fail. The new scheduling mode is driven by a Transition Prediction model that predicts when tests are likely to (newly) fail. It utilizes shallow machine-learning to run a smaller batch of tests predicted to be more likely to be broken at higher frequency in order to find breakages faster. We discuss both the design and constraints on the production Speculative Cycle system and the shallow machine-learning model (Transition Prediction) that powers this predictive test selection.

While the production system is highly tied to Google's unique development environment and organizational constraints (i.e.: large monorepos with centralized CI infrastructure), our ML model presents a very coarse-grain approach to prediction for test selection using test and code-under-test metadata that has equivalents in most development environments and is language/framework agnostic.

Our contributions in this paper are therefore:

Speculative Cycles: A system for frequent / cost-aware batch testing.
Transition Prediction (TRANSPRED): Predicting target status transitions (Pass to Fail) for arbitrary build & test targets. With a budget of 25% of the total targets, Transition Prediction achieves a 85% recall rate in detecting breakages.
A very large scale study (utilizing 3-months of production data: 120 billion test×cycle pairs, 7.7 million breaking targets, and ∼20,000 unique breakages) on the efficacy of Speculative Testing using Transition Prediction evaluating its performance against randomized testing.
A comparative assessment of dataset and training configurations for Transition Prediction, determining that the selection of features and the length of the training window substantially influence the system's performance.

Background

Developer Services

Bazel

Google uses Bazel as its build system monorepo wide. Bazel allows specifying build/test "targets" that depend on other targets. Each target is comprised of a set of "actions" to be executed. Therefore, the execution of a target corresponds to a Directed Acyclic Graph (DAG) of actions. Given Bazel makes dependency management declarative and all dependencies are built at the same version of the codebase, builds are ostensibly held to be hermetic - builds at the same version of the codebase should produce the same output for each action. This opens up the possibility for massive compute savings through caching of intermediate actions across multiple builds.

Forge

Build/Test actions at Google run under a centralized compute cluster called Forge which attempts to cache these intermediate actions to avoid recomputation at the same version of the codebase. For Postsubmit testing, we make use of this caching by batching builds/tests at single versions every hour or so.

Comprehensive Testing Cycles

In the standard case, TAP Postsubmit performs each Comprehensive Testing Cycle once the previous cycle's active consumption of resources drops below a threshold of their allocated build system resources. This cycle consists of picking a recent change(snapshot of the repository), at which we batch and run a "comprehensive" set of tests. Comprehensive cycles run all tests that are "affected" since the previous cycle. "Affected" is our term for dependence as implied by the build system's package graph, allowing us to perform a degree of static test selection at a coarse granularity.

Ancillary Systems For Build Gardening

The development life cycle explained above relies on Postsubmit testing to make up for the lack of comprehensive testing prior to submission. This introduces the concept of Postsubmit breakages which are caught sometime after submission. The process of detecting the breakage, identifying the cause of the breakage (culprit finding), and fixing the breakage (sometimes via a rollback) is called "Build Gardening" at Google.

Culprit Finding

We have previously outlined Google's culprit finding approach in (Henderson et al. 2023, 2024). To formulate the problem, we are given a test/build "target" and must determine at which change C over a range of sequential changes [A, B] did the target start breaking. The change C is referred as a Culprit Change or simply Culprit. Given the presence of build/test non-determinism ("flakes"), we want to find the change at which the test was "truly" failing but "truly" passing at the previous change. Our previous work explores how we do this in a time and run-efficient manner using a historical flake rate aware Bayesian model to sample runs.

Culprit Verifier

Unfortunately, persistent flakiness and non-determinism can still slip by the culprit finders. The pathological case is that of build system non-determinism. Google's build system Bazel assumes that builds are hermetic and should produce the same output at the same version. This assumption means that unlike tests, failed actions are cached by the build system so immediate reruns will continue to assert what could potentially be a flaky build failure. The culprit verifier does extensive reruns on a subset of culprit conclusions produced by the culprit finders. We sample from two different sampling frames: (1) a simple random sample of all conclusions produced and (2) the first conclusion produced for each uniquely blamed culprit. For details on the design of the verification system, see the Flake Aware Culprit Finding paper (Henderson et al. 2023) and the SafeRevert paper (Henderson et al. 2024).

Although the verifier is fallible, the dataset it produces is still the most accurate representation of true breakages at Google. We know it fails because we have seen specific examples of its failures. But, from a measurement perspective, bounding its accuracy is challenging because its failure rate is low enough that it exceeds our ability to accurately measure. We have recently conducted manual verification while launching a new version of our Auto Rollback system on all culprits rolled back during the "dogfood" phase (an opt-in beta). Out of 250+ hand verified culprits: 5 were incorrect culprits from the culprit finder, 2 incorrect culprits were incorrectly verified correct by the verifier. This gives us an estimated culprit finding accuracy of 98% (matching verifier produced accuracy dashboard) and a verifier accuracy of 99.2%. Given the low sample size the confidence interval on that measurement is 99.2 ± 3.3% which is too wide. To get sufficient power to accurately estimate the verifier accuracy, we would need to hand verify at least 5000 culprits.

Thus, while we are confident in the overall accuracy measurements provided by the verifier, we acknowledge that "ground truth" is difficult to obtain. We utilize the dataset produced by the verifier to label the culprits to train our prediction model (detailed below). This approach vastly reduces the impact of flakiness on our model training data.

Autorollback

At Google, there is a developer guideline to prefer rollbacks to fix forwards. Given this context, it is appropriate to automatically roll back a change if someone is confident that the change broke the build. Unfortunately, given the accuracy issue of the culprit finder described above, we can't immediately rollback upon a culprit finding completion for one target. In the SafeRevert paper (Henderson et al. 2024), we discussed how we attempt to determine the amount of evidence needed to proceed with a rollback. In practice, we currently restrict rollbacks to changes that break at least 10 targets.

The Evolution of TAP

Prior to development of TAP, Google had a more traditional continuous integration system. Primarily, the decision to run continuous integration was left up to individual teams and the system that existed was federated: teams brought their own capacity (machines, possibly under their desks) and had to configure and run the CI system(s). TAP, upon launch in 2009, was a massive improvement from setup and continuous maintenance perspective alone. Unlike the previous systems, TAP ran all builds centrally and only required a small amount of configuration: it simply required users to specify which paths they would like to test and where the (failure) notifications should go.

The central builds and low configuration overhead made TAP immediately successful. Within a few years it had completely displaced the prior system. But, success came at a price: it was plagued by build capacity limitations and scalability challenges stemming from design decisions of running tests on each change in the initial implementation. By 2012 (following a full launch in 2010) Google was running low on machine capacity to run TAP. The testing model had naively assumed that the build dependence based test selection would be enough to control demand; It was not.

TAP both pre- and post-submit were thus rewritten (multiple times). TAP Presubmit now has several different modes, adapts to resource availability, uses machine learning driven test selection, and has "advisors" which can ignore or rerun tests on failure (Hoang and Berding 2024). Unfortunately for you (the reader) discussing these innovations is out of scope for this manuscript.

Today, TAP Postsubmit no longer runs on every change. It waits for there to be capacity in the Build System (K. Wang et al. 2020, 2021) and then enqueues all tests that have been affected since their last definitive status. At Google's scale and growth rate, we have had to continuously innovate and rewrite core parts of TAP just to maintain this model. However, we have long known that just using build-graph dependency based test selection in Postsubmit was untenable in the long term (Memon et al. 2017; Leong et al. 2019). It is only now that we have both the high-accuracy culprit finding infrastructure (Henderson et al. 2023, 2024) and the organizational will to fundamentally change (again) core assumptions for how TAP works.

The Current Dilemma

The cost of TAP Postsubmit's comprehensive testing continues to rise due to ever-increasing compute demand, increasing fleet machine diversity, and evolving development practices. This means that with no systemic changes in our approach towards Postsubmit testing:

Teams start needing to wait longer and longer for signal on their projects' health (or "greenness") from TAP blocking their releases which has significant monetary impact
Developers are now alerted hours after they submit of a breaking change, creating developer toil

Speculative Testing Cycles

In order to protect release quality and improve developer productivity, TAP is introducing Speculative Testing Cycles - a process by which we non-deterministically select a smaller subset of targets that we deem more likely to fail, and run this smaller set more frequently than traditional comprehensive cycles. For traditional comprehensive test cycles, we attempt to batch as many builds/tests together at the same change as possible to get the caching benefits from Forge. We do the same here for Speculative Cycles, picking a single change approximately every 20 minutes, Forge build resources allowing. As with Comprehensive Cycles, we first perform static build system level dependency analysis to find all targets which could have a status change.

Determining Breakage Likelihood

We now have a set of statically-filtered targets {T} and a sequence of commits {C} consisting of all commits since the last Speculative Cycle. Our problem is then to determine for a given target t ∈ T: what is the likelihood it was broken by a change c ∈ C affecting it? For compute tractability, given the frequency of change submissions and number of targets in our codebase, we simplify by choosing to aggregate across C and ask: "Was target t broken by any change in C?" We identify this problem formulation and approaches to answer it as Transition Prediction, and discuss it in detail in the next section.

Making Scheduling Decisions

Our Transition Prediction model gives us "scores" from 0 to 1 indicating the propensity of this target to be newly broken. Given that we rely on machine learning models for these predictions, we must be careful to note that these scores do not meaningfully represent probabilities. They are the product of an algorithm attempting to minimize a loss function relative to its training dataset. While they can be colloquially understood to imply likelihood and allow relative comparison, moving the threshold at which scores map onto decisions to (not) schedule will produce non-linear changes to the actual observed probability of finding a breakage ("recall") or scheduling false positives ("false positive rate").

Given this fact and our desire to have stable execution cost/latency under our scheduling scheme, we choose to primarily rank targets and schedule the top-k highest ranking targets under some top K parameter. For the purpose of this paper, this is our main parameter for tuning the Speculative Cycle system while the remaining knobs exist in feature selection and training configuration for the Transition Prediction ML model.

Goal of Finding Test Breakages

As we've discussed, Postsubmit testing feeds into both Release pipelines and the breakage triage/fixing workflow (what we call "Build Gardening"). Release implications of Speculative Cycle(s) depend on many several other projects at play which are outside the scope of this paper. The direct outcome is that newly detected build/test failures will now propagate to Build Gardening consumers faster, namely:

Developers directly through emails sent upon detection of a failure
Automated Culprit Finders that attempt to identify the specific culprit change
Auto-Rollback reverts changes once it acquires enough evidence from culprit finders

Figure 1

A developer or the automated culprit finders may only need a single target breakage (We will denote this criterion as AtLeast(1)) to consider investigating and to identify a breaking change that needs to be reverted. But given the non-determinism issues discussed above, auto-rollback requires a stronger signal - at least N distinct targets were culprit-found to the same change (AtLeast(N)). Both these goals are important to the health of the codebase, and while the former is a simple and more intuitive metric, there is a strong argument to be made for the latter - in that the number of distinct broken targets is a good proxy for the cost of that bad commit. The larger the number of broken targets, the larger the likelihood that it falls under a configured Presubmit testing directory, impacting developer productivity like we discussed above. Targets could belong to multiple release-level project groups, and more targets being broken is a higher cost to overall release health with more automation and production pushes being delayed or blocked. Figure [fig:culprit-prop] presents the total proportion of culprits remaining as we increase N which follows a power law distribution.

Evaluating Performance

Culprit detection is then split between Comprehensive and Speculative Cycles. Under a specific scheduling scheme, Speculative Cycles successfully detect some proportion of culprits before Comprehensive Cycles. This proportion is the recall of the system. This recall in combination with the actual run frequency and execution runtime of the two cycles determines the actual reduction in culprit detection latency, where latency is defined relative to the submission time of the culprit change. Depending on the consumer, latency improvements will have non-linear marginal payoffs. For example, reduction from half a day to an hour may have measurable impacts on development friction caused by failing tests, but reduction from a half hour to 10 minutes may be negligible. Therefore, both metrics can be compared to the cost of a specific scheme in order to justify value / viability of the system.

Transition Prediction

Transition Prediction (or TRANSPRED for short) attempts to predict transition likelihood every 20 minutes for millions of targets with a submission rate of over tens of thousands of changes per day. In order to make this tractable and most importantly cost-beneficial relative to simply running more tests, our predictions must be cheap and low latency. This informs using simple tree-based model techniques like Gradient Boosted Decision Trees and Random Forests. Tree-based models are used across industry for being low cost for inference, avoid over-fitting, and achieve high performance with simple coarse grained features like change and target metadata. We have found consistently throughout our work that Gradient Boosted Tree models perform best for Google's coarse grained test selection problems.

Features

Figure 2

Figure [fig:feature-collection] shows an overview of the structure of the feature vector. In a production system, only features that can be acquired quickly and reliably at scale can be used. The end-to-end latency of deciding which tests to run (including fetching the data, assembling the feature vectors, and running the prediction) needs to be well under 20 minutes such that the current Speculative Cycle finishes before the next one begins. Thus, we are using simple, cheap to compute, coarse grained metadata features from targets and commits. The predictions are done per target so the features for one target are included in each feature vector. However, the predictions are not done for a single commit but across a range of commits. To increase the robustness of the commit based features, we only include commits that affect the target. A commit affects a target if the target is reachable in the build graph from the files modified in the change.

Target level features include static and dynamic characteristics. The static items are unchanging (or very rarely changing) and include target's programming language, whether it is a test or build target, and its Bazel rule type, etc. The dynamic items are based on historical execution data, such as the failure count in Postsubmit and Presubmit over several different time windows. Commit level features include: change metadata like lines of code modified, number of reviewers, number of linked bugs, description, etc. Finally, some features correspond to the relationship between the target and the changes in this specific cycle - specifically its build graph distance to the files modified and whether the target was run at Presubmit time.

Table 1

Table [table:featureDescription] contains details on the features used in the production model ('BASE') and those added for this paper ('AUG'), for augmented).

Labeling

Our data set has a row/vector for each target in each Speculative Cycle. In order to train the model, we need to label each vector with whether or not that target should be run in the given Speculative Cycle. Optimally the targets should only be run when a new failure is introduced into the repository. We use the verified culprits data set described previously (Henderson et al. 2023, 2024) as our source of truth on which commits introduced breakages. Unfortunately, the automated culprit finders de-prioritize comprehensiveness in favor of finding results quickly for active breakages blocking developers. This can leave some target level breakages with no culprit-identifying labels for up to two weeks after the failures was detected.

In addition, some flakiness in tests and builds may be caused by something outside of the code and configuration checked into the repository. For example, a buggy compiler action may non-deterministically miscompile a file causing the linker to fail when it happens. Because the compile action "succeeds" (produces an output and no error code), the output of the compiler (the object file) will be cached. Subsequent builds will re-use the cached output and fail when attempting to link. However, if the cached item expires or the build happens in a different data center with a different cache the linker action may succeed (when the compiler behaved correctly). This type of flakiness is difficult to diagnose real-time with only build re-executions due to the multiple layers of caching. However, build executions separated in time have a better chance of identifying the flaky behavior.

In order to protect against unavailability of data, we perform labeling a full week after the inference examples are acquired. As we've described, this still does not completely shield us from both incorrect positives and lack of coverage but is the best ground truth we have.

Super Extreme Class Imbalance

Postsubmit breakages are exceedingly rare. The only target-vector rows in our dataset that need to be scheduled are targets that are newly broken. This only occurs 0.0001% of the time. In model training, we refer to the class with fewer examples at the minority class while the class with more examples is the majority class. In our problem, the minority class is these newly broken targets just described.

When there are too few examples in the minority class, the effectiveness of the many training algorithms is reduced. The "loss" (e.g. the function model training is optimizing) becomes biased towards the majority class. A classic pathological case is a model that trivially predicts the majority class - that is, predicts the majority class every time and never predicts the minority class. These models will actually minimize the loss function even though they never make useful predictions. Our approach must guard against these biases if we want to usefully predict which tests to run during a Speculative Cycle.

For our system, we downsample the negative class (targets that are not newly failing). This reduces the imbalance between the classes. It also implicitly values the True Positives (new breakages) higher than the True Negatives. Every false negative (missed breakage) leads to developer toil as the detection of the breakage is delayed. Given the super extreme imbalance between the classes, we expect most targets run during a Speculative Cycle are to be False Negatives (and not indicate a new breakage).

Training Configuration

For our model, we use the Yggdrasil Decision Forest library (YDF) (Guillame-Bert et al. 2023). YDF touts superior decision tree sampling and splitting algorithms over prior algorithms (such as XGBoost (chen2016xgboost?)) and provides better integration with Google's machine learning infrastructure.

Conventional training schemes randomly shard their dataset between train, validation, and test splits or perhaps perform cross-validation with a set of parallel splits. For our problem, randomized splits are harmful and lead to us overestimating our performance as we leak time-dependent attributes of the repository across sets. Instead, we create time-ordered splits where the train set consists of the first A days, the validation set the next B days, and the test set the final C days.

Empirical Evaluation

Evaluation

We empirically evaluated Speculative Cycles against a baseline random scheduler on culprit-detection recall and against an optimal algorithm for evaluating culprit detection latency improvements. We evaluated different dataset pruning/selection and training configurations to discover important hyper-parameters for the Transition Prediction model on a standard set of features. We examined three research questions:

Research Questions

What is the performance of Speculative Cycles with Transition Prediction in terms of percentage of novel breakages detected (recall) in comparison to a baseline randomized testing approach?
Speculative cycles compete with full testing cycles for machine resources. Do speculative cycles improve productivity outcomes for Google developers?
What aspects of the model design most impact system performance? Considered aspects include features selected, super-extreme class imbalance corrections, and training environment.

Measuring Performance (Recall)

Speculative Cycles as a system attempts to detect bug introducing changes (culprits) through running and surfacing at least N newly failing targets. Specifically, the goal is to capture the majority of (newly failing) target results that detect novel bugs introduced into the repository with the faster Speculative Cycles instead of the slower Comprehensive Cycles. This naturally lends itself to measurement in the form of recall - the percentage of these events we capture.

We additionally observe that our dataset is imperfect. Not every commit labeled as a "culprit" is indeed a bug introducing change. As discussed above, our datasets contains some inaccuracies particularly when a particular culprit commit only a few targets "blaming" it.² The distribution in the number of targets blaming a culprit follows a power law distribution. The majority of culprits are blamed by fewer than 10 targets. However, the lower the number of blamed targets the more likely it is that the culprit finding was inaccurate and that the commit does not in fact introduce a bug. In our labels, the signal is highest for culprits with many targets blaming them and lowest when there is only a single target. To protect our system from the noise these "small" culprits are filtered out during training.

We use AtLeast(n) to denote the subset of culprits that have at least n targets blaming each culprit. In our evaluation, we provide recall over both (a) AtLeast(1) - the set of all culprits and (b) AtLeast(10) - the set of culprits causing at least 10 target breakages where we successfully detect 10 breakages. AtLeast(10) also ensures that we meet the (traditional) minimum required threshold (i.e. 10 failing targets) to trigger an automatic rollback. Speculative Cycles sees its highest value in terms of mean time-to-fix when automatic rollbacks are correctly triggered (especially for commits that have wider impact radius). Therefore, by assessing Recall using AtLeast(10) we also assess whether or not the Speculative Cycle can find enough evidence to trigger rollbacks.

Measuring Productivity Outcomes (Latency Improvement)

Although recall is a concrete metric, our true goal is to improve productivity by reducing friction caused by breakages that slipped into Postsubmit. Speculative Cycles focuses on early detection of breakages (and their culprit detection and fix) to reduce the friction caused by bad commits in Postsubmit. Its impact is measured by the reduction in "breakage detection latency". Early detection and fix also improves the mean time to fix latency for bad commits. We will evaluate the "breakage detection latency" decrease as a percentage relative to the current latency using empirically observed timings for running both Speculative and Comprehensive over the days present in our test dataset.

High Priority Configuration of Training Scheme

In general, it is considered good practice in machine learning to prioritize dataset quality over attempting to fine tune model level hyper-parameters. We have similarly observed little value in specific training parameters and use YDF's defaults for Gradient Boosted Trees for this work to show the general case performance expected. The more interesting questions that pertain to our dataset configuration are:

Training Split / Window Size

As we use time based splitting to avoid the issue of "time-traveling" across the datasets, our training split configuration determines both the training "window size" and the "delay" between the training and test datasets.

We hypothesized that the training data may have some recency bias. A shorter/ more recent look-back horizon on the training data may contain more useful signal for predicting the likelihood of a target transitioning. A longer window could dilute the usefulness of more recent breakages, while introducing noise from old breakages that have since been fixed and are less likely to reoccur. In order to test this hypothesis, we ran the model five times using the same feature set and configuration, only changing the number of weeks of data used during training. Note that our production pipeline uses a 3:1:1 week split for training:test:validation.

Downsampling and Upweighting

Our extreme class imbalance of 40, 000 : 1 necessitates downsampling our negative class to a ratio that is compute-efficient, but also avoids trivially predicting 0. Upweighting the positive class can be useful to appropriately value positive examples with respect to the loss function of the training algorithm. The ratios of down-sampling listed above reduced our class imbalance ratio to 400 : 1, 200 : 1, and 40 : 1 respectively. For each case, we then upweighted the positive samples of target breaking commits by the new class imbalance ratio to see what effect, if any, it had on improving model performance.

Dataset

Our evaluation dataset consists of roughly 3 months of Transition Prediction data split across Train, Validation, and Test sets. In total, the dataset consisted of 120 Billion target-cycle pairs, 7.7 million of which were breaking targets. This corresponds to ∼20 thousand unique culprits. As mentioned above, we ensure that our splits are entirely sequential and disjoint to avoid cross contamination. The exact split and training configuration is varied while answering RQ3. We then fix our most performant model for evaluating RQ1 against the BASELINE random prediction and for RQ2 against latency in the CI environment with only Comprehensive Cycles.

Results

Summary of Results

Note: For RQ1 and RQ2, the model was trained using the 'AUG' feature set (Table [table:featureDescription]), a 3-week training window, a 0.01 downsampling rate for the majority class, and no upweighting for the minority class. This configuration matches our production model, except for the feature set. RQ3 explores performance across configurations.

RQ1: What is the performance of Speculative Cycles with Transition Prediction in terms of percentage of novel breakages detected (recall) in comparison to a baseline randomized testing approach?

Figure 3

TRANSPRED demonstrates superior recall compared to the BASELINE model across all budgets (see Figure [fig:cl-level-recall]). At a budget of 25% of the total targets, TRANSPRED achieves a recall of 85%, greater than the BASELINE model's recall of 56%. This highlights the model's effectiveness in identifying breaking changes with limited resources.

In the context of Transition Prediction, recall measures the percentage of breaking changes identified using a given budget. The budget parameterizes the recall metric as shown in Figure [fig:cl-level-recall]. To simplify the complex accounting of machine and test costs, we use the percentage of targets scheduled by TRANSPRED (versus a Comprehensive Cycle). Our budget for this assessment is 25% of the targets that would have been used by a Comprehensive Cycle. Recall is then defined as percentage of culprits identified out of the total set of culprits.

Figure [fig:cl-level-recall] provides a visual representation of how recall changes as the percentage of scheduled targets varies, comparing the performance of TRANSPRED against the BASELINE model. For each model the figure shows the recall of both AtLeast(1) and AtLeast(10). AtLeast(1) corresponds to finding any target for a breakage or informally, that breakages are at least "identified". The recall for AtLeast(10) concentrates on finding breakages with at least 10 targets broken - including scheduling at least 10 of those broken targets to ensure auto rollback has enough evidence to trigger.

For TRANSPRED, the AtLeast(10) recall is higher than the AtLeast(1) recall. This indicates the model is better at catching bad commits that break 10 or more targets. It's intuitive that larger breakages would be better handled by the model as they are much better represented in the dataset, but an interesting result that we have is better coverage of them, i.e., consistently finding more individual targets broken by such commits. Culprits that only break a few tests are arguably less consequential than the few culprits that break a large number of tests. This strong performance on commits that break at least 10 targets indicates that when TRANSPRED does find a break, it has a good chance to be rolled back automatically.

Both configurations for TRANSPRED outperform the BASELINE random model, indicating that the model is learning something useful from metadata and that the metadata alone (without considering the content of the changes) can be predictive of breakages. We look forward to future experiments utilizing the content of commits that compare both the efficacy and the model costs.

RQ2: Speculative Cycles compete with full testing cycles for machine resources, what configurations of Speculative Cycles improve productivity outcomes for Google developers?

Speculative Cycles reduce the median (p50) breakage detection latency by 65% (70 minutes) over the existing Comprehensive Cycles.

Figure 4

Figure [fig:detection-latency] visualizes a histogram of the performance of the model in terms of breakage detection latency. We consider this metric to be the "outcome metric" for this work as it is a proxy for improving overall developer productivity. It is known from internal work that developer productivity degrades when they need to debug build and test breakages they did not cause during interactive development. Productivity also degrades when they are interrupted to troubleshoot releases stalled due to breakages. By reducing the total time from culprit commit submission to detection, the model reduces the amount of friction developers experience from breakages.

As with RQ1, we have fixed TRANSPRED to represent Speculative Cycles using the same Transition Prediction model and scheduling 25% of targets. In the visualization, the model (in black) is compared against two alternatives. The first labeled "Only Comprehensive" shows the distribution of time (in minutes) it takes the current TAP Postsubmit scheduling algorithm (Comprehensive Cycles) to detect breakages. The second alternative, "Optimal" imagines a perfect model that always correctly predicts which target need to be scheduled. Our model, TRANSPRED, falls between these two extremes of no-improvement to full-improvement. Observe that in comparison to "Only Comprehensive" the variance of breakage detection latency for TRANSPRED is substantially reduced. While, the new model isn't perfect (as shown in the figure), it is a welcome result that the Speculative Cycles reliably finds culprit commits in 37 minutes in the median case (p50), a 65% latency reduction over traditional Comprehensive Cycles that (p50) take 107 minutes.

RQ3:What aspects of the model design most impact performance? Considered aspects include features selected, super-extreme class imbalance corrections, and training environment.

The feature set used and the training window length had the greatest impact on model performance as measured by ROC AUC. Downsampling is critical for training speed but further downsampling hurts performance while upweighting has negligible impact.

To better understand how model design impacts performance, we ran multiple experiments across the different axes of decision making, including 1) feature selection, 2) training environment. and 3) different ways to correct for the super-extreme class imbalance

Feature Selection

Tables 2 and 3

Table [table:training-length] shows an improvement in performance when the model is trained on the 'AUGMENTED' feature set vs 'BASE' set, regardless of number of weeks considered for the training window. Critically, we see a lot of signal coming from the added target history features over longer time intervals.

Training Environment

Table [table:training-length] indicates our hypothesis was correct - performance starts to degrade as we increase or decrease the training window length from the optimal 3 weeks.

Correcting for Super-Extreme Class Imbalance

Production systems exhibit extremely low Postsubmit breakage rates (40,000:1 in our case). This class imbalance can hinder model training by biasing it towards the majority class. However, due to the asymmetric cost of false positives (unnecessary execution) versus false negatives (late breakage detection), prioritizing catching newly breaking targets at the cost of incorrectly scheduling some healthy targets is acceptable. This lets us experiment with some imbalance-correction techniques without being strictly bound to the true breakage rate, as shown in Table [table:samplingAndWeightTable].

This paper explored the problem of reducing the time to discovery of novel test failures in continuous integration. This is a variation of the well studied problem of Test Case Prioritization and Selection (Bates and Horwitz 1993; Rothermel and Harrold 1997; Elbaum, Malishevsky, and Rothermel 2000; Leon and Podgurski 2003; Engström, Runeson, and Skoglund 2010; Zhou 2010; Singh et al. 2012; Gligoric et al. 2014; Mondal, Hemmati, and Durocher 2015; Musa et al. 2015; de S. Campos Junior et al. 2017; Najafi, Shang, and Rigby 2019; Machalica et al. 2019; De Castro-Cabrera, García-Dominguez, and Medina-Bulo 2020; Pan et al. 2022; Jin and Servant 2023) adapted to the CI environment (Chen et al. 2020; Saidani et al. 2020; Abdalkareem, Mujahid, and Shihab 2021; Abdalkareem et al. 2021; Al-Sabbagh, Staron, and Hebig 2022; Saidani, Ouni, and Mkaouer 2022a, 2022b; Jin and Servant 2023; Jin et al. 2024; Liu et al. 2023; Hong et al. 2024; Sun, Habchi, and McIntosh 2024; G. Wang et al. 2024; Zeng et al. 2024). Our approach is predicated on having highly accurate historical information on what causes breakages in the Google environment. We achieve this through culprit finding and verification (Henderson et al. 2023, 2024). Culprit finding (Couder 2008; Ziftci and Ramavajjala 2013; Ziftci and Reardon 2017; Saha and Gligoric 2017; Najafi, Rigby, and Shang 2019; Beheshtian, Bavand, and Rigby 2022; Keenan 2019; An and Yoo 2021; Ocariza 2022) aims to identify the true "bug introducing commits" (BICs) (Śliwerski, Zimmermann, and Zeller 2005; Rodríguez-Pérez, Robles, and González-Barahona 2018; Borg et al. 2019; Wen et al. 2019; An et al. 2023) by rerunning tests over the search space of possible versions.

The extensive amount of work on test selection and prioritization makes it difficult to fully summarize outside of a survey (Engström, Runeson, and Skoglund 2010; Singh et al. 2012; de S. Campos Junior et al. 2017; De Castro-Cabrera, García-Dominguez, and Medina-Bulo 2020; Pan et al. 2022). Here are some highlights. Early work in test selection focused on selecting tests while maintaining test suite "adequacy" according to some (usually coverage based) criterion. For instance Bates and Horwitz (Bates and Horwitz 1993) define a coverage criterion based on coverage of elements in the Program Dependence Graph (PDG) (Horwitz, Prins, and Reps 1988; Podgurski and Clarke 1989) and Rothermel and Harrold developed a regression selection technique based on the PDG (Rothermel and Harrold 1997). By 2000 papers such as Elbaum et al. (Elbaum, Malishevsky, and Rothermel 2000)'s were looking at prioritizing test cases based statement coverage to speed up test suite execution and evaluating performance based on the now standard average percentage of faults detected (APFD) metric. By 2003, the community was starting to look for other sources of data to prioritize and filter test cases. For instance, Leon and Podgurski (Leon and Podgurski 2003) compared the bug finding efficacy (APFD) of coverage techniques versus techniques that prioritized diversity of basic block execution profiles (that included execution counts). In 2010 Engstrom et al. (Engström, Runeson, and Skoglund 2010) surveyed the state of the test selection and documented 28 techniques.

In 2017 the TAP team at Google began publishing on the problem (Memon et al. 2017; Leong et al. 2019) and separately a partner team had put into production an ML based test filtering system for TAP Presubmit. In contrast to this prior work by TAP, our current work is better grounded because of the verified culprits dataset we created in the last several years. For Memon et al. (Memon et al. 2017) TAP did not yet have a robust culprit finding system to precisely identify the bug introducing changes. Instead, that paper attempted to infer what commits might be the culprit. In Leong et al. (Leong et al. 2019) TAP did have a culprit finder but it was not as robust against flakiness and non-determinism as our current system. Our current culprit finder uses the Flake Aware algorithm (FACF) that adaptively deflakes the tests while performing culprit finding. Additionally, we further verify all culprit commits by doing additional executions in patterns designed to catch flaky behavior (Henderson et al. 2023, 2024). The FACF paper (Henderson et al. 2023) performed experiments that show the FACF algorithm is statistically significantly more accurate than the "deflaked variant" of the standard "bisect" algorithm used in the Leong paper. Specifically the experiments indicate FACF is >10% more accurate at identifying flaky breakages than Bisect with 8 additional deflaking runs. By properly deflaking our training and evaluation labels, our model is much less likely to prioritize flaky tests and our evaluation more accurately reflects our desired productivity outcomes.

Meanwhile in 2019, Machalica et al. (Machalica et al. 2019) reported on a similar effort at Facebook in 2019 to develop a test selection system for their version of Presubmit testing. Conceptually, the Presubmit selection system at Facebook and the Presubmit selection system at Google have similarities. Both use shallow machine learning models with sets of features not dissimilar to the features presented in this work. Both systems (like this work) take a systematic approach to reducing the effect of flaky failures on developer productivity. The primary contribution of this paper versus this past work is the application and evaluation of selection techniques to the Postsubmit environment. In Postsubmit, the features must be necessarily adjusted as the prediction of whether or not a test will change behavior is not against a single commit but against a sequence of commits. Evaluation also changes: the concern is not just about recall but also latency of breakage detection and time-to-fix.

Recently the wider field (outside of Google and Facebook) has produced compelling work (Jin and Servant 2023; Jin et al. 2024; Liu et al. 2023; Hong et al. 2024; Sun, Habchi, and McIntosh 2024; G. Wang et al. 2024; Zeng et al. 2024). We will note a few items here. Jin and Servant (Jin and Servant 2023) presented HybridCISave which uses a shallow machine learning based approach to do both fine-grained (individual tests) and coarse grained (CI workflow builds) selection. By combining both selection types both cost savings and safety are improved. The authors evaluate their results using TravisTorrent (Beller, Gousios, and Zaidman 2022). This work is a good representative of the general approach of using shallow machine learning for selection outside of the mega-corp environment. Wang et al. (G. Wang et al. 2024) address the problem of feature selection by using a transformer based language model, GitSense, that automates statistic feature extraction. The authors evaluate their results using the dataset from Chen et al. (Chen et al. 2020). Zeng et al. (Zeng et al. 2024) use mutation testing to better assess the safety of a commercial CI system (YourBase) that skips tests based on inferred dependencies. This use of mutation testing highlights one weakness of traditional safety evaluation, they only use historical faults and may miss novel bugs. By injecting mutation faults a more holistic view of the performance of the model can be obtained. Determining how to apply mutation testing to shallow machine learning approaches that use metadata features (such as ours) is an open problem.

Conclusion

In this paper, we presented a new scheduling system for Google's main test automation platform (TAP). The new scheduling system, Speculative Cycles, prioritizes finding novel test breakages faster. Speculative Cycles are powered by a new machine learning model, Transition Prediction, that predicts when tests are likely to fail. While the results are not perfect, they indicate that the new system reduces the breakage detection latency by 65% (70 minutes) on average. This reduction in breakage detection time leads to a reduction in developer friction and increased productivity. In terms of model design, the most impactful choice made was total amount of prior history used to train the model. As the training set size went from 1 week to 3 weeks model performance improved. But as the training set size further increased the model performance actually began to degrade. Only high level metadata features were used in this paper. In the future we look forward to experimenting with more complex models.

Abdalkareem, Rabe, Suhaib Mujahid, and Emad Shihab. 2021. "A Machine Learning Approach to Improve the Detection of CI Skip Commits." IEEE Transactions on Software Engineering 47 (12): 2740-54. https://doi.org/10.1109/TSE.2020.2967380.

Abdalkareem, Rabe, Suhaib Mujahid, Emad Shihab, and Juergen Rilling. 2021. "Which Commits Can Be CI Skipped?" IEEE Transactions on Software Engineering 47 (3): 448-63. https://doi.org/10.1109/TSE.2019.2897300.

Al-Sabbagh, Khaled, Miroslaw Staron, and Regina Hebig. 2022. "Predicting Build Outcomes in Continuous Integration Using Textual Analysis of Source Code Commits." In Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering, 42-51. Singapore Singapore: ACM. https://doi.org/10.1145/3558489.3559070.

An, Gabin, Jingun Hong, Naryeong Kim, and Shin Yoo. 2023. "Fonte: Finding Bug Inducing Commits from Failures." arXiv. http://arxiv.org/abs/2212.06376.

An, Gabin, and Shin Yoo. 2021. "Reducing the Search Space of Bug Inducing Commits Using Failure Coverage." In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 1459-62. Athens Greece: ACM. https://doi.org/10.1145/3468264.3473129.

Bates, Samuel, and Susan Horwitz. 1993. "Incremental Program Testing Using Program Dependence Graphs." In Proceedings of the 20th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages - POPL '93, 384-96. Charleston, South Carolina, United States: ACM Press. https://doi.org/10.1145/158511.158694.

Beheshtian, Mohammad Javad, Amir Hossein Bavand, and Peter C. Rigby. 2022. "Software Batch Testing to Save Build Test Resources and to Reduce Feedback Time." IEEE Transactions on Software Engineering 48 (8): 2784-2801. https://doi.org/10.1109/TSE.2021.3070269.

Beller, Moritz, Georgios Gousios, and Andy Zaidman. 2022. "TravisTorrent." figshare. https://doi.org/10.6084/M9.FIGSHARE.19314170.V1.

Bland, Mike. 2012. "The Chris/Jay Continuous Build." Personal {{Website}}. Mike Bland's Blog. https://mike-bland.com/2012/06/21/chris-jay-continuous- build.html.

Borg, Markus, Oscar Svensson, Kristian Berg, and Daniel Hansson. 2019. "SZZ Unleashed: An Open Implementation of the SZZ Algorithm - Featuring Example Usage in a Study of Just-in-Time Bug Prediction for the Jenkins Project." In Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation - MaLTeSQuE 2019, 7-12. Tallinn, Estonia: ACM Press. https://doi.org/10.1145/3340482.3342742.

Chen, Bihuan, Linlin Chen, Chen Zhang, and Xin Peng. 2020. "BUILDFAST: History-aware Build Outcome Prediction for Fast Feedback and Reduced Cost in Continuous Integration." In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE), 42-53.

Couder, Christian. 2008. "Fighting Regressions with Git Bisect." The Linux Kernel Archives 4 (5). https://www. kernel. org/pub/software/scm/git/doc s/git- bisect-lk2009.html.

De Castro-Cabrera, M. Del Carmen, Antonio García-Dominguez, and Inmaculada Medina-Bulo. 2020. "Trends in Prioritization of Test Cases: 2017-2019." Proceedings of the ACM Symposium on Applied Computing, 2005-11. https://doi.org/10.1145/3341105.3374036.

de S. Campos Junior, Heleno, Marco Antônio P Araújo, José Maria N David, Regina Braga, Fernanda Campos, and Victor Ströele. 2017. "Test Case Prioritization: A Systematic Review and Mapping of the Literature." In Proceedings of the 31st Brazilian Symposium on Software Engineering, 34-43. New York, NY, USA: ACM. https://doi.org/10.1145/3131151.3131170.

Elbaum, Sebastian, Alexey G. Malishevsky, and Gregg Rothermel. 2000. "Prioritizing Test Cases for Regression Testing." In Proceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis, 102-12. Issta '00. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/347324.348910.

Engström, Emelie, Per Runeson, and Mats Skoglund. 2010. "A Systematic Review on Regression Test Selection Techniques." Information and Software Technology 52 (1): 14-30. https://doi.org/10.1016/j.infsof.2009.07.001.

Fowler, Martin. 2006. "Continuous Integration." https://martinfowler.com/articles/ continuousIntegration.html.

Gligoric, Milos, Rupak Majumdar, Rohan Sharma, Lamyaa Eloussi, and Darko Marinov. 2014. "Regression Test Selection for Distributed Software Histories." In Computer Aided Verification, edited by David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Alfred Kobsa, Friedemann Mattern, John C. Mitchell, et al., 8559:293-309. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-08867-9_19.

Guillame-Bert, Mathieu, Sebastian Bruch, Richard Stotz, and Jan Pfeifer. 2023. "Yggdrasil Decision Forests: A Fast and Extensible Decision Forests Library." In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 4068-77. Long Beach CA USA: ACM. https://doi.org/10.1145/3580305.3599933.

Gupta, Pooja, Mark Ivey, and John Penix. 2011. "Testing at the Speed and Scale of Google." https://google-engtools.blogspot.com/2011/06/testing-at- speed-and-scale-of-google.html.

Henderson, Tim A. D., Bobby Dorward, Eric Nickell, Collin Johnston, and Avi Kondareddy. 2023. "Flake Aware Culprit Finding." In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE. https://doi.org/10.1109/ICST57152.2023.00041.

Henderson, Tim A. D., Avi Kondareddy, Sushmita Azad, and Eric Nickell. 2024. "SafeRevert: When Can Breaking Changes Be Automatically Reverted?" In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST), 395-406. Toronto, ON, Canada: IEEE. https://doi.org/10.1109/ICST60714.2024.00043.

Hoang, Minh, and Adrian Berding. 2024. "Presubmit Rescue: Automatically Ignoring FlakyTest Executions." In Proceedings of the 1st International Workshop on Flaky Tests, 1-2. Lisbon Portugal: ACM. https://doi.org/10.1145/3643656.3643896.

Hong, Yang, Chakkrit Tantithamthavorn, Jirat Pasuksmit, Patanamon Thongtanunam, Arik Friedman, Xing Zhao, and Anton Krasikov. 2024. "Practitioners' Challenges and Perceptions of CI Build Failure Predictions at Atlassian." In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 370-81. Porto de Galinhas Brazil: ACM. https://doi.org/10.1145/3663529.3663856.

Horwitz, Susan, Jan Prins, and Thomas Reps. 1988. "On the Adequacy of Program Dependence Graphs for Representing Programs." ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), 146-57. https://doi.org/10.1145/73560.73573.

Jin, Xianhao, Yifei Feng, Chen Wang, Yang Liu, Yongning Hu, Yufei Gao, Kun Xia, and Luchuan Guo. 2024. "PIPELINEASCODE: A CI/CD Workflow Management System Through Configuration Files at ByteDance." In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 1011-22. Rovaniemi, Finland: IEEE. https://doi.org/10.1109/SANER60148.2024.00109.

Jin, Xianhao, and Francisco Servant. 2023. "HybridCISave: A Combined Build and Test Selection Approach in Continuous Integration." ACM Transactions on Software Engineering and Methodology 32 (4): 1-39. https://doi.org/10.1145/3576038.

Keenan, James. 2019. "James E. Keenan - "Multisection: When Bisection Isn't Enough to Debug a Problem"." https://www.youtube.com/watch?v=05CwdTRt6AM.

Leon, D., and A. Podgurski. 2003. "A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases." In 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003., 2003-Janua:442-53. IEEE. https://doi.org/10.1109/ISSRE.2003.1251065.

Leong, Claire, Abhayendra Singh, Mike Papadakis, Yves Le Traon, and John Micco. 2019. "Assessing Transition-Based Test Selection Algorithms at Google." In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 101-10. IEEE. https://doi.org/10.1109/ICSE-SEIP.2019.00019.

Liu, Bohan, He Zhang, Weigang Ma, Gongyuan Li, Shanshan Li, and Haifeng Shen. 2023. "The Why, When, What, and How About Predictive Continuous Integration: A Simulation-Based Investigation." IEEE Transactions on Software Engineering 49 (12): 5223-49. https://doi.org/10.1109/TSE.2023.3330510.

Machalica, Mateusz, Alex Samylkin, Meredith Porth, and Satish Chandra. 2019. "Predictive Test Selection." In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 91-100. IEEE. https://doi.org/10.1109/ICSE-SEIP.2019.00018.

Memon, Atif, Zebao Gao, Bao Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. 2017. "Taming Google-scale Continuous Testing." In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), 233-42. Piscataway, NJ, USA: IEEE. https://doi.org/10.1109/ICSE-SEIP.2017.16.

Micco, John. 2012. "Tools for Continuous Integration at Google Scale." Tech {{Talk}}. Google NYC. https://youtu.be/KH2_sB1A6lA.

---. 2013. "Continuous Integration at Google Scale." Lecture. EclipseCon 2013. https://web.archive.org/web/20140705215747/https:// www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/2013- 03-24%20Continuous%20Integration%20at%20Google%20Scale.pdf.

Mondal, Debajyoti, Hadi Hemmati, and Stephane Durocher. 2015. "Exploring Test Suite Diversification and Code Coverage in Multi-Objective Test Case Selection." 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings, 1-10. https://doi.org/10.1109/ICST.2015.7102588.

Musa, Samaila, Abu Bakar Md Sultan, Abdul Azim Bin Abd-Ghani, and Salmi Baharom. 2015. "Regression Test Cases Selection for Object-Oriented Programs Based on Affected Statements." International Journal of Software Engineering and Its Applications 9 (10): 91-108. https://doi.org/10.14257/ijseia.2015.9.10.10.

Najafi, Armin, Peter C. Rigby, and Weiyi Shang. 2019. "Bisecting Commits and Modeling Commit Risk During Testing." In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 279-89. New York, NY, USA: ACM. https://doi.org/10.1145/3338906.3338944.

Najafi, Armin, Weiyi Shang, and Peter C. Rigby. 2019. "Improving Test Effectiveness Using Test Executions History: An Industrial Experience Report." In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 213-22. IEEE. https://doi.org/10.1109/ICSE-SEIP.2019.00031.

Ocariza, Frolin S. 2022. "On the Effectiveness of Bisection in Performance Regression Localization." Empirical Software Engineering 27 (4): 95. https://doi.org/10.1007/s10664-022-10152-3.

Pan, Rongqi, Mojtaba Bagherzadeh, Taher A. Ghaleb, and Lionel Briand. 2022. "Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review." Empirical Software Engineering 27 (2): 29. https://doi.org/10.1007/s10664-021-10066-6.

Podgurski, A, and L Clarke. 1989. "The Implications of Program Dependencies for Software Testing, Debugging, and Maintenance." In Proceedings of the ACM SIGSOFT '89 Third Symposium on Software Testing, Analysis, and Verification, 168-78. New York, NY, USA: ACM. https://doi.org/10.1145/75308.75328.

Potvin, Rachel, and Josh Levenberg. 2016. "Why Google Stores Billions of Lines of Code in a Single Repository." Communications of the ACM 59 (7): 78-87. https://doi.org/10.1145/2854146.

Rodríguez-Pérez, Gema, Gregorio Robles, and Jesús M. González-Barahona. 2018. "Reproducibility and Credibility in Empirical Software Engineering: A Case Study Based on a Systematic Literature Review of the Use of the SZZ Algorithm." Information and Software Technology 99 (July): 164-76. https://doi.org/10.1016/j.infsof.2018.03.009.

Rothermel, Gregg, and Mary Jean Harrold. 1997. "A Safe, Efficient Regression Test Selection Technique." ACM Transactions on Software Engineering and Methodology 6 (2): 173-210. https://doi.org/10.1145/248233.248262.

Sadowski, Caitlin, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. "Modern Code Review: A Case Study at Google." In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, 181-90. Gothenburg Sweden: ACM. https://doi.org/10.1145/3183519.3183525.

Saha, Ripon, and Milos Gligoric. 2017. "Selective Bisection Debugging." In Fundamental Approaches to Software Engineering, edited by Marieke Huisman and Julia Rubin, 10202:60-77. Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-662-54494-5_4.

Saidani, Islem, Ali Ouni, Moataz Chouchen, and Mohamed Wiem Mkaouer. 2020. "Predicting Continuous Integration Build Failures Using Evolutionary Search." Information and Software Technology 128 (December): 106392. https://doi.org/10.1016/j.infsof.2020.106392.

Saidani, Islem, Ali Ouni, and Mohamed Wiem Mkaouer. 2022a. "Improving the Prediction of Continuous Integration Build Failures Using Deep Learning." Automated Software Engineering 29 (1): 21. https://doi.org/10.1007/s10515-021-00319-5.

---. 2022b. "Detecting Continuous Integration Skip Commits Using Multi-Objective Evolutionary Search." IEEE Transactions on Software Engineering 48 (12): 4873-91. https://doi.org/10.1109/TSE.2021.3129165.

Singh, Yogesh, Arvinder Kaur, Bharti Suri, and Shweta Singhal. 2012. "Systematic Literature Review on Regression Test Prioritization Techniques." Informatica (Slovenia) 36 (4): 379-408. https://doi.org/10.31449/inf.v36i4.420.

Śliwerski, Jacek, Thomas Zimmermann, and Andreas Zeller. 2005. "When Do Changes Induce Fixes?" ACM SIGSOFT Software Engineering Notes 30 (4): 1. https://doi.org/10.1145/1082983.1083147.

Sun, Gengyi, Sarra Habchi, and Shane McIntosh. 2024. "RavenBuild: Context, Relevance, and Dependency Aware Build Outcome Prediction." Proceedings of the ACM on Software Engineering 1 (FSE): 996-1018. https://doi.org/10.1145/3643771.

Wang, Guoqing, Zeyu Sun, Yizhou Chen, Yifan Zhao, Qingyuan Liang, and Dan Hao. 2024. "Commit Artifact Preserving Build Prediction." In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 1236-48. Vienna Austria: ACM. https://doi.org/10.1145/3650212.3680356.

Wang, Kaiyuan, Daniel Rall, Greg Tener, Vijay Gullapalli, Xin Huang, and Ahmed Gad. 2021. "Smart Build Targets Batching Service at Google." In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 160-69. Madrid, ES: IEEE. https://doi.org/10.1109/ICSE-SEIP52600.2021.00025.

Wang, Kaiyuan, Greg Tener, Vijay Gullapalli, Xin Huang, Ahmed Gad, and Daniel Rall. 2020. "Scalable Build Service System with Smart Scheduling Service." In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, 452-62. Virtual Event USA: ACM. https://doi.org/10.1145/3395363.3397371.

Wen, Ming, Rongxin Wu, Yepang Liu, Yongqiang Tian, Xuan Xie, Shing-Chi Cheung, and Zhendong Su. 2019. "Exploring and Exploiting the Correlations Between Bug-Inducing and Bug-Fixing Commits." In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 326-37. Tallinn Estonia: ACM. https://doi.org/10.1145/3338906.3338962.

Zeng, Zhili, Tao Xiao, Maxime Lamothe, Hideaki Hata, and Shane Mcintosh. 2024. "A Mutation-Guided Assessment of Acceleration Approaches for Continuous Integration: An Empirical Study of YourBase." In Proceedings of the 21st International Conference on Mining Software Repositories, 556-68. Lisbon Portugal: ACM. https://doi.org/10.1145/3643991.3644914.

Zhou, Zhi Quan. 2010. "Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing." Proceedings - International Computer Software and Applications Conference, 208-13. https://doi.org/10.1109/COMPSACW.2010.43.

Ziftci, Celal, and Vivek Ramavajjala. 2013. "Finding Culprits Automatically in Failing Builds - i.e. Who Broke the Build?" https://www.youtube.com/watch?v=SZLuBYlq3OM.

Ziftci, Celal, and Jim Reardon. 2017. "Who Broke the Build? Automatically Identifying Changes That Induce Test Failures in Continuous Integration at Google Scale." Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017, 113-22. https://doi.org/10.1109/ICSE-SEIP.2017.13.

https://bazel.build/↩︎
When a target breaks we perform culprit finding. The conclusion of the culprit finding is the culprit commit. We colloquially say the target is blaming the culprit.↩︎

SafeRevert: When Can Breaking Changes be Automatically Reverted?

2024-05-27T00:00:00-04:00

Tim A. D. Henderson, Avi Kondareddy, Sushmita Azad, and Eric Nickell. SafeRevert: When Can Breaking Changes be Automatically Reverted?. ICST Industry Track 2024.
DOI. PDF. WEB.

Note

This is a conversion from a latex paper I wrote. If you want all formatting correct you should read the pdf version.

Abstract

When bugs or defects are introduced into a large scale software repository, they reduce productivity. Programmers working on related areas of the code will encounter test failures, compile breakages, or other anomalous behavior. On encountering these issues, they will need to troubleshoot and determine that their changes were not the cause of the error and that another change is at fault. They must then find that change and revert it to return the repository to a healthy state. In the past, our group has identified ways to identify the root cause (or culprit) change that introduced a test failure even when the test is flaky. This paper focuses on a related issue: at what point does the Continuous Integration system have enough evidence to support automatically reverting a change? We will motivate the problem, provide several methods to address it, and empirically evaluate our solution on a large set (25,137) of real-world breaking changes that occurred at Google. SafeRevert improved recall (number of changes recommend for reversion) by 2x over the baseline method while meeting our safety criterion.

Introduction

Large scale software development is enabled by automatically executing tests in a continuous integration environment. Continuous integration (CI) (Fowler 2006) is the industrial practice of using automated systems to automatically integrate changes into the source of truth for the software system or repository. This improves collaboration by helping software developers avoid breaking compilation, tests, or structure of the system that others are relying on.

Corporations and teams may engage in CI using adhoc tools across many independent software repositories. For instance Github (github.com) supports CI “Actions" which can be integrated into any repository. However, many organizations are recognizing the value of using a “mono-repository" (monorepo) development model where many or all teams in the organization use a single shared software repository (Potvin and Levenberg 2016). At the largest organizations such as Google (Potvin and Levenberg 2016; Memon et al. 2017; Henderson et al. 2023), Microsoft (Herzig and Nagappan 2015), and Facebook (Machalica et al. 2019) large repositories are complimented by advanced centralized CI systems.

In these large, modern CI systems, the integration goes beyond just ensuring the code textually merges. Compilations are invoked, tests are executed, and additional static and dynamic verification steps are performed. The demand for machine resources can exceed capacity for build, test, and verification tasks desired by a large-scale CI system. To combat this problem, test case selection or prioritization is used (Gupta, Ivey, and Penix 2011; Micco 2013; Machalica et al. 2019) to select fewer tests to run.

Additionally, CI steps are often invoked at multiple points in the Software Development Life Cycle. In the past, it may have been assumed that the tests were executed by the CI system once, at the time a commit was created in the version control. Today, CI may execute tests multiple times during development: when a change is sent for code review, immediately prior to submission or integration into the main development branch, after one or more changes has been integrated into the development branch, and when a new release is created. This paper is primary concerned with testing that occurs after the code has been integrated into the main development branch.

Why is Testing Necessary After Code Integration?

When deploying CI for the first time, many organizations primarily focus on conducting testing at the time a change is merged into the main branch. For instance, they may test when a Pull Request (PR) on Github is going to be merged into the main branch. The PR is merged if the tests pass and no changes have been made to the main branch since the testing started. This strategy works well until the rate of PR submission exceeds the average amount of time it takes to run all the tests.

At this point, organizations may do a stop gap fix such as adding more machines to run tests in parallel or reducing the size of the test suite. However, at some point, these measures will prove ineffective and the rate that testing can be conducted will become a impediment to an organization’s engineering velocity. To address this, one common solution is to introduce a “submission queue" which batches changes together for testing and merges them all if the tests pass (Ananthanarayanan et al. 2019). If the tests fail, the offending change must be identified and the remaining changes in the batch must be retested (at least doubling the total testing time) (Najafi, Shang, and Rigby 2019).

As the submission rate continues to increase, the organization may add conservative test selection based on file-level dependence analysis (Ananthanarayanan et al. 2019). But at this point, the organization will be reaching the limit of what can be done to completely prevent any breaking changes from being integrated into the main development branch beyond just buying more and more machines to further parallelize the testing.

And what about just buying more machines? Won’t this effectively solve the problem? It would until the submission rate exceeds the time it takes to run the slowest test. At this point, the number of machines purchased can only partially control impact to developer productivity. Even with a very large budget for testing, the number of changes per batch will continue to grow as increases in the number of machines will not reduce the batch size but only keep the testing time per batch relatively constant. The larger the batch size, the more likely that a developer will experience conflicts with another change when attempting to submit. These conflicts (either at the syntactical or semantic level) will make it difficult for developers to reliably submit their changes and require them to constantly monitor the submission process. This monitoring will hurt overall developer productivity.

At the highest submission rates seen in industry today, using submission queues that guarantee zero breaking changes becomes infeasible both from a machine cost perspective and from a developer productivity perspective. Therefore, many organizations relax the requirement that an integrated change will never break a test. Even while using submit queues to gate contributions for a single team, Google has relied on postsubmit testing for mission-critical software since the early 2000s as documented by Mike Bland (Bland 2012).

Post Submission Testing

The purpose of testing after a change has been integrated (called Post-Submission Testing or Postsubmit Testing hereafter) is to identify defects that slipped through testing that occurred prior to integration. Typically, organizations are unable to run all tests at every integrated change. Instead, testing is conducted periodically. In this paper we will refer to this periodic testing as Testing Cycles.

Figure [fig:ci-timeline] is a simplified view of the postsubmit testing strategy used at Google. The test scheduler waits until our Build System (Wang et al. 2020) has capacity to start a test cycle. It then schedules tests and then waits until the system has capacity again. When scheduling tests, certain tests may be temporarily skipped or throttled to conserve resources. As shown in Figure [fig:ci-timeline], this leads to the system usually detecting new failures some time after the version that introduced them had been integrated. This leads to the problem of “Culprit Finding."

Culprit Finding

Culprit finding is conceptually simple: given a list of versions that may have introduced a fault, locate the offending change. One may solve this problem with a variety of techniques: Zeller’s Delta Debugging (Zeller 1999), Git’s bisection algorithm (Couder 2008), or Google’s Flake Aware Culprit Finding (FACF) algorithm (Henderson et al. 2023). Every technique used to identify the offending change will have some error rate in industrial practice.

But wait! How can a binary search or delta debugging have an “error rate"? The answer is that old nemesis of industrial practice: flaky or non-deterministic test behavior (Parry et al. 2022). Flaky tests can be caused both by problems in the production code (ex: race conditions causing rare errors), in the test code (ex: use of “sleep") and by infrastructure problems (ex: unreliable machine with bad ram or network congestion). All of these problems behave in version non-hermetic ways, where failures may not be strictly linked to the version of code executed. Furthermore, even when the test is fully version hermetic the flakiness may not obey a Bernoulli distribution as subsequent executions may not be fully independent of the prior ones.

The above issues mean that even culprit finding algorithms such as FACF (Henderson et al. 2023) that have been purpose-built to mitigate flakiness will have some error rate. While that error rate will be much less than a naive algorithm, it may still be high enough to cause problems when deployed in certain environments.

Automatically Reverting Changes

Once an organization has a reliable and high performance culprit finding system, it is natural to use it to automatically revert (undo, roll back) changes that introduce defects into the main development branch. By automatically undoing these changes, the system decreases the amount of time developers working with impacted tests will experience breakages that are unrelated to the changes they are working on. It will also increase the number of versions that are viable to be used to make software releases. Reducing development friction and increasing the number of release candidates improves developer productivity by reducing the amount of time developers spend troubleshooting tests that were broken by someone else.

Unfortunately, naively integrating FACF directly into a system that immediately reverts all changes it identifies will lead to unhappy developers. This is because even though Google’s FACF has a measured per-test accuracy of 99.54%, when aggregated (grouped) by blamed change, the accuracy per-change is only 77.37% in the last 28 day window as of this writing.¹ This would translate into incorrectly reverting approximately 20-130 changes per day out of an approximately of 300-500 total changes per day reverted (see Figure [fig:baseline]). At Google, we find the cost of incorrectly reverting a change is extremely high in terms of developer toil and frustration. Therefore, we need to reduce the rate of bad reversions as much as possible. In this work, we aim to incorrectly revert fewer than one change per day on average.

Our Contributions and Findings

In this paper we propose a shallow machine-learning based method for using the output of a culprit finding algorithm to automatically revert (undo, rollback) a change that controls for potential errors from the culprit finding algorithm. Our method is generic and can be used with any culprit finding algorithm.

SafeRevert: a method for using the output of any culprit finding algorithm to automatically revert (undo) a change. SafeRevert controls for the error rate while increasing total number of reversions over baseline methods.
An ablation study on the models, designs and features used by the machine learning system used in SafeRevert to identify the most impactful features and best models.
A large scale study on SafeRevert’s efficacy using $\thicksim$25,137 potential culprit changes identified by Google’s production culprit finder over a $\sim$3 month window yields a recall of 55.7%, $\sim2.1 \times$ higher than the baseline method.

Background

Figure 1

The Test Automation Platform (TAP) is a proprietary Continuous Integration (CI) system at Google. It runs tests that execute on a single machine without the use of the network or external resources.² All tests on TAP are required to run in under 15 minutes and exceeding that limit is considered a test failure. TAP executes tests both before a user submits their changes and after the user has submitted their change. This paper is only concerned with the testing that has occurred after a user has submitted a change. At Google, this is referred to as “Postsubmit Testing". The part of TAP that does Postsubmit Testing is called TAP Postsubmit.

TAP has appeared in the literature several times (Gupta, Ivey, and Penix 2011; Micco 2012, 2013; Memon et al. 2017; Leong et al. 2019; Wang et al. 2020; Henderson et al. 2023) and there has been gradual evolution its testing strategy over the years. However, in general TAP uses a combination of static and dynamic Test Selection, execution throttling, and just-in-time scheduling to control testing load. A simplified diagram of TAP Postsubmit testing is visualized in Figure [fig:ci-timeline]. Tests are periodically run in Testing Cycles.³ During a cycle, Projects that are eligible to run their test in the cycle are selected. Tests from those projects are included if some file was modified since that test’s last execution can influence their behavior via inspection of a dependence graph at the Build Target and File level granularity (Gupta, Ivey, and Penix 2011; Micco 2012). When breakages inevitably occur culprit finding is conducted using the FACF algorithm to locate the offending change (Henderson et al. 2023).

FACF operates on a single test target at a time. For simplicity, we use the term “test” both for a target which executes test code or for a “build targets” which verifies that a binary can compile. Conceptually, FACF performs a “Noisy Binary Search” (Rivest, Meyer, and Kleitman 1978) (also called a Rényi-Ulam game (Pelc 2002)) which FACF models under the Bayesian Search Framework (Ben-Or and Hassidim 2008). The input to FACF includes the suspect changes (“suspects”) that may have broken the test and an estimate collected by an independent system of how likely it is to fail non-deterministically without the presence of a deterministic bug, it’s “flakiness”. Much like a normal binary search, it then divides the search space, executes tests, and updates a probability distribution based on the outcomes. Eventually, the system will determine that one of the suspects is above a set probability threshold (.9999) and is the source of the test failure, or that none of the suspects is at fault and the original failure was spurious, due to a flake (non-deterministic failure).

Culprit Finding Accuracy

Now some caveats: The math behind FACF (Henderson et al. 2023) assumes that individual test executions of the same test at either the same version or different versions are independent statistical events. That is, a prior or concurrent execution of a test cannot influence a subsequent execution. Unfortunately, while this assumption is theoretically sound, in practice this property does not always hold. External factors outside a program’s code can also influence a test’s execution behavior. For example, the time of day, day of year, load on the machine, etc., can all influence the outcome of certain tests. Thus, while we have configured FACF to have an accuracy of 99.99%, in practice we do not observe this level of accuracy. As noted in the Introduction, our observed accuracy was 99.54% in the last 28 day window as of this writing. This level of accuracy is consistent with the empirical study conducted in the 2023 paper (Henderson et al. 2023).

How was accuracy “observed”? What was the ground truth used? By what measurement technique? At Google, we continuously randomly sample a subset of culprit finding execution results as they are completed. The selected sample is then cross checked by performing extensive reruns that demand a higher level of accuracy and assume a worse flakiness rate for the test than used for culprit finding. Additionally, more reruns are scheduled for a later time to control for time-based effects. Finally, execution ordering effects are controlled for by ensuring that executions at versions that should fail can fail both before and after versions that should pass are executed. Full details on our methods for verification can be found in the FACF paper (Henderson et al. 2023). Even with extensive reruns there remains a small probability of error. While our method of verification is imperfect, it allows for continuously computing a statistical estimate on the accuracy of the culprit finder.

Measurements of FACF accuracy reported in the 2023 paper were performed per test breakage. That is, culprit finding is performed on a test $t$ when it has an observed failure at some version $\beta$ when it was previously passing at a prior version $\alpha$. The range $(\alpha, \beta]$ is referred to as the search-range and the key $\{t \times (\alpha, \beta]\}$ is the search-key. For a given search-key, the culprit finder first filters the search-range to changes on which $t$ has a transitive build dependence, which we call the suspect-set. A culprit $\kappa$ is identified somewhere between $\beta$ and $\alpha$ (e.g. $\alpha \sqsubset \kappa \sqsubseteq \beta$ where $a \sqsubset b$ indicates version $a$ is before version $b$). It is possible that there are multiple $\kappa_t$ for different $t$ w.r.t. the same search range, which we can call the culprit-set of the search range. What is measured is whether or not $\kappa$ is correct (true or false) for a given $t$ and search range $\alpha \sqsubset \beta$. Accuracy is the total correct measurements over the total number of measurements taken.

Applying Culprit Finding Results to Change Reversion

When breaking changes get merged into the main development branch they impede development productivity. At Google we term impediments to productivity as “friction.” Breaking changes impact three types of friction tracked at Google: Release Friction, Presubmit Friction and Development Friction. Release Friction occurs when breaking changes that merge into the main development branch impede automated releases and require manual intervention by the primary team’s on-duty engineer. Presubmit Friction occurs when pre-submission (“presubmit”) testing in TAP fails due to a broken test in the main development branch. Development Friction occurs when a developer manually triggers the execution of a test broken on the main branch during active code development. All three types of friction can steal time from developers and reduce productivity.

To reduce friction, changes that break tests can be automatically reverted (“rolled back”). However, auto-revert (“auto-rollback”) only improves developer productivity if the changes reverted actually broke tests. Let’s consider for a moment what happens when a change is incorrectly reverted. The developer who authored the change now needs to go on a wholly different troubleshooting journey to determine why their change didn’t stay submitted, and they are faced with the same damning-but-incorrect evidence used to revert their change. The developer is frustrated and must now debug tests that their team doesn’t own and which their changes should not affect.

Some changes are particularly difficult to submit with a buggy auto-revert system: those that change core libraries. These changes have the potential to break a large number of tests and impact a large number of other developers. But, at the same time they are also more likely to be incorrectly blamed (as TAP tracks what tests are affected by each change and uses this as an input to culprit finding). Past auto-revert systems that have been too inaccurate have caused core library authors to opt their changes out of auto reversion as the productivity cost to the core library teams has been too great to bear.

Accuracy of Auto-Revert versus Culprit Finding

A change $\kappa$ that breaks a test may have broken more than one. If so, then multiple independent executions of the culprit finding algorithm (one for each broken test) will blame the version $\kappa$ as the culprit. As explored above, the algorithms have an error rate. A change $\kappa$ that breaks $n$ tests will have $m \le n$ culprit finding conclusions that correctly identify $\kappa$ as the culprit.

Our concern with auto-revert isn’t the accuracy of culprit finding per search-key $\{t \times (\alpha, \beta]\}$ but rather per culprit change $\kappa$. Let $K$ be the set of all changes in the repository blamed as culprits by the culprit finding system. Let $c \subseteq K$ be the subset of the culprits that are correct and $\overline{c} \subseteq K$ be the subset of culprits that are incorrect. Then if every culprit in $K$ was automatically reverted, the accuracy of the auto-revert system would be ${\left|{c}\right|} / {\left|{K}\right|}$. In the introduction, we noted that the accuracy per change was 77.37% which equals ${\left|{c}\right|} / {\left|{K}\right|}$.

Why is the auto-revert accuracy (77.37%) lower than the culprit finding accuracy (99.54%)? Because there are many more tests than culprits and because correctly identified culprit changes may have broken more than one test. The chance of finding that culprit is higher because there are more chances to find it. Additionally, all of the culprit finding conclusions that identify one of those changes that break many tests will be correct. The set of incorrectly identified culprit $\overline{c}$ tends to contain mostly changes that are blamed by a fewer number of tests than those in set $c$.

Problem Statement

Identify a method for selecting as many as possible changes from the set of culprits $K$ that can be safely reverted. A change is safe to revert if it is a true culprit. Safety can be defined in terms of probability as well: A change is safe to revert if the chance it is a true culprit is $>99\%$. The number of changes selected should be maximized (while maintaining safety) to ensure the methods performs reversions rather than safely doing nothing at all.

Automatically Reverting Changes

Figure 2

We will describe several methods for selecting changes to revert while maintaining the intended safety property. The first method is a simple “baseline” method that the more complex methods will be evaluated against. The other methods use shallow machine learning systems to take advantage of metadata available at revert time.

Our baseline method is based on a single observation: a change $\kappa_0$ that is identified as the culprit by culprit finding on many tests is more likely to be a culprit than a change $\kappa_1$ that is only blamed by a few tests. Therefore, the simplest heuristic approach to selecting changes to revert is to threshold on a minimum number of tests identifying the change. BASELINE(N) will refer to this method with selecting changes with at least N blaming targets. Figure [fig:baseline] shows the performance of this method at various minimum number of targets. The most conservative, BASELINE(50) can avoid most false positives while reverting only a $\sim$13% of true culprits. We choose BASELINE(10) for our final evaluation as this is the heuristic Google has historically used. Additionally, 10 is the lowest configuration of BASELINE that generally (although not always) meets the desired safety criteria.

Predicting Auto-Revert Correctness using Machine Learning

Instead of choosing just one suggestive feature – the number of blaming tests – to decide on a reversion, our proposed method uses multiple features and shallow machine learning models to improve performance relative to the baseline method. A selection of easily obtainable coarse-grained metadata on the changes, tests, and culprit finding process are used as features to the models. As these features are mainly simple numerical, categorical, and textual features about the code, we use simple model architectures to control for over-fitting. Architectures examined are: Random Forests (RF) and AdaBoost (ADA) (both provided by scikit-learn⁴ (Pedregosa et al. 2011)), and XGBoost (XGB)⁵ (Chen and Guestrin 2016). These tree-based model architectures are easy to use, have low computational costs, and are robust to the feature representation versus alternatives that require more preprocessing.

Features Used

A suspect change is one that has been identified by at least one culprit-finder in the process of culprit finding a test. We can consider this the test blaming the change. For any suspect, we have one or more blaming tests. Then, we have a few sources of information that may be valuable for predicting a true breakage: characteristics of the change itself and characteristics of the blaming tests or culprit finding results. Table 1 contains the grouping of the features against their logical feature category, arrived at using the Pearson correlation between numerical features and lift analysis for non-numeric features.

Feature Representation

Categorical Features

Given a set of categories, we can create a fixed length representation by encoding a choice of category as a one-hot vector. When we have a variable number of categories per instance, such as language per test, this trivially becomes a multi-hot representation by summing the vectors.

Numerical Features

Singular numerical fields are integrated without preprocessing or normalization, as we expect decision trees to be robust to value-scaling. Variable-length numerical fields are turned into variable-length categorical fields by generating bins representing quantiles of the training set distribution of that value. Then, bin-mapped values can be condensed into a fixed-length multi-hot encoding as above.

Token Set Features

Token Sets correspond to variable-length categorical features where the list of categories is not known in advance. A good example of such a feature is compiler flags which should be dealt with adaptively by the model. We handle this issue by building a vocabulary per token-set feature on the training set parameterized by a minimum and maximum across-instance frequency. The change description, while potentially better dealt with through doc2vec or a more sophisticated NLP embedding approach, is treated like a token set and the resulting multi-hot encoding, having each field normalized by document frequency (TF-IDF).

Features for Machine Learning Model.
Category	Features
BASE	# of Blaming Tests
CHANGE_CONTENT	LOC (changed, added, deleted, total)
	# of Files Changed
	File Extensions
CHANGE_METADATA	Reviewer/Approver Count
	Issue/Bug Count
	Robot Author
CHANGE_TOKENS	Directory Tokens
CHANGE_TOKENS	Description
CULPRIT_FINDER	$\|$Suspect-Set$\|$ of Search-Key
CULPRIT_FINDER	$\|$Culprit-Set$\|$ of Search-Range
FLAKINESS	Historical Flaky Identification Count (Build/Test)
FLAKINESS	Historical Execution Flake Rate
TEST_ATTRIBUTES	Type of Test (Test vs Build)
	Bazel Rule (Ex: “java_test”)
	Machine Types (Ex: gpu)
	Test Language (Ex: java, c++, etc)
	Compiler Flags (ex: sanitizers)
	Average Machine Cost

Feature Grouping

Logical Sub-Grouping for Feature Lift Analysis

In order to avoid crowding out the evaluation and analysis with too many combinations of features, we’ve grouped them in Table 1 into logical groups of features based off of conceptual categories and to reflect cross-feature correlations for numerical features, computed via pearsons correlation coefficient. As our “Token Set" features represent a distinct class of features both in terms of encoding method and granularity (highly specific to individual changes), we separate them from the coarse-grain change features. These logical groups will allow us to clearly convey the individual signal being provided by each class of feature during our feature analysis.

Availability Sub-Grouping for Model Comparison Analysis

Separately, we define the feature sets F = {BASE, AHEAD-OF-TIME, REAL-TIME, ALL} to distinguish between features available at any potential inference time

BASE is just the number of blaming tests. This corresponds to the BASELINE heuristic method.
AHEAD-OF-TIME features are properties of the change under suspicion, and are therefore available immediately. Features under CHANGE_CONTENT, CHANGE_METADATA, and CHANGE_TOKENS from Table 1 belong here.
REAL-TIME features are available only as a result of culprit finding processes at play continuously during the decision window, and can become available at different times based on different sources of generation, often after a change has already been suspected by a culprit finder – all other features from Table 1 fall in this set.
ALL features is the union of AHEAD-OF-TIME and REAL-TIME.

This distinction is important: the usefulness of automatically reverting changes is critically dependent on its latency from the point of discovering that a breakage exists. The longer we take to automatically revert the change, the more friction experienced by developers, and the more likely a developer would have had to manually intervene.

Feature Minimization

Given the demonstrated feature lift, we’re also interested in minimizing the features needed to achieve similar performance levels without over-fitting on redundant features. We’ll elaborate on the exact minimized feature sets in the evaluation section for feature analysis, where we use representative features from each logical sub-groups that provide significant lift individually. Thus we have 3 further feature sets, MIN(f) for $f \in F$. Representative features are determined over the test set based on individual feature comparison, left out for brevity in the analysis, rather than the logical category evaluation presented here. We evaluate RF(f), ADA(f), XGB(f), RF(MIN(f)), ADA(MIN(f)), and XGB(MIN(f)) for each feature set $f \in F$.

Thresholding to use Model Prediction Score for Selection

Models produce an output probability score from 0 to 1. To discretize these scores into actual change selections. We dynamically pick a threshold using the TEST dataset to select the threshold that minimizes the bad revert rate while maximizing a positive, non-zero recall.

Hyperparams for ML Models

Using scikit-learn, we define RF as the RandomForestClassifier parameterized with a depth of 16 and ADA as the AdaBoostClassifier with default parameters, each of which provide discretized predictions with the above threshold selection procedure. XGBoost is configured with: objective=‘binary:logistic’, eta=0.05, and max_delta_step = 1.

Empirical Evaluation

We empirically evaluated SafeRevert (the ML based method) against the BASELINE heuristic method. We evaluate: the overall performance of the different ML models considered, the importance features used, and compare the BASELINE method to a selected ML model.

Research Questions

What is the safety and performance of studied methods in the context of the developer workflow?
What is the marginal benefit provided by each feature?
Did the chosen method significantly improve performance over the baseline method while maintaining required safety levels?

Measuring Safety

As mentioned in section 1.4, an incorrectly reverted change causes unacceptably high developer toil. Safety is the likelihood that a method will produce a false positive result by incorrectly categorizing a change as being a culprit and reverting it. This is a referred to as a Bad Revert while correctly reverting a change that introduced a bug is a Good Revert. The total number of reverts is Bad Reverts + Good Reverts. A change that should have been reverted but wasn’t is a Missed Good Revert and a change that was correctly not reverted is a Avoided Bad Revert.

In terms of classic terminology for evaluating binary classifiers:

Good Revert = True Positive (TP)
Bad Revert = False Positive (FP)
Avoided Bad Revert = True Negative (TN)
Missed Good Revert = False Negative (FN)

The safety properties we are most interested are (a) the total number of bad reverts, (b) total number of bad reverts per day, and (c) the bad revert rate. The bad revert rate (BRR) is the percentage of bad reverts out of the total number of reverts $= \frac{FP}{TP + FP}$. This is otherwise known as the False Discovery Rate (FDR). Note, that $FDR = 1 - Precision = 1 - \frac{TP}{TP + FP}$. Thus, safety can either be stated in terms of precision (ex. precision must be above 99%) or in terms of bad revert rate (ex. bad revert rate must be below 1%).

In this evaluation we will consider a method safe if its bad revert rate is below 1%, there are fewer than 14 bad reverts in the final validation set, and there are no more than 5 bad reverts per day.

Why these numbers? Our end goal is a production SafeRevert. Our small team supports a large number of services and users of TAP. We need to minimize toil for the 2 engineers per week who are “oncall” for them. The numbers above were selected to be manageable for us in terms of the overhead required for communicating with our users and investigating the root cause of bad reverts. While these numbers are subjective and dependent on our context, they are meaningful to our team. We expect other teams supporting central CI systems would make similar choices.

Measuring Performance

If a method is deemed safe (meets above criteria) then it is an eligible method to be used to pick changes to revert. To determine whether one safe reversion method is better than another we look at the how many Good Reverts a method is able to achieve out of the total possible good reverts. This corresponds to the metric known as recall $= \frac{TP}{TP + FN}$. The higher a safe method’s recall the better it performs. As with safety, we prefer methods that are consistent and have low variability in their reversion recall per day. We then define performance as the number of breaking changes successfully reverted both in total and per-day.

Evaluation Dataset

Our evaluation dataset consists of roughly 3.5 months of data split between a training, test, and validation set and contains $\thicksim$25,137 unique changes identified by the production culprit finder as culprits. Each row has a boolean indicating whether followup verification confirmed the change was indeed a culprit. This verification continues to be done in the manner described in our 2023 paper (Henderson et al. 2023).

For training and evaluation we produce a time-based split: TRAIN consists of the first 2.5 months, TEST the next $\sim$2 weeks, and VALIDATION the final $\sim$2 weeks. Time based splits avoid cross-contaminating our training with diffuse information about types of breakages that may be based on time-dependent attributes of the codebase. It is important that the training data is uncontaminated with any data from the time period where the evaluation occurs. If it is, the evaluation will not reflect the performance observed in production.

All comparative model and feature evaluation was performed against the TEST set in order to determine our optimal ML model configuration, OPTIMAL. We then evaluate OPTIMAL and BASELINE against the VALIDATION set.

Results

Tables 2-5

Figures 3-4

RQ1: What is the safety and performance of studied methods in the context of the developer workflow?

Summary: The XGB(ALL) was best overall, with a recall of 61.2%, and a bad revert rate of .3%.

Note: This experiment was conducted on the TEST data set as the VALIDATION set was reserved for RQ3.

Table [table:model-comparison] summarizes the critical metrics for safety (bad revert rate) and performance (recall) for the three model types considered: RandomForest [RF], AdaBoost [ADA], and XGBoost [XGB]. Figure [fig:model-comparison] shows the Receiver Operator Curves (ROC) for all 3 model types and their associated area under the curve (AUC) values. The ROC curve better summarizes model performance over a wider range of target objectives than the table which is focused on the safest configuration (minimizing Bad Revert Rate). For instance if a different application had a higher tolerance for bad reversion/false positives the ROC curves show that you could achieve a 90% true positive rate with a 20% false positive rate.

Safety: In general, XGB and ADA both outperformed the RF in terms of safety. Under none of the test configurations did the RF have a safe configuration. XGB and ADA are safe (under our criteria, see Section 4.2) using all the features or the REAL-TIME subset. The safest configuration was XGB(ALL) which had only 5 bad reverts and a bad revert rate of 0.003. XGB(REAL-TIME) and ADA(ALL) has the same or smaller number of bad reverts but a worse rate due to their lower recall.

Performance: The highest performing configuration was XGB(ALL) which had 1980 Good Reverts in the Test set and a recall of 63.2%. The recall rate of XGB(REAL-TIME) was 52.2% indicating that most of the model performance is gained from the REAL-TIME features, not from the AHEAD-OF-TIME features which performed much worse while being unsafe. ADA(ALL) was safe but its recall rate was only 42.2% which lower than both safe XGB configurations.

RQ2: What is the marginal benefit provided by each feature group?

Summary: Historical flakiness data was the most valuable feature set, boosting model recall by 17%, while change metadata was the least important.

Note: This experiment was conducted on the TEST data set as the VALIDATION set was reserved for RQ3.

To determine marginal benefit we perform an ablation study to look at the marginal benefit provided by each feature (as grouped in Table 1). The ablation study was performed using the XGBoost model which was chosen over AdaBoost due to its higher recall as seen in Table [table:model-comparison]. Three marginal benefit experiments were performed:

Measure the performance of the model trained against only the features in a single feature set. We visualize the results in Figure [fig:feature-group-contrib] as an ROC curve. Table [table:feature-ablation] contains the critical safety and performance metrics of Bad Reverts (safety) and Recall (performance) as well as the AUC which summarizes overall model predictive performance – generally a model with a higher AUC will be more performant than one with a lower AUC.
Measure the Positive(+) Lift provided by adding a single feature group to our BASE feature (the number of blaming tests). For instance, we would add the features in CHANGE_CONTENT to create a XGB(CHANGE_CONTENT + BASE) model. The +Lift for Recall is computed as Recall[XGB(CHANGE_CONTENT + BASE)] - Recall[XGB(BASE)] and similar for AUC of the ROC curve. The results of this analysis are shown in Table [table:feature-ablation].
Measure the Negative(–) Lift provided by removing a single feature group from out ALL feature set. For instance we would remove the features in CHANGE_CONTENT to create a XGB(ALL - CHANGE_CONTENT) model. The –Lift for Recall is computed as Recall[XGB(ALL)] - Recall[XGB(ALL - CHANGE_CONTENT)] and similar for AUC of the ROC curve. The results of this analysis are shown in Table [table:feature-ablation].

As discussed in section 3.4.3, we attempted to minimize the feature subsets while providing comparable performance to the full feature sets. Table [table:featureMinimization] contains a breakdown of the performance for these minimal subsets.

RQ3:Did the chosen method improve performance over the baseline method while maintaining required safety levels?

Summary: XGB(ALL) improved overall recall while meeting our safety requirements with a Bad Revert Rate under 1% and an average of 0.6 Bad Reverts per Day. BASELINE(10) had a recall of 26.3% while XGB(ALL) had a recall of 55.7% – an improvement of $\sim 2.1\times$.

Note: We used the reserved VALIDATION data set for this research question.

Table [table:validation-evaluation] shows per day metrics comparing XGB(ALL) against the heuristic BASELINE(10) method. XGB(ALL) was selected as our “OPTIMAL” model to compare against the BASELINE method given its performance in the TEST set as evaluated in RQ1. As a reminder the BASELINE method thresholds the number of blaming tests to a fixed number. BASELINE(10)’s average bad reverts per day is 0.86 and its total Bad Revert Rate (BRR) is 1.5%. XGB(ALL) achieves the overall safety limit with a BRR of 0.5% and has an average bad reverts per day of 0.6 which meets our safety threshold.

XGB(ALL)’s bad revert rate outperforms BASELINE(10), with a recall 2.1 times higher than BASELINE(10). XGB(ALL) made 1,823 total good reverts in the validation set while BASELINE(10) performed 860 good reverts. Based on the data in Table [table:validation-evaluation] XGB(ALL) is both safer and more performant than BASELINE(10) on the validation data set.

Discussion

The proposed method, SafeRevert, is generic and can be used with any culprit finding algorithm. By grouping individual features from different sources of data into logical feature sets (Table 1), performing a detailed feature ablation study (Fig [fig:feature-group-contrib]), and running the model on a minimal set of features (Table [table:featureMinimization]), we hope to provide a template via which teams in other contexts build on when adopting the approach we outline. In particular, while some features used may be Google specific, our feature ablation study can be replicated on different features in other software development organizations. While we don’t expect a team implementing SafeRevert to achieve the exact Recall and Bad Revert Rates we report we do expect this method to out perform the BASELINE method once an appropriate set of features is identified.

Threats to Validity

Our dataset may misrepresent information available at inference time in the forthcoming service as it may include information not available to us at our decision time. This is due to using offline data in the dataset as the production system based on this paper is currently under construction. We adjust this threat by evaluating performance restricted to features available independent of any culprit finding event and restricting our real-time data to a time bound relative to change submission time.

The data presented in this final manuscript differs slightly than the reviewed manuscript. At the time of review approximated 15% of the dataset was lacking verification results (collected using the method described by Henderson (Henderson et al. 2023)). This was disclosed in this section to the reviewers and we made two conservative assumptions: 1) any culprit change lacking verification results was considered a false positive and 2) we assumed the BASELINE method was correct for those changes (inflating the BASELINE methods performance versus the studied ML methods for SafeRevert). Since the peer review was completed, a bug in the verification system was identified and fixed. The bug caused a proportion of incorrect culprit changes to “get stuck” in a queue waiting for an additional test execution due to a typo in a comparison (using > instead of >=). Once the bug was fixed, the research team was able to rerun the study with the additional label data.

Post-rerun we observed: 1) the BASELINE method performed worse and 2) the studied methods were robust to the change in labeling data. In the reviewed manuscript, XGB(ALL) had the following results in RQ3: 13 Total Bad Reverts, 1817 Good Reverts, 0.7% Bad Revert Rate, and 55.8% Recall; BASELINE(5) was compared against and had 8 Total Bad Reverts, 1286 Good Reverts, 0.6% Bad Revert Rate, and 39.4% Recall. Compare against Table [table:validation-evaluation] and Figure [fig:baseline]. We switched to using BASELINE(10) for this final manuscript as it is the production method currently used at Google.

In this paper, we are presenting what we believe to be a novel problem to the wider software engineering community: how to safely choose changes to revert with incomplete but suggestive evidence. This problem relies on identifying these problematic changes. In our case we identify the problematic changes to revert via automated culprit finding (Couder 2008; Ziftci and Ramavajjala 2013; Ziftci and Reardon 2017; Saha and Gligoric 2017; Najafi, Rigby, and Shang 2019; Beheshtian, Bavand, and Rigby 2022; Keenan 2019; An and Yoo 2021; Ocariza 2022; Henderson et al. 2023).

Culprit finding’s development has often occurred outside of the academic literature. The first reference to the process of using binary search to identify a bug was in Yourdon’s 1975 book on Program Design: Page 286, Figure 8.3 titled “A binary search for bugs” (Yourdon 1975). The process is explained in detail in the BUG-HUNTING file in the Linux source tree in 1996 by Larry McVoy (McVoy 1996; Cox, Ness, and McVoy 2023). By 1997, Brian Ness had created a binary search based system for use at Cray with their “Unicos Source Manager” version control system (Ness and Ngo 1997; Cox, Ness, and McVoy 2023). Previously in (Henderson et al. 2023) we had credited Linus Torvalds for the invention based on lack of findable antecedent for his work on the git bisect command (Couder 2008). We regret the error and recognize the above individuals for their important contributions.

Other methods for identifying buggy code or breaking changes have been studied (some widely) in the literature. Fault Localization looks to identify the buggy code that is causing either operational failure or test failures (Agarwal and Agrawal 2014; Wong et al. 2016). Methods for fault localization include: delta debugging (Zeller 1999), statistical coverage based fault localization (J. a. Jones, Harrold, and Stasko 2002; J. A. Jones and Harrold 2005; Lucia et al. 2014; Henderson and Podgurski 2018; Henderson, Podgurski, and Kucuk 2019; Kucuk, Henderson, and Podgurski 2021), information retrieval (which may include historical information) (J. Zhou, Zhang, and Lo 2012; Youm et al. 2015; Ciborowska and Damevski 2022), and program slicing (Podgurski and Clarke 1990; Horwitz and Reps 1992; Ren et al. 2004).

Bug Prediction attempts to predict if a change, method, class, or file is likely to contain a bug (Lewis et al. 2013; Punitha and Chitra 2013; Osman et al. 2017). This idea has conceptual similarities to the work we are doing in this paper where we are using similar metrics to predict whether or not a change which has been implicated by culprit finding is in fact the change that introduced the bug. Our methods could be potentially improved by incorporating the additional features used in the bug prediction work such as the change complexity, code complexity, and object-oriented complexity metrics.

Test Case Selection (Leon and Podgurski 2003; Engström, Runeson, and Skoglund 2010; Z. Q. Zhou 2010; Mondal, Hemmati, and Durocher 2015; Musa et al. 2015; Pan et al. 2022) and Test Case Prioritization (Singh et al. 2012; de S. Campos Junior et al. 2017; De Castro-Cabrera, García-Dominguez, and Medina-Bulo 2020; Pan et al. 2022) are related problems to the change reversion problem we study. Instead of predicting whether or not a change caused a known test failure in Test Case Selection/Prioritization often the change is used to predict whether a given test will fail before it is run. There is a large body work in that uses dynamic information from past test executions (such as code coverage) to inform the selection process. We believe this hints that such information could be highly informative for the change reversion problem as well.

Finally, there are a family of methods for finding bug inducing commits for the purpose of supporting studies that data mine software repositories (Śliwerski, Zimmermann, and Zeller 2005; Rodríguez-Pérez, Robles, and González-Barahona 2018; Borg et al. 2019; Wen et al. 2019; An et al. 2023). These methods typically used to conduct a historical analysis of a repository rather than as an online detection as in culprit finding.

Conclusion

We presented: SafeRevert a method for improving systems that automatically revert changes that break tests. SafeRevert was developed as a way to improve the number bad changes automatically reverted while maintaining safety (rarely reverting good changes). To evaluate SafeRevert, we performed an empirical evaluation comparing the performance of SafeRevert against the baseline method that is currently utilized in production which utilizes a simple heuristic to determine if a change is safe to revert: the number of tests which “blame” the culprit change. When evaluating RQ3 in Section 5, it was observed that the XGB(ALL) configuration of SafeRevert doubled the number changes reverted while reducing the number of bad reverts performed (see Table [table:validation-evaluation]).

While it is unlikely that a replication study in a different development environment would reproduce our exact results we do expect based on the robust difference observed between SafeRevert and BASELINE that SafeRevert (or similar ML based method) will be able to improve the number of changes eligible for automatic reversion. We hope that by introducing this problem to the larger software engineering community that new and innovative approaches to solving it will be developed.

References

Agarwal, Pragya, and Arun Prakash Agrawal. 2014. “Fault-Localization Techniques for Software Systems: A Literature Review.” SIGSOFT Softw. Eng. Notes 39 (5): 1–8. https://doi.org/10.1145/2659118.2659125.
An, Gabin, Jingun Hong, Naryeong Kim, and Shin Yoo. 2023. “Fonte: Finding Bug Inducing Commits from Failures.” arXiv. http://arxiv.org/abs/2212.06376.
An, Gabin, and Shin Yoo. 2021. “Reducing the Search Space of Bug Inducing Commits Using Failure Coverage.” In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 1459–62. Athens Greece: ACM. https://doi.org/10.1145/3468264.3473129.
Ananthanarayanan, Sundaram, Masoud Saeida Ardekani, Denis Haenikel, Balaji Varadarajan, Simon Soriano, Dhaval Patel, and Ali Reza Adl-Tabatabai. 2019. “Keeping Master Green at Scale.” Proceedings of the 14th EuroSys Conference 2019. https://doi.org/10.1145/3302424.3303970.
Beheshtian, Mohammad Javad, Amir Hossein Bavand, and Peter C. Rigby. 2022. “Software Batch Testing to Save Build Test Resources and to Reduce Feedback Time.” IEEE Transactions on Software Engineering 48 (8): 2784–2801. https://doi.org/10.1109/TSE.2021.3070269.
Ben-Or, Michael, and Avinatan Hassidim. 2008. “The Bayesian Learner Is Optimal for Noisy Binary Search (and Pretty Good for Quantum as Well).” In 2008 49th Annual IEEE Symposium on Foundations of Computer Science, 221–30. Philadelphia, PA, USA: IEEE. https://doi.org/10.1109/FOCS.2008.58.
Bland, Mike. 2012. “The Chris/Jay Continuous Build.” Personal {{Website}}. Mike Bland’s Blog. https://mike-bland.com/2012/06/21/chris-jay-continuous- build.html.
Borg, Markus, Oscar Svensson, Kristian Berg, and Daniel Hansson. 2019. “SZZ Unleashed: An Open Implementation of the SZZ Algorithm - Featuring Example Usage in a Study of Just-in-Time Bug Prediction for the Jenkins Project.” In Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation - MaLTeSQuE 2019, 7–12. Tallinn, Estonia: ACM Press. https://doi.org/10.1145/3340482.3342742.
Chen, Tianqi, and Carlos Guestrin. 2016. “XGBoost: A Scalable Tree Boosting System.” In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–94. San Francisco California USA: ACM. https://doi.org/10.1145/2939672.2939785.
Ciborowska, Agnieszka, and Kostadin Damevski. 2022. “Fast Changeset-Based Bug Localization with BERT.” In Proceedings of the 44th International Conference on Software Engineering, 946–57. Pittsburgh Pennsylvania: ACM. https://doi.org/10.1145/3510003.3510042.
Couder, Christian. 2008. “Fighting Regressions with Git Bisect.” The Linux Kernel Archives 4 (5). https://www. kernel. org/pub/software/scm/git/doc s/git- bisect-lk2009.html.
Cox, Russ, Brian Ness, and Larry McVoy. 2023. “Comp.lang.compilers "Binary Search Debugging of Compilers".” https://groups.google.com/g/comp.compilers/c/vGh4s3HBQ-s/m/ Chvpu7vTAgAJ.
De Castro-Cabrera, M. Del Carmen, Antonio García-Dominguez, and Inmaculada Medina-Bulo. 2020. “Trends in Prioritization of Test Cases: 2017-2019.” Proceedings of the ACM Symposium on Applied Computing, 2005–11. https://doi.org/10.1145/3341105.3374036.
de S. Campos Junior, Heleno, Marco Antônio P Araújo, José Maria N David, Regina Braga, Fernanda Campos, and Victor Ströele. 2017. “Test Case Prioritization: A Systematic Review and Mapping of the Literature.” In Proceedings of the 31st Brazilian Symposium on Software Engineering, 34–43. New York, NY, USA: ACM. https://doi.org/10.1145/3131151.3131170.
Engström, Emelie, Per Runeson, and Mats Skoglund. 2010. “A Systematic Review on Regression Test Selection Techniques.” Information and Software Technology 52 (1): 14–30. https://doi.org/10.1016/j.infsof.2009.07.001.
Fowler, Martin. 2006. “Continuous Integration.” https://martinfowler.com/articles/ continuousIntegration.html.
Gupta, Pooja, Mark Ivey, and John Penix. 2011. “Testing at the Speed and Scale of Google.” http://google-engtools.blogspot.com/2011/06/testing-at- speed-and-scale-of-google.html.
Henderson, Tim A. D., Bobby Dorward, Eric Nickell, Collin Johnston, and Avi Kondareddy. 2023. “Flake Aware Culprit Finding.” In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE. https://hackthology.com/flake-aware-culprit-finding.html. https://doi.org/10.1109/ICST57152.2023.00041.
Henderson, Tim A. D., and Andy Podgurski. 2018. “Behavioral Fault Localization by Sampling Suspicious Dynamic Control Flow Subgraphs.” In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), 93–104. Vasteras: IEEE. https://hackthology.com/behavioral-fault-localization-by-sampling-suspicious-dynamic-control-flow-subgraphs.html. https://doi.org/10.1109/ICST.2018.00019.
Henderson, Tim A. D., Andy Podgurski, and Yigit Kucuk. 2019. “Evaluating Automatic Fault Localization Using Markov Processes.” In 2019 19th International Working Conference on Source Code Analysis and Manipulation (SCAM), 115–26. Cleveland, OH, USA: IEEE. https://hackthology.com/evaluating-automatic-fault-localization-using-markov-processes.html https://doi.org/10.1109/SCAM.2019.00021.
Herzig, Kim, and Nachiappan Nagappan. 2015. “Empirically Detecting False Test Alarms Using Association Rules.” In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, 39–48. Florence, Italy: IEEE. https://doi.org/10.1109/ICSE.2015.133.
Horwitz, S, and T Reps. 1992. “The Use of Program Dependence Graphs in Software Engineering.” In International Conference on Software Engineering, 9:349. Springer. http://portal.acm.org/citation.cfm?id=24041&dl=.
Jones, J.a., M. J. Harrold, and J. Stasko. 2002. “Visualization of Test Information to Assist Fault Localization.” Proceedings of the 24th International Conference on Software Engineering. ICSE 2002. https://doi.org/10.1145/581339.581397.
Jones, James A., and Mary Jean Harrold. 2005. “Empirical Evaluation of the Tarantula Automatic Fault-localization Technique.” In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, 273–82. New York, NY, USA: ACM. https://doi.org/10.1145/1101908.1101949.
Keenan, James. 2019. “James E. Keenan - "Multisection: When Bisection Isn’t Enough to Debug a Problem".” https://www.youtube.com/watch?v=05CwdTRt6AM.
Kucuk, Yigit, Tim A. D. Henderson, and Andy Podgurski. 2021. “Improving Fault Localization by Integrating Value and Predicate Based Causal Inference Techniques.” In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 649–60. Madrid, ES: IEEE. https://doi.org/10.1109/ICSE43902.2021.00066.
Leon, D., and A. Podgurski. 2003. “A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases.” In 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003., 2003-Janua:442–53. IEEE. https://doi.org/10.1109/ISSRE.2003.1251065.
Leong, Claire, Abhayendra Singh, Mike Papadakis, Yves Le Traon, and John Micco. 2019. “Assessing Transition-Based Test Selection Algorithms at Google.” In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 101–10. IEEE. https://doi.org/10.1109/ICSE-SEIP.2019.00019.
Lewis, Chris, Zhongpeng Lin, Caitlin Sadowski, and Xiaoyan Zhu. 2013. “Does Bug Prediction Support Human Developers? Findings from a Google Case Study.” In Proceedings of the …, 372–81. http://dl.acm.org/citation.cfm?id=2486838.
Lucia, David Lo, Lingxiao Jiang, Ferdian Thung, and Aditya Budi. 2014. “Extended Comprehensive Study of Association Measures for Fault Localization.” Journal of Software: Evolution and Process 26 (2): 172–219. https://doi.org/10.1002/smr.1616.
Machalica, Mateusz, Alex Samylkin, Meredith Porth, and Satish Chandra. 2019. “Predictive Test Selection.” In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 91–100. IEEE. https://doi.org/10.1109/ICSE-SEIP.2019.00018.
McVoy, Larry. 1996. “BUG-HUNTING.” linux/1.3.73. https://elixir.bootlin.com/linux/1.3.73/source/ Documentation/BUG-HUNTING.
Memon, Atif, Zebao Gao, Bao Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. 2017. “Taming Google-scale Continuous Testing.” In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), 233–42. Piscataway, NJ, USA: IEEE. https://doi.org/10.1109/ICSE-SEIP.2017.16.
Micco, John. 2012. “Tools for Continuous Integration at Google Scale.” Tech {{Talk}}. Google NYC. https://youtu.be/KH2_sB1A6lA.
———. 2013. “Continuous Integration at Google Scale.” Lecture. EclipseCon 2013. https://web.archive.org/web/20140705215747/https:// www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/2013- 03-24%20Continuous%20Integration%20at%20Google%20Scale.pdf.
Mondal, Debajyoti, Hadi Hemmati, and Stephane Durocher. 2015. “Exploring Test Suite Diversification and Code Coverage in Multi-Objective Test Case Selection.” 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings, 1–10. https://doi.org/10.1109/ICST.2015.7102588.
Musa, Samaila, Abu Bakar Md Sultan, Abdul Azim Bin Abd-Ghani, and Salmi Baharom. 2015. “Regression Test Cases Selection for Object-Oriented Programs Based on Affected Statements.” International Journal of Software Engineering and Its Applications 9 (10): 91–108. https://doi.org/10.14257/ijseia.2015.9.10.10.
Najafi, Armin, Peter C. Rigby, and Weiyi Shang. 2019. “Bisecting Commits and Modeling Commit Risk During Testing.” In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 279–89. New York, NY, USA: ACM. https://doi.org/10.1145/3338906.3338944.
Najafi, Armin, Weiyi Shang, and Peter C. Rigby. 2019. “Improving Test Effectiveness Using Test Executions History: An Industrial Experience Report.” In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 213–22. IEEE. https://doi.org/10.1109/ICSE-SEIP.2019.00031.
Ness, B., and V. Ngo. 1997. “Regression Containment Through Source Change Isolation.” In Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC’97), 616–21. Washington, DC, USA: IEEE Comput. Soc. https://doi.org/10.1109/CMPSAC.1997.625082.
Ocariza, Frolin S. 2022. “On the Effectiveness of Bisection in Performance Regression Localization.” Empirical Software Engineering 27 (4): 95. https://doi.org/10.1007/s10664-022-10152-3.
Osman, Haidar, Mohammad Ghafari, Oscar Nierstrasz, and Mircea Lungu. 2017. “An Extensive Analysis of Efficient Bug Prediction Configurations.” In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering, 107–16. Toronto Canada: ACM. https://doi.org/10.1145/3127005.3127017.
Pan, Rongqi, Mojtaba Bagherzadeh, Taher A. Ghaleb, and Lionel Briand. 2022. “Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review.” Empirical Software Engineering 27 (2): 29. https://doi.org/10.1007/s10664-021-10066-6.
Parry, Owain, Gregory M. Kapfhammer, Michael Hilton, and Phil McMinn. 2022. “A Survey of Flaky Tests.” ACM Transactions on Software Engineering and Methodology 31 (1): 1–74. https://doi.org/10.1145/3476105.
Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, et al. 2011. “Scikit-Learn: Machine Learning in Python.” Journal of Machine Learning Research 12: 2825–30.
Pelc, Andrzej. 2002. “Searching Games with Errorsfifty Years of Coping with Liars.” Theoretical Computer Science 270 (1-2): 71–109. https://doi.org/10.1016/S0304-3975(01)00303-6.
Podgurski, A, and L A Clarke. 1990. “A Formal Model of Program Dependences and Its Implications for Software Testing, Debugging, and Maintenance.” IEEE Transactions of Software Engineering 16 (9): 965–79. https://doi.org/10.1109/32.58784.
Potvin, Rachel, and Josh Levenberg. 2016. “Why Google Stores Billions of Lines of Code in a Single Repository.” Communications of the ACM 59 (7): 78–87. https://doi.org/10.1145/2854146.
Punitha, K., and S. Chitra. 2013. “Software Defect Prediction Using Software Metrics - A Survey.” In 2013 International Conference on Information Communication and Embedded Systems (ICICES), 555–58. Chennai: IEEE. https://doi.org/10.1109/ICICES.2013.6508369.
Ren, Xiaoxia, Fenil Shah, Frank Tip, Barbara G. Ryder, and Ophelia Chesley. 2004. “Chianti: A Tool for Change Impact Analysis of Java Programs.” In Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, 432–48. Vancouver BC Canada: ACM. https://doi.org/10.1145/1028976.1029012.
Rivest, R. L., A. R. Meyer, and D. J. Kleitman. 1978. “Coping with Errors in Binary Search Procedures (Preliminary Report).” In Proceedings of the Tenth Annual ACM Symposium on Theory of Computing - STOC ’78, 227–32. San Diego, California, United States: ACM Press. https://doi.org/10.1145/800133.804351.
Rodríguez-Pérez, Gema, Gregorio Robles, and Jesús M. González-Barahona. 2018. “Reproducibility and Credibility in Empirical Software Engineering: A Case Study Based on a Systematic Literature Review of the Use of the SZZ Algorithm.” Information and Software Technology 99 (July): 164–76. https://doi.org/10.1016/j.infsof.2018.03.009.
Saha, Ripon, and Milos Gligoric. 2017. “Selective Bisection Debugging.” In Fundamental Approaches to Software Engineering, edited by Marieke Huisman and Julia Rubin, 10202:60–77. Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-662-54494-5_4.
Singh, Yogesh, Arvinder Kaur, Bharti Suri, and Shweta Singhal. 2012. “Systematic Literature Review on Regression Test Prioritization Techniques.” Informatica (Slovenia) 36 (4): 379–408. https://doi.org/10.31449/inf.v36i4.420.
Śliwerski, Jacek, Thomas Zimmermann, and Andreas Zeller. 2005. “When Do Changes Induce Fixes?” ACM SIGSOFT Software Engineering Notes 30 (4): 1. https://doi.org/10.1145/1082983.1083147.
Wang, Kaiyuan, Greg Tener, Vijay Gullapalli, Xin Huang, Ahmed Gad, and Daniel Rall. 2020. “Scalable Build Service System with Smart Scheduling Service.” In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, 452–62. Virtual Event USA: ACM. https://doi.org/10.1145/3395363.3397371.
Wen, Ming, Rongxin Wu, Yepang Liu, Yongqiang Tian, Xuan Xie, Shing-Chi Cheung, and Zhendong Su. 2019. “Exploring and Exploiting the Correlations Between Bug-Inducing and Bug-Fixing Commits.” In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 326–37. Tallinn Estonia: ACM. https://doi.org/10.1145/3338906.3338962.
Wong, W. Eric, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. “A Survey on Software Fault Localization.” IEEE Transactions on Software Engineering 42 (8): 707–40. https://doi.org/10.1109/TSE.2016.2521368.
Youm, Klaus Changsun, June Ahn, Jeongho Kim, and Eunseok Lee. 2015. “Bug Localization Based on Code Change Histories and Bug Reports.” In 2015 Asia-Pacific Software Engineering Conference (APSEC), 190–97. New Delhi: IEEE. https://doi.org/10.1109/APSEC.2015.23.
Yourdon, E. 1975. Techniques of Program Design. New Jersey: Prentice-Hall.
Zeller, Andreas. 1999. “Yesterday, My Program Worked. Today, It Does Not. Why?” SIGSOFT Softw. Eng. Notes 24 (6): 253–67. https://doi.org/10.1145/318774.318946.
Zhou, Jian, Hongyu Zhang, and David Lo. 2012. “Where Should the Bugs Be Fixed? More Accurate Information Retrieval-Based Bug Localization Based on Bug Reports.” Proceedings - International Conference on Software Engineering, 14–24. https://doi.org/10.1109/ICSE.2012.6227210.
Zhou, Zhi Quan. 2010. “Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing.” Proceedings - International Computer Software and Applications Conference, 208–13. https://doi.org/10.1109/COMPSACW.2010.43.
Ziftci, Celal, and Vivek Ramavajjala. 2013. “Finding Culprits Automatically in Failing Builds - i.e. Who Broke the Build?” https://www.youtube.com/watch?v=SZLuBYlq3OM.
Ziftci, Celal, and Jim Reardon. 2017. “Who Broke the Build? Automatically Identifying Changes That Induce Test Failures in Continuous Integration at Google Scale.” Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017, 113–22. https://doi.org/10.1109/ICSE-SEIP.2017.13.

Both the per-test error rate and the per-change error rates quoted above are drawn from the same 28 day window. The Postsubmit result which triggers culprit finding may itself have been a flake, so there were spurious suspect ranges in the same period. These rates may differ slightly from the numbers reported in (Henderson et al. 2023) as they reflect our most recent data as of 2023-11-12.↩︎
There are some legacy tests which are allowed to use the network but there is an on-going effort to clean up their usage.↩︎
Previously, we referred to these cycles as “Milestones” in most of the previous literature but we are gradually changing our internal nomenclature as we evolve TAP Postsubmit’s testing model.↩︎
https://scikit-learn.org/↩︎
https://xgboost.readthedocs.io/en/stable/index.html ↩︎

Flake Aware Culprit Finding

2023-04-17T00:00:00-04:00

Tim A. D. Henderson, Bobby Dorward, Eric Nickell, Collin Johnston, and Avi Kondareddy. Flake Aware Culprit Finding. ICST Industry Track 2023.
DOI. PDF. WEB. Google Research Preprint.

Note

This is a conversion from a latex paper I wrote. If you want all formatting correct you should read the pdf version.

Abstract

When a change introduces a bug into a large software repository, there is often a delay between when the change is committed and when bug is detected. This is true even when the bug causes an existing test to fail! These delays are caused by resource constraints which prevent the organization from running all of the tests on every change. Due to the delay, a Continuous Integration system needs to locate buggy commits. Locating them is complicated by flaky tests that pass and fail non-deterministically. The flaky tests introduce noise into the CI system requiring costly reruns to determine if a failure was caused by a bad code change or caused by non-deterministic test behavior. This paper presents an algorithm, Flake Aware Culprit Finding, that locates buggy commits more accurately than a traditional bisection search. The algorithm is based on Bayesian inference and noisy binary search, utilizing prior information about which changes are most likely to contain the bug. A large scale empirical study was conducted at Google on 13,000+ test breakages. The study evaluates the accuracy and cost of the new algorithm versus a traditional deflaked bisection search.

Introduction

Fast, collaborative, large scale development is enabled by the use of Continuous Integration in a mono-repository (monorepo) environment where all of the source code is stored in a single shared repository (Potvin and Levenberg 2016). In a monorepo, all developers share a single source of truth for the state of the code and most builds are made from “head” (the most recent commit to the repository) using the source code for all libraries and dependencies rather than versioned pre-compiled archives. This enables (among other things) unified versioning, atomic changes, large scale refactorings, code re-use, cross team collaboration, and flexible code ownership boundaries.

Very large scale monorepos (such as those at Google (Potvin and Levenberg 2016; Memon et al. 2017), Microsoft (Herzig and Nagappan 2015), and Facebook (Machalica et al. 2019)) require advanced systems for ensuring changes submitted to the repository are properly tested and validated. Continuous Integration (CI) (Fowler 2006) is a system and practice of automatically integrating changes into the code repository that serves as the source of truth for the organization. In modern environments, integration involves not only performing the textual merge required to add the change but also verification tasks such as ensuring the software compiles and the automated tests pass both on the new change before it is integrated and after it has been incorporated into the main development branch.

Due to resource constraints, tests are only run at specific versions (called milestones at Google) and many tests may be skipped because they were not affected by the intervening changes (as determined by static build dependencies) (Gupta, Ivey, and Penix 2011). This process is illustrated in Figure [fig:ci-timeline]. A test is considered passing at a version $b$ if and only if it has a passing result at the most recent preceding affecting version. Formally, if there exists a version $a$ where the test is passing and $a \le b$ and there does not exist an intervening version $c$, $a < c \le b$, which affects the test then the test is passing at version $a$. A project’s status is considered passing at a version $b$ if all tests are passing (using the above definition) at version $b$.

If a test $t$ is failing (consistently) at a milestone version $m$, it may not necessarily mean that version $m$ introduced a change which broke test $t$. As shown in Figure [fig:ci-timeline] one or more changes between $m$ and the previous milestone at which test $t$ was run may have introduced a breaking change. Figuring out which version introduced the breakage is called culprit finding.

Many companies now employ automated culprit finding systems. These systems typically implement a (possibly n-way) “bisect search” similar to the one built into the Git version control system (Couder 2008). Git’s bisect command runs a binary search on the versions between the first known breaking version and the last known passing version to identify the version which broke the test.

In the Google environment, there are two problems with a traditional bisection algorithm (Couder 2008): (1) it does not account for flaky (non-deterministic) tests, and (2) when using $k$-reruns to deflake the search accuracy plateaus while build cost continues to increase linearly. This paper proposes a new method for culprit finding which is both robust to non-deterministic (flaky) test behavior and can identify the culprit with only logarithmic builds in the best case by using prior information to accelerate the search.

Contributions

A Flake Aware Culprit Finding (FACF) algorithm.
A mathematical model for culprit finding adjusting for non-deterministic test behavior.
A large scale empirical study of FACF on Google’s mono repository comparing its performance against the traditional bisect algorithm.
The study also evaluates the effectiveness: (a) optimizations for FACF, and (b) adding deflaking runs to Bisect.

Background

At Google, the Test Automation Platform (TAP) runs most tests that can run on a single machine (and do not use the network except for the loopback interface)¹ and can run in under 15 minutes.² Figure [fig:ci-timeline] illustrates at a high level of how TAP works. First, (not illustrated) tests are grouped into logical projects by the teams that own them. Then, (for accounting and quota allocation purposes) those projects are grouped into very high level groupings called Quota Buckets. Each Quota Bucket is scheduled independently when build resources are available. This means that TAP does not run every test at every version. Instead it only runs tests periodically at select versions called “Milestones” (Memon et al. 2017; Micco 2013; Leong et al. 2019).

When scheduling a Milestone, all the tests that are built and run with the same command line flags to the build system³ are grouped into a set. That set is filtered to remove tests that haven’t been affected by a change to themselves or a build level dependency (Gupta, Ivey, and Penix 2011; Micco 2013, 2012). This is a very coarse grained form of static dependence-based test selection (Binkley 1992; Horwitz and Reps 1992; Tip 1995). Google’s use of build dependence test selection has been influential and it is now also used at a number of other corporations who have adopted Bazel or similar tools (Ananthanarayanan et al. 2019; Machalica et al. 2019). Finally, that (potentially very large) set of “affected” tests is sent to the Build System (Wang et al. 2020) which batches them into efficiently sized groups and then builds and runs them when there is capacity.

Since tests are only run periodically even with accurate build level dependence analysis to drive test selection, there are usually multiple changes which may have introduced a fault in the software or a bug in the test. Finding these “bug inducing changes” – which we call culprits – is the subject of this paper. The process of finding these changes is aggravated by any non-deterministic behavior of the tests and the build system running them.

Flaky Tests

When a test behaves non-deterministically, we refer to it, using the standard term, as a flaky test. A flaky test is one that passes or fails non-deterministically with no changes to the code of either the test or the system under test. At Google, we consider all tests to be at least slightly flaky as the machines that tests are running on could fail in ways that get classified as a test failure (such as resource exhaustion due to a noisy neighbor resulting in a test exceeding its time limit). We encourage everyone to adopt this conservative view for the purposes of identifying culprits.

An important input to the culprit finding algorithm we are presenting is an estimate of how flaky a test might be, as represented by an estimated probability of the test failing due to a flaky failure. We will denote the estimated probability of failure due to a flake for a test $t$ as $\hat{f}_t$. The actual probability of failure due to a flake will be denoted $f_t$ (it is impossible in practice to know $f_t$). At Google, we estimate $\hat{f}_t$ based on the historical failure rate of the test. Apple and Facebook recently published on their work in on estimating flake rates (Kowalczyk et al. 2020; Machalica et al. 2020). One challenge with flake rate estimation is that a single change to the code can completely change the flakiness characteristics of a test in unpredictable ways. Making the flake rate estimation sensitive to recent changes is an ongoing challenge and one we partially solve in this work by doing live re-estimation of the flake rate during the culprit finding process itself.

Culprit Finding

Informally, Culprit Finding is the problem of identifying a version $c$ between two versions $a$ and $b$ such that $a < c \le b$ where a given test $t$ was passing at $a$, failing at $b$, and $c$ is the version which introduced the fault. At Google we consider the passing status of a test to be 100% reliable and therefore version $a$ cannot be a culprit but version $b$ could be the culprit. For simplicity, this paper only considers searching for the culprit for a single test $t$ (which failed at $b$) but in principle it can be extended to a set of tests $T$.

Google’s version control system, Piper (Potvin and Levenberg 2016), stores the mainline history as a linear list of versions. Each version gets a monotonically, but not sequentially, increasing number as its version number called the Changelist Number or CL number for short. Thus, our implementation and experiments do not consider branches and merges as would be necessary for a Distributed Version Control System (DVCS) such as git. However, only trivial changes to the mathematics are needed to apply our approach to such an environment (Couder 2008).

Thus, the versions larger than $a$ and smaller or equal to $b$ are an ordered list of items we will call the search range and denote as $S = \left\{s_1, ..., s_{n-1}, s_n\right\}$ where $s_n = b$. During the search, test executions will be run (and may be run in parallel) at various versions. The unordered set of executions at version $s_i$ will be denoted $X_i$.

For the purposes of culprit finding, based on the status of failure detected, the build/test execution statuses are mapped to one of 2 possible states: PASS or FAIL. We denote (for version $s_i$): the number of statuses that map to PASS as $\texttt{\small PASSES}(X_i)$ and the number of statuses that map to FAIL as $\texttt{\small FAILS}(X_i)$.

Bisection

The Bisect algorithm is a specialized form of binary search. It runs tests (or more generally any operation on the code that can return PASS or FAIL) to attempt to identify which commit introduced a bug. Algorithm [alg:bisect] gives the psuedo-code for this procedure. Bisect is now commonly implemented in most version control systems (see git bisect and hg bisect as common examples). In the case of Distributed Version Control Systems (DVCS) systems such as ‘git‘ and ‘hg‘ the implementation of bisect is somewhat more complex as they model version history as a directed acyclic graph (Couder 2008).

The algorithm presented in Alg [alg:bisect] has two important changes from the standard binary search. First, it takes into account that a test can fail and pass at the same version and the firstFail function specifically looks for a failure after the lastPass. Second, it takes a parameter $k$ which allows the user to specify that each test execution be run $k$ times. This allows the user to “deflake” the test executions by rerunning failures. Our empirical evaluation (Section 5) examines both the cost and effectiveness of this simple modification. While this simple modification to bisect may be reasonably effective for moderately flaky tests, it does not explicitly model the effect of non-deterministic failures on the likelihood of a suspect to be the culprit.

Guessing Games and Noisy Searches

One can view the culprit finding scenario as a kind of guessing game. The questioner guesses the test always fails at a given suspect version. A test execution at that suspect acts as a kind of “responder” or “oracle” which answers. The test execution is an unreliable oracle which may answer FAIL when it should have answered PASS. But, the oracle never answers PASS when it should have answered FAIL – note the asymmetry.

This kind of guessing game is a Rényi-Ulam game, which has been well characterized by the community (Pelc 2002). The main differences from the “classical” games presented by Rényi and separately by Ulam is (a) the format of the questions and (b) the limitations placed on the oracle responding incorrectly. In a common formulation of the Rényi-Ulam game, the questioner is trying to “guess the number” between (for instance) 1 and 1,000,000. Each question is allowed to be any subset of numbers in the search range. The responder or oracle is allowed to lie at most a fixed number of times (for instance once or twice). This formulation has allowed the community to find analytic solutions to determine the number of questions in the optimal strategy. We refer the reader to the survey from Pelc for an overview of the field and its connection with error correcting codes (Pelc 2002). The culprit finding scenario is “noisy binary search” scenario where the oracle only lies for one type of response. Rivest et al. (Rivest, Meyer, and Kleitman 1978) provide a theoretical treatment of this “half-lie” variant (as well as the usual variation).

Bayesian Binary Search

Ben Or and Hassidim (Ben-Or and Hassidim 2008) noted the connection between these “Noisy Binary Search” problems and Bayesian inference. Our work builds off of their formulation and applies it to the culprit finding setting. In the Bayesian setting, consider the probability each suspect is the culprit and the probability there is no culprit (i.e. the original failure was caused by a flake). These probabilities together form a distribution: \[(\textrm{Pr}\left[{\text{there is no culprit}}\right]) + \sum_{i=1}^{n} \textrm{Pr}\left[{s_i \text{ is the culprit}}\right] = 1\]

We represent this distribution with $\textrm{Pr}\left[{C_i}\right]$ with $i=1..(n+1)$ where \[\begin{aligned} \begin{split} \textrm{Pr}\left[{C_i}\right] &= \textrm{Pr}\left[{s_i \text{ is the culprit}}\right], \; \forall \; i \in [1, n] \\ \textrm{Pr}\left[{C_{n+1}}\right] &= \textrm{Pr}\left[{\text{there is no culprit}}\right] \\ \end{split}\end{aligned}\]

Initially when the original failure is first detected at $s_n$ and the search range is created, the probability distribution is uniform: $\textrm{Pr}\left[{C_i}\right] = \frac{1}{(n+1)}$ for all $i \in [1, n+1]$. Now consider new evidence $E$ arriving. In the Bayesian model, this would update our probability given this new evidence: \[\begin{aligned} \textrm{Pr}\left[{C_i | E}\right] = \frac{\textrm{Pr}\left[{E | C_i}\right] \cdot \textrm{Pr}\left[{C_i}\right]}{\textrm{Pr}\left[{E}\right]} \label{eq:bayes}\end{aligned}\]

In the above equation, the $\textrm{Pr}\left[{C_i}\right]$ is the prior probability and the probability $\textrm{Pr}\left[{C_i | E}\right]$ is the posterior probability. In the culprit finding context, the new evidence is typically newly observed test executions. However, there is no reason we cannot take into account other information as well.

In the Bayesian framework, new evidence is applied iteratively. In each iteration, the prior probability is the posterior probability of the previous iteration. Since multiplication is commutative, multiple pieces of evidence can be applied in any order to arrive at the same posterior probability: \[\begin{aligned} \begin{split} \textrm{Pr}\left[{C_i | E_1, E_2}\right] &= \frac{\textrm{Pr}\left[{E_2 | C_i}\right]}{\textrm{Pr}\left[{E_2}\right]} \frac{\textrm{Pr}\left[{E_1 | C_i}\right]}{\textrm{Pr}\left[{E_1}\right]} \textrm{Pr}\left[{Ci}\right] \\ &= \frac{\textrm{Pr}\left[{E_1 | C_i}\right]}{\textrm{Pr}\left[{E_1}\right]} \frac{\textrm{Pr}\left[{E_2 | C_i}\right]}{\textrm{Pr}\left[{E_2}\right]} \textrm{Pr}\left[{Ci}\right] \end{split}\end{aligned}\]

After the current posterior probability has been calculated, it is used to select the next suspect to execute a test at. In the traditional Bayes approach, the suspect to examine is the one with the maximum posterior probability. However, in the context of a noisy binary search, the suspects are ordered so a PASS at suspect $s_i$ indicates a PASS at all suspects prior to $s_i$. Following Ben Or and Hassidim (Ben-Or and Hassidim 2008), we convert the posterior distribution to a cumulative probability distribution and select the first suspect with a cumulative probability $\geq 0.5$.

Flake Aware Culprit Finding

Now we introduce our algorithm, Flake Aware Culprit Finding (FACF), in the Bayesian framework of Ben Or and Hassidim (Ben-Or and Hassidim 2008). As above, let $\textrm{Pr}\left[{C_i}\right]$ be the probability that suspect $s_i$ is the culprit and $\textrm{Pr}\left[{C_{n+1}}\right]$ be the probability there is no culprit. These probabilities are initialized to some suitable initial distribution such as the uniform distribution.

To update the distribution with new evidence $E$ in the form of a pass or fail for some suspect $k$, apply Bayes rule (Equation [eq:bayes]). Note that $\textrm{Pr}\left[{E}\right]$ can be rewritten by Bayes rule in terms of a sum by marginalizing over the likelihood that each suspect $j$ could be the culprit: \[\begin{aligned} \textrm{Pr}\left[{E}\right] = \sum_{j=1}^{n} \textrm{Pr}\left[{E | C_j}\right] \textrm{Pr}\left[{C_j}\right]\end{aligned}\]

There are two types of evidence: test executions with state PASS or FAIL at suspect $i$. We indicate these as the events $P_i$ and $F_i$ respectively. The marginalized probability of a PASS at $i$ given there is a culprit at $j$ is: \[\begin{aligned} \textrm{Pr}\left[{P_i | C_j}\right] = \begin{cases} 1 - \hat{f}_t & \text{if } i < j \\ 0 & \text{otherwise} \end{cases} \label{eq:pass}\end{aligned}\]

The marginalized probability of a FAIL at $i$ given there is a culprit at $j$ is: \[\begin{aligned} \textrm{Pr}\left[{F_i | C_j}\right] = \begin{cases} \hat{f}_t & \text{if } i < j \\ 1 & \text{otherwise} \end{cases} \label{eq:fail}\end{aligned}\]

With Equation [eq:pass], we write a Bayesian update for PASS at suspect $k$ as the following. Given \[\begin{aligned} \textrm{Pr}\left[{C_i | P_k}\right] = \frac{\textrm{Pr}\left[{P_k | C_i}\right] \textrm{Pr}\left[{C_i}\right]}{ \textrm{Pr}\left[{P_k}\right] }\end{aligned}\]

let \[\begin{aligned} \begin{split} \textrm{Pr}\left[{P_k}\right] &= \sum_{j=1}^{n} \textrm{Pr}\left[{P_k | C_j}\right] \textrm{Pr}\left[{C_j}\right] \\ &= \sum_{j=k+1}^{n} (1 - \hat{f}_t) \textrm{Pr}\left[{C_j}\right] \end{split}\end{aligned}\]

cdf = cumulativeDistribution(dist) thresholds = $\left\{ \frac{i}{(k+1)} \; | \; i=1..k \right\}$ tidx = 0 runs = list() runs

then \[\begin{aligned} \textrm{Pr}\left[{C_i | P_k}\right] = \begin{cases} \frac{(1 - \hat{f}_t) \textrm{Pr}\left[{C_i}\right]}{\textrm{Pr}\left[{P_k}\right]} & \text{if } k < i \\ 0 & \text{otherwise} \\ \end{cases}\end{aligned}\]

The update using Equation [eq:fail] for FAIL at suspect $k$ is similar. Given \[\begin{aligned} \textrm{Pr}\left[{C_i | F_k}\right] = \frac{\textrm{Pr}\left[{F_k | C_i}\right] \textrm{Pr}\left[{C_i}\right]}{ \sum_{j=1}^{n} \textrm{Pr}\left[{F_k | C_j}\right] \textrm{Pr}\left[{C_j}\right] }\end{aligned}\]

let \[\begin{aligned} \begin{split} \textrm{Pr}\left[{F_k}\right] &= \sum_{j=1}^{n} \textrm{Pr}\left[{F_k | C_j}\right] \textrm{Pr}\left[{C_j}\right] \\ &= \sum_{j=1}^{k} \textrm{Pr}\left[{C_j}\right] + \sum_{j=k+1}^{n} \hat{f}_t \textrm{Pr}\left[{C_j}\right] \end{split}\end{aligned}\]

then \[\begin{aligned} \textrm{Pr}\left[{C_i | F_k}\right] = \begin{cases} \frac{\hat{f}_t \textrm{Pr}\left[{C_i}\right]}{\textrm{Pr}\left[{F_k}\right]} & \text{if } k < i \\ \frac{\textrm{Pr}\left[{C_i}\right]}{\textrm{Pr}\left[{F_k}\right]} & \text{otherwise} \\ \end{cases}\end{aligned}\]

The FACF Algorithm

Algorithm [alg:facf] presents the pseudo code for a culprit finding algorithm based on the above Bayesian formulation. It takes a prior distribution (the uniform distribution can be used), an estimated flake rate $\hat{f}_t$, and the suspects. It returns the most likely culprit or NONE (in the case that it is most likely there is no culprit). The FACF algorithm builds a set of evidence $X$ containing the observed passes and failures so far. At each step, that evidence is converted to a distribution using the math presented above. Then, FACF calls Algorithm [alg:nextRuns] NextRuns to compute the next suspect to run.

NextRuns converts the probability distribution into a cumulative distribution. It then selects the suspects with cumulative probability over a set of thresholds and returns them. The version of NextRuns we present here has been generalized to allow for $k$ parallel runs and also contains a minor optimization on lines 10-11. If the suspect $s_{i-1}$ prior to the selected suspect $s_i$ hasn’t been selected to run and its cumulative probability is above $0$ then the prior suspect $s_{i-1}$ is selected in favor of $s_i$. This optimization ensures that we look for the passing run preceding the most likely culprit before looking for a failing run at the culprit.

Choosing the Thresholds in NextRuns

Another optimization can be made to NextRuns by changing the thresholds used to select suspects. Instead of simply dividing the probability space up evenly it uses the observation that the information gain from a PASS is more than the gain from a FAIL. Modeling that as $I(P_i) = 1$ and $I(F_i) = 1 - \hat{f}_t$, we can compute the expected information gain for new evidence at suspect $s_i$ as $\textrm{E}\left[{I(s_i)}\right] = I(P_i)P_i + I(F_i)F_i$. Expanding from the marginalized equations [eq:pass] and [eq:fail] gives \[\begin{aligned} \textrm{E}\left[{I(s_i)}\right] = (1 - \hat{f}_t)(n + i \hat{f}_t - 1)\end{aligned}\]

Computing the cumulative expectation over $\{ \textrm{E}\left[{I(s_1)}\right], ..., \textrm{E}\left[{I(s_{n+1})}\right] \}$ and normalizing allows us to select the suspect that maximizes the information gain. The empirical evaluation (Section 5) examines the effect of this optimization. The importance of the supplied flake rate estimate ($\hat{f}_t$) is also examined in Section 5.

Prior Probabilities for FACF

The prior distribution supplied to Algorithm [alg:facf] does not have to be a uniform distribution. Instead, it can be informed by any other data the culprit finding system might find useful. At Google, we are exploring many such sources of data, but so far are only using two primary sources in production.

First, we have always used the static build dependence graph to filter out non-affecting suspects before culprit finding (Gupta, Ivey, and Penix 2011). In FACF, we take that a step further and, using a stale snapshot for scalability, we compute the minimum distance in the build graph between the files changed in each suspect and the test being culprit-found (Memon et al. 2017). Using these distances, we form a probability distribution where the suspects with files closer to the test have a higher probability of being the culprit (Ziftci and Ramavajjala 2013; Ziftci and Reardon 2017). Second, we look at the historical likelihood of a true breakage (versus a flaky one) to determine the probability that there is no culprit. Combining that distribution with the one from the minimum build distance forms a prior distribution we use for culprit finding.

Using this scheme, flaky tests tend to get reruns at the $s_n$ (the suspect where the failure was detected) as their first test executions during culprit finding. This is intentional as a large fraction of the search ranges we culprit find tend to be flaky and this reduces the cost of proving that. In the empirical evaluation (Section 5), we examine the effect of the prior distribution on both cost and accuracy.

Culprit Verification for Accuracy Evaluation

The previous sections detailed our culprit finding algorithm (FACF) and the baseline algorithm we compare against (deflaked bisection). In order to evaluate the accuracy of a culprit finding system for a particular dataset, we need to know which commits are the actual culprit commits. If the actual culprit commits are known, then we can compute the usual measures (True Positive Rate, False Positive Rate, Accuracy, Precision, Recall, etc...) (Tharwat 2021). Previous work in culprit finding, test case selection, fault localization and related research areas have often used curated datasets (e.g. Defects4J (Just, Jalali, and Ernst 2014)), bug databases (Bhattacharya and Neamtiu 2010; Herzig et al. 2015; Najafi, Rigby, and Shang 2019), and synthetic benchmarks with injected bugs (Henderson and Podgurski 2018; Henderson, Podgurski, and Kucuk 2019). But our primary interest is in the accuracy of the algorithms in the Google environment, not the accuracy on open source bug databases, curated datasets, or synthetic bugs.

At Google, we do not have a comprehensive database of developer investigations of every continuous integration test failure – indeed there are far too many failures per day for developers to examine and evaluate every single one. We also didn’t want to completely rely on human feedback to tell us when our culprit finding systems have identified the wrong culprit – as had been previously done (Ziftci and Reardon 2017).

Instead, we wanted to be able to verify whether or not an identified culprit for a particular test and search range is correct or not. The verification needs to be automated such that the we can operationally monitor the accuracy of the deployed system and catch any performance regressions in the culprit finder. Continuous monitoring ensures a high quality user experience by alerting our team to accuracy degradations.

Culprit Finding Conclusions

A culprit finder may draw one of two conclusions:

FI: FLAKE_IDENTIFIED the culprit finder determined the status transition was caused by non-deterministic behavior.
CVI: CULPRIT_VERSION_IDENTIFIED(version) the culprit finder determined a specific version was identified as the culprit.

The verification system will label each conclusion examined with either CORRECT or INCORRECT from which we can define the standard counts for True and False Positives and Negatives. Accuracy, Precision, and Recall can then be directly computed.

When is a Culprit Incorrect?

For ease of discussion, we will use the notation introduced in Section 2.2 for the search range $S$ with $s_0$ indicating the previous passing version, $s_n$ indicating the version where the failure was first detected, and $s_k$ indicating the identified culprit. Now, consider the two cases

FLAKE_IDENTIFIED these are incorrect if no subsequent runs pass at the version where the original failure was detected (no flaky behavior can be reproduced).
CULPRIT_VERSION_IDENTIFIED(version) these are incorrect if we can prove any of the following:
1. A pass at $s_n$ indicates the failure was a flake.
2. A pass at the culprit $s_k$ indicates that the culprit was incorrectly identified.
3. No passes detected at the version prior to the culprit, $s_{k-1}$, indicates that the culprit was incorrectly identified.
Additionally the following conditions indicate the culprit may have been misidentified due to a test dependency on either time or some unknown external factor:
1. No passes are identified at $s_0$, indicating the test failure may be time or environment dependent (and the conclusion cannot be trusted).
2. No pass at $s_0$ is detected prior (in the time dimension) to a failure at $s_n$.
3. No failure at $s_n$ is detected prior (in the time dimension) to a pass at $s_0$.
4. No pass at $s_{k-1}$ is detected prior (in the time dimension) to a failure at $s_k$.
5. No failure at $s_k$ is detected prior (in the time dimension) to a pass at $s_{k-1}$.
6. Verification is not reproducible 17 hours later.⁴

Thus, our approach for determining whether or not a culprit is correct is to do extensive reruns for a particular conclusion.

How Many Reruns to Conduct to Verify a Conclusion

A test run $x$ times fails every time with the rate $\hat{f}_t^x$. If we want to achieve confidence level $C$ (ex. $C = .99999999$) of verifier correctness, we can compute the number of reruns to conduct for each check with $x = \left\lceil{\frac{\,\textrm{log}\!\!\left({1 - C}\right)}{\,\textrm{log}\!\!\left({\hat{f}_t}\right)}}\right\rceil$.

Sampling Strategy

While it would be ideal to verify every single culprit for every single test, the expense, in practice, of such an operation is too high. We conduct verification with a very high confidence level ($C = 0.99999999$) and a minimum on the estimated flakiness rate of $0.1$ to ensure a minimum of 8 reruns for each check even for targets that have not been historically flaky. With such an expensive configuration, we can only afford the following sampling strategy.

Randomly sample at $1\%$ of culprit finding conclusions.
Sample the first conclusion for each unique culprit version.

We tag which sampling strategy was used to sample a particular conclusion allowing us to compute statistics for each sampling frame independently.

The Verified Culprits Dataset

Using our production culprit finders based on the FACF algorithm and a verifier implementing the above verification strategy, we have produced an internal dataset with 144,130 verified conclusions in the last 60 days, which we use in Section 5 to evaluate the performance of the algorithms presented in this paper. Note, because the evaluation of an algorithm is also expensive, we only conduct the evaluation on a subset of the whole dataset.

While we are unable to make the dataset public, we encourage others to use our verification approach on their own culprit finding systems and when possible share these datasets with the external community for use in future studies. Such datasets are not only useful for culprit finding research but also for fault localization, test prioritization, test selection, automatic program repair, and other topics.

Empirical Evaluation

We empirically evaluated the behavior of the probabilistic Flake Aware Culprit Finding (FACF) algorithm, compared it to the traditional Bisect algorithm (Couder 2008), evaluated the effectiveness of adding deflaking runs to Bisect, and evaluated the effectiveness of two optimizations for FACF.

Research Questions

The following research questions were considered. All questions were considered with respect to the Google environment (see Section 5.5 for more details about the dataset used).

Which algorithm (FACF or Bisect) was more accurate in identifying culprit commits?
What difference do algorithmic tweaks (such as adding deflaking runs to Bisect) make to accuracy.
How efficient is each algorithm as measured in number of test executions?
What difference do algorithmic tweaks (such as adding deflaking runs to Bisect) make to efficiency.
Does flakiness affect culprit finding accuracy and cost?

Measuring Culprit Finding Accuracy

To measure the accuracy of the culprit finders we produce a binary variable (“correct”) which is true if the culprit finder identified version matched the culprit in the dataset. Accuracy is then defined as the sample proportion: $\textbf{ACC} = \frac{\text{\# correct}}{\text{Total}}$. Culprit finder “A” is judged more accurate than culprit finder “B” if A’s ACC value is higher than B’s and a statistical test (described in Section 5.4) determines the difference is significant.

Measuring Culprit Finding Cost

We measure cost as the number of test executions performed by the algorithm. Culprit finder “A” is judged lower cost than culprit finder “B” if the mean number of test executions is less than “B” and a statistical test (described in Section 5.4) determines the difference is significant.

On Our Usage of Statistical Tests

For answering RQ1 and RQ2, we observe the distribution per culprit-finding algorithm of whether or not the algorithm identified the culprit (0 or 1). For answering RQ3 and RQ4, we observe the distribution per culprit-finding algorithm of the number of test executions required to complete culprit finding.

We follow the recommendations of McCrum-Gardner (McCrum-Gardner 2008) as well as Walpole et al. (Walpole 2007) as to the proper test statistics for our analysis. Since data points for a specific culprit range have a correspondence across algorithms, we perform paired statistical tests when feasible. The analysis is complicated by the observation that neither the dichotomous distribution of correctness nor the numerical distribution of number of test executions are normally distributed. Following McCrum-Gardner we opt to use non-parametric tests. Additionally different tests are used for accuracy (dichotomous nominal variable) and number of executions (numerical variable) respectively.

As we have more than 2 groups to consider (10 algorithms), we initially perform omnibus tests over all algorithms to justify the existence of statistical significance over all algorithms. In response to the multiple comparisons problem (Walpole 2007), we compute the corrected significance bound using the formula for the family-wise error rate. Our experimental design has us first conduct the ANOVA significance test to determine if any of the algorithms differed in performance. Then we conduct post-hoc testing to determine which algorithm’s performance differed. For each considered question we conduct $\binom{10}{2} + 1$ significance tests. We have $6$ considered questions which we use significance testing to answer, giving us a total $r = 6\left(\binom{10}{2} + 1\right) = 276$ tests.

Solving for the $\alpha$ in the experiment-wise error rate formula from Walpole (pg. 529 (Walpole 2007)) with $p$ standing for the probability of a false rejection of at least one hypothesis in the experiment gives: $\alpha = 1 - (1 - p)^{\frac{1}{r}}$. Given $r = 276$ tests and an intended experiment-wise error rate of $p = 0.001$, we conservatively arrive at $\alpha = 10^{-6}$ after rounding down to control for family-wise error rate. We use $\alpha$ as our significance level in all tests.

Cochran’s Q Test is used for significance testing of the accuracy distributions and Friedman’s $\chi^{2}$ Test for number of executions. Post-hoc testing was conducted with McNemar’s test and Wilcoxon Signed Rank Test respectively.

Evaluation Dataset

For the study we used a subset of the dataset produced by the production Culprit Verifier (described in Section 4). The evaluation dataset consists of 13600 test breakages with their required build flags, search range of versions, and the verified culprit. The verified culprit may be an indicator that there was no culprit and detected transition was caused by a flake. Approximately 60% of the items had a culprit version and 40% did not (as these were caused by a flaky failure).

Multiple tests may be broken by the same version. When looking at unique search ranges, we actually see approximately twice as many flaky ranges as non-flaky ones. This hints towards having tighter bounds on which ranges are worth culprit finding as all of the runs for two-thirds of ranges would have been better utilized on more deflaking runs if these two categories are accurately discernable.

We therefore examine the overall dataset for discrepancies between the two categories. For example, flaky ranges have, on average, 8 tests failing while non-flaky ranges have 224. For the objective of finding unique real culprits, we are interested in the minimum flake rate over targets in a range for flaky and non-flaky ranges, respectively. Fig [fig:min-flake-rate] shows us that, in fact, 99% of real culprit ranges have targets below a flake rate of 75%.

Results

RQ1: Which algorithm (FACF or Bisect) was more accurate in identifying culprit commits?

All versions of FACF were more accurate overall than all versions of Bisect (Figure 2). The difference was significant with a p-value $< 10^{-12}$, below the adjusted $10^{-6}$ significance level required.

RQ2: What difference do algorithmic tweaks (such as adding deflaking runs to Bisect) make to accuracy?

The production version of FACF, FACF(all), was most accurate and significantly different than all other algorithms (Figure 2). This was followed by FACF(fr=.1) which set the floor of 10% on the estimated flake rate for tests. The other FACF variants were equally accurate.

RQ3: How efficient is each algorithm as measured in number of test executions?

Figure 2 shows the most efficient algorithm was unsurprisingly Bisect(1) which does not attempt to do any extra executions to deflake test failures. The next most efficient algorithm was FACF(all) which was also the most accurate. As can be seen in the boxplot in Figure 2, the FACF variants tend to have many more extreme outliers in their cost distribution than the Bisect variants. This is because they account for the prior flake rate. An organization may control their cost by not culprit finding tests which are more than (for instance) 50% flaky. No such upper bound was established for this experiment.

RQ4: What difference do algorithmic tweaks (such as adding deflaking runs to Bisect) make to efficiency?

First, both FACF(all) and FACF(priors) had less cost than the standard FACF. This indicates that while FACF(priors) does not improve accuracy it does improve efficiency of the algorithm. In particular many more culprits were found with just two builds.

Second, FACF(heur-thresholds) had no significant effect on cost nor did it show any improvement accuracy. We conclude that although the mathematical motivation may be sound, in the Google environment, we cannot validate a benefit to this optimization at this time.

RQ5: Does flakiness affect culprit finding accuracy and cost?

Figures 3 and 4 show the accuracy and cost of culprit finding flaky and non-flaky culprits. For search ranges without a culprit, FACF(all) clearly dominates Bisect in accuracy and even has a small advantage in cost over Bisect(1).

When there is a culprit, the FACF variants are all more accurate than the Bisect variants according to the statistical tests. However, the difference is much less pronounced with Bisect(1) accuracy $\sim$96.1% and FACF(all) accuracy $\sim$97.5%. In terms of cost, Bisect(1) and Bisect(2) are the most efficient followed by the FACF variants.

We will note, a production culprit finder cannot know a priori if a breakage was caused by a flake. It would need to do deflaking runs to establish that, costing approximately $\left\lceil{\frac{\,\textrm{log}\!\!\left({p}\right)}{\,\textrm{log}\!\!\left({\hat{f}_t}\right)}}\right\rceil$ where $p$ is your desired confidence level. For $p=10^{-4}$ and $\hat{f} = .2$, this amounts to 6 extra runs, which would significantly increase the cost of bisection. However, as noted in Section 5.5, prior flakiness rates could be used to prune out a significant portion of flaky ranges (lowering cost) with a minimal hit to accuracy.

Culprit Finding

Despite its industrial importance, culprit finding has been relatively less studied in the literature in comparison to topics such as Fault Localization (Agarwal and Agrawal 2014; Wong et al. 2016) (coverage based (J. a. Jones, Harrold, and Stasko 2002; J. A. Jones and Harrold 2005; Lucia et al. 2014; Henderson and Podgurski 2018; Henderson, Podgurski, and Kucuk 2019; Kucuk, Henderson, and Podgurski 2021), information retrieval based (J. Zhou, Zhang, and Lo 2012; Youm et al. 2015; Ciborowska and Damevski 2022), or slicing based (Podgurski and Clarke 1990; Horwitz and Reps 1992; Ren et al. 2004)), Bug Prediction (Lewis et al. 2013; Punitha and Chitra 2013; Osman et al. 2017), Test Selection (Engström, Runeson, and Skoglund 2010; Pan et al. 2022), Test Prioritization (de S. Campos Junior et al. 2017; Pan et al. 2022) and the SZZ algorithm for identifying bug inducing commits in the context of research studies (Śliwerski, Zimmermann, and Zeller 2005; Rodríguez-Pérez, Robles, and González-Barahona 2018; Borg et al. 2019; Wen et al. 2019). This is perhaps understandable as the problem only emerges in environments were it becomes to costly to run every test at every code submission.

The idea of using a binary search to locate the culprit is common, but it is not completely clear who invented it first. The Christian Couder of git bisect wrote a comprehensive paper on its implementation (Couder 2008) but indicates there are other similar tools: “So what are the tools used to fight regressions? [...] The only specific tools are test suites and tools similar to git bisect.” The first version of the git bisect command appears to have been written by Linus Torvalds in 2005.⁵ Torvald’s commit message simply states “This is useful for doing binary searching for problems.” The source code for the mercurial bisect command hg bisect states that it was inspired by the git command.⁶ The Subversion (SVN) command, svn bisect, also cites the git command as inspiration.⁷ Therefore, as far as the authors can tell bisection for finding culprits appears to have been invented by Torvalds.

The Couder paper (Couder 2008) cites a GitHub repository for a program called “BBChop,” which is lightly documented and appears not entirely completed by a user enigmatically named Ealdwulf Wuffinga (likely a pseudonym, as that was a name of the King of East Anglia from 664-713).⁸ This program claims to have also implemented a Bayesian search for culprits. We gladly cede the throne for first invention of applying Bayes rule to culprit finding to Ealdwulf and note that our invention was independent of the work in BBChop.

More recently, work has been emerging on the industrial practice of Culprit Finding. In 2013, Ziftci and Ramavajjala gave a public talk at the Google Test Automation Conference in on Culprit Finding at Google (Ziftci and Ramavajjala 2013). They note, that by 2013 Google has already been using $m$-ary bisection based culprit finding for “small” and “medium” tests. They then give a preview of an approach elaborated on in Ziftci and Reardon’s 2017 paper (Ziftci and Reardon 2017) for culprit finding integration tests where test executions take a long time to run. The authors propose a suspiciousness metric based on the minimum build dependence distance and conduct a case study on the efficacy of using such a metric to surface potential culprits to developers. We utilize a similar build dependence distance as input to construct a prior probability (see Section 3.3) in this work but do not directly surface the metric to our end users. Finally, for ground truth they rely on the developers to report what the culprit version is. In contrast, the empirical evaluation we present in this new work is fully automated and does not rely on developers.

In 2017, Saha and Gligoric (Saha and Gligoric 2017) proposed accelerating the traditional bisect algorithm via Test Selection techniques. They use Coverage Based Test Selection (Z. Q. Zhou 2010; Musa et al. 2015) to choose which commits are most likely to affect the test results at the failing commit. This technique can be viewed as fine grained, dynamic version of the static build dependence selection strategy we employ at Google.

In 2019, Najafi et al. (Najafi, Rigby, and Shang 2019) looked at a similar problem to the traditional culprit finding problem we examine here. In their scenario, there is a submission queue that queues up commits to integrate into the main development branch. It batches these waiting commits and tests them together. If a test fails, they need to culprit find to determine the culprit commit. The authors evaluate using a ML-driven risk based batching approach. In 2022, the same research group re-evaluated the 2019 paper on an open source dataset in Beheshtian et al.’s 2022 paper (Beheshtian, Bavand, and Rigby 2022). They found that in the new environment the risk based batching didn’t do as well as a new and improved combinatorics based batching scheme for identifying the culprit commits.

James Keenan gave a presentation in 2019 at the Perl Conference in which he presented a concept, multisection, for dealing with multiple bugs in the same search range (Keenan 2019). Ideally, industrial culprit finders would automatically detect that there are multiple distinct bug inducing commits.

In 2021, two papers looked at using coverage based data for accelerating bug localization. An and Yoo (An and Yoo 2021) used coverage data to accelerate both bisection and SZZ (Śliwerski, Zimmermann, and Zeller 2005) and found significant speedups. Wen et al. (Wen et al. 2021) combined traditional Coverage Based Statistical Fault Localization (J. a. Jones, Harrold, and Stasko 2002) with historical information to create a new suspiciousness score (Sun and Podgurski 2016). This fault localization method could be used in a culprit finding context by using the line rankings to rank the commits.

Finally, in 2022 Frolin Ocariza (Ocariza 2022) published a paper on bisecting performance regressions. There are strong connections between a flaky test and a performance regression as a performance regression may not happen 100% of the time. Furthermore, the “baseline” version may randomly have poor performance due to environmental or other factors. Ocariza also arrives at a probabilistic approach, using a Bayesian model to determine whether a run at a particular version is exhibiting the performance regression or not. While mathematically related to our work here and complementary, the approach is distinct. The authors note that their technique can be combined with a Noisy Binary Search citing Karp (Karp and Kleinberg 2007) or with a multisection search citing Keenan (Keenan 2019).

Noisy Binary Search

There is a vast literature on Rényi-Ulam games and Noisy Searching problems (doubly so when considering their connection to error correcting codes)! We direct readers to the Pelc survey (Pelc 2002) as a starting point. We will highlight a few articles here. First and foremost, the Ben Or and Hassidim paper (Ben-Or and Hassidim 2008) from 2008 most clearly explains the problem in terms of a Bayesian inference, which is the formulation we use here. Second, Waeber et al. in 2013 (Waeber, Frazier, and Henderson 2013) discuss the theoretical foundations of these probabilistic bisection algorithms. Finally, according to Waeber, Horstein first described this algorithm in the context of error correcting codes in 1963 (Horstein 1963).

Conclusion

Flake Aware Culprit Finding (FACF) is both accurate, efficient, and able to flexibly incorporate prior information on the location of a culprit. A large scale empirical study on real test and build breakages found FACF to be significantly more effective than a traditional bisection search while not increasing cost over a deflaked bisection.

References

Agarwal, Pragya, and Arun Prakash Agrawal. 2014. “Fault-Localization Techniques for Software Systems: A Literature Review.” SIGSOFT Softw. Eng. Notes 39 (5): 1–8. https://doi.org/10.1145/2659118.2659125.
An, Gabin, and Shin Yoo. 2021. “Reducing the Search Space of Bug Inducing Commits Using Failure Coverage.” In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 1459–62. Athens Greece: ACM. https://doi.org/10.1145/3468264.3473129.
Ananthanarayanan, Sundaram, Masoud Saeida Ardekani, Denis Haenikel, Balaji Varadarajan, Simon Soriano, Dhaval Patel, and Ali Reza Adl-Tabatabai. 2019. “Keeping Master Green at Scale.” Proceedings of the 14th EuroSys Conference 2019. https://doi.org/10.1145/3302424.3303970.
Beheshtian, Mohammad Javad, Amir Hossein Bavand, and Peter C. Rigby. 2022. “Software Batch Testing to Save Build Test Resources and to Reduce Feedback Time.” IEEE Transactions on Software Engineering 48 (8): 2784–2801. https://doi.org/10.1109/TSE.2021.3070269.
Ben-Or, Michael, and Avinatan Hassidim. 2008. “The Bayesian Learner Is Optimal for Noisy Binary Search (and Pretty Good for Quantum as Well).” In 2008 49th Annual IEEE Symposium on Foundations of Computer Science, 221–30. Philadelphia, PA, USA: IEEE. https://doi.org/10.1109/FOCS.2008.58.
Bhattacharya, Pamela, and Iulian Neamtiu. 2010. “Fine-Grained Incremental Learning and Multi-Feature Tossing Graphs to Improve Bug Triaging.” In 2010 IEEE International Conference on Software Maintenance, 1–10. Timi oara, Romania: IEEE. https://doi.org/10.1109/ICSM.2010.5609736.
Binkley, D. 1992. “Using Semantic Differencing to Reduce the Cost of Regression Testing.” In Proceedings Conference on Software Maintenance 1992, 41–50. Orlando, FL, USA: IEEE Comput. Soc. Press. https://doi.org/10.1109/ICSM.1992.242560.
Borg, Markus, Oscar Svensson, Kristian Berg, and Daniel Hansson. 2019. “SZZ Unleashed: An Open Implementation of the SZZ Algorithm - Featuring Example Usage in a Study of Just-in-Time Bug Prediction for the Jenkins Project.” In Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation - MaLTeSQuE 2019, 7–12. Tallinn, Estonia: ACM Press. https://doi.org/10.1145/3340482.3342742.
Ciborowska, Agnieszka, and Kostadin Damevski. 2022. “Fast Changeset-Based Bug Localization with BERT.” In Proceedings of the 44th International Conference on Software Engineering, 946–57. Pittsburgh Pennsylvania: ACM. https://doi.org/10.1145/3510003.3510042.
Couder, Christian. 2008. “Fighting Regressions with Git Bisect.” The Linux Kernel Archives 4 (5). https://www. kernel. org/pub/software/scm/git/doc s/git- bisect-lk2009.html.
de S. Campos Junior, Heleno, Marco Antônio P Araújo, José Maria N David, Regina Braga, Fernanda Campos, and Victor Ströele. 2017. “Test Case Prioritization: A Systematic Review and Mapping of the Literature.” In Proceedings of the 31st Brazilian Symposium on Software Engineering, 34–43. New York, NY, USA: ACM. https://doi.org/10.1145/3131151.3131170.
Engström, Emelie, Per Runeson, and Mats Skoglund. 2010. “A Systematic Review on Regression Test Selection Techniques.” Information and Software Technology 52 (1): 14–30. https://doi.org/10.1016/j.infsof.2009.07.001.
Fowler, Martin. 2006. “Continuous Integration.” https://martinfowler.com/articles/ continuousIntegration.html.
Gupta, Pooja, Mark Ivey, and John Penix. 2011. “Testing at the Speed and Scale of Google.” http://google-engtools.blogspot.com/2011/06/testing-at- speed-and-scale-of-google.html.
Henderson, Tim A. D., and Andy Podgurski. 2018. “Behavioral Fault Localization by Sampling Suspicious Dynamic Control Flow Subgraphs.” In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), 93–104. Vasteras: IEEE. https://doi.org/10.1109/ICST.2018.00019.
Henderson, Tim A. D., Andy Podgurski, and Yigit Kucuk. 2019. “Evaluating Automatic Fault Localization Using Markov Processes.” In 2019 19th International Working Conference on Source Code Analysis and Manipulation (SCAM), 115–26. Cleveland, OH, USA: IEEE. https://doi.org/10.1109/SCAM.2019.00021.
Herzig, Kim, Michaela Greiler, Jacek Czerwonka, and Brendan Murphy. 2015. “The Art of Testing Less Without Sacrificing Quality.” In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, 1:483–93. IEEE. https://doi.org/10.1109/ICSE.2015.66.
Herzig, Kim, and Nachiappan Nagappan. 2015. “Empirically Detecting False Test Alarms Using Association Rules.” In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, 39–48. Florence, Italy: IEEE. https://doi.org/10.1109/ICSE.2015.133.
Horstein, M. 1963. “Sequential Transmission Using Noiseless Feedback.” IEEE Transactions on Information Theory 9 (3): 136–43. https://doi.org/10.1109/TIT.1963.1057832.
Horwitz, S, and T Reps. 1992. “The Use of Program Dependence Graphs in Software Engineering.” In International Conference on Software Engineering, 9:349. Springer. http://portal.acm.org/citation.cfm?id=24041&dl=.
Jones, J.a., M. J. Harrold, and J. Stasko. 2002. “Visualization of Test Information to Assist Fault Localization.” Proceedings of the 24th International Conference on Software Engineering. ICSE 2002. https://doi.org/10.1145/581339.581397.
Jones, James A., and Mary Jean Harrold. 2005. “Empirical Evaluation of the Tarantula Automatic Fault-localization Technique.” In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, 273–82. New York, NY, USA: ACM. https://doi.org/10.1145/1101908.1101949.
Just, René, Darioush Jalali, and Michael D Ernst. 2014. “Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs.” In Proceedings of the 2014 International Symposium on Software Testing and Analysis, 437–40. New York, NY, USA: ACM. https://doi.org/10.1145/2610384.2628055.
Karp, Richard M., and Robert Kleinberg. 2007. “Noisy Binary Search and Its Applications.” In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, 881–90. SODA ’07. USA: Society for Industrial and Applied Mathematics. https://dl.acm.org/doi/abs/10.5555/1283383.1283478.
Keenan, James. 2019. “James E. Keenan - "Multisection: When Bisection Isn’t Enough to Debug a Problem".” https://www.youtube.com/watch?v=05CwdTRt6AM.
Kowalczyk, Emily, Karan Nair, Zebao Gao, Leo Silberstein, Teng Long, and Atif Memon. 2020. “Modeling and Ranking Flaky Tests at Apple.” In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice, 110–19. Seoul South Korea: ACM. https://doi.org/10.1145/3377813.3381370.
Kucuk, Yigit, Tim A. D. Henderson, and Andy Podgurski. 2021. “Improving Fault Localization by Integrating Value and Predicate Based Causal Inference Techniques.” In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 649–60. Madrid, ES: IEEE. https://doi.org/10.1109/ICSE43902.2021.00066.
Leong, Claire, Abhayendra Singh, Mike Papadakis, Yves Le Traon, and John Micco. 2019. “Assessing Transition-Based Test Selection Algorithms at Google.” In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 101–10. IEEE. https://doi.org/10.1109/ICSE-SEIP.2019.00019.
Lewis, Chris, Zhongpeng Lin, Caitlin Sadowski, and Xiaoyan Zhu. 2013. “Does Bug Prediction Support Human Developers? Findings from a Google Case Study.” In Proceedings of the …, 372–81. http://dl.acm.org/citation.cfm?id=2486838.
Lucia, David Lo, Lingxiao Jiang, Ferdian Thung, and Aditya Budi. 2014. “Extended Comprehensive Study of Association Measures for Fault Localization.” Journal of Software: Evolution and Process 26 (2): 172–219. https://doi.org/10.1002/smr.1616.
Machalica, Mateusz, Wojtek Chmiel, Stanislaw Swierc, and Ruslan Sakevych. 2020. “Probabilistic Flakiness: How Do You Test Your Tests?” DEVINFRA. https://engineering.fb.com/2020/12/10/developer-tools/ probabilistic-flakiness/.
Machalica, Mateusz, Alex Samylkin, Meredith Porth, and Satish Chandra. 2019. “Predictive Test Selection.” In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 91–100. IEEE. https://doi.org/10.1109/ICSE-SEIP.2019.00018.
McCrum-Gardner, Evie. 2008. “Which Is the Correct Statistical Test to Use?” British Journal of Oral and Maxillofacial Surgery 46 (1): 38–41. https://doi.org/10.1016/j.bjoms.2007.09.002.
Memon, Atif, Zebao Gao, Bao Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. 2017. “Taming Google-scale Continuous Testing.” In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), 233–42. Piscataway, NJ, USA: IEEE. https://doi.org/10.1109/ICSE-SEIP.2017.16.
Micco, John. 2012. “Tools for Continuous Integration at Google Scale.” Tech {{Talk}}. Google NYC. https://youtu.be/KH2_sB1A6lA.
———. 2013. “Continuous Integration at Google Scale.” Lecture. EclipseCon 2013. https://web.archive.org/web/20140705215747/https:// www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/2013- 03-24%20Continuous%20Integration%20at%20Google%20Scale.pdf.
Musa, Samaila, Abu Bakar Md Sultan, Abdul Azim Bin Abd-Ghani, and Salmi Baharom. 2015. “Regression Test Cases Selection for Object-Oriented Programs Based on Affected Statements.” International Journal of Software Engineering and Its Applications 9 (10): 91–108. https://doi.org/10.14257/ijseia.2015.9.10.10.
Najafi, Armin, Peter C. Rigby, and Weiyi Shang. 2019. “Bisecting Commits and Modeling Commit Risk During Testing.” In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 279–89. New York, NY, USA: ACM. https://doi.org/10.1145/3338906.3338944.
Ocariza, Frolin S. 2022. “On the Effectiveness of Bisection in Performance Regression Localization.” Empirical Software Engineering 27 (4): 95. https://doi.org/10.1007/s10664-022-10152-3.
Osman, Haidar, Mohammad Ghafari, Oscar Nierstrasz, and Mircea Lungu. 2017. “An Extensive Analysis of Efficient Bug Prediction Configurations.” In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering, 107–16. Toronto Canada: ACM. https://doi.org/10.1145/3127005.3127017.
Pan, Rongqi, Mojtaba Bagherzadeh, Taher A. Ghaleb, and Lionel Briand. 2022. “Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review.” Empirical Software Engineering 27 (2): 29. https://doi.org/10.1007/s10664-021-10066-6.
Pelc, Andrzej. 2002. “Searching Games with Errorsfifty Years of Coping with Liars.” Theoretical Computer Science 270 (1-2): 71–109. https://doi.org/10.1016/S0304-3975(01)00303-6.
Podgurski, A, and L A Clarke. 1990. “A Formal Model of Program Dependences and Its Implications for Software Testing, Debugging, and Maintenance.” IEEE Transactions of Software Engineering 16 (9): 965–79. https://doi.org/10.1109/32.58784.
Potvin, Rachel, and Josh Levenberg. 2016. “Why Google Stores Billions of Lines of Code in a Single Repository.” Communications of the ACM 59 (7): 78–87. https://doi.org/10.1145/2854146.
Punitha, K., and S. Chitra. 2013. “Software Defect Prediction Using Software Metrics - A Survey.” In 2013 International Conference on Information Communication and Embedded Systems (ICICES), 555–58. Chennai: IEEE. https://doi.org/10.1109/ICICES.2013.6508369.
Ren, Xiaoxia, Fenil Shah, Frank Tip, Barbara G. Ryder, and Ophelia Chesley. 2004. “Chianti: A Tool for Change Impact Analysis of Java Programs.” In Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, 432–48. Vancouver BC Canada: ACM. https://doi.org/10.1145/1028976.1029012.
Rivest, R. L., A. R. Meyer, and D. J. Kleitman. 1978. “Coping with Errors in Binary Search Procedures (Preliminary Report).” In Proceedings of the Tenth Annual ACM Symposium on Theory of Computing - STOC ’78, 227–32. San Diego, California, United States: ACM Press. https://doi.org/10.1145/800133.804351.
Rodríguez-Pérez, Gema, Gregorio Robles, and Jesús M. González-Barahona. 2018. “Reproducibility and Credibility in Empirical Software Engineering: A Case Study Based on a Systematic Literature Review of the Use of the SZZ Algorithm.” Information and Software Technology 99 (July): 164–76. https://doi.org/10.1016/j.infsof.2018.03.009.
Saha, Ripon, and Milos Gligoric. 2017. “Selective Bisection Debugging.” In Fundamental Approaches to Software Engineering, edited by Marieke Huisman and Julia Rubin, 10202:60–77. Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-662-54494-5_4.
Śliwerski, Jacek, Thomas Zimmermann, and Andreas Zeller. 2005. “When Do Changes Induce Fixes?” ACM SIGSOFT Software Engineering Notes 30 (4): 1. https://doi.org/10.1145/1082983.1083147.
Sun, Shih-Feng, and Andy Podgurski. 2016. “Properties of Effective Metrics for Coverage-Based Statistical Fault Localization.” In 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), 124–34. IEEE. https://doi.org/10.1109/ICST.2016.31.
Tharwat, Alaa. 2021. “Classification Assessment Methods.” Applied Computing and Informatics 17 (1): 168–92. https://doi.org/10.1016/j.aci.2018.08.003.
Tip, Frank. 1995. “A Survey of Program Slicing Techniques.” Journal of Programming Languages 3 (3): 121–89.
Waeber, Rolf, Peter I. Frazier, and Shane G. Henderson. 2013. “Bisection Search with Noisy Responses.” SIAM Journal on Control and Optimization 51 (3): 2261–79. https://doi.org/10.1137/120861898.
Walpole, Ronald E., ed. 2007. Probability & Statistics for Engineers & Scientists. 8th ed. Upper Saddle River, NJ: Pearson Prentice Hall.
Wang, Kaiyuan, Greg Tener, Vijay Gullapalli, Xin Huang, Ahmed Gad, and Daniel Rall. 2020. “Scalable Build Service System with Smart Scheduling Service.” In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, 452–62. Virtual Event USA: ACM. https://doi.org/10.1145/3395363.3397371.
Wen, Ming, Junjie Chen, Yongqiang Tian, Rongxin Wu, Dan Hao, Shi Han, and Shing-Chi Cheung. 2021. “Historical Spectrum Based Fault Localization.” IEEE Transactions on Software Engineering 47 (11): 2348–68. https://doi.org/10.1109/TSE.2019.2948158.
Wen, Ming, Rongxin Wu, Yepang Liu, Yongqiang Tian, Xuan Xie, Shing-Chi Cheung, and Zhendong Su. 2019. “Exploring and Exploiting the Correlations Between Bug-Inducing and Bug-Fixing Commits.” In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 326–37. Tallinn Estonia: ACM. https://doi.org/10.1145/3338906.3338962.
Wong, W. Eric, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. “A Survey on Software Fault Localization.” IEEE Transactions on Software Engineering 42 (8): 707–40. https://doi.org/10.1109/TSE.2016.2521368.
Youm, Klaus Changsun, June Ahn, Jeongho Kim, and Eunseok Lee. 2015. “Bug Localization Based on Code Change Histories and Bug Reports.” In 2015 Asia-Pacific Software Engineering Conference (APSEC), 190–97. New Delhi: IEEE. https://doi.org/10.1109/APSEC.2015.23.
Zhou, Jian, Hongyu Zhang, and David Lo. 2012. “Where Should the Bugs Be Fixed? More Accurate Information Retrieval-Based Bug Localization Based on Bug Reports.” Proceedings - International Conference on Software Engineering, 14–24. https://doi.org/10.1109/ICSE.2012.6227210.
Zhou, Zhi Quan. 2010. “Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing.” Proceedings - International Computer Software and Applications Conference, 208–13. https://doi.org/10.1109/COMPSACW.2010.43.
Ziftci, Celal, and Vivek Ramavajjala. 2013. “Finding Culprits Automatically in Failing Builds - i.e. Who Broke the Build?” https://www.youtube.com/watch?v=SZLuBYlq3OM.
Ziftci, Celal, and Jim Reardon. 2017. “Who Broke the Build? Automatically Identifying Changes That Induce Test Failures in Continuous Integration at Google Scale.” Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017, 113–22. https://doi.org/10.1109/ICSE-SEIP.2017.13.

In practice the restriction on network usage on TAP is somewhat loose. There are legacy tests that have been tagged to allow them to utilize the network and there is an on going effort to fully eliminate network usage on TAP.↩︎
Other CI platforms exist for integration tests that span multiple machines, use large amounts of RAM, or take a very long time to run. The algorithm presented in this paper has also been implemented for one of those CI systems but we will not present an empirical evaluation of its performance in that environment in this paper. In general, Culprit Finding is much more difficult in such environments as tests may use a mixture of code from different versions and network resources may be shared across tests.↩︎
An internal version of Bazel https://bazel.build/.↩︎
The exact number of hours isn’t so important. The main thing is to repeat the verification at a suitable interval from the original attempt.↩︎
https://github.com/git/git/commit/8b3a1e056f2107deedfdada86046971c9ad7bb87 ↩︎
https://www.mercurial-scm.org/repo/hg/file/52464a20add0/mercurial/hbisect.py, https://www.mercurial-scm.org/repo/hg/rev/a7678cbd7c28 ↩︎
https://metacpan.org/release/INFINOID/App-SVN-Bisect-1.1/source/README ↩︎
https://github.com/Ealdwulf/bbchop ↩︎

Improving fault localization by integrating value and predicate based causal inference techniques

2021-05-26T00:00:00-04:00

Yiğit Küçük, Tim A. D. Henderson, and Andy Podgurski Improving fault localization by integrating value and predicate based causal inference techniques. ICSE 2021.
DOI. PDF. SUPPLEMENT. ARTIFACT. WEB.

Abstract

Statistical fault localization (SFL) techniques use execution profiles and success/failure information from software executions, in conjunction with statistical inference, to automatically score program elements based on how likely they are to be faulty. SFL techniques typically employ one type of profile data: either coverage data, predicate outcomes, or variable values. Most SFL techniques actually measure correlation, not causation, between profile values and success/failure, and so they are subject to confounding bias that distorts the scores they produce. This paper presents a new SFL technique, named UniVal, that uses causal inference techniques and machine learning to integrate information about both predicate outcomes and variable values to more accurately estimate the true failure-causing effect of program statements. UniVal was empirically compared to several coverage-based, predicate-based, and value-based SFL techniques on 800 program versions with real faults.

Read the paper

The Impact of Rare Failures on Statistical Fault Localization: the Case of the Defects4J Suite

2019-10-03T00:00:00-04:00

Yiğit Küçük, Tim A. D. Henderson, and Andy Podgurski The Impact of Rare Failures on Statistical Fault Localization: the Case of the Defects4J Suite. ICSME 2019.
DOI. PDF. WEB.

Abstract

Statistical Fault Localization (SFL) uses coverage profiles (or "spectra") collected from passing and failing tests, together with statistical metrics, which are typically composed of simple estimators, to identify which elements of a program are most likely to have caused observed failures. Previous SFL research has not thoroughly examined how the effectiveness of SFL metrics is related to the proportion of failures in test suites and related quantities. To address this issue, we studied the Defects4J benchmark suite of programs and test suites and found that if a test suite has very few failures, SFL performs poorly. To better understand this phenomenon, we investigated the precision of some statistical estimators of which SFL metrics are composed, as measured by their coefficients of variation. The precision of an embedded estimator, which depends on the dataset, was found to correlate with the effectiveness of a metric containing it: low precision is associated with poor effectiveness. Boosting precision by adding test cases was found to improve overall SFL effectiveness. We present our findings and discuss their implications for the evaluation and use of SFL metrics.

(Read the rest of the paper as a pdf)

Evaluating Automatic Fault Localization Using Markov Processes

2019-09-30T00:00:00-04:00

Tim A. D. Henderson, Yiğit Küçük, and Andy Podgurski Evaluating Automatic Fault Localization Using Markov Processes. SCAM 2019.
DOI. PDF. SUPPLEMENT. WEB.

Note

This is a conversion from a latex paper I wrote. If you want all formatting correct you should read the pdf version.

Abstract

Statistical fault localization (SFL) techniques are commonly compared and evaluated using a measure known as "Rank Score" and its associated evaluation process. In the latter process each SFL technique under comparison is used to produce a list of program locations, ranked by their suspiciousness scores. Each technique then receives a Rank Score for each faulty program it is applied to, which is equal to the rank of the first faulty location in the corresponding list. The SFL technique whose average Rank Score is lowest is judged the best overall, based on the assumption that a programmer will examine each location in rank order until a fault is found. However, this assumption *oversimplifies* how an SFL technique would be used in practice. Programmers are likely to regard suspiciousness ranks as just one source of information among several that are relevant to locating faults. This paper provides a new evaluation approach using first-order Markov models of debugging processes, which can incorporate multiple additional kinds of information, e.g., about code locality, dependences, or even intuition. Our approach, $ \textrm{HT}_{\textrm{Rank}} $, scores SFL techniques based on the expected number of steps a programmer would take through the Markov model before reaching a faulty location. Unlike previous evaluation methods, $\textrm{HT}_{\textrm{Rank}}$ can compare techniques even when they produce fault localization reports differing in structure or information granularity. To illustrate the approach, we present a case study comparing two existing fault localization techniques that produce results varying in form and granularity.

Introduction

Automatic fault localization is a software engineering technique to assist a programmer during the debugging process by suggesting "suspicious" locations that may contain or overlap with a fault (bug, defect) that is the root cause of observed failures. The big idea behind automatic fault localization (or just fault localization) is that pointing the programmer towards the right area of the program will enable them to find the relevant fault more quickly.

A much-investigated approach to fault localization is Coverage-Based Statistical Fault Localization (CBSFL), which is also known as Spectrum-Based Fault Localization [1]-[3]. This approach uses code-coverage profiles and success/failure information from testing or field use of software to rank statements or other generic program locations (e.g., basic blocks, methods, or classes) from most "suspicious" (likely to be faulty) to least suspicious. To perform CBSFL, each test case is run using a version of the program being debugged that has been instrumented to record which potential fault locations were actually executed on that test. A human or automated test oracle labels each test to indicate whether it passed or failed. The coverage profiles are also referred to as coverage spectra.

In the usage scenario typically envisioned for CBSFL, a programmer uses the ranked list of program locations to guide debugging. Starting at the top of the list and moving down, they examine each location to determine if it is faulty. If the location of a fault is near the top of the list, the programmer saves time by avoiding consideration of most of the non-faulty locations in the program. However, if there is no fault near the top of the list, the programmer instead wastes time examining many more locations than necessary. CBSFL techniques are typically evaluated empirically in terms of their ability to rank faulty locations near the top of the list [4], [5], as measured by each technique's "Rank Score", which is the location of the first faulty location in the list. A CBSFL technique that consistently ranks faulty statements from a wide range of faulty programs near the top of the corresponding lists is considered a good technique.

One pitfall of using the ranked-list evaluation regime outlined above is that it can be applied fairly only when the techniques being compared provide results as a prioritized list of program elements of the same granularity. This means that if technique A produces a prioritized list of basic blocks, technique B produces an unordered set of sub-expressions, and technique C produces a prioritized list of classes then it is not valid to use the Standard Rank Score to compare them. The Standard Rank Score can only be applied to ordered lists and thus cannot be used directly to evaluate technique B, and it requires the techniques being compared to have the same granularity. Our new evaluation metric (called $\textrm{HT}_{\textrm{Rank}}$) accounts for these differences in granularity and report construction, allowing a direct comparison between different styles of fault localization. We present a case study in Section 27 comparing behavioral fault localization (which produces fragments of the dynamic control flow graph) to standard CBSFL.

Second, it is evident that the imagined usage scenario for CBSFL, in which a programmer examines each possible fault location in rank order until a fault is found, is an oversimplification of programmer behavior [6]. Programmers are likely to deviate from this scenario, e.g.: by choosing not to re-examine locations that have already been carefully examined and have not changed; by examining the locations around a highly ranked one, regardless of their ranks; by examining locations that a highly ranked location is dependent upon or that are dependent upon it [7]; by employing a conventional debugger (such as gdb, WinDB, or Visual Studio's debugger); or simply by using their knowledge and intuition about the program.

To support more flexible and nuanced evaluation criteria for CBSFL and other fault localization techniques, we present a new approach to evaluating them that is based on constructing and analyzing first-order Markov models of debugging processes and that uses a new evaluation metric ($\textrm{HT}_{\textrm{Rank}}$) based on the "hitting time" of a Markov process. This approach allows researchers to directly compare different fault localization techniques by incorporating their results and other relevant information into an appropriate model. To illustrate the approach, we present models for two classes of fault localization techniques: CBSFL and Suspicious Behavior Based Fault Localization (SBBFL) [8], [9]. The models we present are also easy to update, allowing researchers to incorporate results of future studies of programmers' behavior during debugging.

Our new debugging model (and its $\textrm{HT}_{\textrm{Rank}}$ metric) can be thought of as a first-order simulation of the debugging process as conducted by a programmer. As such, we intend for it to be a practical alternative to conducting an expensive user study. The model is capable of incorporating a variety of behaviors a programmer may exhibit while debugging, allowing researchers to evaluate the performance of their tool against multiple debugging "styles."

Coverage Based Statistical Fault Localization (CBSFL) [1], [4] techniques attempt to quantify the likelihood that individual program locations are faulty using sample statistics, called suspiciousness metrics or fault localization metrics, which are computed from PASS/FAIL labels assigned to test executions and from coverage profiles (coverage spectra) collected from those executions. A CBSFL suspiciousness metric (of which there are a great many [2], [3]) measures the statistical association between the occurrence of test failures and the coverage of individual program locations (program elements) of a certain kind.

Some statistical fault localization techniques use additional information beyond basic coverage information to either improve accuracy or provide more explainable results. For instance, work on Causal Statistical Fault Localization uses information about the execution of program dependence predecessors of a target statement to adjust for confounding bias that can distort suspiciousness scores [10]. By contrast, Suspicious-Behavior-Based Fault Localization (SBBFL) techniques use runtime control-flow information (the behavior) to identify groups of "collaborating" suspicious elements [9]. These techniques typically leverage data mining techniques [11] such as frequent pattern mining [12]-[14] or significant pattern mining [9], [15]. Unlike CBSFL techniques, SBBFL techniques output a ranked list of patterns (subgraphs, itemsets, trees) which each contain multiple program locations. This makes it difficult to directly compare SBBFL and CBSFL techniques using traditional evaluation methods.

Finally, a variety of non-statistical (or hybrid) approaches to fault localization have been explored [16]-[19]. These approaches range from delta debugging [20] to nearest neighbor queries [7] to program slicing [21], [22] to information retrieval [23]-[25] to test case generation [26]-[28]. Despite technical and theoretical differences in these approaches, they all suggest locations (or groups of locations) for programmers to consider when debugging.

The Tarantula Evaluation

Some of the earliest papers on fault localization do not provide a quantitative method for evaluating performance (as is seen in later papers [5]). For instance, the earliest CBSFL paper [1], by Jones et al., proposes a technique and evaluates it qualitatively using data visualization. At the time, this was entirely appropriate as Jones was proposing a technique for visualizing the relative suspiciousness of different statements, as estimated with what is now called a suspiciousness metric (Tarantula). The visualization used for evaluating this technique aggregated the visualizations for all of the subject programs included in the study.

While the evaluation method used in the Jones et al. paper [1] effectively communicated the potential of CBSFL (and interested many researchers in the idea) it was not good way to compare multiple fault localization techniques. In 2005 Jones and Harrold [4] published a study that compared their Tarantula technique to three other techniques: Set Union and Intersection [29], Nearest Neighbor [7], and Cause-Transitions [30]. These techniques involved different approaches toward the fault localization problem and originally had been evaluated in different ways. Jones and Harrold re-evaluated all of the techniques under a new common evaluation framework.

In their 2005 paper, Jones and Harrold evaluated the effectiveness of each fault localization technique by using it to rank the statements in each subject program version from most likely to be the root cause of observed program failures to least likely. For their technique Tarantula, the statements were ranked using the Tarantula suspiciousness score.¹ To compare the effectiveness of the techniques, another kind of score was assigned to each faulty version of each subject program. This score is based on the "rank score":

Definition. Tarantula Rank Score [4]

Given a set of locations $L$ with their suspiciousness scores $s(l)$ for $l \in L$ the Rank Score $r(l)$ for a faulty location $l \in L$ is: \[\begin{aligned} {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) \ge s(l) \right\} }\right|}} \end{aligned}\]

For Set Union and Intersection, Nearest Neighbor, and Cause-Transitions, Jones and Harrold used an idea of Renieres and Reiss [7] and ranked a program's statements based on consideration of its System Dependence Graph (SDG) [31]. The surrogate suspiciousness score of a program location $L$ is the inverse of the size of the smallest dependence sphere around $L$ that contains a faulty location. The surrogate scores are then used to calculate the Tarantula Rank Score (Def. 14).

In Jones's and Harrold's evaluation the authors did not use the Tarantula Rank Score directly but instead used a version of it that is normalized by program size:

Definition. Tarantula Effectiveness Score (Expense) [4]

This score is the proportion of program locations that do not need to be examined to find a fault when the locations are examined in rank order. Formally, let $n$ be the total number of program locations, and let $r(f)$ be the Tarantula Rank Score (Def. 14) of the faulty location $f$. Then the score is: \[\begin{aligned} \frac{n-r(f)}{n} \end{aligned}\]

Using the normalized effectiveness score, Jones and Harrold directly compared the fault localization effectiveness of the techniques they considered. They did this in two ways. First, they presented a table that bucketed all the buggy versions of all the programs by their Tarantula Effectiveness Scores (given as percentages). Second, they presented a figure that showed the same data as a cumulative curve.

The core ideas of Jones' and Harrold's Effectivness/Expense score now underlie most evaluations of CBSFL techniques. Faulty statements are scored, ranked, rank-scored, normalized, and then aggregated over all programs and versions to provide an overall representation of a fault localization method's performance (e.g., [53], [2], [3], [32]-[34]). However, some refinements have been made to both the Rank Score and the Effectiveness Score.

The Implied Debugging Models

It is worth (re)stating here the debugging model implied in the Jones and Harrold evaluation [4]. The programmer receives from the fault localization tool a ranked list of statements with the most suspicious statements at the top. The programmer then moves down the list examining each location in turn. If multiple statements have the same rank (the same suspiciousness score) all of those statements are examined before the programmer makes a determination on whether or not the bug has been located. This rule is captured in the mathematical definition of the Tarantula Rank Score (Definition 14).

For the non-CBSFL methods which Jones compared CBSFL against, the ranks of the program locations were once again compared using the method of Renieres and Reiss [7], [30], [35] which is sometimes called T-Score. As a reminder, this method computes a surrogate suspiciousness score based on the size of smallest dependence sphere² centered around the locations indicated in the fault localization report that contain the faulty code. This implies a debugging model in which the programmer examines each "shell" of the dependence sphere in turn before moving onto the next larger shell (see Figure 7 in [30] for a visualization).

Neither of these debugging models are realistic. Programmers may be reasonably expected to deviate from the ordering implied by the ranking. During the debugging process a programmer may use a variety of information sources — including intuition — to decide on the next element to examine. They may examine the same element multiple times. They may take a highly circuitous route to the buggy code or via intuition jump immediately to the fault. The models described above allow for none of these subtleties.

Wong et al. [32] introduced the most commonly used effectiveness score, which is called the $\mathcal{EXAM}$ score. This score is essentially the same as the Expense score (Def. 15) except that it gives the percentage of locations that need to be examined rather than those avoided.

Definition. $\mathcal{EXAM}$ Score [32]

\[\begin{aligned} \frac{r(f)}{n} \end{aligned}\]

Ali et al. [37] identified an important problem with Jones' and Harrold's evaluation method: some fault localization techniques always assign different locations distinct suspiciousness scores, but others do not. Ali et al. pointed out that when comparing techniques, the Tarantula Effectiveness Score may favor a technique that generates more distinct suspiciousness scores than the other techniques. The fix they propose is to assign to a location in a group of locations with the same suspiciousness score a rank score that reflects developers having to examine half the locations in the group on average.

Definition. Standard Rank Score

This score is the expected number of locations a programmer would inspect before locating a fault. Formally, given a set of locations $L$ with their suspiciousness scores $s(l)$ for $l \in L$, the Rank Score for a location $l \in L$ is [37]: \[\begin{aligned} {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) > s(l) \right\} }\right|}} + \frac{ {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) = s(l) \right\} }\right|}} }{ 2 } \end{aligned}\] Note: when we refer to the "Standard Rank Score" this is the definition we are referring to.

Parnin and Orso [6] conducted a study of programmers' actual use of a statistical fault localization tool (Tarantula [1]). One of their findings was that programmers did not look deeply through the ranked list of locations and would instead only consider the first few locations. Consequently, they encouraged researchers to no longer report effectiveness scores as percentages. Most CBSFL studies now report absolute (non-percentage) rank scores. This is desirable for another reason: larger programs can have much larger absolute ranks than small programs, for the same percentage rank. Consider, for instance a program with 100,000 lines. If a fault's Rank Score is 10,000 its percentage Exam Score would be 10%. A 10% Exam Score might look like a reasonably good localization (and would be if the program had 100 lines) but no programmer will be willing to look through 10,000 lines. By themselves, percentage evaluation metrics (like Exam Score) produce inherently misleading results for large programs.

Steinmann et al. [53] identified a number of threats to validity in CBSFL studies, including: heterogeneous subject programs, poor test suites, small sample sizes, unclear sample spaces, flaky tests, total number of faults, and masked faults. For evaluation they used the Standard Rank Score of Definition 17 modified to deal with $k$ faults tied at the same rank.

Definition. Steinmann Rank Score

This score is the expected number of locations a programmer would inspect before finding a fault when multiple faulty statements may have the same rank. Formally, given a set of locations $L$ with their suspiciousness scores $s(l)$ for $l \in L$, the Rank Score for a location $l \in L$ is [53]: \[\begin{aligned} & {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) > s(l) \right\} }\right|}}\\ & + \frac{ {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) = s(l) \right\} }\right|}} + 1 }{ {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) = s(l) \wedge x \text{ is a faulty location} \right\}}\right|}} + 1 } \end{aligned}\]

Moon et al. [38] proposed Locality Information Loss (LIL) as an alternative evaluation framework. LIL models the localization result as a probability distribution constructed from the suspiciousness scores:

Definition. LIL Probability Distribution

Let $\tau$ be a suspicious metric normalized to the $[0,1]$ range of reals. Let $n$ be the number of locations in the program and let $L = \{l_1,\ldots, l_n\}$ be the set of locations. The constructed probability distribution is given by: \[\begin{aligned} P_{\tau}(l_i) = \frac{\tau(l_i)}{\sum^{n}_{j=1} \tau(l_j)} \end{aligned}\]

LIL uses the Kullback-Leibler measure of divergence between distributions to compute a score indicating how different the distribution constructed for a suspiciousness metric of interest is from the distribution constructed from an "ideal" metric, which gives a score of 1 to the faulty location(s) and gives negligible scores to every other location. The advantage of the LIL framework is that it does not depend on a list of ranked statements and can be applied to non-statistical methods (using a synthetic $\tau$). The disadvantage of LIL is that it does not reflect programmer effort (as the Rank Scores do). However, it may be a better metric to use when evaluating fault localization systems as components of automated fault repair systems.

Pearson et al. [5] re-evaluated a number of previous results using new real world subject programs with real defects and test suites. In contrast to previous work they made use of statistical hypothesis testing and confidence intervals to characterize the uncertainty of the results. To evaluate the performance of each technique under study they used the $\mathcal{EXAM}$ score, reporting best, average, and worst case results for multi-statement faults.

A New Approach to Evaluation

Programmers consider multiple sources of information when performing debugging tasks and use them to guide their exploration of the source code. In our new approach to evaluating fault localization techniques, a model is constructed for each technique $T$ and each program $P$ of how a programmer using $T$ might move from examining one location in $P$ to examining another. The model for $T$ and $P$ is used to compute a statistical estimate of the expected number of moves a programmer using $T$ would make before encountering a fault in $P$. This estimate is used to compute a "hitting-time rank score" for technique $T$. The scores for all the techniques can then be compared to determine which performed best on program $P$. This section presents the general approach and specific example models. The models make use of CBSFL reports and information about static program structure and dynamic control flow.

In order to support very flexible modeling and tractable analysis of debugging processes, we use first-order Markov chains (described below) to model them. Our first example models the debugging process assumed in previous work, in which a programmer follows a ranked list of suspicious program locations until a fault is found. Then we describe how to incorporate structural information about the program (which could influence a programmer's debugging behavior). Finally, we show how to model and compare CBSFL to a recent Suspicious Behavioral Based Fault Localization (SBBFL) algorithm [9] that identifies suspicious subgraphs of dynamic control flow graphs. This third model demonstrates the flexibility of our approach and could be adapted to evaluate other SBBFL techniques [8], [39]-[47]. It also demonstrates that our approach can be used to compare statistical and non-statistical fault localization techniques under the same assumptions about programmer debugging behavior.

It is important to emphasize that the quality and value of the evaluation results obtained with our approach depend primarily on the appropriateness of the model of the debugging process that is created. This model represents the evaluators' knowledge about likely programmer behavior during debugging and the factors that influence it. To avoid biasing their evaluation, evaluators must commit to an evaluation model and refrain from "tweaking" it after applying it to the data. [48].

Background on Ergodic Markov Chains

A finite state Markov chain consists of a set of states $S = \{s_1, s_2, ..., s_n \}$ and an $n \times n$ matrix ${\bf P}$, called the transition matrix [49]. Entry ${\bf P}_{i,j}$ gives the probability for a Markov process in state $s_i$ to move to state $s_j$. The probability of a Markov process moving from one state to another only depends on the state the process is currently in. This is known as the Markov property.

A Markov chain is said to be ergodic if, given enough steps, it can move from any state $s_i$ to any state $s_j$, i.e. ${\textrm{Pr}\left[{s_i \xrightarrow{*} s_j}\right]} > 0$. Thus, there are no states in an ergodic chain that the process can never leave.

Ergodic Markov chains have stationary distributions. Let ${\bf v}$ be an arbitrary probability vector. The stationary distribution is a probability vector ${\bf w}$ such that \[\lim_{n \rightarrow \infty} {\bf v}{\bf P}^{n} = {\bf w}\] The vector ${\bf w}$ is a fixed point on ${\bf P}$ implying ${\bf w}{\bf P} = {\bf w}$. Stationary distributions give the long term behavior of a Markov chain - meaning that after many steps the chance a Markov process ends in state $s_i$ is given by ${\bf w}_i$.

The expected hitting time of a state in a Markov chain is the expected number of steps (transitions) a Markov process will make before it encounters the state for the first time. Our new evaluation metric ($\text{HT}_{\text{Rank}}$) uses the expected hitting time of the state representing a faulty program location to score a fault localization technique's performance. Lower expected hitting times yield better localization scores.

Definition. Expected Hitting Time

Consider a Markov chain with transition matrix ${\bf P}$. Let $T_{i,j}$ be a random variable denoting the time at which a Markov process that starts at state $s_i$ reaches state $s_j$. The expected hitting time (or just hitting time) of state $s_j$ for such a process is the expected value of $T_{i,j}$ \[\begin{aligned} {\textrm{E}\left[{T_{i,j}}\right]} = \sum_{k=1}^{\infty} k \cdot {\textrm{Pr}\left[{T_{i,j}=k}\right]} \end{aligned}\]

In general, a hitting time for a single state may be computed in $\mathcal{O}(n^3)$ steps [50]. Somewhat less time is required for sparse transition matrices [51]. Chapter 11 of Grinstead and Snell [49] provides an accessible introduction to hitting time computation.

Some programs may have too many elements for exact hitting time computations (our case study contains one such program). To deal with large programs the expected hitting time can also be estimated by taking repeated random walks through the Markov chain to obtain a sample of hitting times. The sample can then be used to estimate the expected hitting time by computing the sample mean.³

Expected Hitting Time Rank Score ($\textrm{HT}_{\textrm{Rank}}$)

    func (n *Node) Has(k int) bool {
        if n == nil {
            return false
++      } else if k == n.Key {
--      } else if k != n.Key {
            return true
        } else if (k < n.Key) {
            return n.left.Has(k)
        } else {
            return n.right.Has(k)
        }
    }

Listing 1. A bug in the implementation of the "Has" method in an AVL tree.

Table I: Reduced CBSFL Results for Listing 1
rank	R. F1	Function (Basic Block)	rank	R. F1	Function (BB)
1.5	0.98	Node.Has (2)	6	0.95	Node.Has (3)
1.5	0.98	Node.String (3)	7.5	0.94	main (1)
1.5	0.98	Node.Verify (4)	7.5	0.94	Scanner.Scan (24)
4	0.97	Node.Has (4)	7.5	0.94	Node.Verify (0)
4	0.97	Node.Has (6)

Fig. 1: A simplified version of the Markov model for evaluating the ranked list of suspicious locations for the bug in Listing 1.

Fig. 2: An example Markov model showing how “jump” edges can be added to represent how a programmer might examine locations which are near the location they are currently reviewing. Compare to Figure 1.

This section introduces our new evaluation metric $\textrm{HT}_{\textrm{Rank}}$. $\textrm{HT}_{\textrm{Rank}}$ produces a "rank score" similar to the score produced by the Standard Rank Score of Definition 17. In the standard score, program locations are ranked by their CBSFL suspiciousness scores. A location's position in the ordered list is that location's Rank Score (see the definition for details).

The new $\textrm{HT}_{\textrm{Rank}}$ score is obtained as follows:

A Markov debugging model is supplied (as a Markov chain).
The expected hitting times (Def. 22) for each location in the program are computed.
The locations are ordered by their expected hitting times.
The $\textrm{HT}_{\textrm{Rank}}$ for a location is its position in the ordered list.

Definition. $\textrm{HT}_{\textrm{Rank}}$

Given a set of locations $L$ and a Markov chain $(S, {\bf P})$ that represents the debugging process and has start state $0$, the Hitting-Time Rank Score $\textrm{HT}_{\textrm{Rank}}$ for a location $l \in L\cap S$ is: \[\begin{aligned} & {{\left|{ \left\{ x ~:~ x \in L \cap S \wedge {\textrm{E}\left[{T_{0,x}}\right]} < {\textrm{E}\left[{T_{0,l}}\right]} \right\} }\right|}} + \\ & \frac{ {{\left|{ \left\{ x ~:~ x \in L \cap S \wedge {\textrm{E}\left[{T_{0,x}}\right]} = {\textrm{E}\left[{T_{0,l}}\right]} \right\} }\right|}} }{ 2 } \end{aligned}\] Note: this is almost identical to Definition 17, but it replaces the suspiciousness score with the expected hitting time. Definition 18 can also be modified in a similar way for multi-fault programs.

Markov Debugging Models

$\textrm{HT}_{\textrm{Rank}}$ is parameterized by a "debugging model" expressed as a Markov chain. As noted above, a Markov chain is made up a of set of states $S$ and transition matrix ${\bf P}$. In a debugging model, there are two types of states: 1) textual locations in the source code of a program and 2) synthetic states. Figures 6 and 7 show examples of debugging models constructed for an implementation of an AVL tree. In the figures, the square nodes are Markov states representing basic blocks in the source code. (Note that CBSFL techniques typically compute the same suspiciousness score for all statements in a basic block.) The smaller, circular nodes are Markov states which are synthetic. They are there for structural reasons but do not represent particular locations in the source code. The edges in the graphs represent possible transitions between states, and they are annotated with transition probabilities. All outgoing edges from a node should sum to 1.

In a Markov debugging model a programmer is represented by the Markov process. When the Markov process is simulated the debugging actions of a programmer are being simulated. This is a "first order" simulation, which means the actions of the simulated programmer (Markov process) only depend on the current location being examined. Thus, the Markov model provides a simple and easy-to-construct mathematical model of a programmer looking through the source code of a program to find the faulty statement(s). The simulated programmer begins at some starting state and moves from state to state until the faulty location is found. We require that all Markov models are ergodic, ensuring that every state (program location) is eventually reachable in the model.

An Extensible Markov Model for CBSFL

As described in Section 12 a CBSFL report is made up of a ranked list of locations in the subject program. Our extensible Markov model includes a representation of the CBSFL ranked list. By itself, using $\textrm{HT}_{\textrm{Rank}}$ with a Markov model of a ranked list is just a mathematically complex way of restating Definition 17. However, with a Markov model of a CBSFL report in hand we can add further connections between program locations (represented as Markov states) to represent other actions a programmer might take besides traversing down the ranked list. For instance, in Section 25 we note that programmers use graphical debuggers to traverse the dynamic control flow in a program - allowing them to visit statements in the order they are executed. We embed the dynamic control flow graph into the transition matrix of the Markov model to reflect this observation.

A Ranked List as a Markov Chain

Figure 6 provides a graphical example of a Markov chain for a ranked list. Since the nodes in a graphical representation of a Markov chain represent states and the edges represent the transition matrix, the probabilities associated with outgoing edges of each node should sum to 1. In Figure 6, each circular node represents a rank in the list and the square nodes represent associated program locations, which are identified by their function names and static basic-block id number. The square nodes that are grouped together all have the same suspiciousness scores. We will provide here a brief, informal description of the structure of the transition matrix. A formal description of the chain is provided in Definition 28 in the Appendix. The exact choice of transition matrix in the formal chain was driven by a proof of equivalence between $\textrm{HT}_{\textrm{Rank}}$ with this Markov model (as defined in Definition 28) and the Standard Rank Score (Definition 17).⁴

The transition matrix boils down to a couple of simple connections. The nodes representing groups form a doubly linked list (see the circular nodes in Figure 6). The ordering of the "group nodes" matches the ordering of the ranks in the ranked list. The links between the node are weighted so that a Markov process will tend to end up in a highly ranked node. More formally, the model was constructed so that if you ordered the Markov states by the probabilities in the stationary distribution (which characterizes the long term behavior of the Markov process) from highest to lowest that ordering would match the order of the ranked list.

The second type of connection is from a group node to its program location nodes. Each location node connects to exactly one group node. The transition probabilities (as shown in Figure 6) are set up such that there is an equal chance of moving to any of the locations in the group.

The final connection is from a location node back to its group node. This is always assigned probability $1$ (see Figure 6). Again, see Definition 28 in the Appendix for the formal description.

Adding Local Jumps to the CBSFL Chain

Figure 7 shows a modified version of the model in Figure 6. The modified model allows the Markov process to jump or "teleport" between program locations which are not adjacent in the CBSFL Ranked List. These "jump" connections are setup to model other ways a programmer might move through the source code of a program when debugging (see below). In the figure, these jumps are shown with added red and blue edges. For the Markov model depicted in the figure, the Markov process will with probability $\frac{1}{2}$ move back to the rank list and with probability $\frac{1}{2}$ teleport or jump to an alternative program location that is structurally adjacent (in the program's source code) to the current one.

Informally, the modified model is set up so that if the Markov process is in a state $s_i$ which represents program location $l_x$ it will with probability $p_{\text{jump}}$ move to a state $s_j$ which represents program location $l_y$ instead of returning to the rank list. The locations that the process can move to from $s_i$ are defined in a jump matrix ${\bf J}$ which is parameter to the Markov model. The matrix ${\bf J}$ encodes assumptions or observations about how programmers behave when they are debugging. A formal definition of the CBSFL chain with jumps is presented in the Appendix (see Definition 29).

Definition 26 defines one general-purpose jump matrix ${\bf J}$. It encodes two assumptions about programmer behavior. First, when a programmer considers a location inside a function they will also potentially examine other locations in that function. Second, when a programmer is examining a location they may examine locations preceding or succeeding it the dynamic control flow (e.g., with the assistance of a graphical debugger). Definition 26 encodes both assumptions by setting the relevant ${\bf J}_{i,j}$ entries to $1$.

Definition. Spacial + Behavioral Jumps

\[\begin{aligned} {\bf J}_{i,j} &= \left\{ \begin{array}{cll} 1 & \text{if} & \text{$s_i$ and $s_j$ represent locations} \\ & & \text{in the same function} \\ \\ 1 & \text{if} & \text{$s_i$ and $s_j$ are adjacent locations in the} \\ & & \text{program's dynamic control flow graph} \\ \\ 0 & \text{otherwise} \end{array} \right. \end{aligned}\]

In addition to ${\bf J}$, the new chain is parameterized by the probability $p_{\text{jump}}$ of a jump occurring when the process visits a state representing a program location. As $p_{\text{jump}} \rightarrow 0$ the transition matrix of new chain approaches the transition matrix for the chain in Definition 28 (see the Appendix). We suggest setting $p_{\text{jump}}$ to $0.5$ in the absence of data from a user study.

Modeling SBBFL

As noted earlier, Markov models can be constructed for alternative fault localization techniques. Suspicious Behavior Based Fault Localization (SBBFL) [8], [9] techniques return a report containing a ranked list of subgraphs or subtrees. Each subgraph contains multiple program locations usually drawn from a dynamic control flow graph of the program. Comparing this output to CBSFL using the Standard Rank Score can be difficult as a location may appear multiple times in graphs returned by the SBBFL algorithm. However, $\textrm{HT}_{\textrm{Rank}}$ can produce an accurate and comparable score by utilizing the expected hitting times of the states representing the faulty location. Definition 30 in the Appendix provides an example Markov chain which models a ranked list of suspicious subgraphs. It can be extended (not shown due to space constraints) to add a Jump matrix in the manner of Definition 29 (see the Appendix).

Case Study

To illustrate our new approach to evaluation, we performed a case study in which it was used to evaluate several fault localization techniques of two different types: CBSFL [3], [4] and Suspicious-Behavior-Based Fault Localization (SBBFL) [8], [9]. We investigated the following research questions:

RQ1: How Accurate is $\textrm{HT}_{\textrm{Rank}}$ Estimation? (Table III)
RQ2: Does it make a difference whether $\textrm{HT}_{\textrm{Rank}}$ or the Standard Rank Score is used? (Table IV, Figs. 5 and 4)

We also considered an important general question about fault localization that a typical research study might investigate.

RQ3: Which kind of fault localization technique performs better, SBBFL or CBSFL? (Table IV)

In order to use an SBBFL technique, more complex profiling data (dynamic control flow graphs or DCFGs) are needed than for CBSFL (which requires only coverage data). Therefore, a more specialized profiler was required for this study. We used Dynagrok (https://github.com/timtadh/dynagrok) [9] - a profiling tool for the Go Programming Language. We also reused subject programs used in a previous study [9] (see Table II). They are all real-world Go programs of various sizes that were injected with mutation faults. The test cases for the programs are all either real-world inputs or test cases from the system regression testing suites that are distributed with the programs. Six representative suspiciousness metrics were considered: Ochiai, F1, Jaccard, Relative Ochiai, Relative F1, and Relative Jaccard [3]. These measures were all previously adapted to the SBBFL context [9].

Table II: Datasets used in the evaluation
Program	L.O.C.	Mutants	Description
AVL (github.com/timtadh/dynagrok)	483	19	An AVL tree
Blackfriday (github.com/russross/blackfriday)	8,887	19	Markdown processor
HTML (golang.org/x/net/html)	9,540	20	An HTML parser
Otto (github.com/robertkrimen/otto)	39,426	20	Javascript interpreter
`gc` (go.googlesource.com/go)	51,873	16	The Go compiler

Note: The AVL tree is in the examples directory.

All applications of techniques were replicated to address random variation in the results as the SBBFL algorithm that we used employs sampling [9]. The results from the replications were averaged and, unless otherwise noted, the average Rank Scores are presented. Finally, the exact $\textrm{HT}_{\textrm{Rank}}$ score was computed for 4 of the 5 programs. For the fifth program, the Go compiler, although $\textrm{HT}_{\textrm{Rank}}$ can be computed exactly, this requires so much time for each run to complete ($\approx$ 4 hours) that it precluded computing exact results for each program version (including all models for both SBBFL and CBSFL). Therefore, expected hitting times for the Go compiler were estimated using the method outlined in Section 21. Specifically, we collected 500 samples of hitting times of all states in the Markov debugging model by taking random walks (with a maximum walk length of 1,000,000 steps). The expected hitting times were estimated by taking the sample mean.

The Chosen $\textrm{HT}_{\textrm{Rank}}$ Model

To avoid bias, it was important to choose the $\textrm{HT}_{\textrm{Rank}}$ model before the case study was conducted, which we did. We used $\textrm{HT}_{\textrm{Rank}}$ with jumps (Definition 29) together with the jump matrix specified in Definition 26. We chose this matrix because we believe the order in which programmers examine program elements during debugging is driven in part by the structure of the program, even when they are debugging with the assistance of a fault localization report. The $p_{\text{jump}}$ probability was set to $0.5$ to indicate an equal chance of the programmer using or not using the fault localization report. A future large scale user study could empirically characterize the behavior of programmers while performing debugging to inform the choice of the $p_{\text{jump}}$ parameter. However, without such a study it is reasonable to use $0.5$, which indicates "no information."

RQ1: How Accurate is $\textrm{HT}_{\textrm{Rank}}$ Estimation?

Table III: $\textrm{HT}_{\textrm{Rank}}$ Estimation Error
Subject Program	avl	blackfriday	html	otto
mean median stdev p-value	0.781	0.469	0.473	0.476

Percentage error of the estimated $\textrm{HT}_{\textrm{Rank}}$ versus the exact $\textrm{HT}_{\textrm{Rank}}$ (see note under Def 22). Letting $y$ be the exact $\textrm{HT}_{\textrm{Rank}}$ and $\hat{y}$ be the estimated $\textrm{HT}_{\textrm{Rank}}$, the percentage error is $\frac{|\hat{y} - y|}{y} * 100$. -2.0em

For large programs the cost of computing the expected hitting times for $\textrm{HT}_{\textrm{Rank}}$ may be too high. Therefore, we have suggested estimating the expected hitting times (see note under Definition 22). To assess the accuracy of estimating the expected hitting time instead of computing it exactly, we did both for four subject programs, using the Relative F1 measure. For each program version, SFL technique, and Markov chain type the estimation error was computed as a percentage of the true value. Table III presents descriptive statistics characterizing these errors for each subject program. The last row of the table gives p-values from a independent two-sample T-test comparing the estimated $\textrm{HT}_{\textrm{Rank}}$ values and the exact $\textrm{HT}_{\textrm{Rank}}$ values. The null hypothesis is that the expected estimate $\textrm{HT}_{\textrm{Rank}}$ is the same as the expected exact $\textrm{HT}_{\textrm{Rank}}$. All p-values are well above the $0.05$ significance level that would suggest rejecting the null hypothesis. Therefore, we accept the null hypotheses that the $\textrm{HT}_{\textrm{Rank}}$ estimates are not significantly different from the exact $\textrm{HT}_{\textrm{Rank}}$ values. As indicated in the table, the maximum estimation error was around 3.7%. Therefore, if estimation is used we recommend considering $\textrm{HT}_{\textrm{Rank}}$ scores that are within 5% of each other to be equivalent.

RQ2: Does it make a difference whether $\textrm{HT}_{\textrm{Rank}}$ or the Standard Rank Score is used?

Table IV: Fault Localization Performance
		Standard Rank Score			HTRank Score
score	subject	CBSFL	SBBFL	%$\Delta$	CBSFL	SBBFL	%$\Delta$
RelativeF1	avl	4.6	2.0	-56 %	9.1	4.9	-47 %
RelativeF1	blackfriday	8.8	4.4	-50 %	42.6	11.0	-74 %
RelativeF1	html	12.8	5.0	-61 %	40.7	25.3	-38 %
RelativeF1	otto	8.2	6.2	-24 %	102.2	21.5	-79 %
RelativeF1	compiler	264.8	98.9	-63 %	1148.9	1475.1	28 %
RelativeOchiai	avl	4.6	2.0	-56 %	7.6	6.3	-16 %
RelativeOchiai	blackfriday	8.8	4.4	-50 %	43.6	9.8	-78 %
RelativeOchiai	html	12.8	5.0	-61 %	38.2	22.4	-41 %
RelativeOchiai	otto	8.7	6.8	-22 %	99.2	102.6	3 %
RelativeOchiai	compiler	262.2	101.6	-61 %	2888.8	2984.1	3 %
RelativeJaccard	avl	4.6	2.0	-56 %	10.1	5.1	-50 %
RelativeJaccard	blackfriday	16.0	12.1	-24 %	80.0	176.2	120 %
RelativeJaccard	html	12.8	5.0	-61 %	33.3	24.4	-27 %
RelativeJaccard	otto	8.2	6.3	-23 %	75.2	19.9	-74 %
RelativeJaccard	compiler	747.9	482.0	-36 %	1074.1	1540.5	43 %
F1	avl	4.6	2.0	-56 %	10.4	5.1	-51 %
F1	blackfriday	16.0	12.1	-24 %	81.9	218.8	167 %
F1	html	12.8	5.0	-61 %	33.9	18.6	-45 %
F1	otto	8.2	6.2	-24 %	81.0	28.5	-65 %
F1	compiler	746.8	470.4	-37 %	1047.0	1537.9	47 %
Ochiai	avl	4.6	2.0	-56 %	5.9	6.3	7 %
Ochiai	blackfriday	14.8	10.3	-30 %	51.2	108.7	112 %
Ochiai	html	12.8	5.0	-61 %	30.7	25.3	-18 %
Ochiai	otto	10.3	8.4	-19 %	312.8	607.7	94 %
Ochiai	compiler	1091.6	773.5	-29 %	3137.2	2988.8	-5 %
Jaccard	avl	4.6	2.0	-56 %	10.1	5.1	-50 %
Jaccard	blackfriday	16.0	12.1	-24 %	77.8	187.7	141 %
Jaccard	html	12.8	5.0	-61 %	33.3	24.6	-26 %
Jaccard	otto	8.2	6.2	-24 %	75.3	24.7	-67 %
Jaccard	compiler	747.9	485.4	-35 %	1078.8	1435.6	33 %

Summarizes the fault localization performance for CBSFL and SBBFL. Each fault localization technique is evaluated using both the Standard Rank Score and the $\textrm{HT}_{\textrm{Rank}}$ Score (Defs. 28 and 25). The mean rank scores are shown as well as the percentage changes from CBSFL to SBBFL. Lower ranks scores indicate better fault localization performance. A negative percentage difference (%$\Delta$) indicates SBBFL improved on CBSFL. A positive %$\Delta$ indicates CBSFL outperformed SBBFL.

Fig 3: Empirical probability density plots for the subject program Otto - showing the distributions of both the Standard Rank Scores and the $\textrm{HT}_{\textrm{Rank}}$ Scores across all runs of all versions of Otto. The CBSFL technique is shown in blue and the SBBFL technique is shown in orange. The only suspciousness score used in this plot is RelativeF1.

Fig 4: Comparison of CBSFL vs SBBFL average ranks (on log scale), under the Standard Rank Score (top) and $\textrm{HT}_{\textrm{Rank}}$ (bottom), for each version of the program Otto; suspiciousness metric is RelativeF1.

Table IV details the fault localization performance we observed for CBSFL and SBBFL under the Standard Rank Score and $\textrm{HT}_{\textrm{Rank}}$ using six different suspiciousness metrics. The results obtained with $\textrm{HT}_{\textrm{Rank}}$ were substantially different from those obtained with the Standard Rank Score, in terms of absolute ranks and percentage differences (%$\Delta$). The mean Standard Rank Score for SBBFL is lower than that for CBSFL for every suspiciousness metric and subject program. By contrast, for some programs and metrics, the mean $\textrm{HT}_{\textrm{Rank}}$ scores for CBSFL are lower than those for SBBFL. The mean $\textrm{HT}_{\textrm{Rank}}$ scores are also higher than the mean Standard Rank Scores overall.

Another way to look at the same phenomenon is shown in Figure 5, which displays empirical probability density plots for the program Otto. Each plot compares the performance of CBSFL to SBBFL using a different evaluation method. The top plot uses the Standard Rank Score while the other plot uses $\textrm{HT}_{\textrm{Rank}}$. As shown in the top plot, the Standard Rank Scores for both CBSFL and SBBFL are concentrated mainly between 0 and 15, with no values over 60. In the $\textrm{HT}_{\textrm{Rank}}$ plot, by contrast, the scores for both CBSFL and SBBFL are much more widely dispersed. Figure 4 compares CBSFL to SBBFL with respect to average ranks (on a log scale), under the Standard Rank Score (top) and $\textrm{HT}_{\textrm{Rank}}$ (bottom), for each version of the program Otto. The suspiciousness metric is RelativeF1. The results for $\textrm{HT}_{\textrm{Rank}}$ are quite distict from those for the Standard Rank Score, with CBSFL showing much more variability under $\textrm{HT}_{\textrm{Rank}}$.

These results provide confirmation for the theoretical motivation for $\textrm{HT}_{\textrm{Rank}}$. Recall that one of the problems with the Standard Rank Score is that it does not account for either the differing structures of fault localization reports or differences in report granularity. SBBFL and CBSFL differ in both structure (ranked CFG fragments vs. ranked basic blocks) and granularity (multiple basic blocks vs. lone basic blocks). We expected the Standard Rank Score to unfairly favor SBBFL because it does not account for these differences and that is exactly what we see in Table IV. Under the Standard Rank Score SBBFL outperforms CBSFL on every program using every suspiciousness score.

To be explicit, SBBFL reports ranked CFG fragments, each of which contains multiple basic blocks. Under the Standard Rank Score those fragments are ranked and the score is the rank of the first fragment in which the bug appears. CBSFL will, by contrast, rank each basic block independently. Now, consider the case where SBBFL reports a CFG fragment at rank 1 that contains 10 basic blocks, one of which contains the bug. Suppose CBSFL also reports each of those basic blocks as maximally suspicious. The Standard Rank Score for CBSFL will be 5 while for SBBFL it will be 1. Thus, the Standard Rank Score is unfairly biased toward SBBFL and against CBSFL. This is once again reflected in Table IV. The new metric $\textrm{HT}_{\textrm{Rank}}$ does not suffer from this problem. As shown in the table and discussed below, SBBFL often but not always outperforms CBSFL under $\textrm{HT}_{\textrm{Rank}}$, suggesting that the theoretical correction we expect is indeed occurring. We therefore conclude that $\textrm{HT}_{\textrm{Rank}}$ provides a better metric for comparison when reports differ in structure and granularity.

RQ3: Which kind of fault localization technique performs better, SBBFL or CBSFL?

Referring once again to Table IV, the $\textrm{HT}_{\textrm{Rank}}$ results indicate that SBBFL often but not always outperformed CBSFL. In particular, when the measure used was Relative F1 (which was found to be the best-performing measure for SBBFL in [9]), SBBFL performed better for all programs but the compiler. However, when Ochiai was used CBSFL outperformed SBBFL, although CBSFL with Relative F1 outperforms CBSFL with Ochiai. This indicates that SBBFL and Relative F1 may be the best combination tested with one caveat. For the compiler, SBBFL never outperforms CBSFL. However, CBSFL also performs very badly on this large program. This indicates that while CBSFL beats SBBFL on this program neither technique is effective at localizing faults in it.

Study Limitations

The purpose of our case study is to illustrate the application of $\textrm{HT}_{\textrm{Rank}}$. It was not designed to settle controversies about the suspiciousness metrics we employed. Second, this study used real programs and tests but used synthetic bugs. In the future, we intend to provide an automated analysis system that builds upon the Defects4J dataset [52]. Third, our study included a representative but not exhaustive set of suspiciousness metrics [2] and may not generalize to other metrics.

Finally, no user-study was performed to validate the chosen debugging model against actions an actual programmer would take. Although we would have liked to perform such a study at this time, our resources do not permit us to do so. To be conclusive, such a study would need to be large-scale and to utilize professional programmers who are skilled in traditional debugging and are able to spend significant time learning to use SFL techniques with comparable skill.

Discussion

A new, flexible approach to evaluating automatic fault localization techniques was presented. The new $\textrm{HT}_{\textrm{Rank}}$ Score provides a principled and flexible way of comparing different fault localization techniques. It is robust to differences in granularity and allows complex fault localization reports (such as those produced by SBBFL) to be incorporated. Unlike previous attempts at cross-technique comparison (see Section 12) the final scores are based on the expected number of steps through a Markov model. The model can incorporate both information from the chosen fault localization technique as well as other information available to the programmer (such as the structure of the program). The $\textrm{HT}_{\textrm{Rank}}$ Score is sensitive to the model used (see Fig 5) and hence the choice of model is important. The choice should be made based on 1) the fault localization technique(s) being evaluated and 2) observations of programmer behavior. Choosing a model after viewing the results (and picking the model that gives the "best" results) leads to a biased outcome.

Recommendations for Researchers

Report both Standard Rank Score and $\textrm{HT}_{\textrm{Rank}}$ Score.
If evaluating multi-line faults the Steinmann Rank Score (Def. 18) should be used as the basis for defining the $\textrm{HT}_{\textrm{Rank}}$ Score.
Report which model is being used and why it was chosen, and include a description of the model.
In the absence of user study data, set $p_{\text{jump}} = .5$ and set the weights in ${\bf J}$ uniformly.

By evaluating fault localization methods with $\textrm{HT}_{\textrm{Rank}}$ in addition to the Standard Rank Score researchers will be able to make valid cross-technique comparisons. This will enable the community to better understand the relationships between techniques and report-granularity while taking into account potential programmer behavior during debugging.

Acknowledgement

This work was partially supported by NSF award CCF-1525178 to Case Western Reserve University.

Markov Chain Definitions

Definition. CBSFL Ranked List Markov Chain

To construct a Markov chain representing a list of program locations ranked in descending order by their suspiciousness scores:

Let $L$ be the set of locations in the program.

For a location $l \in L$ let $s(l)$ be its CBSFL suspiciousness score.

Partition the locations in $L$ into a list of groups $G = \left\{ g_1 \subseteq L, g_2 \subseteq L, ..., g_n \subseteq L \right\}$ such that for each group $g_i$ all of the locations it contains have the same score: $ \forall ~ {g_i \in G}, ~ \forall ~ {l, l' \in g_i} \left[ s(l) = s(l') \right]$

The score of a group $s(g_i)$ is defined to be the common score of its members: $ \forall ~ {l \in g_i} \left[ s(g_i) = s(l) \right]$

Order $G$ by the scores of its groups, such that $g_0$ has the highest score and $g_n$ has the lowest: $s(g_0) > s(g_1) > ... > s(g_n)$

Now construct the set of states. There is one state for each group $g \in G$ and for each location $l \in L$. \[S = \left\{ g : g \in G \right\} \cup \left\{ l : l \in L \right\}\]

Finally construct the transition matrix ${\bf P}$ for the states $S$. \[{\bf P}_{i,j} = \left\{ \begin{array}{lll} 1 & \text{if} & s_i \in L \wedge s_j \in G \wedge s_i \in s_j \\ \\ \frac{{{\left|{L}\right|}} - 1}{2{{\left|{L}\right|}}} & \text{if} & s_i = g_0 \wedge s_j = s_i \\ \\ \frac{1}{2{{\left|{L}\right|}}} & \text{if} & s_i = g_n \wedge s_j = s_i \\ \\ \frac{{{\left|{L}\right|}} - 1}{2{{\left|{L}\right|}}} & \text{if} & s_i \in G \wedge s_j \in G \wedge s_i - 1 = s_j \\ \\ \frac{1}{2{{\left|{L}\right|}}} & \text{if} & s_i \in G \wedge s_j \in G \wedge s_i + 1 = s_j \\ \\ \frac{1}{2{{\left|{s_i}\right|}}} & \text{if} & s_i \in G \wedge s_j \in L \wedge s_j \in s_i \\ \\ 0 & \text{otherwise} \end{array} \right.\]

Definition. CBSFL with Jumps Markov Chain

This definition augments Definition 28.

Let ${\bf J}$ be a "jump" matrix representing ways a programmer might move through the program during debugging.

If ${\bf J}_{x,y} > 0$ then locations $x,y \in L$ are "connected" by ${\bf J}$.

Let $p_{\text{jump}}$ be the probability that when visiting a location $l \in L$ the Markov process "jumps" to another location. Let $1 - p_{\text{jump}}$ be the probability that the process returns to the state which represents its group instead of jumping. As $p_\text{{jump}} \rightarrow 0$ the behavior of the chain approaches the behavior of the chain in Definition 28.

The new transition matrix ${\bf P}$ for the states $S$ is

\[\begin{aligned} {\bf P}_{i,j} &= \left\{ \begin{array}{cll} 1 - p_{\text{jump}} & \text{if} & s_i \in L \wedge s_j \in G \wedge s_i \in s_j \\ & & \wedge \left( \sum_{k}{{\bf J}_{i,k}} \right) > 0 \\ \\ p_{\text{jump}} \left( \frac{{\bf J}_{i,j}}{\sum_{k}{{\bf J}_{i,k}}} \right) & \text{if} & s_i \in L \wedge s_j \in L \wedge {\bf J}_{i,j} > 0 \\ \\ \frac{{{\left|{L}\right|}} - 1}{2{{\left|{L}\right|}}} & \text{if} & s_i = g_0 \wedge s_j = s_i \\ \\ \frac{1}{2{{\left|{L}\right|}}} & \text{if} & s_i = g_n \wedge s_j = s_i \\ \\ \frac{{{\left|{L}\right|}} - 1}{2{{\left|{L}\right|}}} & \text{if} & s_i \in G \wedge s_j \in G \wedge s_i - 1 = s_j \\ \\ \frac{1}{2{{\left|{L}\right|}}} & \text{if} & s_i \in G \wedge s_j \in G \wedge s_i + 1 = s_j \\ \\ \frac{1}{2{{\left|{s_i}\right|}}} & \text{if} & s_i \in G \wedge s_j \in L \wedge s_j \in s_i \\ \\ 0 & \text{otherwise} \end{array} \right. \end{aligned}\]

Definition. Suspicious Behavior Markov Chain

A chain that models a ranked list of suspicious subgraphs:

Let $H$ be a set of suspicious subgraphs (behaviors).

For a subgraph $h \in H$ let $\varsigma(h)$ be its suspiciousness score [9].

Let $L$ be the set of locations in the program.

Partition the subgraphs in $H$ into a list of groups $G = \left\{ g_1 \subseteq H, g_2 \subseteq H, ..., g_n \subseteq H \right\}$ such that for each group $g_i$ all of the locations in $g_i$ have the same score: $\forall ~ {g_i \in G} ~ \forall ~ {a, b \in g_i} \left[ \varsigma(a) = \varsigma(b) \right]$

Let the score of a group $\varsigma(g_i)$ be the same as the scores of its members: $ \forall ~ {g_i \in G} ~ \forall ~ {h \in g_i} \left[ \varsigma(g_i) = \varsigma(h) \right]$

Order $G$ by the scores of its groups, such that $g_0$ has the highest score and $g_n$ has the lowest: $ \varsigma(g_0) > \varsigma(g_1) > ... > \varsigma(g_n)$

Now construct the set of states. One state for each group $g \in G$, one state for each subgraph $h \in H$, and one state for each location $l \in V_{h}$ for all $h \in H$. \[S = \left\{ g : g \in G \right\} \cup \left\{ h : h \in H \right\} \cup \left\{ l : l \in V_h,~ \forall~ h \in H \right\}\]

Let $c: L \rightarrow \mathbb{N}^{+}$ be a function that gives the number of subgraphs $h \in H$ which a location $l$ appears in.

Finally construct the transition matrix ${\bf P}$ for the states $S$. \[{\bf P}_{i,j} = \left\{ \begin{array}{cll} \frac{1}{c(s_i)} & \text{if} & s_i \in L \wedge s_j \in H \wedge s_i \in V_{s_j} \\ \\ \frac{1}{2}\frac{1}{{{\left|{V_{s_i}}\right|}}} & \text{if} & s_i \in H \wedge s_j \in L \wedge s_j \in V_{s_i} \\ \\ \frac{1}{2} & \text{if} & s_i \in H \wedge s_j \in G \wedge s_i \in s_j \\ \\ \frac{{{\left|{H}\right|}} - 1}{2{{\left|{H}\right|}}} & \text{if} & s_i = g_0 \wedge s_j = s_i \\ \\ \frac{1}{2{{\left|{H}\right|}}} & \text{if} & s_i = g_n \wedge s_j = s_i \\ \\ \frac{{{\left|{H}\right|}} - 1}{2{{\left|{H}\right|}}} & \text{if} & s_i \in G \wedge s_j \in G \wedge s_i - 1 = s_j \\ \\ \frac{1}{2{{\left|{H}\right|}}} & \text{if} & s_i \in G \wedge s_j \in G \wedge s_i + 1 = s_j \\ \\ \frac{1}{2{{\left|{s_i}\right|}}} & \text{if} & s_i \in G \wedge s_j \in H \wedge s_j \in s_i \\ \\ 0 & \text{otherwise} \end{array} \right.\]

References

[1] J. Jones, M. Harrold, and J. Stasko, "Visualization of test information to assist fault localization," Proceedings of the 24th International Conference on Software Engineering. ICSE 2002, 2002, doi:10.1145/581339.581397.

[2] Lucia, D. Lo, L. Jiang, F. Thung, and A. Budi, "Extended comprehensive study of association measures for fault localization," Journal of Software: Evolution and Process, vol. 26, no. 2, pp. 172-219, Feb. 2014, doi:10.1002/smr.1616.

[3] S.-F. Sun and A. Podgurski, "Properties of Effective Metrics for Coverage-Based Statistical Fault Localization," in 2016 ieee international conference on software testing, verification and validation (icst), 2016, pp. 124-134, doi:10.1109/ICST.2016.31.

[4] J. A. Jones and M. J. Harrold, "Empirical Evaluation of the Tarantula Automatic Fault-localization Technique," in Proceedings of the 20th ieee/acm international conference on automated software engineering, 2005, pp. 273-282, doi:10.1145/1101908.1101949.

[5] S. Pearson, J. Campos, R. Just, G. Fraser, R. Abreu, M. D. Ernst, D. Pang, and B. Keller, "Evaluating and Improving Fault Localization," in Proceedings of the 39th international conference on software engineering, 2017, pp. 609-620, doi:10.1109/ICSE.2017.62.

[6] C. Parnin and A. Orso, "Are Automated Debugging Techniques Actually Helping Programmers?" in ISSTA, 2011, pp. 199-209.

[7] M. Renieres and S. Reiss, "Fault localization with nearest neighbor queries," in 18th ieee international conference on automated software engineering, 2003. proceedings., 2003, pp. 30-39, doi:10.1109/ASE.2003.1240292.

[8] H. Cheng, D. Lo, Y. Zhou, X. Wang, and X. Yan, "Identifying Bug Signatures Using Discriminative Graph Mining," in Proceedings of the eighteenth international symposium on software testing and analysis, 2009, pp. 141-152, doi:10.1145/1572272.1572290.

[9] T. A. D. Henderson and A. Podgurski, "Behavioral Fault Localization by Sampling Suspicious Dynamic Control Flow Subgraphs," in IEEE conference on software testing, validation and verification, 2018.

[10] G. G. K. Baah, A. Podgurski, and M. J. M. Harrold, "Causal inference for statistical fault localization," in Proceedings of the 19th international symposium on software testing and analysis, 2010, pp. 73-84, doi:10.1145/1831708.1831717.

[11] C. C. Aggarwal and J. Han, Eds., Frequent Pattern Mining. Cham: Springer International Publishing, 2014.

[12] R. Agrawal, T. Imieliński, and A. Swami, "Mining association rules between sets of items in large databases," ACM SIGMOD Record, vol. 22, no. 2, pp. 207-216, Jun. 1993, doi:10.1145/170036.170072.

[13] X. Yan and J. Han, "gSpan: graph-based substructure pattern mining," in 2002 ieee international conference on data mining, 2002. proceedings., 2002, pp. 721-724, doi:10.1109/ICDM.2002.1184038.

[14] C. C. Aggarwal, M. A. Bhuiyan, and M. A. Hasan, "Frequent Pattern Mining Algorithms: A Survey," in Frequent pattern mining, Cham: Springer International Publishing, 2014, pp. 19-64.

[15] X. Yan, H. Cheng, J. Han, and P. S. Yu, "Mining Significant Graph Patterns by Leap Search," in Proceedings of the 2008 acm sigmod international conference on management of data, 2008, pp. 433-444, doi:10.1145/1376616.1376662.

[16] R. Abreu, P. Zoeteweij, and A. Van Gemund, "An Evaluation of Similarity Coefficients for Software Fault Localization," in 2006 12th pacific rim international symposium on dependable computing (prdc'06), 2006, pp. 39-46, doi:10.1109/PRDC.2006.18.

[17] R. Abreu, P. Zoeteweij, R. Golsteijn, and A. J. C. van Gemund, "A practical evaluation of spectrum-based fault localization," Journal of Systems and Software, vol. 82, no. 11, pp. 1780-1792, 2009, doi:10.1016/j.jss.2009.06.035.

[18] P. Agarwal and A. P. Agrawal, "Fault-localization Techniques for Software Systems: A Literature Review," SIGSOFT Softw. Eng. Notes, vol. 39, no. 5, pp. 1-8, Sep. 2014, doi:10.1145/2659118.2659125.

[19] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, "A Survey on Software Fault Localization," IEEE Transactions on Software Engineering, vol. 42, no. 8, pp. 707-740, Aug. 2016, doi:10.1109/TSE.2016.2521368.

[20] A. Zeller, "Yesterday, My Program Worked. Today, It Does Not. Why?" SIGSOFT Softw. Eng. Notes, vol. 24, no. 6, pp. 253-267, Oct. 1999, doi:10.1145/318774.318946.

[21] F. Tip, "A survey of program slicing techniques," Journal of programming languages, vol. 3, no. 3, pp. 121-189, 1995.

[22] X. Mao, Y. Lei, Z. Dai, Y. Qi, and C. Wang, "Slice-based statistical fault localization," Journal of Systems and Software, vol. 89, no. 1, pp. 51-62, 2014, doi:10.1016/j.jss.2013.08.031.

[23] A. Marcus, A. Sergeyev, V. Rajlieh, and J. I. Maletic, "An information retrieval approach to concept location in source code," Proceedings - Working Conference on Reverse Engineering, WCRE, pp. 214-223, 2004, doi:10.1109/WCRE.2004.10.

[24] J. Zhou, H. Zhang, and D. Lo, "Where should the bugs be fixed? More accurate information retrieval-based bug localization based on bug reports," Proceedings - International Conference on Software Engineering, pp. 14-24, 2012, doi:10.1109/ICSE.2012.6227210.

[25] T.-D. B. Le, R. J. Oentaryo, and D. Lo, "Information retrieval and spectrum based bug localization: better together," in Proceedings of the 2015 10th joint meeting on foundations of software engineering - esec/fse 2015, 2015, pp. 579-590, doi:10.1145/2786805.2786880.

[26] S. Artzi, J. Dolby, F. Tip, and M. Pistoia, "Directed Test Generation for Effective Fault Localization," in Proceedings of the 19th international symposium on software testing and analysis, 2010, pp. 49-60, doi:10.1145/1831708.1831715.

[27] S. K. Sahoo, J. Criswell, C. Geigle, and V. Adve, "Using likely invariants for automated software fault localization," in Proceedings of the eighteenth international conference on architectural support for programming languages and operating systems, 2013, vol. 41, p. 139, doi:10.1145/2451116.2451131.

[28] A. Perez, R. Abreu, and A. Riboira, "A Dynamic Code Coverage Approach to Maximize Fault Localization Efficiency," J. Syst. Softw., vol. 90, pp. 18-28, Apr. 2014, doi:10.1016/j.jss.2013.12.036.

[29] H. Agrawal, J. Horgan, S. London, and W. Wong, "Fault localization using execution slices and dataflow tests," in Proceedings of sixth international symposium on software reliability engineering. issre'95, 1995, pp. 143-151, doi:10.1109/ISSRE.1995.497652.

[30] H. Cleve and A. Zeller, "Locating causes of program failures," Proceedings of the 27th international conference on Software engineering - ICSE '05, p. 342, 2005, doi:10.1145/1062455.1062522.

[31] S. Horwitz, "Identifying the Semantic and Textual Differences Between Two Versions of a Program," SIGPLAN Not., vol. 25, no. 6, pp. 234-245, Jun. 1990, doi:10.1145/93548.93574.

[32] E. Wong, T. Wei, Y. Qi, and L. Zhao, "A Crosstab-based Statistical Method for Effective Fault Localization," in 2008 international conference on software testing, verification, and validation, 2008, pp. 42-51, doi:10.1109/ICST.2008.65.

[33] D. Landsberg, H. Chockler, D. Kroening, and M. Lewis, "Evaluation of Measures for Statistical Fault Localisation and an Optimising Scheme," in International conference on fundamental approaches to software engineering, 2015, vol. 9033, pp. 115-129, doi:10.1007/978-3-662-46675-9.

[34] Y. Zheng, Z. Wang, X. Fan, X. Chen, and Z. Yang, "Localizing multiple software faults based on evolution algorithm," Journal of Systems and Software, vol. 139, pp. 107-123, 2018, doi:10.1016/j.jss.2018.02.001.

[35] C. Liu, L. Fei, X. Yan, J. Han, and S. P. Midkiff, "Statistical debugging: A hypothesis testing-based approach," IEEE Transactions on Software Engineering, vol. 32, no. 10, pp. 831-847, Oct. 2006, doi:10.1109/TSE.2006.105.

[36] J. Ferrante, K. J. Ottenstein, and J. D. Warren, "The program dependence graph and its use in optimization," vol. 9. pp. 319-349, Jul-1987.

[37] S. Ali, J. H. Andrews, T. Dhandapani, and W. Wang, "Evaluating the Accuracy of Fault Localization Techniques," 2009 IEEE/ACM International Conference on Automated Software Engineering, pp. 76-87, 2009, doi:10.1109/ASE.2009.89.

[38] S. Moon, Y. Kim, M. Kim, and S. Yoo, "Ask the Mutants: Mutating faulty programs for fault localization," Proceedings - IEEE 7th International Conference on Software Testing, Verification and Validation, ICST 2014, pp. 153-162, 2014, doi:10.1109/ICST.2014.28.

[39] C. Liu, H. Yu, P. S. Yu, X. Yan, H. Yu, J. Han, and P. S. Yu, "Mining Behavior Graphs for ‘Backtrace' of Noncrashing Bugs," in Proceedings of the 2005 siam international conference on data mining, 2005, pp. 286-297, doi:10.1137/1.9781611972757.26.

[40] G. Di Fatta, S. Leue, and E. Stegantova, "Discriminative Pattern Mining in Software Fault Detection," in Proceedings of the 3rd international workshop on software quality assurance, 2006, pp. 62-69, doi:10.1145/1188895.1188910.

[41] F. Eichinger, K. Böhm, and M. Huber, "Mining Edge-Weighted Call Graphs to Localise Software Bugs," in European conference machine learning and knowledge discovery in databases, 2008, pp. 333-348, doi:10.1007/978-3-540-87479-9_40.

[42] F. Eichinger, K. Krogmann, R. Klug, and K. Böhm, "Software-defect Localisation by Mining Dataflow-enabled Call Graphs," in Proceedings of the 2010 european conference on machine learning and knowledge discovery in databases: Part i, 2010, pp. 425-441.

[43] Z. Mousavian, M. Vahidi-Asl, and S. Parsa, "Scalable Graph Analyzing Approach for Software Fault-localization," in Proceedings of the 6th international workshop on automation of software test, 2011, pp. 15-21, doi:10.1145/1982595.1982599.

[44] F. Eichinger, C. Oßner, and K. Böhm, "Scalable software-defect localisation by hierarchical mining of dynamic call graphs," Proceedings of the 11th SIAM International Conference on Data Mining, SDM 2011, no. c, pp. 723-734, 2011.

[45] S. Parsa, S. A. Naree, and N. E. Koopaei, "Software Fault Localization via Mining Execution Graphs," in Proceedings of the 2011 international conference on computational science and its applications - volume part ii, 2011, pp. 610-623.

[46] L. Mariani, F. Pastore, and M. Pezze, "Dynamic Analysis for Diagnosing Integration Faults," IEEE Trans. Softw. Eng., vol. 37, no. 4, pp. 486-508, Jul. 2011, doi:10.1109/TSE.2010.93.

[47] A. Yousefi and A. Wassyng, "A Call Graph Mining and Matching Based Defect Localization Technique," in 2013 ieee sixth international conference on software testing, verification and validation workshops, 2013, pp. 86-95, doi:10.1109/ICSTW.2013.17.

[48] J. P. A. Ioannidis, "Why most published research findings are false," PLoS Medicine, vol. 2, no. 8, pp. 0696-0701, 2005, doi:10.1371/journal.pmed.0020124.

[49] C. M. Grinstead and J. L. Snell, Introduction to Probability, 2nd ed. Providence, RI: American Mathematical Society, 1997.

[50] J. G. Kemeny and J. L. Snell, Finite Markov Chains, First. Princeton, NJ: Van Nostrand, 1960.

[51] T. A. Davis, "Algorithm 832: UMFPACK V4.3—an Unsymmetric-pattern Multifrontal Method," ACM Trans. Math. Softw., vol. 30, no. 2, pp. 196-199, Jun. 2004, doi:10.1145/992200.992206.

[52] R. Just, D. Jalali, and M. D. Ernst, "Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs," in Proceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437-440, doi:10.1145/2610384.2628055.

[53] F. Steimann, M. Frenkel, and R. Abreu, "Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators," Proceedings of the 2013 International Symposium on Software Testing and Analysis - ISSTA 2013, p. 314, 2013, doi:10.1145/2483760.2483767.

Jones et al. did not use the term "suspiciousness score" or "suspiciousness metric" in their 2002 paper [1]. They introduced the term "suspiciousness score" in their 2005 paper [4], in the context of ranking statements. Both terms are now in common use.↩
A dependence sphere is computed from the Program Dependence Graph (PDG) [31], [36]. In a PDG, program elements are nodes and their control and data dependencies are represented as edges. Given two nodes $\alpha$ and $\beta$ a graph with a shortest path (ignoring edge directionality) $\pi$ between them, the sphere is all those nodes in the graph have have a path from $\alpha$ as short (or shorter) than $\pi$ (once again ignoring edge directionality).↩
Implementations of these two computations maybe found in https://github.com/timtadh/expected-hitting-times.↩
See https://hackthology.com/pdfs/scam-2019-supplement.pdf↩

Gödel's Incompleteness Theorem Does Have Philosophical Implications

2018-11-05T00:00:00-05:00

In 1931 Kurt Gödel published a paper in German that has continued to reverberate through the scientific community:

K. Gödel, "On Formally Undecidable Propositions of Principia Mathematica and Related Systems," Monatshefte für Math. und Phys., vol. 38, pp. 173–198, Jan. 1931.

Translation: doi pdf

This paper sets clear limits on what is "rationally computable" and what that means. There have been those that argue this work has little to no philosophical significance. I believe it does have philosophical significance as I explain at the end.

First some definitions.

By rational I mean reasoning (that is thinking) through the means of logical inference without outside facts or "intuition." Such a manner of thought would have been foreign to most in 1931 (except those pursuing formal mathematics). Today it has become explicit through our interaction with computers -- who can only reason rationally.*

By computable I mean a question (such as does 2 + 2 = 4?) can be provably answered in a finite number of discrete steps in the logical system of choice. By logical system of choice I mean the allowed methods of reasoning and the set of base "facts" (called axioms). For example:

Facts:

0 == 0

0 + 1 != 0

x + 1 == y + 1 ---> x == y

x + (y + 1) == (x + y) + 1

0 + x == x

allowed deductions:

if A is true and A implies B then B is true

2 + 2 == 4?

2 ==> 0 + 1 + 1 ("de-sugaring the natural number")

4 ==> 0 + 1 + 1 + 1 + 1 ("de-sugaring the natural number")

(0 + 1 + 1) + (0 + 1 + 1) == (0 + 1 + 1 + 1 + 1) ("restatement of lemma")

((0 + 1 + 1) + (0 + 1)) + 1 (rule 4)

((0 + 1 + 1) + (0)) + 1 + 1 (rule 4)

(0 + 1 + 1) + 1 + 1 (rule 5)

(0 + 1) + 1 + 1 + 1 (rule 4)

0 + 1 + 1 + 1 + 1 (rule 4)

(0 + 1 + 1 + 1 + 1) == (0 + 1 + 1 + 1 + 1) ("restatement of lemma using previous step")

(0 + 1 + 1 + 1) == (0 + 1 + 1 + 1) (rule 3)

(0 + 1 + 1) == (0 + 1 + 1) (rule 3)

(0 + 1) == (0 + 1) (rule 3)

0 == 0 (rule 3)

true (rule 1)

What did Gödel prove?

In laymen's terms, Gödel proved that any logical system with finite "axioms" (read: starting facts) could not be both "complete" and "consistent." By complete logicians mean the system can answer any question. By consistent logicians mean the system never gives two answers (both true and false) to a single question.**

Why does this matter?

It matters because, like the Halting Theorem, the Incompleteness Theorem shows there are computations (answers to logical questions) which cannot be decided by mechanistic means -- that is they are uncomputable.

Some say that these results are both ordinary and without profound implications. For instance consider this comment from an interview with Tim Maudlin:

Does Gödel's incompleteness theorem have implications beyond mathematics? Is it a worm in the apple of rationality?

No. Absolutely no one should have ever been surprised that mathematical truth cannot be equated with theoremhood in some finite axiomatic system. An infinitude of mathematical truths are uninteresting trivia, with no obvious route to being proved. Example: let's say that the decimal expressions of the square root of 17 and pi to the 27th power “match” just in case either they have the same digit in the tenths place, or the same two digits in the next two places, or the same three digits in the next three places, etc. If we treat these decimal expressions as essentially random sequences of digits, then the a priori chance that these two numbers match is one out of nine.

Now: how do we tell if they match or not? Well, we can just calculate out the sequences of digits and check. And if they match we will eventually find the match and prove that they match. But what if, as is likely, they don't match? No amount of just grinding out the digits and checking will ever prove it: there are always more digits to check. And I see zero prospect of any other way to prove that they don't match. So if they don't match, that is an unprovable mathematical fact. It is also a very, very, very uninteresting one. All Gödel did was find a clever way to construct a provably unprovable mathematical fact, given any consistent and finite set of axioms to work with. The work is clever but in no way profound. It should have come as no surprise at all.

However, it is (particularly in the historical context) profound to be able to prove that a particular question has no computable answer. The answer may be out there but if we (humans) discover it will be by luck not will. There are limits and important ones about what we as humans and machines can and cannot prove and know. From a philosophical point of view this argues for the concept of mystery. That there are and will be those areas of truth for which we can grasp but never hold, for we reach for but do not touch. Thus, we must strive on in our collective quest to know who and what we. All the while knowing there are parts of the world and ourselves for which there is no final a HA!

* - Of course those who work on stochastic algorithms and machine learning may quibble on this point. However, I believe even those algorithms and systems -- and indeed our brains at a fundamental level -- reason rationally. Indeed this is why Gödel's theorem is of such interest.

** - This and this article necessarily simplifies and collapses much of the nuance in the work. For instance there are actually 2 incompleteness theorems that work in concert with each other. I am interested in this article with the effect of the theorems not in giving a rigorous introduction to them.

How to Evaluate Statistical Fault Localization

2018-07-27T00:00:00-04:00

Cite as:

Tim A. D. Henderson. How To Evaluate Statistical Fault Localization. Blog. 2018. https://hackthology.com/how-to-evaluate-statistical-fault-localization.html
PDF. WEB.

Note

This is a conversion from a latex paper I wrote. If you want all formatting correct you should read the pdf version.

Note

I now have a pulished paper on this topic. You probably want to read that as well.

Tim A. D. Henderson, Yiğit Küçük, and Andy Podgurski. Evaluating Automatic Fault Localization Using Markov Processes. SCAM 2019.

Introduction

Automatic fault localization is a software engineering technique to assist a programmer during the debugging process by suggesting suspicious locations that may be related to the root cause of the bug. The big idea behind behind automatic fault localization (or just fault localization) is by pointing the programmer towards the right area of the program the programmer will find the cause of the bug more quickly.

One approach to fault localization is Spectrum Based Fault Localization which is also known as Coverage Based Statistical Fault Localization (CBSFL) [1]-[3]. This approach uses test coverage information to rank the statements from most "suspicious" to least suspicious. To perform CBSFL, test cases are run through an instrumented program. The instrumentation collects coverage profiles which report each statement¹ executed during the test run. A test oracle is used to label each execution profile with whether or not the test passed or failed. Such oracles can be either automatic or manual (i.e. a human). The labeled execution profiles are referred to as the coverage spectra.

CBSFL techniques score each program element (location in the program) by its "statistical suspiciousness" such that the most suspicious element has the highest score. The scores are computed by suspiciousness metrics which attempt to quantify the relationship between the execution each program element and the occurrence of program failure.

There have been a great many statistical fault localization suspiciousness metrics proposed [2] since the idea was first proposed by Jones, Harrold and Stasko in 2002 [1]. The majority of the metrics are computed from just a few values: the number of tests $n$, the number of passing tests $p$, the number failing tests $f$, the number of test runs an element $e$ was executed in $n_e$, the number of passing test runs an element was executed in $p_e$, and the number of failing test runs an element was executed in $f_e$. For instance, using these simple statistics one can estimate the conditional probability of program failure $F$ given that a particular element $e$ was executed: \[\begin{aligned} {\textrm{Pr}\left[{F|e}\right]} &= \frac{{\textrm{Pr}\left[{F \cap e}\right]}}{{\textrm{Pr}\left[{e}\right]}} \approx \frac{\frac{f_{e}}{n}}{\frac{n_{e}}{n}} = \frac{f_e}{n_e}\end{aligned}\] While many of the studies in statistical fault localization use more complex formulas [2], [3] (with various technical motivations) most of the metrics are measures statistical association as in the equation above.

The runtime information used to compute the above statistics is commonly referred to as the coverage spectra of the program. Coverage spectra is a matrix ${\bf C}$ where ${\bf C}_{i,j} > 0$ indicates that the program element $\mathcal{E}_i$ was executed at least once during test $\mathcal{T}_j$. Additionally, there is an additional "test status" vector ${\bf S}$ where ${\bf S}_{j} = \text{"F''}$ if test $\mathcal{T}_j$ failed and ${\bf S}_{j} = \text{"P''}$ if test $j$ passed.

To collect the coverage spectra the program is instrumented to collect a runtime profile of the executed elements. This instrumentation can be done (in principle) at any granularity including: expression, statement, basic block², function, class, file, or package. The statistical methods which make use of spectra are agnostic to their granularity. While the granularity does not effect the statistical computation it changes how the localization results are perceived by the programmer. In the past, programmers have indicated a desire for finer grained results: at statement, basic block, or function level [5] over very coarse grained results at the class, file, or package level.

Some statistical fault localization techniques use additional information to either improve accuracy or provide more explainable results. For instance, work on Causal Fault Localization uses additional static and dynamic information to control for statistical confounding [6]. In contrast Suspicious Behavior Based Fault Localization (SBBFL) uses runtime control flow information (the behavior) to identify groups of collaborating suspicious elements [7]. These techniques leverage data mining techniques [8] such as frequent [9]-[11] or significant pattern mining [7], [12]. When significant patterns are mined metrics (such as statistical fault localization suspiciousness metrics) are used to identify the most significant patterns [7], [13].

Finally, a variety of non-statistical (or mixed methods) techniques for fault localization have been explored [14]-[17]. These range from delta debugging [18] to nearest neighbor queries [19] to program slicing [20], [21] to information retrieval [22]-[24] to test case generation [25]-[27]. Despite differences in the technical and theoretical approach of these alternate methods they also suggest locations (or groups of locations) for the programmer to consider when debugging.

Evaluation Methods

Some of the earliest papers in fault localization do not provide a quantitative method for evaluating performance (as is seen in later papers [28]). For instance, in the earliest CBSFL paper [1] (by Jones et al.'s) the technique is evaluated using a qualitative visualization. At the time, this was entirely appropriate as Jones was proposing a technique for visualizing test coverage for assisting the debugging process. The test coverage visualization was driven by what is now called a statistical fault localization metric (Tarantula). The evaluation visualization aggregated the visualizations all of the programs included in the study.

While, the evaluation method used in the Jones paper effectively communicated the potential of CBSFL (and got many researchers excited about the idea) it was not good way to compare multiple fault localization techniques. In 2005 Jones and Harrold [29] conducted a study which compared their Tarantula technique to 3 other techniques: Set Union and Intersection [30], Nearest Neighbor [19], and Cause-Transitions [31]. These techniques all took unique approaches toward the fault localization problem and were originally evaluated in different ways. Jones and Harrold re-evaluated all 5 methods under a new common evaluation framework.

In the 2005 paper, Jones and Harrold evaluate the effectiveness of each technique by using the technique to rank the statements in the subject programs. Each technique ranked the statements from most likely to be the cause of the fault to least likely. For Tarantula, the statements are ranked using the Tarantula suspiciousness score:³

Definition. Tarantula Rank Score [29]

Given a set of locations $L$ with their suspiciousness scores $s(l)$ for $l \in L$ the Rank Score for a location $l \in L$ is: \[\begin{aligned} {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) > s(l) \right\} }\right|}} + {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) = s(l) \right\} }\right|}} \end{aligned}\]

For Set Union and Intersection, Nearest Neighbor and Cause-Transitions the statements are ranked using a System Dependence Graph (SDG) [32] technique from Renieres and Reiss [19] who first suggested the ranking idea. The ranks are then used to calculate the Tarantula Rank Score.

In the Jones and Harrold evaluation the authors do not use the Tarantula Rank Score directly but instead use a version normalized by program size:

Definition. Tarantula Effectiveness Score (Expense) [29]

The percentage of program elements that do not need to be examined to find the fault when the elements are arranged according to their rank. Formally: let $n$ be the total number of program elements, and let $r(f)$ be the Tarantula Rank Score of the faulty element $f$ then the score is: \[\begin{aligned} \frac{n-r(f)}{n} \end{aligned}\]

Using the normalized effectiveness score Jones and Harrold directly compare the fault localization effectiveness of each of the considered methods. They did this in two ways. First, they presented a table (Table 2) which bucketed all the buggy versions form all the programs by the percentage given by the Tarantula Effectiveness Score. Second, they presented a figure (Figure 2) which showed the data in Table 2 as a cumulative curve.

The basic evaluation method presented by Jones and Harrold has become the standard evaluation method. Faulty statements are scored, ranked, rank-scored, normalized, and then aggregated over all versions and programs to provide an overall representation of the fault localization method's performance (a few examples: [2], [3], [33]-[36]). While the basic method has stayed fairly consistent, there has been some innovation in the scoring (both the Rank Score and the Effectiveness Scores).

For instance, Wong et al. [34] introduced the most commonly used Effectiveness Score the $\mathcal{EXAM}$ score. This score is essentially the same as the Expense score except it gives the percentage of elements which need to be examined rather than those avoided.

Definition. $\mathcal{EXAM}$ Score [34]

The percentage of program elements that need to be examined to find the fault when the elements are arranged according to their rank. Formally: let $n$ be the total number of program elements, and let $r(f)$ be the Tarantula Rank Score of the faulty element $f$ then the score is: \[\begin{aligned} \frac{r(f)}{n} \end{aligned}\]

Ali et al. [37] identified an important problem with the Jones and Harrold evaluation: some fault localization metrics and algorithms rank statements equally. This is captured in the second term in the definition for the Tarantula Rank Score. However, Ali points out that this introduces bias towards algorithms that always assign unique scores (that are close together) rather than those that would score the same group of statement equally. The fix is to instead compute the expected number of statements the programmer would examine if they chose the next equally scored element at random.

Definition. Rank Score

Gives the expected number of locations a programmer would inspect before finding the bug. Formally, given a set of locations $L$ with their suspiciousness scores $s(l)$ for $l \in L$ the Rank Score for a location $l \in L$ is [37]: \[\begin{aligned} {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) > s(l) \right\} }\right|}} + \frac{ {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) = s(l) \right\} }\right|}} }{ 2 } \end{aligned}\]

Following Ali, we recommend utilizing the above definition for Rank Score over the Tarantula definition.

Parin and Orso [38] conducted a user study which looked at the programmer experience of using a statistical fault localization tool (Tarantula [1]). Among their findings they found that programmers would not look deeply through the list of locations and would instead only consider the first few items. As a result they encouraged studies to no longer report scores as percentages. While some studies still report the percentages most studies are now reporting the absolute (non-percentage) rank scores. Reporting as absolute scores is important for another reason, if percentage ranks are reported larger programs can have much larger absolute ranks for the same percentage rank. This biases the evaluation toward large programs even when the actual localization result is poor.

Steimann et al. [33] identified a number of threats to validity in CBSFL studies including: heterogeneous subject programs, poor test suites, small sample sizes, unclear sample spaces, flaky tests, total number of faults, and masked faults. For evaluation they used the Rank Score modified to deal with $k$ faults tied at the same rank.

Definition. Steimann Rank Score

Gives the expected number of locations a programmer would inspect before finding the bug when multiple faulty statements have the same rank. Formally, given a set of locations $L$ with their suspiciousness scores $s(l)$ for $l \in L$ the Rank Score for a location $l \in L$ is [33]: \[\begin{aligned} & {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) > s(l) \right\} }\right|}}\\ & + \frac{ {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) = s(l) \right\} }\right|}} + 1 }{ {{\left|{ \left\{ x ~:~ x \in L \wedge s(x) = s(l) \wedge x \text{ is a faulty location} \right\}}\right|}} + 1 } \end{aligned}\]

Moon et al. [39] proposed Locality Information Loss (LIL) as an alternative evaluation framework. LIL models the localization result as a probability distribution constructed from the suspiciousness scores:

Definition. LIL Probability Distribution

Let $\tau$ be a suspicious metric normalized to the $[0,1]$ range of reals. Let $n$ be the number statements in the program. Let $S$ be the set of statements. For all $1 \le i \le n$ let $s_i \in S$. The constructed probability distribution is: \[\begin{aligned} P_{\tau}(s_i) = \frac{\tau(s_i)}{\sum^{n}_{j=1} \tau(s_j)} \end{aligned}\]

LIL uses a measure of distribution divergence (Kullback-Leibler) to compute a score of how different the constructed distribution is from the "perfect" expected distribution. The advantage of the LIL framework is it does not depend on a list of ranked statements and can be applied to non-statistical methods (using a synthetic $\tau$). The disadvantage of LIL is it does not indicate programmer effort (as indicated by the Rank Score). However, it may be a better metric to use when evaluating fault localization systems as a component for automated bug repair systems.

Pearson et al. [28] re-evaluated a number of previous results using new real world subject programs with real defects and test suites. In contrast to previous work they made use of statistical hypothesis testing and confidence intervals to test the significance of the results. To evaluate the performance of each technique under study they used the $\mathcal{EXAM}$ score reporting best, average, and worst case results for multi-statement faults.

T-Score [40] is designed for non-statistical fault localization methods which produce a small set of suspicious statements in the program. To evaluate how helpful these reports are T-Score uses the Program Dependence Graph (PDG) [32], [41] to compute a set of vertices in the graph that must be examined in order to reach any faulty vertex. This set is computed via a breadth first search from the set of vertices in the report. Finally, the score is computed as the percentage of examined vertices out of the total number of vertices in the graph.

Multiple Fault Evaluations

With the exception of LIL, the evaluation methods discussed so far are generally defined to operate with a single faulty location. However, there may be multiple faults or multiple locations associated with a single fault or both. Multiple faults can interact [42] and interfere with the performance of the fault localizer. For evaluation purposes one of the most popular methods is to take either best result [43], the average result [14], [44], or the worst result [43].

Evaluating Other Techniques

One of the challenges with the methods presented so far is they may not work well for evaluating alternate fault localization methods. For instance, information retrieval based localization methods do not necessarily score and rank every program location. Instead they produce a report of associated regions. Jones and Harrold [29] used a synthetic ranking system [19] based on the SDG [32] which in principle could be used in such situation. However, like the T-Score it uses an arbitrary method (minimal dependence spheres) to compute the number of SDG nodes which must be examined.

The LIL method could also potentially be used to evaluate alternative methods. It does not rely on ranking but instead on the suspiciousness scores which it converts into a probability distribution. To support evaluating report based localization the reports are converted to a probability distribution with all locations in the report set to a equal high probability and all locations not in the report set to a tiny probability.

Suspicious Behavior Based Fault Localization [7] requires particular care. These methods are produce a ranked set of "behaviors" which are structured groups of interacting program locations. The structure could be a call invocation structure [45]-[49], a general control flow structure [7], [13], [50], or even an information flow structure [51]. The structures are scored and ranked similar to CBSFL. However, unlike in CBSFL all program locations are not necessarily included. In the past, studies have used a variety of techniques to evaluated the effectiveness including precision and recall [13] and scores based off of the $\mathcal{EXAM}$ score [7].

Another subtle special case involves comparing statistical techniques which operate on different granularity levels. As mentioned previously, coverage can be collected at any granularity level: expression, statement, basic block, method or function, class, file, and even non-structural elements such as paths. Any of the CBSFL metrics can be used with any of these granularities. However, a single study using any of the previous evaluation methods must keep the granularity consistent. This makes it impossible to compare across granularity levels. This is makes it particularly difficult to accurately compare method level behavioral approaches [45] to CBSFL.

Assumptions

The biggest assumption that all evaluation models make is so-called perfect bug understanding which assumes programmers will recognize a bug as soon as they "examine" the faulty location. This assumption is obviously false [38]. However, it continues to be a useful simplifying assumption for evaluation purposes of the localization algorithms. From the standpoint of automated fault localization there are really two tasks: 1) finding the fault and 2) explaining the fault. Assuming perfect bug understanding is reasonable for evaluating a tools performance on task 1. However, there is the important caveat that programmers need more assistance at task 2. As a research community we do not currently have a standard method for evaluating our algorithmic performance on task 2.

The second assumption is that programmers will follow the rank list or suspiciousness scores when debugging a program using a fault localization tool. This assumption is obviously false as well [38]. A programmer may follow the list for the very first item and even the second but where they go from there is likely only partially influenced by the list. The bigger influence will be from the conclusions they are drawing from what they learn upon inspecting each location.

References

[4] A. Aho, R. Sethi, M. S. Lam, and J. D. Ullman, Compilers: principles, techniques, and tools. 2007.

[5] P. S. Kochhar, X. Xia, D. Lo, and S. Li, "Practitioners' expectations on automated fault localization," in Proceedings of the 25th international symposium on software testing and analysis - issta 2016, 2016, pp. 165-176, doi:10.1145/2931037.2931051.

[6] G. G. K. Baah, A. Podgurski, and M. J. M. Harrold, "Causal inference for statistical fault localization," in Proceedings of the 19th international symposium on software testing and analysis, 2010, pp. 73-84, doi:10.1145/1831708.1831717.

[7] T. A. D. Henderson and A. Podgurski, "Behavioral Fault Localization by Sampling Suspicious Dynamic Control Flow Subgraphs," in IEEE conference on software testing, validation and verification, 2018.

[8] C. C. Aggarwal and J. Han, Eds., Frequent Pattern Mining. Cham: Springer International Publishing, 2014.

[9] R. Agrawal, T. Imieliński, and A. Swami, "Mining association rules between sets of items in large databases," ACM SIGMOD Record, vol. 22, no. 2, pp. 207-216, Jun. 1993, doi:10.1145/170036.170072.

[10] X. Yan and J. Han, "gSpan: graph-based substructure pattern mining," in 2002 ieee international conference on data mining, 2002. proceedings., 2002, pp. 721-724, doi:10.1109/ICDM.2002.1184038.

[11] C. C. Aggarwal, M. A. Bhuiyan, and M. A. Hasan, "Frequent Pattern Mining Algorithms: A Survey," in Frequent pattern mining, Cham: Springer International Publishing, 2014, pp. 19-64.

[12] X. Yan, H. Cheng, J. Han, and P. S. Yu, "Mining Significant Graph Patterns by Leap Search," in Proceedings of the 2008 acm sigmod international conference on management of data, 2008, pp. 433-444, doi:10.1145/1376616.1376662.

[13] H. Cheng, D. Lo, Y. Zhou, X. Wang, and X. Yan, "Identifying Bug Signatures Using Discriminative Graph Mining," in Proceedings of the eighteenth international symposium on software testing and analysis, 2009, pp. 141-152, doi:10.1145/1572272.1572290.

[14] R. Abreu, P. Zoeteweij, and A. Van Gemund, "An Evaluation of Similarity Coefficients for Software Fault Localization," in 2006 12th pacific rim international symposium on dependable computing (prdc'06), 2006, pp. 39-46, doi:10.1109/PRDC.2006.18.

[15] R. Abreu, P. Zoeteweij, R. Golsteijn, and A. J. C. van Gemund, "A practical evaluation of spectrum-based fault localization," Journal of Systems and Software, vol. 82, no. 11, pp. 1780-1792, 2009, doi:10.1016/j.jss.2009.06.035.

[16] P. Agarwal and A. P. Agrawal, "Fault-localization Techniques for Software Systems: A Literature Review," SIGSOFT Softw. Eng. Notes, vol. 39, no. 5, pp. 1-8, Sep. 2014, doi:10.1145/2659118.2659125.

[17] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, "A Survey on Software Fault Localization," IEEE Transactions on Software Engineering, vol. 42, no. 8, pp. 707-740, Aug. 2016, doi:10.1109/TSE.2016.2521368.

[18] A. Zeller, "Yesterday, My Program Worked. Today, It Does Not. Why?" SIGSOFT Softw. Eng. Notes, vol. 24, no. 6, pp. 253-267, Oct. 1999, doi:10.1145/318774.318946.

[19] M. Renieres and S. Reiss, "Fault localization with nearest neighbor queries," in 18th ieee international conference on automated software engineering, 2003. proceedings., 2003, pp. 30-39, doi:10.1109/ASE.2003.1240292.

[20] F. Tip, "A survey of program slicing techniques," Journal of programming languages, vol. 3, no. 3, pp. 121-189, 1995.

[21] X. Mao, Y. Lei, Z. Dai, Y. Qi, and C. Wang, "Slice-based statistical fault localization," Journal of Systems and Software, vol. 89, no. 1, pp. 51-62, 2014, doi:10.1016/j.jss.2013.08.031.

[22] A. Marcus, A. Sergeyev, V. Rajlieh, and J. I. Maletic, "An information retrieval approach to concept location in source code," Proceedings - Working Conference on Reverse Engineering, WCRE, pp. 214-223, 2004, doi:10.1109/WCRE.2004.10.

[23] J. Zhou, H. Zhang, and D. Lo, "Where should the bugs be fixed? More accurate information retrieval-based bug localization based on bug reports," Proceedings - International Conference on Software Engineering, pp. 14-24, 2012, doi:10.1109/ICSE.2012.6227210.

[24] T.-D. B. Le, R. J. Oentaryo, and D. Lo, "Information retrieval and spectrum based bug localization: better together," in Proceedings of the 2015 10th joint meeting on foundations of software engineering - esec/fse 2015, 2015, pp. 579-590, doi:10.1145/2786805.2786880.

[25] S. Artzi, J. Dolby, F. Tip, and M. Pistoia, "Directed Test Generation for Effective Fault Localization," in Proceedings of the 19th international symposium on software testing and analysis, 2010, pp. 49-60, doi:10.1145/1831708.1831715.

[26] S. K. Sahoo, J. Criswell, C. Geigle, and V. Adve, "Using likely invariants for automated software fault localization," in Proceedings of the eighteenth international conference on architectural support for programming languages and operating systems, 2013, vol. 41, p. 139, doi:10.1145/2451116.2451131.

[27] A. Perez, R. Abreu, and A. Riboira, "A Dynamic Code Coverage Approach to Maximize Fault Localization Efficiency," J. Syst. Softw., vol. 90, pp. 18-28, Apr. 2014, doi:10.1016/j.jss.2013.12.036.

[28] S. Pearson, J. Campos, R. Just, G. Fraser, R. Abreu, M. D. Ernst, D. Pang, and B. Keller, "Evaluating and Improving Fault Localization," in Proceedings of the 39th international conference on software engineering, 2017, pp. 609-620, doi:10.1109/ICSE.2017.62.

[29] J. A. Jones and M. J. Harrold, "Empirical Evaluation of the Tarantula Automatic Fault-localization Technique," in Proceedings of the 20th ieee/acm international conference on automated software engineering, 2005, pp. 273-282, doi:10.1145/1101908.1101949.

[30] H. Agrawal, J. Horgan, S. London, and W. Wong, "Fault localization using execution slices and dataflow tests," in Proceedings of sixth international symposium on software reliability engineering. issre'95, 1995, pp. 143-151, doi:10.1109/ISSRE.1995.497652.

[31] H. Cleve and A. Zeller, "Locating causes of program failures," Proceedings of the 27th international conference on Software engineering - ICSE '05, p. 342, 2005, doi:10.1145/1062455.1062522.

[32] S. Horwitz, "Identifying the Semantic and Textual Differences Between Two Versions of a Program," SIGPLAN Not., vol. 25, no. 6, pp. 234-245, Jun. 1990, doi:10.1145/93548.93574.

[33] F. Steimann, M. Frenkel, and R. Abreu, "Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators," Proceedings of the 2013 International Symposium on Software Testing and Analysis - ISSTA 2013, p. 314, 2013, doi:10.1145/2483760.2483767.

[34] E. Wong, T. Wei, Y. Qi, and L. Zhao, "A Crosstab-based Statistical Method for Effective Fault Localization," in 2008 international conference on software testing, verification, and validation, 2008, pp. 42-51, doi:10.1109/ICST.2008.65.

[35] D. Landsberg, H. Chockler, D. Kroening, and M. Lewis, "Evaluation of Measures for Statistical Fault Localisation and an Optimising Scheme," in International conference on fundamental approaches to software engineering, 2015, vol. 9033, pp. 115-129, doi:10.1007/978-3-662-46675-9.

[36] Y. Zheng, Z. Wang, X. Fan, X. Chen, and Z. Yang, "Localizing multiple software faults based on evolution algorithm," Journal of Systems and Software, vol. 139, pp. 107-123, 2018, doi:10.1016/j.jss.2018.02.001.

[38] C. Parnin and A. Orso, "Are Automated Debugging Techniques Actually Helping Programmers?" in ISSTA, 2011, pp. 199-209.

[39] S. Moon, Y. Kim, M. Kim, and S. Yoo, "Ask the Mutants: Mutating faulty programs for fault localization," Proceedings - IEEE 7th International Conference on Software Testing, Verification and Validation, ICST 2014, pp. 153-162, 2014, doi:10.1109/ICST.2014.28.

[40] C. Liu, C. Chen, J. Han, and P. S. Yu, "GPLAG: Detection of Software Plagiarism by Program Dependence Graph Analysis," in Proceedings of the 12th acm sigkdd international conference on knowledge discovery and data mining, 2006, pp. 872-881, doi:10.1145/1150402.1150522.

[41] J. Ferrante, K. J. Ottenstein, and J. D. Warren, "The program dependence graph and its use in optimization," vol. 9. pp. 319-349, Jul-1987.

[42] N. DiGiuseppe and J. A. Jones, "On the influence of multiple faults on coverage-based fault localization," in Proceedings of the 2011 international symposium on software testing and analysis - issta '11, 2011, p. 210, doi:10.1145/2001420.2001446.

[43] W. E. Wong, Y. Qi, L. Zhao, and K. Y. Cai, "Effective fault localization using code coverage," Proceedings - International Computer Software and Applications Conference, vol. 1, no. Compsac, pp. 449-456, 2007, doi:10.1109/COMPSAC.2007.109.

[44] L. Naish, H. J. Lee, and K. Ramamohanarao, "A model for spectra-based software diagnosis," ACM Transactions on Software Engineering and Methodology, vol. 20, no. 3, pp. 1-32, 2011, doi:10.1145/2000791.2000795.

[45] C. Liu, H. Yu, P. S. Yu, X. Yan, H. Yu, J. Han, and P. S. Yu, "Mining Behavior Graphs for ‘Backtrace' of Noncrashing Bugs," in Proceedings of the 2005 siam international conference on data mining, 2005, pp. 286-297, doi:10.1137/1.9781611972757.26.

[46] F. Eichinger, K. Böhm, and M. Huber, "Mining Edge-Weighted Call Graphs to Localise Software Bugs," in European conference machine learning and knowledge discovery in databases, 2008, pp. 333-348, doi:10.1007/978-3-540-87479-9_40.

[47] T. M. Chilimbi, B. Liblit, K. Mehra, A. V. Nori, and K. Vaswani, "HOLMES: Effective Statistical Debugging via Efficient Path Profiling," in Proceedings of the 31st international conference on software engineering, 2009, pp. 34-44, doi:10.1109/ICSE.2009.5070506.

[48] A. Yousefi and A. Wassyng, "A Call Graph Mining and Matching Based Defect Localization Technique," in 2013 ieee sixth international conference on software testing, verification and validation workshops, 2013, pp. 86-95, doi:10.1109/ICSTW.2013.17.

[49] T. Diamantopoulos and A. Symeonidis, "Localizing Software Bugs using the Edit Distance of Call Traces," International Journal on Advances in Software, vol. 7, no. 1 & 2, pp. 277-288, 2014.

[50] Z. Mousavian, M. Vahidi-Asl, and S. Parsa, "Scalable Graph Analyzing Approach for Software Fault-localization," in Proceedings of the 6th international workshop on automation of software test, 2011, pp. 15-21, doi:10.1145/1982595.1982599.

[51] F. Eichinger, K. Krogmann, R. Klug, and K. Böhm, "Software-defect Localisation by Mining Dataflow-enabled Call Graphs," in Proceedings of the 2010 european conference on machine learning and knowledge discovery in databases: Part i, 2010, pp. 425-441.

The coverage can be collected at other levels as well. For instance there has been work which collects it at the class, method, and basic block levels.↩
A basic block is a sequence of sequential instructions, always entered from the first instruction and exited from the last [4].↩
It is ahistorical to call Tarantula metric a suspiciousness score when referring to the 2002 paper [1]. Jones introduced the term suspiciousness score in the 2005 paper [29] for the purpose of ranking the statements. However, the term is now in common use and it was explained above.↩

Behavioral Fault Localization by Sampling Suspicious Dynamic Control Flow Subgraphs

2018-04-03T00:00:00-04:00

Tim A. D. Henderson and Andy Podgurski. Behavioral Fault Localization by Sampling Suspicious Dynamic Control Flow Subgraphs. ICST 2018.
DOI. PDF. WEB.

Abstract

We present a new algorithm, Score Weighted Random Walks (SWRW), for behavioral fault localization. Behavioral fault localization localizes faults (bugs) in programs to a group of interacting program elements such as basic blocks or functions. SWRW samples suspicious (or discriminative) subgraphs from basic-block level dynamic control flow graphs collected during the execution of passing and failing tests. The suspiciousness of a subgraph may be measured by any one of a family of new metrics adapted from probabilistic formulations of existing coverage-based statistical fault localization metrics. We conducted an empirical evaluation of SWRW with nine subgraph-suspiciousness measures on five real-world subject programs. The results indicate that SWRW outperforms previous fault localization techniques based on discriminative subgraph mining.

Introduction

Automated fault localization techniques have been developed to help programmers locate software faults (bugs) responsible for observed software failures. Many of these techniques are statistical in nature (e.g., {Jones 2002, Liblit 2005, Liu 2005}). They employ statistical measures of the association, if any, between the occurrence of failures and the execution of particular program elements like statements or conditional branches. The program elements that are most strongly associated with failures are identified as "suspicious", so that developers can examine them to see if they are faulty. The association measures that are used are often called suspiciousness metrics {Jones 2004}. Such statistical fault localization (SFL) techniques typically require execution profiles (or spectra) and PASS/FAIL labels for a set of both passing and failing program runs. Each profile entry characterizes the execution of a particular program element during a run. For example, a statement-coverage profile for a run indicates which statements were executed at least once. The profiles are collected with program instrumentation, while the labels are typically supplied by software testers or end users.

Coverage Based Statistical Fault Localization (CBSFL) techniques compute a suspiciousness score for each program element. The elements are presented to the programmer as a ranked list with the most suspicious elements at the top {Liblit 2005}. Programmers utilize the list to guide their debugging effort by starting at the most suspicious element and moving down. The effectiveness of a CBSFL suspiciousness metric is judged by how accurately it ranks a bug's (or bugs) location in a program. The higher in the list the bug's location appears the better the metric performs {Lucia 2014}.

Kochhar et al. {Kochhar 2016} recently surveyed 386 software engineering practitioners about their expectations for automated fault localization. While practitioners indicated their preference for very accurate algorithms, over 85% of respondents also indicated their preference for tools which help them understand the output of fault localization algorithms. This is an important finding as most statistical approaches do not provide an explanation of their results. The SFL techniques often simply compute suspiciousness measures and rank the program elements accordingly. These rankings may be helpful, but without more information programmers could overlook the faulty element even when it is ranked highly.

Figure 1. Process for localizing faults with discriminative graph mining.

Suspicious-Behavior Based Fault Localization (SBBFL) is a statistical fault localization technique that aids the programmer in understanding suspiciousness scores by providing a context of interacting elements. (NB: Dynamic slicing {Tip 1995} also provides such a context, but does not in itself involve suspiciousness measures) Instead of implicating a single element, SBBFL implicates a larger runtime behavior (see process in Figure 1). The implicated control flow paths (or subgraphs) may help the programmer understand the nature of a bug {Cheng 2009}.

We present a new algorithm, Score-Weighted Random Walks (SWRW), for behavioral fault-localization. SWRW belongs to a family of discriminative graph-mining algorithms that have previously been used for behavioral fault localization { Liu 2005, DiFatta 2006, Eichinger 2008, Cheng 2009, Eichinger 2010, Eichinger 2011, Mousavian 2011, Mariani 2011, Parsa 2011, Yousefi 2013 }. Graph mining is very powerful in principle but algorithms must make trade-offs to address the challenging combinatorics of the graph mining problem. Our new algorithm, SWRW, mitigates the combinatorics by randomly sampling "suspicious" subgraphs from dynamic control flow graphs. During the sampling process, the most suspicious subgraphs (as judged by a suitable suspiciousness metric) are favored for selection. Unlike previous algorithms, SWRW can be used with a wide variety of suspiciousness metrics --- which allows it to use better metrics than available to previous work. Even when using the same metric as similar algorithms, SWRW localizes faults more accurately than they do.

Summary of Contributions

A new behavioral fault localization algorithm, SWRW, that samples suspicious subgraphs from dynamic control flow graphs. Unlike similar algorithms, SWRW can be used with a variety of suspiciousness metrics.
New generalizations of existing suspiciousness metrics that allow them to be applied to behaviors represented by subgraphs of dynamic control flow graphs.
An empirical study whose results suggest that SWRW is more accurate than similar algorithms.
Dynagrok, a new instrumentation, mutation, and analysis tool for the Go programming language.

Dynagrok: A New Profiling Tool

All Coverage-Based Statistical Fault Localization (CBSFL) techniques use coverage profiles to gather information on how software behaved when executed on a set of test inputs. A coverage profile typically contains an entry for each program element of a given kind (e.g., statement, basic block, branch, or function), which records whether (and possibly how many times) the element was executed during the corresponding program run. The profiles and PASS/FAIL labels for all tests are then used to compute a statistical suspiciousness score for each program element.

The process of gathering the coverage information from running programs is called profiling and there are many different varieties of profilers and profiling techniques available. Coverage profiling is a simple and widely implemented technique, which is why it has been widely used by the fault localization community. Another technique is tracing, which logs the sequence of program locations as they are executed. The traces provide detailed information on the behavior of the program but could grow to be very large for long running programs. This paper uses execution flow profiling which computes the dynamic interprocedural control flow graph of a program's execution. This provides some of the benefits of tracing without recording an excessive amount of data.

To capture execution flow profiles we developed Dynagrok, a new analysis, instrumentation and mutation platform for the Go programming language. Go is a newer language (2009) from Google that has been seeing increasing adoption in industry. It has been adopted for web programming, systems programming, "DevOps," network programing, and databases {http://tiobe.com/tiobe-index/, http://blog.golang.org/survey2017-results}. Dynagrok builds upon the abstract syntax tree (AST) representation provided by the Go standard library.

Dynagrok collects profiles by inserting instrumentation into the AST of the subject program. The profiles currently collected are dynamic control flow graphs (DCFGs) whose vertices represent basic blocks. A basic block is a sequence of program operations that can only be entered at the start of the sequence and can only be exited after the last operation in the sequence {Aho 2007}. A basic-block level control flow graph (CFG) is a directed labeled graph $g = (V, E, l)$ comprised of a finite set of vertices $V$, a set of edges $E \subseteq V \times V$, and a labeling function $l$ mapping vertices and edges to labels. Each vertex $v \in V$ represents a basic block of the program. Each edge $(u, v) \in E$ represents a transition in program execution from block $u$ to block $v$. The labeling function $l$ labels the basic blocks with a unique identifier (e.g. function-name:block-id), which is consistently applied across multiple executions but is never repeated in the same execution.

Figure 2 shows an example DCFG collected by Dynagrok for a simple program that computes terms of the Fibonacci sequence. To collect such graphs Dynagrok parses the program into an AST using Go's standard library. Dynagrok then uses a custom control flow analysis to build static control flow graphs. Each basic block holds pointers to the statements inside of the AST. The blocks also have a pointer to the enclosing lexical block in the AST. Using this information, Dynagrok inserts profiling instructions into the AST at the beginning of each basic block. The instructions inserted by Dynagrok use its dgruntime library to track the control flow of each thread (which is called a goroutine in Go). When the program shuts down (either normally or abnormally) the dgruntime library merges the flow graphs from all the threads together and writes out the result.

(Read the rest of the paper as a pdf)

Figure 2. The dynamic control flow graph (DCFG) for the program
in Listing 1 (see below).

Each vertex is a basic block with a basic block identifer (e.g. b1) that, in conjunction with the name of the containing function, serves as the label for the block (e.g. main:b1). Each edge shows the number of traversals taken during the execution of the program. Note that the loop update blocks (main:b3 and fib:b7) will not be in the profiles because Dynagrok instruments the Go source code and profiling instructions cannot be syntactically inserted in those locations. The instrumented program is shown on the right.

package main

func main() {
    for i := 10; i >= -2; i-- {
        println(fib(i))
    }
}

func fib(x int) int {
    if x < 0 {
        return 0
    }
    p, c := 0, 1
    for i := 0; i < x; i++ {
        n := p + c
        p, c = c, n
    }
    return c
}

Listing 1. An example Go program to compute the Fibonacci Sequence.

package main

import "dgruntime"

func main() {
    defer func() { dgruntime.Shutdown() }()
    dgruntime.EnterFunc("main")
    defer func() { dgruntime.ExitFunc("main") }()
    for i := 10; dgruntime.EnterBlkFromCond(2) && i >= -2; i-- {
        dgruntime.EnterBlk(4)
        println(fib(i))
    }
}

func fib(x int) int {
    dgruntime.EnterFunc("fib")
    defer func() { dgruntime.ExitFunc("fib") }()
    if x < 0 {
        dgruntime.EnterBlk(2)
        return 0
    }
    dgruntime.EnterBlk(3)
    p, c := 0, 1
    for i := 0; dgruntime.EnterBlkFromCond(4) && i < x; i++ {
        dgruntime.EnterBlk(6)
        n := p + c
        p, c = c, n
    }
    dgruntime.EnterBlk(5)
    return c
}

Listing 2. The program shown in Listing 1 after the instrumentation has been inserted by dynagrok.

How to Tokenize Complex Strings with Lexmachine

2017-12-08T00:00:00-05:00

This article is about using lexmachine to tokenize strings (split up into component parts) in the Go (golang) programming language. If you find yourself processing a complex file format or network protocol this article will walk you through how to use lexmachine to process both accurately and quickly. If you need more help after reading this article take a look at the documentation, a tutorial, or an explainer article on how it all works.

What is string tokenization?

String tokenization takes a piece of text and splits it into categorized substrings. When conducted as part of parsing or compiling a programming language it may also be called lexical analysis. Depending on the kind of text being processed splitting it up might be very simple or extremely complicated. For instance, some simple text formats (such as basic CSV or TSV files) separate each record by newline characters (\n or ASCII 0x0A). At the other end of the spectrum, natural language (such as English text) may be very difficult to correctly tokenize.

Let's look at a quick example of tokenizing an English sentence.

Mary had a little lamb and its fleece was white as snow.

Each word will be a separate token. A token is a pair made up of the lexeme (or substring) and the token type (or category). When tokenizing English, often the token type will be the part of speech.

 Mary           had     a          little      lamb    and            ...
<Proper Noun>  <Verb>  <Article>  <Adjetive>  <Noun>  <Conjunction>   ...

Tokenizing English is very difficult in general because determining the part of speech of a particular word is not always obvious. The same word can play multiple semantic roles in a sentence. Humans consider the sentence in its entirety when determining the parts of speech for each word. This allows us to "noun verbs" and "verb nouns."

String tokenization as part of parsing

There are many common applications of tokenization which are not as difficult as assigning parts of speech to words in English. For instance, computer languages, configuration files, log files, network protocols, data interchange formats, etc... are all easy to tokenize. In these applications (as with natural language), tokenization is the first step in parsing or compilation and is called lexical analysis.

Modern compilers are designed as a "pipeline" or series of steps (called passes) which operate over a program. They take the source code and transform it step by step into the desired output (machine code, assembly, another programming language). Breaking compilation into steps simplifies each stage and makes the individual passes reusable. The start of the pipeline is shown in Figure 1.

Figure 1. The start of a compiler's pipeline of passes.
A compiler starts with source code and then preforms lexical analysis to split the code up into *tokens*. Then, the *parser* transforms that stream or list of tokens into a structured format (often an Abstract Syntax Tree).

But why tokenize before parsing? Surely, the parser could operate on the bytes of the source code rather than on a list of tokens. Some parsers do operate directly on the bytes. However, the parser can be simplified and made shorter and often more robust by defining it over tokens rather than bytes.

Consider the following, programming languages often have keywords: func, return, if. They also have "identifiers" or names. The keywords are set aside as special names and are treated by a parser the same way other names are treated as they often denote the start of special syntactic regions. In this way, they really operate in a similar manner to punctuation in written English.

However, since keywords would be valid names if they were not reserved special care must be taken in a parser which operates on bytes directly not to accidentally classify a keyword as a name. Parsers which work on tokens rather than bytes can ignore this problem as it is handled at the lexical analysis stage.

Processing a custom configuration file format

As a motivating example for why you might need to perform a tokenization operation in your daily life as a programmer let's take a look at processing a customer configuration file format. Linux has a library libsensor which allows it to read and understand the output of hardware sensors (such as CPU temperature probes). It has a custom configuration file format, sensors.conf, which describes how Linux should translate the raw readings hardware monitoring chips to real-world values -- such as voltage and temperature. The first part of my laptop's sensors.conf file begins with the following:

# It is recommended not to modify this file, but to drop your local
# changes in /etc/sensors.d/. File with names that start with a dot
# are ignored.

chip "lm78-*" "lm79-*" "lm80-*" "lm96080-*"

    label temp1 "M/B Temp"


chip "w83792d-*"

    label in0 "VcoreA"
    label in1 "VcoreB"
    label in6 "+5V"
    label in7 "5VSB"
    label in8 "Vbat"

    set in6_min  5.0 * 0.90
    set in6_max  5.0 * 1.10
    set in7_min  5.0 * 0.90
    set in7_max  5.0 * 1.10
    set in8_min  3.0 * 0.90
    set in8_max  3.0 * 1.10

Let's pretend we want to extract some information from sensors.conf files. For instance, maybe we want to know for each chip what labels have been defined. Straightforward enough and definitely doable "by hand" if necessary but much easier if the file is preprocessed to pull out the tokens:

Type    | Lexeme
-------------------
COMMENT | # It is recommended not to modify this file, but to drop your local
COMMENT | # changes in /etc/sensors.d/. File with names that start with a dot
COMMENT | # are ignored.
CHIP    | chip
NAME    | "lm78-*"
NAME    | "lm78-*"
NAME    | "lm79-*"
NAME    | "lm80-*"
NAME    | "lm96080-*"
LABEL   | label
NAME    | temp1
NAME    | "M/B Temp"
CHIP    | chip
NAME    | "w83792d-*"
...
SET     | set
NAME    | in8_max
FLOAT   | 3.0
STAR    | *
FLOAT   | 1.10

According to the man page the chip and label statements have the following structure (other bits left out):

chip Statement

A chip statement selects for which chips all following compute, label, ignore and set statements are meant for.

label Statement

A label statement describes how a feature should be called.

chip NAME-LIST
label NAME NAME

A NAME is a string. If it only contains letters, digits and underscores, it does not have to be quoted; in all other cases, you must use double quotes around it. Within quotes, you can use the normal escape-codes from C.

A NAME-LIST is one or more NAME items behind each other, separated by whitespace.

So to find all of the labeled "features" of a chip we need to do two things. First, find a chip statement and collect all the names associated with it. These names are actually patterns which match multiple chips but we will ignore that for this example. Second, collect all the label statements (and the two associated names). The labels go with the chips which immediately proceed them.

To implement the above idea we are going to use lexmachine to construct a lexer (or tokenizer) which split up the file into the appropriate tokens.

Defining a lexer with lexmachine

A lexer (or tokenizer) is defined by a set of categories (token types) which are defined by patterns. The patterns (in a traditional lexer) are expressed as regular expressions. As a quick and incomplete review, regular expressions (regex) are a "pattern" which describe a set of strings. A regex is made up of characters (a, b, 0, @, ...) which combine with the operators: concatenation abc, alternation a|b, grouping a(b|c)d, and repetition a*. Some examples:

abc matches {"abc"}
a|b matches {"a", "b"}
a(b|c)d matches {"abd", "acd"}
a* matches {"", "a", "aa", ...}
a(b(c|d))* matches {"a", "abc", "abd", "abcbc", "abcbd", ...}

A fairly close reading of the man page for sensors.conf gives the following regular expressions to tokenize the file. (Note: [] define character classes which express a number of alternative characters to match)

AT: @
PLUS: \+
STAR: \*
DASH: -
SLASH: /
BACKSLASH: \\
CARROT: \^
BACKTICK: `
COMMA: ,
LPAREN: \(
RPAREN: \)
BUS: bus
CHIP: chip
LABEL: label
COMPUTE: compute
IGNORE: ignore
SET: set
NUMBER: [0-9]*\.?[0-9]+
COMMENT: #[^\n]*
SPACE: \s+
NAME: [a-zA-Z_][a-zA-Z0-9_]*
NAME: "((\\\\)|(\\.)|([^"\\]))*"

With these regular expressions in hand let's define a lexer using lexmachine and use it to tokenize an example sensors.conf.

Aside, can't we just use regexp?

A lexical analysis engine is very similar to a standard regular expression engine. However, the problem which regular expression engines solve (matching whole strings or finding patterns inside of strings) is slightly different than the lexial analysis problem. You could implement a lexical analyzer with the regexp package but it would be much slower than lexmachine. Lexical analysis engines are fast because they use the same theoretical framework as is used to implement regular expression engines with the following adjustments:

Prefixes of a string are matched

All patterns are matched at the same time

The pattern which matches the longest prefix wins

In case of ties, the pattern with the highest precedence wins

Constructing a `*lexmachine.Lexer`

Importing the packages

import (
    "github.com/timtadh/lexmachine"
    "github.com/timtadh/lexmachine/machines"
)

Defining the tokens types

There are many ways to represent the token types (categories). In this example, the names of the types are defined as a list of strings. The types themselves (ids) will be the index of the names. An init function is used to construct a reverse mapping from name to id.

var tokens = []string{
    "AT", "PLUS", "STAR", "DASH", "SLASH", "BACKSLASH", "CARROT", "BACKTICK",
    "COMMA", "LPAREN", "RPAREN", "BUS", "COMPUTE", "CHIP", "IGNORE", "LABEL",
    "SET", "NUMBER", "NAME", "COMMENT", "SPACE",
}
var tokmap map[string]int

func init() {
    tokmap = make(map[string]int)
    for id, name := range tokens {
        tokmap[name] = id
    }
}

Constructing a new lexer object

lexer := lexmachine.NewLexer()

Adding a single pattern

Patterns are added using the lexer.Add method. Let's add the pattern for the chip keyword:

lexer.Add([]byte(`chip`), func(s *lexmachine.Scanner, m *machines.Match) (interface{}, error) {
    return 0, nil
})

The Add method takes two arguments. The first is the regular expression and the second is the lexing action function. The action is called on pattern discovery. This action typically takes a low-level *machines.Match object and turns it into a token representation of your (the user's) choice.

For users who do not have strict requirements for how tokens are represented the *lexmachine.Token object provides a useful implementation. *lexmachine.Scanner has a utility method for constructing the tokens which helps us write a simple getToken action:

func getToken(tokenType int) lexmachine.Action {
    return func(s *lexmachine.Scanner, m *machines.Match) (interface{}, error) {
        return s.Token(tokenType, string(m.Bytes), m), nil
    }
}

Adding all the patterns

func newLexer() *lexmachine.Lexer {
    lexer := lexmachine.NewLexer()
    lexer.Add([]byte("@"), getToken(tokmap["AT"]))
    lexer.Add([]byte(`\+`), getToken(tokmap["PLUS"]))
    lexer.Add([]byte(`\*`), getToken(tokmap["STAR"]))
    lexer.Add([]byte("-"), getToken(tokmap["DASH"]))
    lexer.Add([]byte("/"), getToken(tokmap["SLASH"]))
    lexer.Add([]byte("\\"), getToken(tokmap["BACKSLASH"]))
    lexer.Add([]byte(`\^`), getToken(tokmap["CARROT"]))
    lexer.Add([]byte("`"), getToken(tokmap["BACKTICK"]))
    lexer.Add([]byte(","), getToken(tokmap["COMMA"]))
    lexer.Add([]byte(`\(`), getToken(tokmap["LPAREN"]))
    lexer.Add([]byte(`\)`), getToken(tokmap["RPAREN"]))
    lexer.Add([]byte("bus"), getToken(tokmap["BUS"]))
    lexer.Add([]byte("chip"), getToken(tokmap["CHIP"]))
    lexer.Add([]byte("label"), getToken(tokmap["LABEL"]))
    lexer.Add([]byte("compute"), getToken(tokmap["COMPUTE"]))
    lexer.Add([]byte("ignore"), getToken(tokmap["IGNORE"]))
    lexer.Add([]byte("set"), getToken(tokmap["SET"]))
    lexer.Add([]byte(`[0-9]*\.?[0-9]+`), getToken(tokmap["NUMBER"]))
    lexer.Add([]byte(`[a-zA-Z_][a-zA-Z0-9_]*`), getToken(tokmap["NAME"]))
    lexer.Add([]byte(`"((\\.)|([^"\\]))*"`), func(s *lexmachine.Scanner, m *machines.Match) (interface{}, error) {
        return s.Token(tokmap["NAME"], string(m.Bytes[1:len(m.Bytes)-1]), m), nil
    })
    lexer.Add([]byte(`#[^\n]*`), getToken(tokmap["COMMENT"]))
    lexer.Add([]byte(`( |\t|\f)+`), getToken(tokmap["SPACE"]))
    lexer.Add([]byte(`\\\n`), getToken(tokmap["SPACE"]))
    lexer.Add([]byte(`\n|\r|\n\r`), getToken(tokmap["NEWLINE"]))
    err := lexer.Compile()
    if err != nil {
        panic(err)
    }
    return lexer
}

Skipping a pattern

Sometimes it is advantageous to not emit tokens for certain patterns and to instead skip them. Commonly this occurs for whitespace and comments. To skip a pattern simply have the action return nil, nil:

func skip(scan *Scanner, match *machines.Match) (interface{}, error) {
    return nil, nil
}

lexer.Add([]byte(`\s+`), skip)     // skip whitespace
lexer.Add([]byte(`#[^\n]*`), skip) // skip comments

In our example for sensors.conf we only care about the keywords and the NAME tokens. Let's skip the rest:

func newLexer() *lexmachine.Lexer {
    lexer := lexmachine.NewLexer()
    lexer.Add([]byte(`@`), skip)
    lexer.Add([]byte(`\+`), skip)
    lexer.Add([]byte(`\*`), skip)
    lexer.Add([]byte("-"), skip)
    lexer.Add([]byte("/"), skip)
    lexer.Add([]byte("\\"), skip)
    lexer.Add([]byte(`\^`), skip)
    lexer.Add([]byte("`"), skip)
    lexer.Add([]byte(","), skip)
    lexer.Add([]byte(`\(`), skip)
    lexer.Add([]byte(`\)`), skip)
    lexer.Add([]byte("bus"), skip)
    lexer.Add([]byte("chip"), getToken(tokmap["CHIP"]))
    lexer.Add([]byte("label"), getToken(tokmap["LABEL"]))
    lexer.Add([]byte("compute"), skip)
    lexer.Add([]byte("ignore"), skip)
    lexer.Add([]byte("set"), skip)
    lexer.Add([]byte(`[0-9]*\.?[0-9]+`), skip)
    lexer.Add([]byte(`[a-zA-Z_][a-zA-Z0-9_]*`), getToken(tokmap["NAME"]))
    lexer.Add([]byte(`"((\\.)|([^"\\]))*"`), func(s *lexmachine.Scanner, m *machines.Match) (interface{}, error) {
        return s.Token(tokmap["NAME"], string(m.Bytes[1:len(m.Bytes)-1]), m), nil
    })
    lexer.Add([]byte(`#[^\n]*`), skip)
    lexer.Add([]byte(`( |\t|\f)+`), skip)
    lexer.Add([]byte(`\\\n`), skip)
    lexer.Add([]byte(`\n|\r|\n\r`), getToken(tokmap["NEWLINE"]))
    err := lexer.Compile()
    if err != nil {
        panic(err)
    }
    return lexer
}

Constructing the lexer exactly once

The lexer should be constructed and compiled exactly once: on program startup (unless the regular expressions are defined dynamically). This ensures that you only pay the compilation costs at program startup and get the maximum benefit from the efficient DFA representation produced by compilation.

var lexer *lexmachine.Lexer

func init() {
    lexer = newLexer()
}

Example tokenization

scanner, err := lexer.Scanner([]byte("chip chip1 chip2 label name value"))
if err != nil {
    panic(err)
}
for tk, err, eof := scanner.Next(); !eof; tk, err, eof = scanner.Next() {
    if err != nil {
        panic(err)
    }
    token := tk.(*lexmachine.Token)
    fmt.Printf("%-7v | %-25q | %v:%v-%v:%v\n",
        tokens[token.Type],
        token.Value.(string),
        token.StartLine,
        token.StartColumn,
        token.EndLine,
        token.EndColumn)
}

Output

CHIP    | "chip"                    | 1:1-1:4
NAME    | "chip1"                   | 1:6-1:10
NAME    | "chip2"                   | 1:12-1:16
LABEL   | "label"                   | 1:18-1:22
NAME    | "name"                    | 1:24-1:27
NAME    | "value"                   | 1:29-1:33

For more details on the scanner interface consult the narrative documentation.

Associating `label`s with `chip`s

There are several ways we could approach taking the output of the scanner above and using it to find what labels go with what chip names. In this example, a very simple, short, but ugly method is going to be demonstrated. Inside of the scanning for-loop a simple state machine will track what the current statement a particular NAME token belongs to. If it belongs to a chip statement or a label statement it is appropriately associated. Finally, whenever a new CHIP token is encountered the current buffer of labels are associated with the previous chip statement.

func printChipLabels(text []byte) error {
    scanner, err := lexer.Scanner(text)
    if err != nil {
        return err
    }
    order := make([]string, 0, 10)
    chipLabels := make(map[string][]string)
    curChips := make([]string, 0, 10)
    curLabels := make([]string, 0, 10)
    curLabel := make([]string, 0, 2)
    state := "none"
    addChips := func() {
        for _, chip := range curChips {
            if _, has := chipLabels[chip]; !has {
                order = append(order, chip)
            }
            chipLabels[chip] = append(chipLabels[chip], curLabels...)
        }
        curChips = make([]string, 0, 10)
        curLabels = make([]string, 0, 10)
        curLabel = make([]string, 0, 2)
    }
    for tk, err, eof := scanner.Next(); !eof; tk, err, eof = scanner.Next() {
        if err != nil {
            return err
        }
        token := tk.(*lexmachine.Token)
        switch token.Type {
        case tokmap["CHIP"]:
            addChips()
            state = "chip"
        case tokmap["LABEL"]:
            state = "label"
        case tokmap["NAME"]:
            switch state {
            case "chip":
                curChips = append(curChips, token.Value.(string))
            case "label":
                curLabel = append(curLabel, token.Value.(string))
                if len(curLabel) >= 2 {
                    curLabels = append(curLabels, strings.Join(curLabel, ":"))
                    curLabel = make([]string, 0, 2)
                }
            }
        case tokmap["NEWLINE"]:
            state = "none"
        }
    }
    addChips() // close the final chip statement
    for _, chip := range order {
        fmt.Printf("chip %v: %v\n", chip, strings.Join(chipLabels[chip], ", "))
    }
    return nil
}

Output for the sensors.conf file on my machine:

chip lm78-*: temp1:M/B Temp
chip lm79-*: temp1:M/B Temp
chip lm80-*: temp1:M/B Temp
chip lm96080-*: temp1:M/B Temp
chip w83792d-*: in0:VcoreA, in1:VcoreB, in6:+5V, in7:5VSB, in8:Vbat
chip w83793-*: in0:VcoreA, in1:VcoreB, in7:+5V, in8:5VSB, in9:Vbat
chip w83795g-*: in12:+3.3V, in13:3VSB, in14:Vbat
chip w83795adg-*: in12:+3.3V, in13:3VSB, in14:Vbat
chip via686a-*: in0:Vcore, in2:+3.3V, in3:+5V, in4:+12V
chip adm1025-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, in5:VCC, temp1:CPU Temp, temp2:M/B Temp
chip ne1619-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, in5:VCC, temp1:CPU Temp, temp2:M/B Temp
chip lm87-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp1:M/B Temp, temp2:CPU Temp
chip adm1024-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp1:M/B Temp, temp2:CPU Temp
chip it87-*: in8:Vbat
chip it8712-*: in8:Vbat
chip it8716-*: in8:Vbat
chip it8718-*: in8:Vbat
chip it8720-*: in8:Vbat
chip fscpos-*: in0:+12V, in1:+5V, in2:Vbat, temp1:CPU Temp, temp2:M/B Temp, temp3:Aux Temp
chip fscher-*: in0:+12V, in1:+5V, in2:Vbat, temp1:CPU Temp, temp2:M/B Temp, temp3:Aux Temp
chip fscscy-*: in0:+12V, in1:+5V, in2:+3.3V, temp1:CPU0 Temp, temp2:CPU1 Temp, temp3:M/B Temp, temp4:Aux Temp
chip fschds-*: temp1:CPU Temp, temp2:Super I/O Temp, temp3:System Temp, an1:PSU Fan, an2:CPU Fan, an3:System FAN2, an4:System FAN3, an5:System FAN4, in0:+12V, in1:+5V, in2:Vbat
chip fscsyl-*: temp1:CPU Temp, temp4:Super I/O Temp, temp5:Northbridge Temp, an1:CPU Fan, an2:System FAN2, an3:System FAN3, an4:System FAN4, an7:PSU Fan, in0:+12V, in1:+5V, in2:Vbat, in3:+3.3V, in5:+3.3V-Aux
chip vt1211-*: in5:+3.3V, temp2:SIO Temp
chip vt8231-*: in5:+3.3V
chip smsc47m192-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, in5:VCC, temp1:SIO Temp
chip lm85-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip lm85b-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip lm85c-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip adm1027-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip adt7463-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip adt7468-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip emc6d100-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip emc6d102-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip emc6d103-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip emc6d103s-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip emc6w201-*: in2:+3.3V, in3:+5V, temp6:M/B Temp
chip pc87365-*: in7:3VSB, in8:VDD, in9:Vbat, in10:AVDD, temp3:SIO Temp
chip pc87366-*: in7:3VSB, in8:VDD, in9:Vbat, in10:AVDD, temp3:SIO Temp
chip adm1030-*: temp1:M/B Temp
chip adm1031-*: temp1:M/B Temp
chip w83627thf-*: in3:+5V, in7:5VSB, in8:Vbat
chip w83627ehf-*: in0:Vcore, in2:AVCC, in3:+3.3V, in7:3VSB, in8:Vbat
chip w83627dhg-*: in0:Vcore, in2:AVCC, in3:+3.3V, in7:3VSB, in8:Vbat
chip w83667hg-*: in0:Vcore, in2:AVCC, in3:+3.3V, in7:3VSB, in8:Vbat
chip nct6775-*: in0:Vcore, in2:AVCC, in3:+3.3V, in7:3VSB, in8:Vbat
chip nct6776-*: in0:Vcore, in2:AVCC, in3:+3.3V, in7:3VSB, in8:Vbat
chip w83627uhg-*: in2:AVCC, in3:+5V, in7:5VSB, in8:Vbat
chip f71805f-*: in0:+3.3V
chip f71872f-*: in0:+3.3V, in9:Vbat, in10:3VSB
chip k8temp-*: temp1:Core0 Temp, temp2:Core0 Temp, temp3:Core1 Temp, temp4:Core1 Temp
chip dme1737-*: in0:5VSB, in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, in5:3VSB, in6:Vbat, temp2:SIO Temp
chip sch311x-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, in5:3VSB, in6:Vbat, temp2:SIO Temp
chip sch5027-*: in0:5VSB, in1:Vcore, in2:+3.3V, in5:3VSB, in6:Vbat, temp2:SIO Temp
chip sch5127-*: in2:+3.3V, in5:3VSB, in6:Vbat
chip f71808e-*: in0:+3.3V, in7:3VSB, in8:Vbat
chip f71808a-*: in0:+3.3V, in7:3VSB, in8:Vbat
chip f71862fg-*: in0:+3.3V, in7:3VSB, in8:Vbat
chip f71869-*: in0:+3.3V, in7:3VSB, in8:Vbat
chip f71869a-*: in0:+3.3V, in7:3VSB, in8:Vbat
chip f71882fg-*: in0:+3.3V, in7:3VSB, in8:Vbat
chip f71889fg-*: in0:+3.3V, in7:3VSB, in8:Vbat
chip f71889ed-*: in0:+3.3V, in7:3VSB, in8:Vbat
chip f71889a-*: in0:+3.3V, in7:3VSB, in8:Vbat
chip f71858fg-*: in0:+3.3V, in1:3VSB, in2:Vbat
chip f8000-*: in0:+3.3V, in1:3VSB, in2:Vbat
chip f81865f-*: in0:+3.3V, in5:3VSB, in6:Vbat
chip adt7473-*: in2:+3.3V, temp2:Board Temp
chip adt7475-*: in2:+3.3V, temp2:Board Temp
chip adt7476-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp
chip adt7490-*: in1:Vcore, in2:+3.3V, in3:+5V, in4:+12V, temp2:M/B Temp

Conclusion

This article presented lexmachine a library that helps you break up complex strings into their component parts safely, reliably, and quickly. To learn more visit the project page, browse the documentation, look at another tutorial, or read an explainer article on how it all works.

Faster Tokenization with a DFA Backend for Lexmachine

2017-11-27T00:00:00-05:00

Lexmachine is a lexical analysis (or tokenization) library for Go (golang) and it just got faster thanks to a new backend I wrote. This article is going to explain the new backend, how it works, and why it is faster. If you want to use lexmachine checkout the documentation or the tutorial.

Lexical analysis is the process of breaking strings up into substrings, called lexemes, and putting them into categories called tokens. This process is also called tokenization and is the first step in parsing complex file formats, programming languages, network protocols, and other information. Let's look at a quick example, suppose we wanted to process a custom configuration file format. For instance, sensors.conf describes how Linux should translate the raw readings hardware monitoring chips to real-world values -- such as voltage and temperature. The first part of my laptop's sensors.conf file begins with the following:

# It is recommended not to modify this file, but to drop your local
# changes in /etc/sensors.d/. File with names that start with a dot
# are ignored.

chip "lm78-*" "lm79-*" "lm80-*" "lm96080-*"

    label temp1 "M/B Temp"


chip "w83792d-*"

    label in0 "VcoreA"
    label in1 "VcoreB"
    label in6 "+5V"
    label in7 "5VSB"
    label in8 "Vbat"

    set in6_min  5.0 * 0.90
    set in6_max  5.0 * 1.10
    set in7_min  5.0 * 0.90
    set in7_max  5.0 * 1.10
    set in8_min  3.0 * 0.90
    set in8_max  3.0 * 1.10

According to the man page, there are several syntactic elements in this file: keywords (chip, label, set, ...), names (in0, in6_min, ..., "lm78-*", "VcoreA", ...), floats (5.0, 0.90, ...), operators (+, -, *, ...), and comments which start with #. A lexical analysis would break down the file as follows:

Type    | Lexeme
-------------------
COMMENT | # It is recommended not to modify this file, but to drop your local
COMMENT | # changes in /etc/sensors.d/. File with names that start with a dot
COMMENT | # are ignored.
CHIP    | chip
NAME    | "lm78-*"
NAME    | "lm78-*"
NAME    | "lm79-*"
NAME    | "lm80-*"
NAME    | "lm96080-*"
LABEL   | label
NAME    | temp1
NAME    | "M/B Temp"
CHIP    | chip
NAME    | "w83792d-*"
...
SET     | set
NAME    | in8_max
FLOAT   | 3.0
STAR    | *
FLOAT   | 1.10

The categories are defined by patterns which are expressed as regular expressions. As a quick review, regular expressions (regex) are a "pattern" which describe a set of strings. A regex is made up of characters (a, b, 0, @, ...) which combine with the operators: concatenation abc, alternation a|b, grouping a(b|c)d, and repetition a*. Some examples:

abc matches {"abc"}
a|b matches {"a", "b"}
a(b|c)d matches {"abd", "acd"}
a* matches {"", "a", "aa", ...}
a(b(c|d))* matches {"a", "abc", "abd", "abcbc", "abcbd", ...}

In the running example of the sensors.conf file one might define the following patterns:

AT: @
PLUS: +
STAR: *
DASH: -
SLASH: /
CARROT: ^
BACKTICK: `
LPAREN: (
RPAREN: )
BUS: bus
CHIP: chip
LABEL: label
COMPUTE: compute
IGNORE: ignore
SET: set
NUMBER: [0-9]*\.?[0-9]+
NAME: [a-zA-Z_][a-zA-Z0-9_]*|"[^"]*"

These patterns define how text in the input should categorized according to the rules for the sensors.conf file.

Efficiently Categorizing Substrings with Automata

Now we turn our attention from what lexical analysis is to how it works. Specifically we are going to concern ourselves with the following matters:

How the lexical analysis problem is different than the one solved by standard implementations of regular expression engines (such as the regexp standard library package).
How to solve the lexical analysis problem with non-deterministic finite automatons (NFA).
How deterministic automatons (DFAs) improve on the solution presented by NFAs

The Lexical Analysis Problem

The regular expression problem decides whether a pattern (as specified by a regular expression) matches a given string. If the pattern does describe the string a regular expression engine will answer: yes it matches. Otherwise, it answers: no it does not.

The lexical analysis problem on the other hand seeks to break up (tokenize) a long piece of text based on an ordered set of regular expressions. It does this by proceeding from the start of the string and finding the longest prefix of the string that matches at least one pattern. It then chooses a matching pattern with the highest precedence and outputs the matched text (lexeme) and the category the pattern defines (token type). Solving the lexical analysis problem with a standard regular expression engine is possible but not efficient. With a standard regex engine, the whole piece of text has to be repeatedly scanned for each token to decide on the longest prefix match and highest precedence token type to use.

However, we can use the theory that is used to implement efficient regular expression engines to implement efficient lexical analysis engines. The following adjustments must be made:

Prefixes of a string are matched
Several patterns are matched at once
The pattern which matches the longest prefix wins
In case of ties, the pattern with the highest precedence wins

Finite Automata

Efficient regular expression engines are implemented using the theory of finite automata. Automata or automatons are theoretical mathematical constructs which specify a machine. Specifically, a machine that reads a string, character by character, and decides whether it is in some language or not (see Figure 1). The language is a type of formal language called a regular language or a Type-3 language in the Chomsky Formal Language Hierarchy.

Figure 1. A Simple Finite State Automaton.
If the string matches the Accept light lights up, otherwise the error light lights up.

Finite state automatons encode their matching "program" as a set of states $S$, an alphabet $\Sigma$, and a transition function which maps the current state and next symbol to the next state $ T: S \times \Sigma \rightarrow S $. There are several distinguished states: the starting state, and one or more accepting states. At each step through the input string the machine reads a character and consults the transition function to determine the next state. If at the end of the string the state has reached an accepting state the string is said to match the automaton.

When the transition function $T$ produces multiple states for the same input combination the automaton is said to be non-deterministic. Figure 2 shows a non-deterministic finite automaton (NFA) for our running example for sensors.conf. In the figure, the starting state (state 0) may transition on character c to states 12, 21, and 40. This may be interpreted as the machine transitioning to all of these states simultaneously and maintaining multiple threads of execution. Simulations of NFAs must keep track of each independent thread of execution. If an NFA has $n$ states and a string has $m$ characters it can be matched in $\mathcal{O}(n \cdot m)$ steps.

Figure 2. A Non-deterministic Finite Automaton for sensors.conf.
States with double circles are accepting states. Each accepting state is labeled with the category it corresponds to. The starting state is marked.

To use an NFA for the lexical analysis problem several adjustments need to be made.

To match prefixes, keep scanning until no new states are found (that is all threads of execution have either reached end of the string or the error state).
Track all accepting states reached during the scanning.
Return the "most relevant" match on state exhaustion by returning the accepting state which matched the longest prefix. Ties are broken by user supplied precedence.
When the user requests the next token, reset the simulation to the starting state and rerun starting at the next character after the previously returned prefix.
After all input has been consumed check all of the input was successfully matched. Return an error otherwise.

As languages get more complex the cost of tokenization goes up when using NFAs because the cost for matching a regular expression is dependent in the size of the NFA ($\mathcal{O}(n \cdot m)$ where $n$ is the number of states in the NFA and $m$ is the length of the string). Can we do better? We can! If we do not allow the transition function $T$ to produce more than one state for each input combination the automaton will be deterministic finite automaton (DFA). If the automaton is deterministic then simulation is greatly simplified versus a non-deterministic automaton. Recall, in the case of NFA simulation non-deterministic steps are modeled by having the machine transition to all possible next states at once and keeping track of multiple threads of execution. In a DFA, there will always be one (and only one) state to transition to -- and therefore only one thread of execution. If there is only one thread of execution the simulation can be run in $\mathcal{O}(m)$ steps (where $m$ is the length of string).

Figure 3 shows the minimal DFA for the NFA given in Figure 2. By coincidence there is one fewer states in the DFA than in the NFA. In general, this will not be the case. Indeed, in the worst case if the NFA has $n$ states the DFA could have as many as $2^{n}$ states. However, on average the DFA will only have $n^3$ states. Thus, a DFA makes the classic space vs. time tradeoff. It uses more space (in the form of a larger transition table on average) to get faster execution time.

Figure 3. A Minimal Deterministic Finite Automaton for sensors.conf.
States with double circles are accepting states. Each accepting state is labeled with the category it corresponds to. The starting state is marked.

Practical Results

Does the new DFA backend outperform the NFA backend when tokenizing many strings? The answer is yes. As an quick (non-rigorous) example, let's tokenize my sensors.conf file 1000 times using both the NFA backend and the DFA backend.

Using the NFA backend

$ go install github.com/timtadh/lexmachine/examples/sensors &&
> cat /etc/sensors*.conf | (/usr/bin/time -v sensors --nfa )
        Command being timed: "sensors --nfa"
        User time (seconds): 3.0
        System time (seconds): 0.00
        Percent of CPU this job got: 102%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.97
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 6564
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 1749
        Voluntary context switches: 1866
        Involuntary context switches: 473
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Using the DFA backend

$ go install github.com/timtadh/lexmachine/examples/sensors &&
> cat /etc/sensors*.conf | (/usr/bin/time -v sensors --dfa )
        Command being timed: "sensors --dfa"
        User time (seconds): 2.01
        System time (seconds): 0.00
        Percent of CPU this job got: 104%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.93
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 7680
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 2172
        Voluntary context switches: 2231
        Involuntary context switches: 359
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Note as predicted by the theory presented above, the NFA backend used less memory but took longer to tokenize the file 1000 times. The DFA backend used more memory (but not a lot more) and tokenized the file faster than the NFA backend.

Intrigued? Try out lexmachine today!

Frequent Subgraph Analysis and its Software Engineering Applications

2017-08-18T00:00:00-04:00

Tim A. D. Henderson. Frequent Subgraph Analysis and its Software Engineering Applications. Case Western Reserve University. Doctoral Dissertation. 2017.
PDF. WEB.

Abstract

Frequent subgraph analysis is a class of techniques and algorithms to find repeated sub-structures in graphs known as frequent subgraphs or graph patterns. In the field of Software Engineering, graph pattern discovery can help detect semantic code duplication, locate the root cause of bugs, infer program specifications, and even recommend intelligent auto-complete suggestions. Outside of Software Engineering, discovering graph patterns has enabled important applications in personalized medicine, computer aided drug design, computer vision, and multimedia.

As promising as much of the previous work in areas such as semantic code duplication detection has been, finding all of the patterns in graphs of a large program's code has previously proven intractable. Part of what makes discovering all graphs patterns in a graph of a large program difficult is the very large number of frequent subgraphs contained in graphs of large programs. Another impediment arises when graphs contain frequent patterns with many automorphisms and overlapping embeddings. Such patterns are pathologically difficult to mine and are found in real programs.

I present a family of algorithms and techniques for frequent subgraph analysis with two specific aims. One, address pathological structures. Two, enable important software engineering applications such as code clone detection and fault localization without analyzing all frequent subgraphs. The first aim is addressed by novel optimizations making the system faster and more scalable than previously published work on both program graphs and other difficult to mine graphs. The second aim is addressed by new algorithms for sampling, ranking, and grouping frequent patterns. Experiments and theoretical results show the tractability of these new techniques.

The power of frequent subgraph mining in Software Engineering is demonstrated with studies on duplicate code (code clone) identification and fault localization. Identifying code clones from program dependence graphs allows the identification of potential semantic clones. The proposed sampling techniques enable tractable dependence clone identification and analysis. Fault localization identifies potential locations for the root cause of bugs in programs. Frequent substructures in dynamic program behavior graphs to identify suspect behaviors which are further isolated with fully automatic test case minimization and generation.

Introduction

Frequent subgraph analysis (FSA) is a family of techniques to discover recurring subgraphs in graph databases. The databases can either be composed of many individual graphs or a single large connected graph. This dissertation discusses my contributions to frequent subgraph analysis and applies the technique to address two pressing problems in software engineering: code clone detection and automatic fault localization.

The work on frequent subgraph analysis was motivated by the software engineering problems. Large programs are composed of repeated patterns arising organically through the process of program construction. Some regions of programs are duplicated (intentionally or unintentionally). The duplicated regions are referred to as code clones (or just clones). Other regions are similar to each other because they perform similar tasks or share development histories.

Code clones may arise from programmers copying and pasting code, from limitations of a programming language, from using certain APIs, from following coding conventions, or from a variety of other causes. Whatever their causes, the existing clones in a code base need to be managed. When a programmer modifies a region of code that is cloned in another location in the program they should make an active decision whether or not to modify the other location. Clearly, such decisions can only be made if the programmer is aware of the other location.

One type of duplication which is particularly difficult to detect is so called Type-4 clones or semantic clones. Semantic clones are semantically equivalent regions of code which may or may not be textually similar. Differences could be small changes such as different variable names or large changes such as a different algorithms which perform the same function. In general identifying semantically equivalent regions is undecidable as a reduction from the halting problem.

Frequent subgraph analysis (FSA) can be used to identify some Type-4 clones (as well as easier to identify clone classes). I use FSA to analyze a graphical representation of the program called the Program Dependence Graph (PDG) {Ferrante 1987}. Dependence graphs strip away syntactic information and focus on the semantic relationships between operations. Non-semantic re-orderings of operations in a program do not effect the structure of its dependence graph {Horwitz 1990, Podgurski 1989, Podgurski 1990}. Since PDGs are not sensitive to unimportant syntactic changes some of the Type-4 clones in a program may be identified with FSA.

The other motivating application I applied FSA to is automatic fault localization. When programs have faults, defects, or bugs it is often time consuming and sometimes difficult to find the cause of the bug. To address this the software engineering community has been working on a variety of techniques for automatic fault localization. The family of statistical fault localization techniques analyzes the behavior of the program when the faults manifest and when they do not. These techniques then identify statistical associations between execution of particular program elements and the occurrence of program failures.

While statistical measures can identify suspicious elements of a program they are blind to the relationships between the elements. If program behavior is modeled through Dynamic Control Flow Graphs, then execution relationships between operations can be analyzed using FSA to identify suspicious interactions. These suspicious interactions represent larger behaviors of the program which are statistically associated with program failure. The behaviors serve as a context of interacting suspicious program elements which potentially makes it easier for programmers to comprehend localization results.

Much of the previous work in frequent subgraph analysis has focused on finding all of the frequent subgraphs in a graph database (called frequent subgraph mining {Inokuchi 2000}). I have shown that finding all of the frequent subgraphs in a database of graphs is not an efficient or effective way to either detect code clones or automatically localize faults. Program dependence graphs of large programs have huge numbers of recurring subgraphs. Experiments on a number of open source projects (see Chapter 4) showed that moderately sized Java programs (~70 KLOC) have more than a hundred of million subgraphs that recur five or more times. Mining all recurring subgraphs is an impractical way to either identify code clones or localize faults.

Furthermore, it turns out that program dependence graphs are particularly difficult to analyze for recurring subgraphs. These graphs often have certain structures which contain many automorphisms. A structure with an automorphism can be rotated upon itself. Each rotation appears to be a recurrence to traditional frequent subgraph mining algorithms. However, because it is merely a rotation, humans (e.g. programmers) do not perceive these rotations as instances of duplication.

To enable scalable frequent subgraph analysis of large programs new techniques were needed. I developed novel optimizations for mining frequent subgraphs and created a state of the art miner (REGRAX) for connected graphs (Chapter 3). To detect code clones from program dependence graphs, I developed an algorithm (GRAPLE) to collect a representative sample of recurring subgraphs (Chapter 4). Finally, a new algorithm (SWRW) was created for localizing faults from dynamic control flow graphs, which outperforms previous algorithms (Chapter 6).

REGRAX contains low level optimizations to the process of identifying frequent subgraphs. Chapter 2 provides the necessary background on frequent subgraph mining for understanding these optimizations. An extensive empirical study was conducted on REGRAX to quantify the effect of each of the new optimizations on databases from the SUBDUE corpus {Cook 1994}, on program dependence graphs, and on random graphs.

GRAPLE is a new algorithm to sample a representative set of frequent subgraphs and estimate statistics characterizing properties of the set of all frequent subgraphs. The sampling algorithm uses the theory of absorbing Markov chains to model the process of extracting recurring subgraphs from a large connected graph. By sampling a representative set of recurring subgraphs GRAPLE is able to conduct frequent subgraph analysis on large programs which normally would not be amenable to such analysis.

One of the questions in code clone detection is: "are code clones detected from program dependence graphs understandable to programmers?" GRAPLE was used to answer this question, as it not only collects a sample of frequent subgraphs but allows researchers to estimate the prevalence of features across the entire population of frequent subgraphs (including those which were not sampled). Chapter 4 details a case study which was conducted at a software company to determine whether their programmers could make use of code clones detected from program dependence graphs. The study would not have been possible without the estimation framework in GRAPLE, as the software contained too many code clones to be reviewed in the allocated budget.

To apply frequent subgraph analysis to automatic fault localization, a new algorithm named Score Weighted Random Walks (SWRW) was developed. SWRW samples discriminative, suspicious, or significant subgraphs from a database of graphs. The database is split into multiple classes where some graphs are labeled "positive" and others "negative." In fault localization the "positive" graphs were those dynamic control flow graphs collected from program executions which exhibited a failure of some type. The "negative" graphs are from executions which did not fail.

SWRW, like GRAPLE, models the problem using the theory of absorbing Markov chains. Unlike GRAPLE, it uses an objective function (drawn from the statistical fault localization literature {Lucia 2014}) to guide the sampling process. In comparison to previous work in fault localization using graph mining, a much wider variety of objective functions can applied. This allows for functions better suited to statistical fault localization to be used as the objective function. SWRW outperforms previous approaches which used discriminative mining to localize faults in terms of fault localization accuracy.

Summary

This dissertation makes important and novel contributions to frequent subgraph analysis which enable scalable semantic code clone detection and behavioral fault localization. These advances can help programmers maintain their software more efficiently leading to more stable and secure software for everyone. The software engineering advances are built on new frequent subgraph analysis algorithms. The new algorithms improve code clone detection time, fault localization latency and accuracy, and enable analysis of larger and more complex programs.

Read the full dissertation.

Rethinking Dependence Clones

2017-02-21T00:00:00-05:00

Tim A. D. Henderson and Andy Podgurski. Rethinking Dependence Clones. IWSC 2017.
DOI. PDF. WEB.

Abstract

Semantic code clones are regions of duplicated code that may appear dissimilar but compute similar functions. Since in general it is algorithmically undecidable whether two or more programs compute the same function, locating all semantic code clones is infeasible. One way to dodge the undecidability issue and find potential semantic clones, using only static information, is to search for recurring subgraphs of a program dependence graph (PDG). PDGs represent control and data dependence relationships between statements or operations in a program. PDG-based clone detection techniques, unlike syntactically-based techniques, do not distinguish between code fragments that differ only because of dependence-preserving statement re-orderings, which also preserve semantics. Consequently, they detect clones that are difficult to find by other means. Despite this very desirable property, work on PDG-based clone detection has largely stalled, apparently because of concerns about the scalability of the approach. We argue, however, that the time has come to reconsider PDG-based clone detection, as a part of a holistic strategy for clone management. We present evidence that its scalability problems are not as severe as previously thought. This suggests the possibility of developing integrated clone management systems that fuse information from multiple clone detection methods, including PDG-based ones.

Introduction

Fragments of similar code are typically scattered throughout large code bases. These repeated fragments or code clones often result from programmers copying and pasting code. Code clones (or just clones) may also result from limitations of a programming language, use of certain APIs or design patterns, following coding conventions, or a variety of other causes. Whatever their causes, existing clones need to be managed. When a programmer modifies a region of code that is cloned in another location in the program, they should make an active decision whether or not to modify the other location. Clearly, such decisions can only be made if the programmer is aware of the other location.

In general, there are 4 types of code clones:

Type-1 Clones - Identical regions of code (excepting whitespace and comments).
Type-2 Clones - Syntactically equivalent regions (excepting names, literals, types, and comments).
Type-3 Clones - Syntactically similar regions (as in Type-2) but with minor differences such as statement additions or deletions.
Type-4 Clones - Regions of code with functionally equivalent behavior but possibly with different syntactic structures.

Much of the research on code clone detection and maintenance has been geared toward Type-1 and Type-2 clone, as they are easier to detect and validate than Type-3 and Type-4 clones. The two most popular detection methods involve searching for clones in token streams and abstract syntax trees (ASTs).

An alternative approach to clone detection is to search for them in a Program Dependence Graph (PDG), which represents the control and data dependences between statements or operations in a program. Recurring subgraphs in PDGs represent potential dependence clones. Some of the previous work on PDG-based clone detection used forward and backward path-slicing to find clones. This method can detect matching slices, but it cannot detect all recurring subgraphs. The latter can be identified using frequent subgraph mining (FSM). However, for low frequency thresholds, the number of PDG subgraphs discovered by FSM may be enormous. For example, we found that for a Java program with 70,000 lines of code (LOC), over 700 million PDG subgraphs with 5 or more instances were discovered by FSM.

Since it is infeasible for developers to examine so many subgraphs, we previously developed GRAPLE, an algorithm to select representative samples of maximal frequent subgraphs. In this paper, the core sampling process remains the same as in GRAPLE but we present a new algorithm for traversing the k-frequent subgraph lattice. One tricky aspect of FSM is how to define exactly what "frequency" means in a large connected graph. In order to handle pathological cases that occur in real programs, we introduce a new metric to measure subgraph frequency (or "support"), called the Greedy Independent Subgraphs (GIS) measure. The results section details the first empirical examination of the scalability and speed of sampling dependence clones from large programs. The study showed that our new system can quickly sample from programs with 500 KLOC of code and successfully sample from programs with perhaps 2 MLOC. Finally, since at times the sampling algorithm may return several potential clones, which are quite similar to each other, we evaluate the performance of a density-based clustering algorithm on the samples collected.

Note

See the PDF for the complete paper. IEEE has the exclusive rights to publish this paper. Follow the DOI for the IEEE copy.

A Job Queue in BASH

2017-02-11T00:00:00-05:00

A Job Queue is typically a first in first out queue of "work items" or "jobs" to be processed. Ideally, a good job queue should support multiple workers (also called readers) so multiple jobs can be processed at one time. For production systems and clusters there are many robust options availble. Sometimes you need a job queue for a local system but cannot install (or do not want to install) one of the many networked job queues. But, if you are running Linux you probably have GNU BASH installed which can be used to create a relatively simple and robust job queue.

Below is an example BASH Job Queue made with fifos and flocks. A fifo is a first in first out UNIX pipe (see man fifo). A flock (see man flock) is a "file lock" which lets the queue support multiple readers. This queue may or may not work for other shells as it relies on the BASH built in read. You will need to consult the man pages for your shell to determine if this will work for you.

#!/usr/bin/env bash

## this is the "job" function which is does whatever work
## the queue workers are supposed to be doing
job() {
  i=$1
  work=$2
  ## run the work ....
}

# make the files
START=$(mktemp -t start-XXXX)
FIFO=$(mktemp -t fifo-XXXX)
FIFO_LOCK=$(mktemp -t lock-XXXX)
START_LOCK=$(mktemp -t lock-XXXX)

## mktemp makes a regular file. Delete that an make a fifo.
rm $FIFO
mkfifo $FIFO
echo $FIFO

## create a trap to cleanup on exit if we fail in the middle.
cleanup() {
  rm $FIFO
  rm $START
  rm $FIFO_LOCK
  rm $START_LOCK
}
trap cleanup 0

## This is the worker to read from the queue.
work() {
  ID=$1
  ## first open the fifo and locks for reading.
  exec 3<$FIFO
  exec 4<$FIFO_LOCK
  exec 5<$START_LOCK

  ## signal the worker has started.
  flock 5                 # obtain the start lock
  echo $ID >> $START      # put my worker ID in the start file
  flock -u 5              # release the start lock
  exec 5<&-               # close the start lock file
  echo worker $ID started

  while true; do
    ## try to read the queue
    flock 4                      # obtain the fifo lock
    read -su 3 work_id work_item # read into work_id and work_item
    read_status=$?               # save the exit status of read
    flock -u 4                   # release the fifo lock

    ## check the line read.
    if [[ $read_status -eq 0 ]]; then
      ## If read gives an exit code of 0 the read succeeded.
      # got a work item. do the work
      echo $ID got work_id=$work_id work_item=$work_item
      ## Run the job in a subshell. That way any exit calls do not kill
      ## the worker process.
      ( job "$work_id" "$work_item" )
    else
      ## Any other exit code indicates an EOF.
      break
    fi
  done
  # clean up the fd(s)
  exec 3<&-
  exec 4<&-
  echo $ID "done working"
}

## Start the workers.
WORKERS=4
for ((i=1;i<=$WORKERS;i++)); do
  echo will start $i
  work $i &
done

## Open the fifo for writing.
exec 3>$FIFO
## Open the start lock for reading
exec 4<$START_LOCK

## Wait for the workers to start
while true; do
  flock 4
  started=$(wc -l $START | cut -d \  -f 1)
  flock -u 4
  if [[ $started -eq $WORKERS ]]; then
    break
  else
    echo waiting, started $started of $WORKERS
  fi
done
exec 4<&-

## utility function to send the jobs to the workers
send() {
  work_id=$1
  work_item=$2
  echo sending $work_id $work_item
  echo "$work_id" "$work_item" 1>&3 ## the fifo is fd 3
}

## Produce the jobs to run.
i=0
for item in {dataset-A,dataset-B,dataset-C,dataset-D}; do
  send $i $data
  i=$((i+1))
done
## close the filo
exec 3<&-
## disable the cleanup trap
trap '' 0
## It is safe to delete the files because the workers
## already opened them. Thus, only the names are going away
## the actual files will stay there until the workers
## all finish.
cleanup
## now wait for all the workers.
wait

References

`xrandr` Script to Setup External Monitor

2017-02-08T00:00:00-05:00

I have a quirky adapter for my external monitor. The adapter doesn't correctly report the display's size or refresh rate to the laptop. To solve this problem, I wrote the following script to automatically setup the external monitor:

#!/usr/bin/env bash

## size and refresh rate of the screen
X=1920
Y=1200
R=60

## mode string
MODE="$X"x"$Y"_"$R".00

## turn the output off in case it is on
xrandr --output DP1 --off

## delete the mode from the output in case it exists
xrandr --delmode DP1 "$MODE"

## delete the mode
xrandr --rmmode "$MODE"

## create the mode, you need to compute the actual mode line using the `gtf`
## program which computes VESA GTF mode lines from size and refresh rates
gtf $X $Y $R | grep Modeline | sed 's/ *Modeline *//' | xargs xrandr --newmode

## add your newly created mode to the output
xrandr --addmode DP1 "$MODE"

## turn on the output with the new mode. Place it to the right of the laptop
## display but displaced -600 pixels above the laptop display
xrandr --output DP1 --mode $MODE --pos 1920x-600

This script also has the advantage that I never have to run the GUI tool to change the position of the monitor in relationship to my laptop's monitor.

Frequent Subgraph Mining of Personalized Signaling Pathway Networks Groups Patients with Frequently Dysregulated Disease Pathways and Predicts Prognosis

2017-01-03T00:00:00-05:00

A. Durmaz*, T. A. D. Henderson*, D. Brubaker, and G. Bebek. Frequent Subgraph Mining of Personalized Signaling Pathway Networks Groups Patients with Frequently Dysregulated Disease Pathways and Predicts Prognosis. PSB 2017. * Co-First Author
DOI. PUBMED. PDF. SUPPLEMENT. WEB.

Abstract

Motivation

Large scale genomics studies have generated comprehensive molecular characterization of numerous cancer types. Subtypes for many tumor types have been established; however, these classifications are based on molecular characteristics of a small gene sets with limited power to detect dysregulation at the patient level. We hypothesize that frequent graph mining of pathways to gather pathways functionally relevant to tumors can characterize tumor types and provide opportunities for personalized therapies.

Results

In this study we present an integrative omics approach to group patients based on their altered pathway characteristics and show prognostic differences within breast cancer (p < 9.57e-10) and glioblastoma multiforme (p < 0.05) patients. We were able validate this approach in secondary RNA-Seq datasets with p < 0.05 and p < 0.01 respectively. We also performed pathway enrichment analysis to further investigate the biological relevance of dysregulated pathways. We compared our approach with network-based classifier algorithms and showed that our unsupervised approach generates more robust and biologically relevant clustering whereas previous approaches failed to report specific functions for similar patient groups or classify patients into prognostic groups.

Conclusions

These results could serve as a means to improve prognosis for future cancer patients, and to provide opportunities for improved treatment options and personalized interventions. The proposed novel graph mining approach is able to integrate PPI networks with gene expression in a biologically sound approach and cluster patients in to clinically distinct groups. We have utilized breast cancer and glioblastoma multiforme datasets from microarray and RNA-Seq platforms and identified disease mechanisms differentiating samples.

Note

See the PDF for the complete paper. Follow the DOI for the authoritative version.

Sampling Code Clones from Program Dependence Graphs with GRAPLE

2016-11-13T00:00:00-05:00

Tim A. D. Henderson and Andy Podgurski. Sampling Code Clones from Program Dependence Graphs with GRAPLE. SWAN 2016.
DOI. PDF. SUPPLEMENT. CODE. WEB.

Abstract

We present GRAPLE, a method to generate a representative sample of recurring (frequent) subgraphs of any directed labeled graph(s). GRAPLE is based on frequent subgraph mining, absorbing Markov chains, and Horvitz-Thompson estimation. It can be used to sample any kind of graph representation for programs. One of many software engineering applications for finding recurring subgraphs is detecting duplicated code (code clones) from representations such as program dependence graphs (PDGs) and abstract syntax trees. To assess the usefulness of clones detected from PDGs, we conducted a case study on a 73 KLOC commercial Android application developed over 5 years. Nine of the application's developers participated. To our knowledge, it is the first study to have professional developers examine code clones detected from PDGs. We describe a new PDG generation tool jpdg for JVM languages, which was used to generate the dependence graphs used in the study.

Introduction

Code clones are similar fragments of program code. They can arise from copying and pasting, using certain design patterns or certain APIs, or adhering to coding conventions, among other causes. Code clones create maintenance hazards, because they often require subtle context-dependent adaptation and because other changes must be applied to each member of a clone class. To manage clone evolution the clones must first be found. Clones can be detected using any program representation: source code text, tokens, abstract syntax trees (ASTs), flow graphs, dependence graphs, etc. Each representation has advantages and disadvantages for clone detection.

PDG-based clone detection finds dependence clones corresponding to recurring subgraphs of a program dependence graph (PDG). Since PDGs are oblivious to semantics preserving statement reorderings they are well suited to detect semantic (functionally equivalent) clones. A number of algorithms find clones from PDGs. However, as Bellon notes, "PDG based techniques are computationally expensive and often report non-contiguous clones that may not be perceived as clones by a human evaluator." Most PDG-based clone detection tools are biased, detecting certain clones but not others.

The root cause of scalability problems with PDG-based clone detection is the number of dependence clones. The Background Section (in the pdf) illustrates this with an example in which we used an unbiased frequent subgraph mining algorithm to detect all dependence clones in Java programs. In programs with about 70 KLOC it detected around 10 million clones before disk space was exhausted. Processing all dependence clones is impractical even for modestly sized programs.

Instead of exhaustively enumerating all dependence clones, an unbiased random sample can be used to statistically estimate parameters of the whole "population" of clones, such as the prevalence of clones exhibiting properties of interest. For these reasons, we developed a statistically unbiased method for sampling dependence clones and for estimating parameters of the whole clone population.

We present GRAPLE (GRAph samPLE), a method to generate a representative sample of recurring subgraphs of any directed labeled graph(s). It can be used to sample subgraphs from any kind of program graph representation. GRAPLE is not a general purpose clone detector but it can answer questions about dependence clones that other PDG-based clone detection tools cannot. We conducted a preliminary case study on a commercial application and had its developers evaluate whether the sampled subgraphs represented code duplication. To our knowledge, it is the first study to have professional programmers examine dependence clones. GRAPLE has applications in bug mining, test case selection, and bioinformatics. The sampling algorithm also applies to frequent item sets, subsequences, and subtrees allowing code clone sampling from tokens and ASTs.

Note

See the PDF for the complete paper. ACM has the exclusive rights to publish this paper. Tim Henderson and Andy Podgurski own the copyright. Follow the DOI for the ACM copy. This copy is posted here with the permission of ACM.

Managing Infrastructure with Python, Fabric and Ansible

2016-08-01T00:00:00-04:00

For PyOhio 2016 I gave a short talk on managing infrastructure. You can watch the talk on youtube or look at the slides. The talk starts with a short poem printed below for posterity. I also created a super simple demo which I did not have time to show. You can download a tarball of the demo. To run the demo you are going to need vagrant and the "hasicorp/precise64" box. No guarantees it works, I wrote it in an hour or so to make sure everything in the presentation was sensible.

The World is Changing

by Tim Henderson

death of a sys-admin

One person.
Alone.
Do the many keyboards
Make the work light?

In the dead of the night,
How the flickers
Of the tubes
Shine so bright.

Two hands
Ten fingers
Typing so swift.
On keys, mechanical,
You can hear them click.

A machine.
One then another,
Fixed.
The backlog clean.

Now it is June,
An evening star in the sky.
The sys-admin feels the breeze
As seagulls fly by.

Back in the office
Are the keyboards lonely?
A customer wonders:
Why do my pages
Load slowly?

there are too many machines

The flickering screens,
Can not keeps pace
With the whirring fans.

The blades spin
As the AC hums.
The metal racks gleam
The floor it glows.

But. Not enough,
Not enough dollars.
For too few cents,
Per page view is made.

Where is the sys-admin?
They left last May.
None to replace them,
For we couldn’t pay.

now, we have become root

The humble programmer.
One keyboard,
One screen!

No rituals on approach.
No doors to knock.
Just expensive headphones
Fuzzing with static.

In the fields of desks
By the window so small
Sits the humble programmer
Trying to stay away
From it all.

But alas!
The servers are down.
Sys-admins?
None to be found!

To the command-line?
Neigh!
To the ansible, we pray!

Provision
Config
Deploy

The chant
Doth resound.

Writing a Lexer in Go with LexMachine

2016-05-25T00:00:00-04:00

This article is about lexmachine, a library I wrote to help you write great lexers in Go. If you are looking to write a golang lexer or a lexer in golang this article is for you.

A lexer is a software component that analyzes a string and breaks it up into its component parts. Each part is tagged with what type of thing it is. This is called lexical analysis. For natural languages (such as English) lexical analysis can be difficult to do automatically but is usually easy for a human to do. Let's look at an example of lexically analyzing the following English sentence often used in typing practice (because it uses every letter in the alphabet).

The quick brown fox jumped over the lazy dog.

The sentence breaks down into individual word each of which has a part of speech

<article>, "The"
<adjective>, "quick"
<adjective>, "brown"
<noun>, "fox"
<verb>, "jumped"
<adverb>, "over"
<article>, "the"
<adjective>, "lazy"
<noun>, "dog"

Note, the word "over" can actually be either a preposition or an adverb. In this case the context of the sentence (it modifies the verb) makes it an adverb. Many words in natural languages have this property where the context of the overall sentence or paragraph determines the role they play in the sentence.

Luckily, the situation is much simpler for computer languages. Most compilers, which are programs that translate computer languages into other computer languages, start by lexically analyzing their input. They can also be used in a number of other scenarios where a program needs to understand data in textual forms.

In this article, I am going to write and explain a lexer for the graphviz dot language. graphviz is a tool to visualize graphs. graphviz takes a string such as:

digraph {
  rankdir=LR;
  a [label="a" shape=box];
  c [<label>=<<u>C</u>>];
  b [label="bb"];
  a -> c;
  c -> b;
  d -> c;
  b -> a;
  b -> e;
  e -> f;
}

And produces :

Lexing Graphviz's `dot` language

Before one can "lex" (short for lexically analyze) a language one needs to know what it is made up of. English is made up of punctuation marks, nouns, verbs and that sort of thing. Computer languages also have punctuation but also have keywords, strings, numbers, comments and so forth.

When a lexer splits up a string into parts the parts are called tokens. The process of splitting up is also called tokenizing. Let's take a look at how the previous example would get tokenized:

Type    | Lexeme
-------------------
DIGRAPH | "digraph"
LCURLY  | "{"
ID      | "rankdir"
EQUAL   | "="
ID      | "LR"
SEMI    | ";"
ID      | "\"a\""
LSQUARE | "["
ID      | "label"
EQUAL   | "="
ID      | "a"
ID      | "shape"
EQUAL   | "="
ID      | "box"
RSQUARE | "]"
SEMI    | ";"
ID      | "c"
LSQUARE | "["
ID      | "<label>"
EQUAL   | "="
ID      | "<<u>C</u>>"
RSQUARE | "]"
SEMI    | ";"
ID      | "b"
.
.
.
RCURLY  | "}"

Note, that like when the English sentence was analyzed, spaces, newlines, tabs and other extraneous characters were dropped. Only the syntactically important characters are output.

Each token has two parts: the type and the lexeme. The type indicates the role the token plays. The lexeme is the string the token was extracted from.

Specifying Tokens

To specify how a string should be tokenized a formalism called regular expressions is used. If you don't already know about regular expressions you could start with the Wikipedia page. For a more advanced introduction see Russ Cox's articles or Alex Aiken's video lectures on the subject.

To review, a regular expression is a way of specifying a "pattern" which matches certain strings. For instance, a+b*a matches aaa and abbbba but not aab. To see why, note that the pattern says a string must start with 1 or more a characters. So all three strings aaa, abbbba and aab satisfy the first requirement. Next, the pattern says a string can have 0 or more b characters. The first string, aaa has none (and that is ok). The second string, abbbba has 4 b characters. The third string, aab has 1 b. So all three strings satisfy the second requirement. Finally the pattern says a string must end in an a. The first and the second string both end in a. However, the third string, aab, does not. Therefore, the first and second strings match the pattern but the third string does not.

The Token's for the `dot` Language

The dot language has keywords, punctuation, comments, and a rather unusual definition for identifier (called ID). In the listing below, the token type is on the left side and the regular expression or literal (in quotation marks) is on the right.

NODE = "node"
EDGE = "edge"
GRAPH = "graph"
DIGRAPH = "digraph"
SUBGRAPH = "subgraph"
STRICT = "strict"
LSQUARE = "["
RSQUARE = "]"
LCURLY = "{"
RCURLY = "}"
EQUAL = "="
COMMA = ","
SEMI = ";"
COLON = ":"
ARROW = "->"
DDASH = "--"
COMMENT = (/\*([^*]|[\r\n]|(\*+([^*/]|[\r\n])))*\*+/)|(//.*$)
ID = ([a-zA-Z_][a-zA-Z0-9_]*)|("([^\"]|(\\.))*")|ID-HTML

Every token type but ID and COMMENT are literals: either a keyword or a punctuation mark. A comment is defined by a complicated regular expression which defines "c-style" range comments or line comments. Since the comment expression is defined by a regular expression no nesting is allowed.

The ID token is more complicated. It consists of three parts:

The "usual form" as a name [a-zA-Z_][a-zA-Z0-9_]*. The pattern means a letter (lower case or capital) or underscore followed by 0 or more letters, numbers, or under-scores.
A string, "([^\"]|(\\.))*". Thus "\\\"" and "asdf\"" are valid but "\\"" is not.

A HTML string, which is non-regular (specified here in BNF):

CHAR = [^<>]
ID-HTML = IdHTML

IdHTML : Tag ;
Tag : < Body > ;
Body : CHAR Body ;
     | Tag Body
     | e                    // denotes epsilon, the empty string
     ;

Thus <<xyz<xy>xyz><asdf>> is valid but <<> is not

The reason the HTML string is not-regular (and therefore cannot be matched by regular expressions) is the angle brackets, < and >, have to be properly matched. That is, every opening bracket < must be matched with a closing bracket >. This is not possible to specify regular expressions because they cannot "count." For a formal explanation see the Pumping Lemma.

Consequences of Non-Regular Tokens

If all tokens were regular (that is specifiable by a regular expression) then the full implementation of the lexer could be generated from the regular expressions for each of the tokens. Since the dot language contains at least one token which is non-regular, special consideration needs to be taken.

This turns out to be a fairly common situation in lexer implementation. For instance, if you want to support c-style comments such as /* comment */ which support properly nested comments /* asdf /* asdf */ asdf*/ then the comment token will no longer be regular. Furthermore, many languages (such as C) require collaboration between the parser and lexer to properly identify whether symbols should be variable names or type names. This can also introduce a degree of non-regularity.

Thus, to properly lexically analyze such languages our framework must have an "escape hatch" that allows the analysis of non-regular tokens on demand while still leveraging theory for most of the work.

The LexMachine

To create a lexer for the dot language I am going to use lexmachine a library I wrote for creating lexers. lexmachine handles all the tricky bits of converting regular expressions into Non-Deterministic Finite Automata (NFA) and using the the NFA to tokenize strings. It also provides the aformentioned "escape hatch" to deal with non-regular Token specifications.

Let's get started!

The Implementation

As a reminder, the implementation is written in Go. In your workspace, create a new package called dot:

$ mkdir dot
$ cd dot

Now create a file for the lexer:

$ touch lexer.go

All of the code for the lexer is going in lexer.go.

Preamble

To begin, put in the package directive and import the lexmachine and machines packages. I also import the standard library packages fmt and strings.

package dot

import (
    "fmt"
    "strings"
)

import (
    lex "github.com/timtadh/lexmachine"
    "github.com/timtadh/lexmachine/machines"
)

Note how I group the imports. In general, have three groups of imports. The first group is for standard library packages, the second for third party, and the third group is for other packages in your code.

Defining the Tokens

Next, I create global variables and initialize them. They contain the literal tokens, the keywords, the token names, and a mapping from the names of the tokens to their type ids. Finally, there is a variable Lexer *lex.Lexer which will hold our Lexer object once constructed.

var Literals []string // The tokens representing literal strings
var Keywords []string // The keyword tokens
var Tokens []string // All of the tokens (including literals and keywords)
var TokenIds map[string]int // A map from the token names to their int ids
var Lexer *lex.Lexer // The lexer object. Use this to construct a Scanner

To initialize the lists of tokens we are going to need a function. The reason is although, Literals and Keywords could be defined in place the rest of the variables cannot be.

func initTokens() {
    Literals = []string{
        "[",
        "]",
        "{",
        "}",
        "=",
        ",",
        ";",
        ":",
        "->",
        "--",
    }
    Keywords = []string{
        "NODE",
        "EDGE",
        "GRAPH",
        "DIGRAPH",
        "SUBGRAPH",
        "STRICT",
    }
    Tokens = []string{
        "COMMENT",
        "ID",
    }
    Tokens = append(Tokens, Keywords...)
    Tokens = append(Tokens, Literals...)
    TokenIds = make(map[string]int)
    for i, tok := range Tokens {
        TokenIds[tok] = i
    }
}

Right now, the initTokens() function is not being called. Later, I will show you how to call it on package initialization inside of an init function.

Defining the Lexer

Creating a new lexer object is straight forward.

lexer := lex.NewLexer()

The Lexer object has three methods:

func (self *Lexer) Add(regex []byte, action Action)
func (self *Lexer) Compile() error
func (self *Lexer) Scanner(text []byte) (*Scanner, error)

The Add method is what we are interested in right now. It adds a new token to the lexer. The token is defined by a pattern expressed as a regular expression and an Action function. When the pattern is matched the Action function gets called.

type Action func(scan *Scanner, match *machines.Match) (token interface{}, err error)

An Action takes a *Scanner (which is a object which is scanning a particular string using the *Lexer object), and a *Match (which represents the string that was matched by the Regular expression. It returns a token and an error.

If the token return value is nil, the *Match is skipped. This can be used to skip whitespace and other things you would rather ignore. Let's go head and code up the the skip Action:

func skip(*lex.Scanner, *machines.Match) (interface{}, error) {
    return nil, nil
}

Super simple! It is a no op!

However, most of the time you will want to create a token. First, we need to have a *Token object to construct. Luckily, lexmachine defines one (although you don't have to use it). Let's take a look at the definition:

type Token struct {
    Type        int // the token type
    Value       interface{} // a value associate with the token
    Lexeme      []byte // the string that was matched
    TC          int // the index (text counter) in the string
    StartLine   int
    StartColumn int
    EndLine     int
    EndColumn   int
}
func (self *Token) Equals(other *Token) bool
func (self *Token) String() string

The *Scanner object provides a convience function Token which constructs a token for you. Here is the definition:

func (self *Scanner) Token(typ int, value interface{}, m *machines.Match) *Token

So, with this in mind, here is a simple Action which will construct a Token with a string version of the lexeme as the Value.

func token(name string) lex.Action {
    return func(s *lex.Scanner, m *machines.Match) (interface{}, error) {
        return s.Token(TokenIds[name], string(m.Bytes), m), nil
    }
}

The name paramter is the name of the token (eg. COMMENT, ID, STRICT, {, ...). The token function will constuct a *Token of correct type (eg. the one you specified with name) and return it.

Adding Patterns to the Lexer

Now that we have Action functions to work with (skip and token) we are ready to add patterns to the lexer. Since lexmachine is built on automata theory patterns are matched with these priorities:

Patterns match prefixes of string being scanned. Normally, a regular expression matches the entire string or the first substring (depending on the mode). After a prefix is matched, the lexer is restarted at the end of the previously matched prefix and matches another prefix until the string is consumed.
Through use of automata theory, all patterns are matched in parallel. Currently, lexmachine uses an Non-Deterministic Finite Automata (NFA) simulation "under-the-hood" to do the matching. NFA simulations take O(P*S) where S is the size of the string and P is the size of the pattern (or in the case of a lexer the sum of the sizes of all of the patterns). There is a Deterministic Finite Automata (DFA) code generator under development (but not ready at this time) which will be able to generate Go code to lex a string in linear time O(S).
The pattern which matches the longest prefix is chosen as the "matching pattern". The matching pattern determines which lexing action gets run (and thus what kind of token gets created).
In case of tie, the pattern which was defined first is chosen.

Since order that patterns are added to the lexer matters, literals and keywords should be added first. This is important as other token's patterns (such as ID) could match them. Since, the literals and keywords are both stored in their own lists this is easy to do in a loop:

for _, lit := range Literals {
    r := "\\" + strings.Join(strings.Split(lit, ""), "\\")
    lexer.Add([]byte(r), token(lit))
}
for _, name := range Keywords {
    lexer.Add([]byte(strings.ToLower(name)), token(name))
}

Note: I add escapes to every character in the literals. So that literals like [ have patterns such as \[. This ensures those characters (which have syntactic meaning in regular expressions) are interpreted as themselves. Otherwise, they would be parsed by the regular expression parser incorrectly.

Second Note: The patterns constructed for the keywords is the lower case version of the token name. I could have had the token names for the keywords be in lower case but it is tradition for token names to be capitalized. This helps distinguish token names from production names in context free grammar.

Adding More Complex Patterns

The simple patterns are now added to the lexer. For the COMMENT and ID tokens I will add a separate pattern for each of the alternative construction options:

lexer.Add([]byte(`//[^\n]*\n?`), token("COMMENT"))
lexer.Add([]byte(`/\*([^*]|\r|\n|(\*+([^*/]|\r|\n)))*\*+/`), token("COMMENT"))
lexer.Add([]byte(`([a-z]|[A-Z])([a-z]|[A-Z]|[0-9]|_)*`), token("ID"))
lexer.Add([]byte(`"([^\\"]|(\\.))*"`), token("ID"))

Note: I left out the last form of ID the HTML string. We will get back to that in a second. The final pattern to add is for whitespace: spaces, tabs, newlines, and carriage returns. I don't want tokens produced for these characters to I will use the skip function as the lexer Action:

lexer.Add([]byte("( |\t|\n|\r)+"), skip)

Using the "Escape Hatch"

Now, to deal with the third form of the ID token: the HTML string. I need a pattern that fires when the beginning of the string is found. This is easy as the HTML strings always start with a < character and the < is not found elsewhere in the language. Then, I write a very special Action function. It turns out, that Actions are allowed to make modifications to the internal state of the *Scanner. In particular they are allowed to change where the index into the string being tokenized is located. That index is called the text counter and is stored in the TC variable. Let's take a look at what *Scanner exports:

type Scanner struct {
    Text []byte
    TC   int
    // contains filtered or unexported fields
}
func (self *Scanner) Next() (tok interface{}, err error, eof bool)
func (self *Scanner) Token(typ int, value interface{}, m *machines.Match) *Token

The Text variable can be read but you should not modify it. Modifying it will have no effect on the tokenization as the NFA simulation keeps its own pointer to the text being scanned. The TC variable is the text counter and we can both read and write it inside of an Action. What this allows us to do is find the starting point of an HTML string with the pattern < and then scan along manually counting the opening and closing angle brackets. Once, the initial open bracket has been closed by a matching > the HTML string has been found.

The only trick is we need to keep track of the text counter and update it. We also have to update the *Match object to contain the correct values for the end lines and columns for our token.

Let's see how it works:

lexer.Add([]byte(`\<`),
    func(scan *lex.Scanner, match *machines.Match) (interface{}, error) {
        str := make([]byte, 0, 10)
        str = append(str, match.Bytes...)
        brackets := 1
        match.EndLine = match.StartLine
        match.EndColumn = match.StartColumn
        for tc := scan.TC; tc < len(scan.Text); tc++ {
            str = append(str, scan.Text[tc])
            match.EndColumn += 1
            if scan.Text[tc] == '\n' {
                match.EndLine += 1
            }
            if scan.Text[tc] == '<' {
                brackets += 1
            } else if scan.Text[tc] == '>' {
                brackets -= 1
            }
            if brackets == 0 {
                match.TC = scan.TC
                scan.TC = tc + 1
                match.Bytes = str
                return token("ID")(scan, match)
            }
        }
        return nil,
            fmt.Errorf("unclosed HTML literal starting at %d, (%d, %d)",
                match.TC, match.StartLine, match.StartColumn)
    },
)

Here, I defined the action in-line since it will not be reused using Go's support for anonymous functions. The text counter, scan.TC, is initially pointing at the character directly following the matched pattern. Thus, the bracket count in brackets is initialized to 1.

When brackets reaches 0 through incrementing and decrementing everytime a < or > is seen the match is found. When the match is found, the scan.TC variable must be updated to communicate back to the scanner where to look for the next token. The *Match is also updated to reflect the full lexeme that was found. Finally, an ID token is constructed using the token function.

If the function runs out of text before brackets reaches 0 an error is returned reporting an unclosed HTML literal.

Compiling the NFA

The last step in *Lexer construction is to compile the NFA. This will be done automatically when a *Scanner is constructed to tokenize the string. However, we can have the NFA precomputed by calling Compile. This is important so that we don't spend time parsing regular expressions every time we want to lex a string

err := lexer.Compile()
if err != nil {
    return nil, err
}
return lexer, n

Putting it all Together

// Creates the lexer object and compiles the NFA.
func initLexer() (*lex.Lexer, error) {
    lexer := lex.NewLexer()

    for _, lit := range Literals {
        r := "\\" + strings.Join(strings.Split(lit, ""), "\\")
        lexer.Add([]byte(r), token(lit))
    }
    for _, name := range Keywords {
        lexer.Add([]byte(strings.ToLower(name)), token(name))
    }

    lexer.Add([]byte(`//[^\n]*\n?`), token("COMMENT"))
    lexer.Add([]byte(`/\*([^*]|\r|\n|(\*+([^*/]|\r|\n)))*\*+/`), token("COMMENT"))
    lexer.Add([]byte(`([a-z]|[A-Z])([a-z]|[A-Z]|[0-9]|_)*`), token("ID"))
    lexer.Add([]byte(`"([^\\"]|(\\.))*"`), token("ID"))
    lexer.Add([]byte("( |\t|\n|\r)+"), skip)
    lexer.Add([]byte(`\<`),
        func(scan *lex.Scanner, match *machines.Match) (interface{}, error) {
            str := make([]byte, 0, 10)
            str = append(str, match.Bytes...)
            brackets := 1
            match.EndLine = match.StartLine
            match.EndColumn = match.StartColumn
            for tc := scan.TC; tc < len(scan.Text); tc++ {
                str = append(str, scan.Text[tc])
                match.EndColumn += 1
                if scan.Text[tc] == '\n' {
                    match.EndLine += 1
                }
                if scan.Text[tc] == '<' {
                    brackets += 1
                } else if scan.Text[tc] == '>' {
                    brackets -= 1
                }
                if brackets == 0 {
                    match.TC = scan.TC
                    scan.TC = tc + 1
                    match.Bytes = str
                    return token("ID")(scan, match)
                }
            }
            return nil,
                fmt.Errorf("unclosed HTML literal starting at %d, (%d, %d)",
                    match.TC, match.StartLine, match.StartColumn)
        },
    )

    err := lexer.Compile()
    if err != nil {
        return nil, err
    }
    return lexer, nil
}

Initializing the Package

To ensure that the regular expressions are only compiled once, we are going to call initLexer once at the start of the program. To do this we put the call inside an init function. Init functions get run once on program start up.

// Called at package initialization. Creates the lexer and populates token lists.
func init() {
    initTokens()
    var err error
    Lexer, err = initLexer()
    if err != nil {
        panic(err)
    }
}

And that is it. Here is the source code for the full lexer.

Using the Lexer

Let's put it all together. Here is a simple example which uses the lexer:

package main

import (
    "fmt"
    "log"
)

import (
    "github.com/timtadh/dot"
    lex "github.com/timtadh/lexmachine"
)

func main() {
    s, err := dot.Lexer.Scanner([]byte(`digraph {
  rankdir=LR;
  a [label="a" shape=box];
  c [<label>=<<u>C</u>>];
  b [label="bb"];
  a -> c;
  c -> b;
  d -> c;
  b -> a;
  b -> e;
  e -> f;
}`))
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println("Type    | Lexeme     | Position")
    fmt.Println("--------+------------+------------")
    for tok, err, eof := s.Next(); !eof; tok, err, eof = s.Next() {
        if err != nil {
            log.Fatal(err)
        }
        token := tok.(*lex.Token)
        fmt.Printf("%-7v | %-10v | %v:%v-%v:%v\n",
            dot.Tokens[token.Type],
            string(token.Lexeme),
            token.StartLine,
            token.StartColumn,
            token.EndLine,
            token.EndColumn)
    }
}

Output:

Type    | Lexeme     | Position
--------+------------+------------
DIGRAPH | digraph    | 1:1-1:7
{       | {          | 1:9-1:9
ID      | rankdir    | 2:3-2:9
=       | =          | 2:10-2:10
ID      | LR         | 2:11-2:12
;       | ;          | 2:13-2:13
ID      | a          | 3:3-3:3
[       | [          | 3:5-3:5
ID      | label      | 3:6-3:10
=       | =          | 3:11-3:11
ID      | "a"        | 3:12-3:14
ID      | shape      | 3:16-3:20
=       | =          | 3:21-3:21
ID      | box        | 3:22-3:24
]       | ]          | 3:25-3:25
;       | ;          | 3:26-3:26
ID      | c          | 4:3-4:3
[       | [          | 4:5-4:5
ID      | <label>    | 4:6-4:12
=       | =          | 4:13-4:13
ID      | <<u>C</u>> | 4:14-4:23
]       | ]          | 4:24-4:24
;       | ;          | 4:25-4:25
ID      | b          | 5:3-5:3
[       | [          | 5:5-5:5
ID      | label      | 5:6-5:10
=       | =          | 5:11-5:11
ID      | "bb"       | 5:12-5:15
]       | ]          | 5:16-5:16
;       | ;          | 5:17-5:17
ID      | a          | 6:3-6:3
->      | ->         | 6:5-6:6
ID      | c          | 6:8-6:8
;       | ;          | 6:9-6:9
ID      | c          | 7:3-7:3
->      | ->         | 7:5-7:6
ID      | b          | 7:8-7:8
;       | ;          | 7:9-7:9
ID      | d          | 8:3-8:3
->      | ->         | 8:5-8:6
ID      | c          | 8:8-8:8
;       | ;          | 8:9-8:9
ID      | b          | 9:3-9:3
->      | ->         | 9:5-9:6
ID      | a          | 9:8-9:8
;       | ;          | 9:9-9:9
ID      | b          | 10:3-10:3
->      | ->         | 10:5-10:6
ID      | e          | 10:8-10:8
;       | ;          | 10:9-10:9
ID      | e          | 11:3-11:3
->      | ->         | 11:5-11:6
ID      | f          | 11:8-11:8
;       | ;          | 11:9-11:9
}       | }          | 12:1-12:1

Conclusion

Believe it or not 3500 words later, we have only scratched the surface on this topic. Testing, custom token representations, automata construction, and more will have to wait for another post. While still in an early state I hope you find lexmachine useful and this article helpful for constructing lexers whatever language you are using.

Golang中的面向对象继承

2016-05-20T00:00:00-04:00

Golang的面向对象机制与Smalltalk或者Java等传统面向对象的编程语言不同。传统面向对象编程语言的一个重要特性是继承机制。因为继承机制支持在关联对象间进行代码复用和数据共享。继承机制曾在代码复用和数据共享的设计模式占据主导地位，但是目前组合这一古老的技术重新焕发了活力。

本篇文章转自Tim Henderson的 "Object Oriented Inheritance in Go"，原文地址是 http://hackthology.com/object-oriented-inheritance-in-go.html 。非常感谢李浩和骏奇对于这篇文章的翻译。

在我们探讨如何在Go中实现继承机制之前（Golong中的继承机制和其他语言(Java)的继承机制有区别），我们先看一下Java中如何实现继承机制。

继承与组合

让我们先看一下我最喜欢的话题之一：编译器！编译器由管道转换构成，该管道读取text文本并将其转化为机器代码、汇编语言、字节码或者其他的编程语言。管道首先会使用语法分析器对目标变成语言进行语法分析。一般情况下文本会被分解为不同的组成部分，例如：关键词、标识符、标点和数字等等。每个组成部分都会被相应的数据类型标记。例如下面这个Java数据类型：

public class Main {}

这些组成部分（可以称作标记）如下所示：

<public keyword>, "public"
<class keyword>, "class"
<idenitifier>, "Main"
<left-bracket>, "{"
<right-bracket>, "}"

这些标记可以划分为两个部分：

标记类型
语义部分

这会导致我们进行如下的Java设计方式：

public enum TokenType {
    KEYWORD, IDENTIFIER, LBRACKET, RBRACKET, ...
}

public class Token {
    public TokenToken type;
    **Here, I think TokenToken should be "TokenType"**
    public String lexeme;
}

对于一些标记类型来说，例如数值常量，标记类型最好能够将包含这些属性信息。就数值常量来说，在他的标记类型里应该包括常量值这一属性。实现这一设计的传统方式是使用继承机制产生Token子类。

public class IntegerConstant extends Token {
    public long value;
}

另外一种完成该设计的方式是利用组合方式产生IntegerConstant，IntegerConstant包含token的引用：

public class IntegerConstant {
    public Token type;
    public long value;
}

在这个例子中，继承机制是一个比较恰当的选择。理由是语法分析器需要返回一个通用类型。考虑一下语法分析器的接口设计：

public class Lexer {
    public Lexer(InputStream in)
    public boolean EOF()
    public Token peek() throws Error
    public Token next() throws Error
}

在继承机制中，IntegerConstant 属于Token类型，所以它可以在Lexer中调用。这不是唯一可用或者最好的设计，但是这种设计方式是有效的。让我们看一下Go是如何完成这一目的的。

Inheritance and Composition in Go

Go中实现组合是一件十分容易的事情。简单组合两个结构体就能够构造一个新的数据类型。

type TokenType uint16

const (
    KEYWORD TokenType = iota
    IDENTIFIER
    LBRACKET
    RBRACKET
    INT
)

type Token struct {
  Type   TokenType
  Lexeme string
}

type IntegerConstant struct {
  Token *Token
  Value uint64
}

这就是Go中实现代码和数据共享的常用方式。然而如果你想实现继承机制，我们该如何去做？

Why would you want to use inheritance in go

一个可选的方案是将Token设计成接口类型。这种方案在Java和Go都适用：

type Token interface {
  Type()   TokenType
  Lexeme() string
}

type Match struct {
  toktype TokenType
  lexeme  string
}

type IntegerConstant struct {
  token Token
  value uint64
}

func (m *Match) Type() TokenType {
  return m.toktype
}

func (m *Match) Lexeme() string {
  return m.lexeme
}

func (i *IntegerConstant) Type() TokenType {
  return i.token.Type()
}

func (i *IntegerConstant) Lexeme() string {
  return i.token.Lexeme()
}

func (i *IntegerConstant) Value() uint64 {
  return i.value
}

这样分析器就可以返回满足Match 和 IntegerConstant类型的Token接口。

继承机制的简化版

上面的实现方案的一个问题是*IntegerConstant的方法调用中，出现了重复造轮子的问题。但是我们可以使用Go内建的嵌入机制来避免此类情况的出现。嵌入机制(匿名嵌入)允许类型之前共享代码和数据。

type IntegerConstant struct {
  Token
  value uint64
}

func (i *IntegerConstant) Value() uint64 {
  return i.value
}

IntegerConstant中匿名嵌入了Token类型，使得IntegerConstant"继承"了Token的字段和方法。很酷的方法！我们可以这样写代码：

t := IntegerConstant{&Match{KEYWORD, "wizard"}, 2}
fmt.Println(t.Type(), t.Lexeme(), t.Value())
x := Token(t)
fmt.Println(x.Type(), x.Lexeme())

(可以在这里试一下：https://play.golang.org/p/PJW7VShpE0)

我们没有编写Type()和Value()方法的代码，但是*IntegerConstant也实现了Token接口，非常棒。

结构体的"继承"机制

Go中有三种方式完成”继承“机制，您已经看到了第一种实现方式：在结构体的第一个字段匿名嵌入接口类型。你还可以利用结构体实现其他两种”继承“机制：

1. 匿名嵌入结构体实例

type IntegerConstant struct {
  Match
  value uint64
}

2. 匿名嵌入结构体实例指针

type IntegerConstant struct {
  *Match
  value uint64
}

在所有的例子中，与正常嵌入类型不同的是我们使用匿名嵌入。然而，这个字段还是有字段名称的，名称是嵌入类型名称。在IntegerConstant 的Match字段中，字段名称是Match，无论嵌入类型是实例还是指针。

在以上的方案中，你不能嵌入与嵌入类型相同的方法名。例如结构体Bar匿名嵌入结构体Foo后，就不能拥有名称为Foo的方法，同样也不能实现type Fooer interface { Foo() }接口类型。

共享代码、共享数据或者两者兼得

相比于Java，Go在继承和聚合之间的界限是很模糊的。Go中没有extends关键词。在语法的层次上，继承看上去与聚合没有什么区别。Go中聚合跟继承唯一的不同在于，继承自其他结构体的struct类型可以直接访问父类结构体的字段和方法。

type Pet struct {
  name string
}

type Dog struct {
  Pet
  Breed string
}

func (p *Pet) Speak() string {
  return fmt.Sprintf("my name is %v", p.name)
}

func (p *Pet) Name() string {
  return p.name
}

func (d *Dog) Speak() string {
  return fmt.Sprintf("%v and I am a %v", d.Pet.Speak(), d.Breed)
}

func main() {
  d := Dog{Pet: Pet{name: "spot"}, Breed: "pointer"}
  fmt.Println(d.Name())
  fmt.Println(d.Speak())
}

(可以试一下 https://play.golang.org/p/Pmkd27Nqqy)

输出：

spot
my name is spot and I am a pointer

嵌入式继承机制的的局限

相比于Java， Go的继承机制的作用是非常有限的。有很多的设计方案可以在Java轻松实现，但是Go却不可能完成同样的工作。让我们看一下：

Overriding Methods

上面的Pet例子中，Dog类型重载了Speak()方法。然而如果Pet有另外一个方法Play()被调用，但是Dog没有实现Play()的时候，Dog类型的Speak()方法则不会被调用。

package main

import (
    "fmt"
)

type Pet struct {
    name string
}

type Dog struct {
    Pet
    Breed string
}

func (p *Pet) Play() {
    fmt.Println(p.Speak())
}

func (p *Pet) Speak() string {
    return fmt.Sprintf("my name is %v", p.name)
}

func (p *Pet) Name() string {
    return p.name
}

func (d *Dog) Speak() string {
    return fmt.Sprintf("%v and I am a %v", d.Pet.Speak(), d.Breed)
}

func main() {
    d := Dog{Pet: Pet{name: "spot"}, Breed: "pointer"}
    fmt.Println(d.Name())
    fmt.Println(d.Speak())
    d.Play()
}

(试一下 https://play.golang.org/p/id-aDKW8L6)

输出：

spot
my name is spot and I am a pointer
my name is spot

但是Java中就会像我们预想的那样工作：

public class Main {
  public static void main(String[] args) {
    Dog d = new Dog("spot", "pointer");
    System.out.println(d.Name());
    System.out.println(d.Speak());
    d.Play();
  }
}

class Pet {
  public String name;

  public Pet(String name) {
    this.name = name;
  }

  public void Play() {
    System.out.println(Speak());
  }

  public String Speak() {
    return String.format("my name is %s", name);
  }

  public String Name() {
    return name;
  }
}

class Dog extends Pet {
  public String breed;

  public Dog(String name, String breed) {
    super(name);
    this.breed = breed;
  }

  public String Speak() {
    return String.format("my name is %s and I am a %s", name, breed);
  }
}

输出：

$ javac Main.java && java Main
spot
my name is spot and I am a pointer
my name is spot and I am a pointer

这个明显的区别是因为Go从根本上阻止了抽象方法的使用。让我们看看下面这个例子：

package main

import (
    "fmt"
)

type Pet struct {
    speaker func() string
    name    string
}

type Dog struct {
    Pet
    Breed string
}

func NewPet(name string) *Pet {
    p := &Pet{
        name: name,
    }
    p.speaker = p.speak
    return p
}

func (p *Pet) Play() {
    fmt.Println(p.Speak())
}

func (p *Pet) Speak() string {
    return p.speaker()
}

func (p *Pet) speak() string {
    return fmt.Sprintf("my name is %v", p.name)
}

func (p *Pet) Name() string {
    return p.name
}

func NewDog(name, breed string) *Dog {
    d := &Dog{
        Pet:   Pet{name: name},
        Breed: breed,
    }
    d.speaker = d.speak
    return d
}

func (d *Dog) speak() string {
    return fmt.Sprintf("%v and I am a %v", d.Pet.speak(), d.Breed)
}

func main() {
    d := NewDog("spot", "pointer")
    fmt.Println(d.Name())
    fmt.Println(d.Speak())
    d.Play()
}

(试一下 https://play.golang.org/p/9iIb2px7jH)

输出：

spot
my name is spot and I am a pointer
my name is spot and I am a pointer

现在跟我们预想的一样了，但是跟Java相比略显冗长和晦涩。你必须手工重载方法签名。而且，代码在结构体未正确初始化的情况下会崩溃，例如当调用Speak()时，speaker()却没有完成初始化工作的时候。

Subtyping

在Java中，Dog继承自Pet，那么Dog类型就是Pet子类。这意味着在任何需要调用Pet类型的场景都可以使用Dog类型替换。这种关系称作多态性，但Go的结构体类型不存在这种机制。

让我们看下面的例子：

package main

import (
    "fmt"
)

type Pet struct {
    speaker func() string
    name    string
}

type Dog struct {
    Pet
    Breed string
}

func NewPet(name string) *Pet {
    p := &Pet{
        name: name,
    }
    p.speaker = p.speak
    return p
}

func (p *Pet) Play() {
    fmt.Println(p.Speak())
}

func (p *Pet) Speak() string {
    return p.speaker()
}

func (p *Pet) speak() string {
    return fmt.Sprintf("my name is %v", p.name)
}

func (p *Pet) Name() string {
    return p.name
}

func NewDog(name, breed string) *Dog {
    d := &Dog{
        Pet:   Pet{name: name},
        Breed: breed,
    }
    d.speaker = d.speak
    return d
}

func (d *Dog) speak() string {
    return fmt.Sprintf("%v and I am a %v", d.Pet.speak(), d.Breed)
}

func Play(p *Pet) {
    p.Play()
}

func main() {
    d := NewDog("spot", "pointer")
    fmt.Println(d.Name())
    fmt.Println(d.Speak())
    Play(d)
}

(试一下 https://play.golang.org/p/e1Ujx0VhwK)

输出：

prog.go:62: cannot use d (type *Dog) as type *Pet in argument to Play

然而，接口类型中存在子类化的多态机制！

package main

import (
    "fmt"
)

type Pet interface {
    Name() string
    Speak() string
    Play()
}

type pet struct {
    speaker func() string
    name    string
}

type Dog interface {
    Pet
    Breed() string
}

type dog struct {
    pet
    breed string
}

func NewPet(name string) *pet {
    p := &pet{
        name: name,
    }
    p.speaker = p.speak
    return p
}

func (p *pet) Play() {
    fmt.Println(p.Speak())
}

func (p *pet) Speak() string {
    return p.speaker()
}

func (p *pet) speak() string {
    return fmt.Sprintf("my name is %v", p.name)
}

func (p *pet) Name() string {
    return p.name
}

func NewDog(name, breed string) *dog {
    d := &dog{
        pet:   pet{name: name},
        breed: breed,
    }
    d.speaker = d.speak
    return d
}

func (d *dog) speak() string {
    return fmt.Sprintf("%v and I am a %v", d.pet.speak(), d.breed)
}

func Play(p Pet) {
    p.Play()
}

func main() {
    d := NewDog("spot", "pointer")
    fmt.Println(d.Name())
    fmt.Println(d.Speak())
    Play(d)
}

(试一下 https://play.golang.org/p/WMH-cr4AJf)

输出：

spot
my name is spot and I am a pointer
my name is spot and I am a pointer

所以接口类型可以用来实现子类化的机制。但是如果你想正确的实现方法重载，需要了解以上的技巧。

Conclusion

事实上，虽然这不是Go的主打特性，但是Go语言在结构体嵌入结构体或者接口方面的能力确实为实际工作增加了很大的灵活性。Go的这些特性为我们解决实际问题提供了新的解决方案。但是相较于Java等语言，由于Go缺少子类化和方法重载支持还有存在一些局限性。Go含有一项Java没有的特性--接口嵌入。关于接口嵌入的细节请参考Golang的官方文档的Embedding部分。

非常感谢echlebek, Alexander Staubo, spriggan3和breerly对这篇文章提供的支持！

Object Oriented Inheritance in Go

2016-05-20T00:00:00-04:00

The Go (golang) programming language is not a traditional object oriented language like Smalltalk or Java. A key feature supporting traditional object oriented design is inheritance. Inheritance supports sharing of code and data between related objects. It used to be that inheritance was the dominant design for sharing code and data but today another (older) technique called composition has seen a resurgence.

Thank you 李浩 (Hao Li) for translating this article into Mandarin: Golang中的面向对象继承

Before we jump into how to use inheritance in Go (which has some interesting edge cases) let's see how it is used in Java.

Inheritance vs. Composition in Java

Let's look at an example from one of my favorite topics: compilers! A compiler is made up of a pipeline of transformations that take "plain text" and transform it either to machine code, assembly language, bytecode, or another programming language. The first stage of the pipeline the lexer performs what is called lexical analysis of the programming language. It traditionally splits the text up into the different components as: keywords, identifiers, punctuation, numbers, etc... Each component is tagged with the class of component it is. So for this fragment of Java:

public class Main {}

The "components" (called tokens) would be:

<public keyword>, "public"
<class keyword>, "class"
<idenitifier>, "Main"
<left-bracket>, "{"
<right-bracket>, "}"

The tokens have two parts:

The token type
The lexeme, or the string the token was extracted from

This leads to the following Java design

public enum TokenType {
    KEYWORD, IDENTIFIER, LBRACKET, RBRACKET, ...
}

public class Token {
    public TokenToken type;
    public String lexeme;
}

For some tokens, such as numeric constants, it is convenient to specialize the Token object to contain some extra information. In the case of numeric constants the numerical value of the lexeme is convenient to store directly in the Token. The traditional way to accomplish this is to have the numeric tokens inherit from the Token class.

public class IntegerConstant extends Token {
    public long value;
}

Another way this can be achieved is to use composition where the IntegerConstant instead of extending the Token class contains a reference to the token.

public class IntegerConstant {
    public Token type;
    public long value;
}

It turns out, in this particular case, inheritance is the better choice. The reason is the Lexer which produces the tokens needs to return a common type. Consider the interface of the Lexer:

public class Lexer {
    public Lexer(InputStream in)
    public boolean EOF()
    public Token peek() throws Error
    public Token next() throws Error
}

Since in the first design (which uses inheritance) a IntegerConstant is a Token it can be used in the Lexer. Now, this isn't the only design that can be used and maybe isn't even the best design but it is valid. Let's take a look at how it translates to into Go.

Inheritance and Composition in Go

Composition is very natural in Go (as it is in most languages). To compose two structures simply provide a pointer or embedding to the collaborating structure.

type TokenType uint16

const (
    KEYWORD TokenType = iota
    IDENTIFIER
    LBRACKET
    RBRACKET
    INT
)

type Token struct {
  Type   TokenType
  Lexeme string
}

type IntegerConstant struct {
  Token *Token
  Value uint64
}

This would be the usual way to share code and data in Go. However, if you feel the need for inheritance then how can we use it?

Why would you want to use inheritance in go?

One of the obvious alternative designs for the Token is to make it an interface. This works equally well in both Java and Go:

type Token interface {
  Type()   TokenType
  Lexeme() string
}

type Match struct {
  toktype TokenType
  lexeme  string
}

type IntegerConstant struct {
  token Token
  value uint64
}

func (m *Match) Type() TokenType {
  return m.toktype
}

func (m *Match) Lexeme() string {
  return m.lexeme
}

func (i *IntegerConstant) Type() TokenType {
  return i.token.Type()
}

func (i *IntegerConstant) Lexeme() string {
  return i.token.Lexeme()
}

func (i *IntegerConstant) Value() uint64 {
  return i.value
}

The Lexer can then easily return the Token interface which both *Match and *IntegerConstant satisfy.

Simplifying with inheritance

One of the problems with the previous design is the manual work in *IntegerConstant calling i.token.Type() and i.token.Lexeme(). It turns out we can use Go's built in support for embedding to avoid this work. Embedding is a limited form of inheritance which allows types to share data and code.

type IntegerConstant struct {
  Token
  value uint64
}

func (i *IntegerConstant) Value() uint64 {
  return i.value
}

By not giving the Token field a name in IntegerConstant, it "inherits" the methods (and fields if Token was a struct) from Token. This pretty cool! We can write code like this:

t := IntegerConstant{&Match{KEYWORD, "wizard"}, 2}
fmt.Println(t.Type(), t.Lexeme(), t.Value())
x := Token(t)
fmt.Println(x.Type(), x.Lexeme())

(try it in the playground https://play.golang.org/p/PJW7VShpE0)

So wow! Not only did we not have implement Type() and Value() but *IntegerConstant also implements the Token interface. Pretty nice.

"Inheriting" from `structs`

There are three ways to do "inheritance" in Go. You have already seen one, "inheriting" from an interface by putting it as the first member without a field name. It turns out your can do the same thing with structs and you have two choices

Inherit by embedding the struct by value

type IntegerConstant struct {
  Match
  value uint64
}

Inherit by embedding a pointer to a struct

type IntegerConstant struct {
  *Match
  value uint64
}

In all cases, the difference from a regular field is the lack of an explicit name. However, the field still has a name. It is the name of the embedded type. In the case of IntegerConstant the Match field is named Match. This is true whether one embeds a pointer to a struct or a struct by value.

On gotcha of all of these options, you can't have a Field and a method with the same name. A struct Bar is embedding a struct Foo precludes Bar from having a method called Foo. It also prevents Bar from implementing type Fooer interface { Foo() }.

Sharing Data, Code or Both

In Go the line between inheritance and composition is pretty blurry in comparison with Java. There is no extends keyword. Syntactically, inheritance looks almost identical to composition. The only difference between composition and inheritance in Go, is a struct which inherits from another struct can directly access the methods and fields of the parent struct.

type Pet struct {
  name string
}

type Dog struct {
  Pet
  Breed string
}

func (p *Pet) Speak() string {
  return fmt.Sprintf("my name is %v", p.name)
}

func (p *Pet) Name() string {
  return p.name
}

func (d *Dog) Speak() string {
  return fmt.Sprintf("%v and I am a %v", d.Pet.Speak(), d.Breed)
}

func main() {
  d := Dog{Pet: Pet{name: "spot"}, Breed: "pointer"}
  fmt.Println(d.Name())
  fmt.Println(d.Speak())
}

(try in the playground https://play.golang.org/p/Pmkd27Nqqy)

Output:

spot
my name is spot and I am a pointer

Limitations of Embedding as Inheritance

In comparison to a language like Java, Go's form of inheritance is quite limited. There are multiple designs which can be easily accomplished in Java which are not possible in Go. Let's look at some of them.

Overriding Methods

In the pet example above, Dog "overrides" the Speak() method. However, if Pet had another method Play() which invokes Speak() that Dog does not override the Dog's implementation of Speak() would not be used:

package main

import (
    "fmt"
)

type Pet struct {
    name string
}

type Dog struct {
    Pet
    Breed string
}

func (p *Pet) Play() {
    fmt.Println(p.Speak())
}

func (p *Pet) Speak() string {
    return fmt.Sprintf("my name is %v", p.name)
}

func (p *Pet) Name() string {
    return p.name
}

func (d *Dog) Speak() string {
    return fmt.Sprintf("%v and I am a %v", d.Pet.Speak(), d.Breed)
}

func main() {
    d := Dog{Pet: Pet{name: "spot"}, Breed: "pointer"}
    fmt.Println(d.Name())
    fmt.Println(d.Speak())
    d.Play()
}

(try it on the playground: https://play.golang.org/p/id-aDKW8L6)

Output:

spot
my name is spot and I am a pointer
my name is spot

Contrast this to Java, in Java it would work as expected!

public class Main {
  public static void main(String[] args) {
    Dog d = new Dog("spot", "pointer");
    System.out.println(d.Name());
    System.out.println(d.Speak());
    d.Play();
  }
}

class Pet {
  public String name;

  public Pet(String name) {
    this.name = name;
  }

  public void Play() {
    System.out.println(Speak());
  }

  public String Speak() {
    return String.format("my name is %s", name);
  }

  public String Name() {
    return name;
  }
}

class Dog extends Pet {
  public String breed;

  public Dog(String name, String breed) {
    super(name);
    this.breed = breed;
  }

  public String Speak() {
    return String.format("my name is %s and I am a %s", name, breed);
  }
}

Output

$ javac Main.java && java Main
spot
my name is spot and I am a pointer
my name is spot and I am a pointer

This is a pretty big difference as it essentially precludes the use of abstract methods as you might want to define them. However, there is a work around:

package main

import (
    "fmt"
)

type Pet struct {
    speaker func() string
    name    string
}

type Dog struct {
    Pet
    Breed string
}

func NewPet(name string) *Pet {
    p := &Pet{
        name: name,
    }
    p.speaker = p.speak
    return p
}

func (p *Pet) Play() {
    fmt.Println(p.Speak())
}

func (p *Pet) Speak() string {
    return p.speaker()
}

func (p *Pet) speak() string {
    return fmt.Sprintf("my name is %v", p.name)
}

func (p *Pet) Name() string {
    return p.name
}

func NewDog(name, breed string) *Dog {
    d := &Dog{
        Pet:   Pet{name: name},
        Breed: breed,
    }
    d.speaker = d.speak
    return d
}

func (d *Dog) speak() string {
    return fmt.Sprintf("%v and I am a %v", d.Pet.speak(), d.Breed)
}

func main() {
    d := NewDog("spot", "pointer")
    fmt.Println(d.Name())
    fmt.Println(d.Speak())
    d.Play()
}

(try it on the playground https://play.golang.org/p/9iIb2px7jH)

Output:

spot
my name is spot and I am a pointer
my name is spot and I am a pointer

Now, it works "as expected" but it is much more verbose and difficult than in Java. You manually have to override the method yourself. Furthermore, it is rather fragile because if the struct is initialized incorrectly it will crash when Speak() is called because speaker() will not have been correctly initialized.

Subtyping

In Java, when the class Dog extends Pet it is a Pet. That means in every place you need an object of type Pet you can use a Dog object. Dog is said to substitute for Pet. This relationship is known as subtyping (Dog is a subtype of Pet). This relationship is also called subtype polymorphism and it does not exist in the Go programming language for struct types.

Let's look at an example:

package main

import (
    "fmt"
)

type Pet struct {
    speaker func() string
    name    string
}

type Dog struct {
    Pet
    Breed string
}

func NewPet(name string) *Pet {
    p := &Pet{
        name: name,
    }
    p.speaker = p.speak
    return p
}

func (p *Pet) Play() {
    fmt.Println(p.Speak())
}

func (p *Pet) Speak() string {
    return p.speaker()
}

func (p *Pet) speak() string {
    return fmt.Sprintf("my name is %v", p.name)
}

func (p *Pet) Name() string {
    return p.name
}

func NewDog(name, breed string) *Dog {
    d := &Dog{
        Pet:   Pet{name: name},
        Breed: breed,
    }
    d.speaker = d.speak
    return d
}

func (d *Dog) speak() string {
    return fmt.Sprintf("%v and I am a %v", d.Pet.speak(), d.Breed)
}

func Play(p *Pet) {
    p.Play()
}

func main() {
    d := NewDog("spot", "pointer")
    fmt.Println(d.Name())
    fmt.Println(d.Speak())
    Play(d)
}

(try it out on the playground https://play.golang.org/p/e1Ujx0VhwK)

Output:

prog.go:62: cannot use d (type *Dog) as type *Pet in argument to Play

However, not all is lost because subtyping does exist for interface types! Let's try it out:

package main

import (
    "fmt"
)

type Pet interface {
    Name() string
    Speak() string
    Play()
}

type pet struct {
    speaker func() string
    name    string
}

type Dog interface {
    Pet
    Breed() string
}

type dog struct {
    pet
    breed string
}

func NewPet(name string) *pet {
    p := &pet{
        name: name,
    }
    p.speaker = p.speak
    return p
}

func (p *pet) Play() {
    fmt.Println(p.Speak())
}

func (p *pet) Speak() string {
    return p.speaker()
}

func (p *pet) speak() string {
    return fmt.Sprintf("my name is %v", p.name)
}

func (p *pet) Name() string {
    return p.name
}

func NewDog(name, breed string) *dog {
    d := &dog{
        pet:   pet{name: name},
        breed: breed,
    }
    d.speaker = d.speak
    return d
}

func (d *dog) speak() string {
    return fmt.Sprintf("%v and I am a %v", d.pet.speak(), d.breed)
}

func Play(p Pet) {
    p.Play()
}

func main() {
    d := NewDog("spot", "pointer")
    fmt.Println(d.Name())
    fmt.Println(d.Speak())
    Play(d)
}

(try it on the playground https://play.golang.org/p/WMH-cr4AJf)

Output:

spot
my name is spot and I am a pointer
my name is spot and I am a pointer

Thus, interfaces can be used to acheive a form of subtyping. However, if they do not change the equation on method overriding. If you want a method overridden correctly you still have to use the "trick" I presented above.

Conclusion

So it turns out, while it isn't a headline feature of Go, its ability for structs to embed struct pointers, structs, and interfaces is powerful and flexible. It allows innovative designs that can solve real problems. However, in comparison to Java it is limited because of a lack of direct support for subtyping and method overriding. It does contain one feature that Java does not, the ability to embed an interface. For more details on embedding checkout the Embedding section of Effective Go.

Thank you to echlebek, Alexander Staubo, spriggan3, and breerly for reading and providing thoughtful feedback on this post.

Exceptions for Go as a Library

2016-05-19T00:00:00-04:00

Go (golang) lacks support for exceptions found in many other languages. There are good reasons for Go to not include exceptions. For instance, by making error handling explicit the programmer is forced to think concretely about the correct action to take. Fined grained control over the handling of errors using multiple return parameters is one of Go's strengths.

However, there are cases where Go programs do not universally benefit from the explicit handling of errors. For instance, consider the following code:

func DoStuff(a, b, c interface{}) error {
    x, err := foo(a)
    if err != nil {
        return err
    }
    y, err := bar(b, x)
    if err != nil {
        return err
    }
    z, err := bax(c, x, y)
    if err != nil {
        return err
    }
    return baz(x, y, z)
}

If Go had exceptions such code could be easily simplified:

func DoStuff(a, b, c interface{}) throws error {
    x := foo(a)
    y := bar(b, x)
    baz(x, y, bax(c, x, y)
}

Adding Exceptions with a Library

I created a library which adds support for exceptions to the go programming language. This library allow you to write go with exceptions and try-catch-finally blocks. It is not appropriate for all situations but can simplify some application code. Libraries and external APIs should continue to conform to the Go standard of returning error values.

Here is an example of the DoStuff function where foo, bar and baz all throw exceptions instead of returning errors. (We will look at the case where they return errors that you want to turn into exceptions next). We want DoStuff to be an public API function and return an error:

func DoStuff(a, b, c interface{}) error {
    return exc.Try(func() {
        x := foo(a)
        y := bar(b, x)
        baz(x, y, bax(c, x, y)
    }).Error()
}

Catch Blocks

Now let's consider the case where we want to catch the exception log and reraise it:

func DoStuff(a, b, c interface{}) error {
    return exc.Try(func() {
        x := foo(a)
        y := bar(b, x)
        baz(x, y, bax(c, x, y)
    }).Catch(&exc.Exception{}, func(t exc.Throwable) {
        log.Print(t)
        exc.Rethrow(t, exc.Errorf("rethrow after logging"))
    }).Error()
}

Rethrow will chain the Throwable t with the new *Error created such that if/when the exception reaches the top level you know exactly how it was created and where it was re-thrown.

Throwing Errors

Ok, what about interacting with regular Go APIs which return errors? How can we turn those errors into exceptions? The easy was is to use the ThrowOnError function which is a sugar for:

if err != nil {
    ThrowErr(ErrorFrom(err)
}

So converting out original DoStuff function we get

func DoStuff(a, b, c interface{}) { // Throws
    x, err := foo(a)
    exc.ThrowOnError(err)
    y, err := bar(b, x)
    exc.ThrowOnError(err)
    z, err := bax(c, x, y)
    exc.ThrowOnError(err)
    exc.ThrowOnError(baz(x, y, z))
}

This package also supports: catching user defined exceptions, catching multiple exception types, Close which works like the "try with resources" construct in Java 7+, (multiple) finally blocks, and a choice between propagating exceptions with Unwind or retrieving the error/exception with Error and Exception functions.

One Gotcha! The Try() function creates a *Block struct. To execute the block you must either call: Unwind, Error, or Exception. Unwind executes the block, if there is an exception coming out of the block it continues to cause the program stack unwind. Error and Exception execute the block, but return the exception as a value to deal with in the usual Go way.

Finally Blocks

Finally blocks are a great feature of exceptions. They allow you to have a block of code run unconditionally after a try and catch block even if there was an un-handled exception or re-raised exception. In this example I use a finally block to log the timing information of the DoStuff function.

func DoStuff(a, b, c interface{}) error {
    start := time.Now()
    return exc.Try(func() {
        x := foo(a)
        y := bar(b, x)
        baz(x, y, bax(c, x, y)
    }).Catch(&exc.Exception{}, func(t exc.Throwable) {
        log.Print(t)
        exc.Rethrow(t, exc.Errorf("rethrow after logging"))
    }).Finally(func() {
        end := time.Now()
        log.Printf("Do stuff took: %v", end.Sub(start))
    }).Error()
}

Automatically Closing Resources

One common situation is to open a resource (like a file) and want it to close no matter what. In Go this is often accomplished with a defer function. However, we can accomplish the same thing with finer granularity using the Close function which acts like a Try but automatically closes a created resource when the block exits:

Close(func() io.Closer {
    f, err := os.Create("/tmp/wizard")
    ThrowOnError(err)
    return f
}, func(c io.Closer) {
    f := c.(*os.File)
    _, err := f.WriteString("wizardry")
    ThrowOnError(err)
}).Unwind()

In the above code, the file created in the first function will always be closed at the end of the *Block even if WriteString had an error. This get more interesting if you imagine passing the resource to other functions which could throw exceptions.

User Defined Exception Types

To create a user defined exception simply create a struct which inherits from the exc.Exception struct by embedding it as the first member.

type MyException struct {
    exc.Exception
}

Then you can catch MyException with Exception. eg:

exc.Try(func() {
    Throw(&MyException{*Errorf("My Exception").Exception()})
}).Catch(&Exception{}, func(t Throwable) {
    log.Log("caught!")
}).Unwind()

This should work with heirarchies of exceptions allowing your code to declare specific exceptions for specific errors.

How does it work?

Go provides panic and recover as built-in functions. panic(.) allows you to cause the program to halt and the stack to "unwind". This means stack frame by stack frame the each function is halted and any deferred functions are run. Let's look at an example:

package main

import (
    "fmt"
)

func a() {
    defer func() {
        fmt.Println("defer a")
    }()
    fmt.Println("a")
    b()
}

func b() {
    defer func() {
        fmt.Println("defer b")
    }()
    fmt.Println("b")
    c()
}

func c() {
    defer func() {
        fmt.Println("defer c")
    }()
    fmt.Println("c")
    panic("c panic")
}

func main() {
    a()
}

Output:

a
b
c
defer c
defer b
defer a
panic: c panic

goroutine 1 [running]:
panic(0x128360, 0x1040a140)
    /usr/local/go/src/runtime/panic.go:481 +0x700
main.c()
    /tmp/sandbox797297488/main.go:28 +0x180
main.b()
    /tmp/sandbox797297488/main.go:20 +0x140
main.a()
    /tmp/sandbox797297488/main.go:12 +0x140
main.main()
    /tmp/sandbox797297488/main.go:32 +0x20

(try it on the playground https://play.golang.org/p/0Vp4jq0978)

So panic by default kills your program but gives you a nice stack trace. It also runs your defered functions which allows some cleanup to happen. Recognizing this was rather limited and their are times even in Go when it is nice to recover from what is usually a fatal error (like in webservers) Go also provides recover.

The recover function can be thought of as a limited panic catching function. It can only meaingfully be used inside of defer functions. When a recover is found it will stop the panic (that is stop unwinding the stack) and allow the function the defered recover is in to exit normally. Let's look at another example:

package main

import (
    "fmt"
)

func a() {
    defer func() {
        fmt.Println("defer a")
    }()
    fmt.Println("a")
    b()
    fmt.Println("end a")
}

func b() {
    defer func() {
        fmt.Println("defer b")
        recover()
    }()
    fmt.Println("b")
    c()
    fmt.Println("end b")
}

func c() {
    defer func() {
        fmt.Println("defer c")
    }()
    fmt.Println("c")
    panic("c panic")
    fmt.Println("end c")
}

func main() {
    fmt.Println("start")
    a()
    fmt.Println("end")
}

Output:

start
a
b
c
defer c
defer b
end a
defer a
end

(Try it on the playground: https://play.golang.org/p/WTzKywhey2)

This time, instead of the program crashing it exited normally. Functions, a and main, who invocations occurred before b which invoked recover exited normally. However, using recover like this will stop all errors and not explain what the problem is. Luckily, recover reports whether or not a panic was recovered and what the argument to the panic was. Here is an example:

package main

import (
    "fmt"
)

func a() {
    defer func() {
        fmt.Println("defer a")
        if e := recover(); e != nil {
            fmt.Println("recovered", e)
        }
        fmt.Println("end defer of a")
    }()
    fmt.Println("a")
    b()
    fmt.Println("end a")
}

func b() {
    defer func() {
        fmt.Println("defer b")
        if e := recover(); e != nil {
            fmt.Println("recovered", e)
        }
        fmt.Println("end defer of b")
    }()
    fmt.Println("b")
    c()
    fmt.Println("end b")
}

func c() {
    defer func() {
        fmt.Println("defer c")
    }()
    fmt.Println("c")
    panic("c panic")
    fmt.Println("end c")
}

func main() {
    fmt.Println("start")
    a()
    fmt.Println("end")
}

Output:

start
a
b
c
defer c
defer b
recovered c panic
end defer of b
end a
defer a
end defer of a
end

(Try it on the playground: https://play.golang.org/p/agxBcqgCb8)

This time the value passed into panic was returned by the call to recover in b. In a which contains the same code as b to recover the panic nothing is returned by recover.

Implementation

To implement exceptions, panic is used to throw the Throwable objects. Then a special function exec is created to execute the try functions:

func (b *Block) exec() (err Throwable) {
    defer func() {
        if e := recover(); e != nil {
            switch exc := e.(type) {
            case Throwable:
                err = exc
            default:
                panic(e)
            }
        }
    }()
    b.try()
    return
}

This function simply calls the function passed into Try(). However, exec also registers a defer which changes the returned value of exec if a Throwable is discovered. Thus, we can throw exceptions with panic (no changes necessary!) and catch them with recover. The final piece is implementing the semantics of try, catch, finally. This is accomplished by a function called run:

func (b *Block) run() (Throwable) {
    err := b.exec()
    if err != nil {
        t := reflect.TypeOf(err)
        for _, c := range b.catches {
            if isa(t, c.exception) {
                err = Try(func(){c.catch(err)}).exec()
                break
            }
        }
    }
    for _, finally := range b.finallies {
        finally()
    }
    return err
}

The run function first executes the try and gets the error if any. If there is an error it tries to find a catch function to handle it (using the isa helper function to identify the catcher). Then whether or not a catch function was found, all finally functions are run in order of declaration. The final trick is the catch functions are run inside of a Try block and exec directly in case they re-throw exceptions. That is it! Check out the source code for more details.

Conclusion

There are good reasons why Go does not include exceptions at the language level. However, at times their absence can become annoying. So I created this little library that adds them to Go. You probably shouldn't make public APIs based on it, but it may help you write certain internal code more clearly. One clear use case is during a refactor adding Throw call in a deep internal function which should have always returned an error. Then at the entry points to the API simply wrap up in a Try().Catch().Error() which would accomplish the same thing as adding a returned error throughout the library.

I hope you enjoy this little experiment of mine and don't send me too much hate mail! I know that exceptions are controversal in the Go community. This library tries to demonstrate how close panic and recover are to exceptions and how to emulate exceptions using panic and recover.

Check it out!.

First Rust Macro

2014-09-20T00:00:00-04:00

I started learning Rust yesterday. Today as I was writing input handling code I wanted to print to the standard err. The way to do that seems to be:

(writeln![io::stderr(), "{}", err]).ok().expect("write to stderr failed");

That seems a bit verbose to me versus printing to the standard out, which is:

println!("{}", err)

So I decided to fix that with the following macro:

macro_rules! log(($fmt:expr$(, $msg:expr)*) => {
    (writeln![io::stderr(), $fmt $(, $msg)*]).ok().expect("log failed")
})

Which you can use like so:

log!("{}", err)

log!("expected {} got {}", expected, actual)

Scala Notes

2014-08-18T00:00:00-04:00

My notes as I relearn Scala. I am reading through Programming in Scala (2nd ed.)

Quibble with the Functional List

> from page 43

Why not append to lists?

Class List does offer an append operation -- it's written :+ and is explained in Chapter 24 -- but this operation is rarely used, because the time it takes to append to a list grows linearly with the size of the list, whereas prepending with :: take constant time. Your options if you want to build a list efficiently by append elements is to prepend them, then when you're done call reverse; or use a ListBuffer a mutable list that does offer an append operation, and when you're done call toList. ListBuffer will be described in Section 22.2.

I have always found the functional linked list to be a bit of an anachronism when appearing as core concept in a modern language. It is a limited data structure in terms of functionality, especially when compared to it non-functional mutable counter part (the doubly linked list with pointers to the head and the tail). The mutable doubly linked list can prepend, append, and swap elements in constant time. This makes it a highly useful data structure for a variety of tasks, including the wonderful Least Recently Used (LRU) cache replacement algorithm. In comparison, the traditional functional list of cons cells really only supports constant time prepend.

Scala should provide a immutable list implementation with all of the features one can expect of an ArrayList, such as the one found in Python or Ruby. Like the Python or Ruby list Scala can simply refer to the list as list and provide details about the algorithms used to achieve the functionality as documentation. In fact a quick Google reveals Scala has such a structure and it calls it a Vector. Why not simply call the Vector a list? Or at least introduce Vectors as the primary list like structure to use.

Using the List class as the example general purpose container type is just asking for trouble. Beginners would be better served by being introduced to Vector or a similar structure right away.

Other than this quibble about the book, I think the Scala collections library is great. A wide variety of data-structures with solid implementations.

Values vs. Variables

> from page 62

¹ The reason parameters are vals is that vals are easier to reason about. You needn't look further to determine if a val is reassigned, as you must do with a var.

Perfectly true, but why should I care? The compiler will have no trouble determining whether or not a var is re-assigned. In fact if this was an imperative language the first step of the optimizer would likely be to convert the code into Single Static Assignment (SSA) form. Once in (SSA) form every variable has exactly one definition and code re-arrangement optimizations can proceed without difficulty.

So using vals instead of vars must be for (in)convenience of the programmer. Scala does recommend using val over var when possible. So the choice is consistent with the philosophy of the language.

My question is why prioritize the immutable against the mutable? The benefits of this choice are not clearly explained. It is true that it is easier to write certain proofs if you don't have to deal with changing data on the heap however today we have the (mathematical) tools to deal with that situation. There are real benefits of mutable data-structures and algorithms, state mutation can be very efficient. For instance, the Union-Find algorithm has a straight forward and optimal imperative implementation. It is significantly harder to achieve an optimal immutable version as demonstrated by this paper from 2007.

Always Go To Editable Command Mode in Vim

2014-06-06T00:00:00-04:00

A useful trick in vim is an editable command mode with previous commands in the buffer. To get to this mode go to normal mode (ie. hit ESC) and then type q: this places you in editable command mode. If you always want to go to that mode you can remap : with the following command:

" Always go to history mode
nnoremap : q:i
nnoremap / q/i
nnoremap ? q?i

Happy Hacking!

Private Methods and Functions Should be Tested

2014-03-20T00:00:00-04:00

Recently I became engaged in a debate around whether or not private methods should usually have unit tests. Rather than writing up a quick off hand reply to the question I decided to write up a detailed stance on the matter.

Testing Theory

Why is software tested? It is tested for two reasons:

To reveal defects and deviations from a specification. Does the software function properly? This is called Software Verification.
To ensure the software meets the needs of the end user. Are we meeting the needs of our users? This is called Software Validation.

To validate software, engineers employ a variety of techniques such as user testing, case studies on user needs, surveys, runtime observations, alpha/beta testing, and acceptance testing. These techniques not only look for bugs and defects in the software but also try and understand whether it effectively solves problems for the user. This process informs the iterative development of the software and can lead to changes in functional requirements.

In order to verify the correct functioning of software, engineers check the software to ensure it satisfies the specification. A specification states how the software should function. Unfortunately, a complete and accurate specification never exists. Engineers carefully determine what to check and based on their estimations on the accuracy and completeness of a specification.

The modern engineer has many techniques that can be used to verify software. Broadly they fit into two categories: testing and program analysis. Program analysis uses various techniques to prove specific properties about a program. For exampled, in statically type checked languages (such as Java or Haskell) the type checker proves the absence of type errors. I will not discuss these techniques in this essay as there availability varies by language and environment.

Testing runs a program with specified inputs and checks that the result is correct. It is the primary way software is verified. While the advanced program analysis techniques are neat almost all software should be rigorously tested. Well what does "rigorously" mean. If software is rigorously tested does that then mean: the software will be dependable (and/or correct)?

Unfortunately, no matter what techniques are used to test software, tests can never ensure software is correct. However, it can give the development team confidence (with the right techniques) in the dependability of the software.

Rigorous Testing

What are some suitable criteria that can be used to determine if a test suite is adequate for verifying the software? Designing a adequate testing criteria has many pitfalls. Consider the following requirement:

A test suite must cover (execute) every statement in the program.

This requirement only looks useful. For instance, it doesn't say that the individual tests should check each component functioned properly. Rather, it only says that the component was executed.

The abstract requirements I often use for tests at the unit (function or method level) is:

A method should checked that it behaves properly in both nominal and exceptional cases.
If a model of how the method should function is available it should be checked against that model.
Path coverage, how many control flow paths through a method are taken, can be used a heuristic to guide the tester towards methods which may require further testing.
When required methods should be checked for security and language level faults such as: buffer overflows, SQL injection errors, cross site scripting errors, etc...
Public methods should be checked against all elements of their specified API contract.
Unit tests should assume the methods used by the method under test behave correctly if each of those methods has good tests.

These requirements will not ensure that a program has no bugs. However, it gives confidence that all methods in the program work properly because it forces the programmer to think about how each method should work. Not only think about it, but think how to test it, how to to check every control flow path, how to check all parts of the specified API. By engaging the programmer in this way bugs can be caught not only in the method but in the specification for the method.

Should Private Methods be Tested?

According to my theory on testing methods at the unit level private methods should be tested. My theory has the inductive hypothesis that methods used by a method under test can only be assumed to work if each of those methods has good tests. If that hypothesis is not met, one loses confidence in the test suite. Furthermore, the programmer is no longer forced to think critically about the function of every method.

I have received some important counter points to this point of view. I am going to address those individually.

If private methods are tested then how can we locate dead code?

One can solve this problem in several ways. My recommendation is to mark tests for private methods in such a way that they can be excluded from test suite runs.

Tests on private methods make the test suite brittle

Here the tester needs to use their best judgement on whether the specification on some area is in too much flux to formalize a test. They should also use their judgement on what to check. If the output format is likely to change rather than checking the output exactly matches some expected output one should check that the output has certain properties.

negative example

This test is very brittle. Slight changes to the code can easily break by re-ordering fields or adding and removing them.

String E = "edge {\"src\":0,\"targ\":1,\"label\":\"cfg\",\"src_label\":\"a\",\"targ_label\":\"b\"}";

@Test
public void Edge() {
    Graph g = new Graph();
    int a = g.addNode("a", "x.y", "c", "m", "t", 1, -1, 2, -1);
    int b = g.addNode("b", "x.y", "c", "m", "t", 1, -1, 2, -1);
    int c = g.addNode("c", "x.y", "c", "m", "t", 1, -1, 2, -1);
    Edge e = new Edge(a, b, "cfg", g);
    assertThat(e.Serialize(), is(E));
}

positive example

This test is better because it looks for just the properties of the serialization which are important (the source, target, and edge label are set correctly).

@Test
public void Edge() {
    Graph g = new Graph();
    int a = g.addNode("a", "x.y", "c", "m", "t", 1, -1, 2, -1);
    int b = g.addNode("b", "x.y", "c", "m", "t", 1, -1, 2, -1);
    int c = g.addNode("c", "x.y", "c", "m", "t", 1, -1, 2, -1);
    Edge e = new Edge(a, b, "cfg", g);
    String E = e.Serialize();
    assertThat(E.contains("\"src\":0"), is(true));
    assertThat(E.contains("\"targ\":1"), is(true));
    assertThat(E.contains("\"label\":\"cfg\""), is(true));
}

Testing private methods pulls away from BDD

BDD (Behavior Driven Development), tries to help projects stay on target by keeping them aligned to user needs. In this sense it is a validation technique which integrates validation into the development process. However, high level behavioural tests rely on the software underneath the high level functionality functioning properly. Verification of that functionality is needed regardless of its relationship to customer requirements.

For instance a customer would never specify that items from a sorted array should be retrieved with a binary search. But, if a programmer needs to retrieve an item from a sorted array and uses a binary search that search must work in order to meet whatever functional requirement the programmer is implementing. Furthermore, security requirements are typically not functional or behavioural but none the less they must be checked.

Testing private methods and methods which implement basic functionality does not conflict with BDD it enables it.

Testing private methods is time consuming

It is often difficult to know when the proper time is to write a test for a method. TDD (Test Driven Development) advocates for writing tests before a method is written. The TDD method can be useful in some circumstances but I would not advocate for universal application of it. I will give the advice that it is easier to write a test for a method at the time of development rather than later on. If the method needs to be modified in some way later on (especially in a way that does not break existing functionality) then the tests serve as a "safety belt" when extending it. If the change to the method changes existing functionality then the old tests can be safely deleted (but new tests should likely be created).

I don't think testing private methods is in particular time consuming. Rather, program verification and validation is in general time consuming. However, performing these activities is what separates high quality software from low quality software.

How to Re-Use a Sub-Project's Buildfile in Apache Buildr

2014-03-14T00:00:00-04:00

Apache Buildr is a nice build system for Java and other JVM langauges. It is based on Ruby's Rake system and makes it easy to setup a multi-project build. However, by default, there is no support for re-using a Buildfile from another project. The intreprepid hacker can fix that problem by loading the sub-project's Buildfile like this:

if File::exists?('sub-project/Buildfile')
  pwd = Dir.pwd
  Dir.chdir File::join(pwd, 'sub-project')
  load "./Buildfile"
  puts project('sub-project').base_dir  ## HACK: pins the project to the
                                        ## proper directory!
  Dir.chdir pwd
end

If you don't access base_dir before you reset the current working directory then it will eventually get set to the wrong directory causing build failures.

After you have loaded the Buildfile you can run:

$ pwd
/path/to/project
$ buildr help:projects
(in /path/to/project, development)
/path/to/project/sub-project
  main-project
  sub-project
  other-project
  another-project

All of the tasks from you sub-project should now work as expected from the parent project. Happy Buildring!

Vi Mode in Python Shell, IRB, and Elsewhere!

2014-02-04T00:00:00-05:00

Enable vi mode in irb, python shell, and other readline based systems:

cat <<END >>~/.inputrc
> set -o vi
> set editing-mode vi
> END

Now everything has vi key bindings! You are welcome.

Functional Iteration in Go

2013-12-13T00:00:00-05:00

Go provides a built in way to iterate easily over built-in collection types: maps, slices and chans. Iterating with these types is much like iterating using for-each type loops in languages like Python, Ruby and Java.

for key, value := range map {
    // key, value
}

for index, item := range slice {
    // index, item
}

for item := range channel {
    // item
}

Unfortunately, Go does not provide a way to iterate over user defined collections using range. I have been experimenting with how to solve this problem since the initial public release of Go and I have finally settled on a solution I like. This new solution uses a functional programming style.

Functional Iterators

type Iterator func()(item interface{}, next Iterator)

This iterator is a function. Calling the function yields the next item in the collection and a continuation function pointer. When the continuation function pointer becomes nil, there are no more items left in the collection. Here's an example of how to use a functional iterator from a test case in my data-structures repository:

var data []int = []int{
    1, 5, 7, 9, 12, 13, 17, 18, 19, 20,
}

j := 0
for k, v, next := tree.Iterate()(); next != nil; k, v, next = next() {
    if !k.Equals(types.Int(data[j])) {
        t.Error("Wrong key")
    }
    if v.(int) != j {
        t.Error("Wrong value")
    }
    j += 1
}

The functional approach allows the user to have a nice "linked-list" style interface while avoiding the problems with using channels to implement iteration. Using channels make iterators quite easy to write, consider this post-order traversal:

package main

import "fmt"

type Node struct {
    label int
    children []Node
}

func (self *Node) String() string {
    return fmt.Sprintf("<%v %v>", self.label, self.children)
}

func (self *Node) PostOrder() (<-chan int) {
    ch := make(chan int)
    go func(yield chan<- int) {
        var iter func(node *Node)
        iter = func(node *Node) {
            if node == nil {
                return
            }
            for _, child := range node.children {
                iter(&child)
            }
            yield<-node.label
        }
        iter(self)
        close(yield)
    }(ch)
    return ch
}

func main() {
    root := &Node{5, []Node{Node{3, []Node{Node{1, nil}, Node{4, nil}}}, Node{7, []Node{Node{6, nil}, Node{10, []Node{Node{8, nil}, Node{12, nil}}}}}}}
    fmt.Println(root)
    for i := range root.PostOrder() {
        fmt.Println(i)
    }
}

(try it on the playground http://play.golang.org/p/kUGDQvWhkb)

However, what happens if the consumer of the iterator needs to bail out early? Like so:

for i := range root.PostOrder() {
    fmt.Println(i)
    if i % 2 == 0 {
        break
    }
}

In this case the go-routine PostOrder is running in will leak. Since, it will be blocked on sending to the consumer who will never consume the item. The fix for this is to make the API more complicated by adding two way communication such that the consuming thread can indicate to the producing thread it no longer wants any more items. This can be a significant complication as it would be in the cute little post-order traversal presented above.

Functional Iteration to the Rescue (almost)

Functional iteration solves several of the problems with iterating using channels while maintaining an easy to use interface. Because all the code runs on the same go-routine no go-routines can leak. If a consumer stops using an iterator the garbage collector can clean it up. In the PostOrder traversal above the channel can leak because the producing go-routine maintains a reference. However, functional iterators are a bit tricky to write. Let's look at a few from the data-structures repository.

Hash Table Iteration.

func (self *Hash) Iterate() KVIterator {
    table := self.table
    i := -1
    var e *entry
    var kv_iterator KVIterator
    kv_iterator = func()(key Equatable, val interface{}, next KVIterator) {
        for e == nil {
            i++
            if i >= len(table) {
                return nil, nil, nil
            }
            e = table[i]
        }
        key = e.key
        val = e.value
        e = e.next
        return key, val, kv_iterator
    }
    return kv_iterator
}

How does the function work? The first thing to notice is the Iterate function returns a function kv_iterator which closes over some state: table, i and e. The table variable is the array backing the hash table. The i variable refers to which bucket is currently being examined. Finally, the e variable refers to which entry in the bucket is next to be returned to the consumer.

When the consumer calls the kv_iterator function for the first time e will be nil and the top loop will run until and entry is found or it has run through the entire table. When and entry is found, the key and value are stored and e is set to e.next. This assignment modifies the closed state of kv_iterator and will be preserved on the next call to kv_iterator. Finally, key, val and kv_iterator are returned to the consumer.

The main thing to understand about the kv_iterator function and all iterators of this style is they implement a tail-recursive approach to iteration. Thus, just like writing a tail-recursive function in Scheme or Lisp one must write the iterator such that no actions need to be taken after the data is returned to the user. This sometimes gets tricky, for instance in post-order iteration.

Post Order Iteration

package main

import "fmt"

type Node struct {
    label int
    children []Node
}

type IntIterator func()(label int, next IntIterator)

func (self *Node) String() string {
    return fmt.Sprintf("<%v %v>", self.label, self.children)
}

func (self *Node) PostOrder() IntIterator {
    type entry struct {
        tn *Node
        i int
    }

    pop := func (stack []entry) ([]entry, *Node, int) {
        if len(stack) <= 0 {
            return stack, nil, 0
        } else {
            e := stack[len(stack)-1]
            return stack[0:len(stack)-1], e.tn, e.i
        }
    }

    stack := append(make([]entry, 0, 10), entry{self, 0})

    var iterator IntIterator
    iterator = func()(label int, next IntIterator) {
        var i int
        var tn *Node

        if len(stack) <= 0 {
            return 0, nil
        }

        stack, tn, i = pop(stack)
        if tn == nil {
            return 0, nil
        }
        for i < len(tn.children) {
            kid := &tn.children[i]
            stack = append(stack, entry{tn, i+1})
            tn = kid
            i = 0
        }
        return tn.label, iterator
    }
    return iterator

}

func main() {
    root := &Node{5, []Node{Node{3, []Node{Node{1, nil}, Node{4, nil}}}, Node{7, []Node{Node{6, nil}, Node{10, []Node{Node{8, nil}, Node{12, nil}}}}}}}
    fmt.Println(root)
    for i, next := root.PostOrder()(); next != nil; i, next = next() {
        fmt.Println(i)
    }
}

(try it out yourself on the playground http://play.golang.org/p/M4ivyGL7GM)

This time, the iterator must keep a stack of items in order to function. The top entry contains the next item to process and location in its child list next to process. The function pops a node, tn, and child index, i, off of the stack. If the node, tn, has unprocessed children it gets the next child, and pushes tn back onto the stack (and increments i). The child then becomes tn. It continues traversing down the left most side of the tree until it gets to a leaf node. At that point, the function exits the loop and returns the final value for tn and the tn_iterator.

The next time the function is called, it will pop the stack and find the parent of the last node it returned to the consumer. If there are more children of parent left to processed it will repeat the process described above. Otherwise, the parent gets returned to the consumer and the stack shrinks.

Conclusion

Functional iterators represent a flexible way to implement generic iterators in Go. They are easy to use and do not leak resources. There only draw back in comparison to other approaches is greater difficulty in constructing the iterators. In order solve this libraries of data-structures and iterator generators should be constructed.

How to Compile MySQL's "ON DUPLICATE KEY UPDATE" in SQL Alchemy

2013-12-05T00:00:00-05:00

This snippet contains a well tested way to compile an "Upsert" statement for MySQL in SQL Alchemy. I have found it useful for several project and I am making it available via this gist for others to enjoy and improve. Unlike some other examples out there this one supports doing a bulk insert.

Innovation isn't always the answer.

2013-11-25T00:00:00-05:00

From, http://www.wired.com/wiredscience/2013/11/vaclav-smil-wired/, at the bottom.

So the answers are not technological but political: better economic policies, better education, better trade policies.

Right. Today, as you know, everything is "innovation." We have problems, and people are looking for fairy-tale solutions—innovation like manna from heaven falling on the Israelites and saving them from the desert. It’s like, "Let's not reform the education system, the tax system. Let's not improve our dysfunctional government. Just wait for this innovation manna from a little group of people in Silicon Valley, preferably of Indian origin."

You people at WIRED - you're the guilty ones! You support these people, you write about them, you elevate them onto the cover! You really messed it up. I tell you, you pushed this on the American public, right? And people believe it now.

Dr. Smil is right about education. We cannot "innovate" our way out of poor educational policy - which is intimately tied up with many of our other policies. The Massive Online Open Course (MOOC) is the new textbook not the new classroom. The motivated student could always visit the library. These courses make material more accessible but the classroom is where the student is not where the material is. Not every person who needs help will be able to seek it. To help them you first have to find them, they will not find you.

The Ruby For Loop

2013-11-21T00:00:00-05:00

I am taking some time to learn Ruby this fall from Eloquent Ruby. The second chapter on choosing the right control structures has this paragraph (emphasis mine):

Since the two versions of the “print my fonts” code are essentially equivalent,[1] why prefer one over the other? Mainly it is a question of eliminating one level of indirection. Ruby actually defines the for loop in terms of the each method: When you say for font in fonts, Ruby will actually conjure up a call to fonts.each. Given that the for statement is really a call to each in disguise, why not just pull the mask off and write what you mean?

[1] Almost. The code block in the each version actually introduces a new scope. Any variables introduced into a code block are local to that block and go away at the end of the block. The more traditional for version of the loop does not introduce a new scope, so that variables introduced inside a for are also visible outside the loop.

Why not just pull the mask off?

We should not avoid constructs that potentially clarify the meaning of our code because under the hood they become something else. By that argument one would always write x86 machine code, memnonic assemblers are not for real programmers! Real programmers assemble the code in their heads!

I think that a for-each loop of the style of Ruby or Python or ____ are a very clear idiom. It reads nicely, every programmer who has a background in a C family language will know instantly what you mean. I don't see any benefit in avoiding the for loop in Ruby any more than I can see benefit in avoiding in C when I could be using a goto and an if statement. Which reads better to you? (Say the words out loud!)

fonts.each do |font|
# fonts each do font :-(

for font in fonts
# for font in fonts :-)

The for loop gets my vote.

An in Memory Go Implementation of Linear Hashing

2013-11-15T00:00:00-05:00

In the last post I described how linear hashing works. If you haven't read it yet, you should go read it first. I can wait, this post presents a new implementation which is a lot simpler than the disk based version referenced in the last post.

In the demonstration version (that would be this one) I represent the buckets as binary search trees. Find the exact code for this article in my goplay repository. An updated and slightly improved version is available in my data-structures repository along with several other interesting algorithms.

Profiling Results

              iterations, ns per operation
BenchmarkGoMap    100000,      26292 ns/op -- native map
BenchmarkHash      20000,      79526 ns/op -- classical separate chaining
BenchmarkMLHash    20000,      80820 ns/op -- in memory LH
BenchmarkLHash       500,    5733882 ns/op -- disk based LH, LRU cached in memory

Structs

type bst struct {
    hash int
    key Hashable
    value interface{}
    left *bst
    right *bst
}

type linearhash struct {
    table []*bst
    n uint
    r uint
    i uint
}

Bucket

func (self *linearhash) bucket(key Hashable) uint {
    m := uint(key.Hash() & ((1<<self.i)-1))
    if m < self.n {
        return m
    } else {
        return m ^ (1<<(self.i-1))
    }
}

Has, Put, Get and Remove

func (self *linearhash) Put(key Hashable, value interface{}) (err error) {
    var updated bool
    bkt_idx := self.bucket(key)
    self.table[bkt_idx], updated = self.table[bkt_idx].Put(key, value)
    if !updated {
        self.r += 1
    }
    if float64(self.r) > UTILIZATION * float64(self.n) * float64(RECORDS_PER_BLOCK) {
        return self.split()
    }
    return nil
}

func (self *linearhash) Get(key Hashable) (value interface{}, err error) {
    bkt_idx := self.bucket(key)
    return self.table[bkt_idx].Get(key)
}

func (self *linearhash) Has(key Hashable) (bool) {
    bkt_idx := self.bucket(key)
    return self.table[bkt_idx].Has(key)
}

func (self *linearhash) Remove(key Hashable) (value interface{}, err error) {
    bkt_idx := self.bucket(key)
    self.table[bkt_idx], value, err = self.table[bkt_idx].Remove(key)
    if err == nil {
        self.r -= 1
    }
    return
}

Split

func (self *linearhash) split() (err error) {
    bkt_idx := self.n % (1 << (self.i - 1))
    old_bkt := self.table[bkt_idx]
    var bkt_a, bkt_b *bst
    self.n += 1
    if self.n > (1 << self.i) {
        self.i += 1
    }
    for k, v, next := old_bkt.Iterate()(); next != nil; k, v, next = next() {
        if self.bucket(k) == bkt_idx {
            bkt_a, _ = bkt_a.Put(k, v)
        } else {
            bkt_b, _ = bkt_b.Put(k, v)
        }
    }
    self.table[bkt_idx] = bkt_a
    self.table = append(self.table, bkt_b)
    return nil
}

Buckets

One cool thing about my BST implementation is it has a functional iterator. As you may know you can't make a generic generator function in Go without using a separate goroutine, which isn't really appropriate here. The Iterate function makes a BSTIterator which is a function which yields the current key, value pair and a function which provides the next item. You iterate by calling next over and over again until the function pointer is nil. I hadn't written an iterator like this before but I like the way it works so I will probably re-use this pattern for future Go iterators.

type BSTIterator func()(key Hashable, value interface{}, next BSTIterator)
func (self *bst) Iterate() BSTIterator {
    pop := func(stack []*bst) ([]*bst, *bst) {
        if len(stack) <= 0 {
            return stack, nil
        } else {
            return stack[0:len(stack)-1], stack[len(stack)-1]
        }
    }
    procnode := func(stack []*bst, node *bst) []*bst {
        if node == nil {
            return stack
        }
        if node.right != nil {
            stack = append(stack, node.right)
        }
        if node.left != nil {
            stack = append(stack, node.left)
        }
        return stack
    }
    var make_iterator func(stack []*bst) BSTIterator
    make_iterator = func(stack []*bst) BSTIterator {
        return func()(Hashable, interface{}, BSTIterator){

            var node *bst
            stack, node = pop(stack)
            if node == nil {
                return nil, nil, nil
            }
            stack = procnode(stack, node)
            return node.key, node.value, make_iterator(stack)
        }
    }
    return make_iterator([]*bst{self})
}

Standard Has, Get, Put, Remove, Size

func (self *bst) Has(key Hashable) (has bool) {
    if self == nil {
        return false
    }
    if self.key.Equals(key) {
        return true
    } else if key.Less(self.key) {
        return self.left.Has(key)
    } else {
        return self.right.Has(key)
    }
}

func (self *bst) Get(key Hashable) (value interface{}, err error) {
    if self == nil {
        return nil, Errors["not-found"]
    }
    if self.key.Equals(key) {
        return self.value, nil
    } else if key.Less(self.key) {
        return self.left.Get(key)
    } else {
        return self.right.Get(key)
    }
}

func (self *bst) Put(key Hashable, value interface{}) (_ *bst, updated bool) {
    if self == nil {
        return &bst{hash: key.Hash(), key: key, value: value}, false
    }
    if self.key.Equals(key) {
        self.value = value
        return self, true
    }

    if key.Less(self.key) {
        self.left, updated = self.left.Put(key, value)
    } else {
        self.right, updated = self.right.Put(key, value)
    }
    return self, updated
}

func (self *bst) Remove(key Hashable) (_ *bst, value interface{}, err error) {
    if self == nil {
        return nil, nil, Errors["not-found"]
    }

    if self.key.Equals(key) {
        if self.left != nil && self.right != nil {
            if self.left.Size() < self.right.Size() {
                lmd := self.right.lmd()
                lmd.left = self.left
                return self.right, self.value, nil
            } else {
                rmd := self.left.rmd()
                rmd.right = self.right
                return self.left, self.value, nil
            }
        } else if self.left == nil {
            return self.right, self.value, nil
        } else if self.right == nil {
            return self.left, self.value, nil
        } else {
            return nil, self.value, nil
        }
    }
    if key.Less(self.key) {
        self.left, value, err = self.left.Remove(key)
    } else {
        self.right, value, err = self.right.Remove(key)
    }
    return self, value, err
}

func (self *bst) Size() int {
    if self == nil {
        return 0
    }
    return 1 + self.left.Size() + self.right.Size()
}

The _md, lmr, rmd

These implement "left most descendent" and "right most descendent". Used by remove to hook up the nodes correctly.

func (self *bst) _md(side func(*bst)*bst) (*bst) {
    if self == nil {
        return nil
    } else if side(self) != nil {
        return side(self)._md(side)
    } else {
        return self
    }
}

func (self *bst) lmd() (*bst) {
    return self._md(func(node *bst)*bst { return node.left })
}

func (self *bst) rmd() (*bst) {
    return self._md(func(node *bst)*bst { return node.right })
}

Linear Hashing

2013-11-13T00:00:00-05:00

Linear Hashing²^,³ is a hash table algorithm suitable for secondary storage. It is often used to implement hash indices in databases and file systems. Linear Hashing was invented by Witold Litwin in 1980 and has been in widespread use since that time. I implemented this file-structure earlier this year. You can find my implementation on github. This article is based off a talk¹ I gave at CWRU Hacker Society.

I also implemented an in memory version of this algorithm. I give a detailed break down of each element of the implementation in the follow up post. You should read that article after you have finished this one.

Hashing, A Refresher

In order to understand Linear Hashing one should take a moment to review Classical Hashing.⁴ Briefly, a hash table is a "symbol table" which maps keys to values. By "maps" I mean if you give it a key it will give the associated value if one exists. Keys must be hashable, which means there must be some way to turn them into integers. For a discussion on constructing such hash functions I recommend the Hashing Tutorial.⁴

Abstract Data Type

If the above description was confusing perhaps this ADT will clarify what I mean:

HashTable
  size():int -- how many entries?
  has(key:Hashable):boolean -- is the key in the table?
  get(key:Hashable):Object throws KeyNotFound -- get the associated value.
  put(key:Hashable, value:Object) -- associate a key and value.


  remove(key:Hashable) throws KeyNotFound -- remove the key from the table.

Hashable
  hash():int

You may have used hash tables by another name in you programming language. For instance in Python they are called dictionaries, in Ruby hashs, and in Java they are called HashMaps.

Implementation

Let's walk through a simple hash table implementation using separate chaining (also called open hashing). We will do this in the Go programming language.

Structs

Here is how we are going to represent a hash table:

type Hashable interface {
    Equals(b Hashable) bool
    Hash() int
}

type entry struct {
    key Hashable


    value interface{}
    next *entry
}

type hash struct {
    table []*entry


    size int
}

A hash is a struct with to elements. An array of pointers to entry. The entries hold our key value pairs. The way the table works is we convert the key into a number which we then clamp to the size of our table. That number will be the index of some entry in our table from which we can add, lookup, or remove the key.

The entry struct represents the key value pair and represents a linked list. Since covering linked list operations is a bit beyond the scope of this paper, let me just present operations on the *entry but with no explanation. The function should be obvious even if the implementation is obscure.

func (self *entry) Put(key Hashable, value interface{}) (e *entry, appended bool) {
    if self == nil {
        return &entry{key, value, nil}, true
    }
    if self.key.Equals(key) {
        self.value = value
        return self, false
    } else {
        self.next, appended = self.next.Put(key, value)
        return self, appended
    }
}

func (self *entry) Get(key Hashable) (has bool, value interface{}) {
    if self == nil {
        return false, nil
    } else if self.key.Equals(key) {
        return true, self.value
    } else {
        return self.next.Get(key)
    }
}



func (self *entry) Remove(key Hashable) *entry {
    if self == nil {
        panic(Errors["list-not-found"])
    }
    if self.key.Equals(key) {
        return self.next
    } else {
        self.next = self.next.Remove(key)
        return self
    }
}

Operations on the Hash Table

Now for how to implement the different operations. As a reminder we are going to convert our key to an index into the table as our first step. Let's make a function for that and call it bucket

func (self *hash) bucket(key Hashable) int {


    return key.Hash() % len(self.table)
}

Insertion

Putting an object into a hash table is very simple. We grab the bucket and use the associate Put method to place our key value pair into the list. If it was actually appended onto the list (rather than updating and existing entry) we increment the size field.

func (self *hash) Put(key Hashable, value interface{}) (err error) {
    bucket := self.bucket(key)
    var appended bool


    self.table[bucket], appended = self.table[bucket].Put(key, value)
    if appended {
        self.size += 1
    }
}

Now there is one more wrinkle I will return to in a moment which is resizing the table when it gets too full.

Retrieval

Retrieval is just as easy. We grab the bucket and look in the linked list to see if it is there or not. If it is, return it.

func (self *hash) Get(key Hashable) (value interface{}, err error) {
    bucket := self.bucket(key)
    if has, value := self.table[bucket].Get(key); has {
        return value, nil
    } else {
        return nil, Errors["not-found"]
    }
}

Removal

Removal is almost the same as insertion except we call Remove on the linked list instead of Put and update the head as before. We check to make sure it is in the linked list first as this slightly simplifies the removal algorithm above.

func (self *hash) Remove(key Hashable) (value interface{}, err error) {
    bucket := self.bucket(key)
    has, value := self.table[bucket].Get(key)

    if !has {
        return nil, Errors["not-found"]
    }
    self.table[bucket] = self.table[bucket].Remove(key)
    self.size -= 1
    return value, nil
}

Expansion

The performance of a hash table degrades as it gets too full. Therefore, we have to periodically expand the size of the hash table. As long we double the size each time all of our operations are asymptotically linear (on average). To double the size of the table, we allocate a new table and copy all of the entries from the old table to the new. We must be careful when we do this and rehash each element. If we don't, we will be unable to find the elements in the new table since the bucket function depends on the table size.

func (self *hash) expand() error {
    table := self.table

    self.table = make([]*entry, len(table)*2)
    self.size = 0
    for _, E := range table {
        for e := E; e != nil; e = e.next {
            if err := self.Put(e.key, e.value); err != nil {
                return err
            }
        }
    }
    return nil
}

Insert Revisited

So when should we expand the hash table? We should expand on insert when the number of elements is over a certain threshold. The threshold is often set at 60% but this setting varies.

func (self *hash) Put(key Hashable, value interface{}) (err error) {
    bucket := self.bucket(key)
    var appended bool
    self.table[bucket], appended = self.table[bucket].Put(key, value)
    if appended {
        self.size += 1
    }
    if self.size * 2 > len(self.table) {
        return self.expand()
    }
    return nil
}

Adapting Hash Tables for Secondary Storage

Figure 1. CPU and Storage

Unfortunately, the algorithm presented above does work well when using secondary storage mediums like hard disks and solid state drives. There are several reasons for this:

Secondary Storage is slower than RAM
The bus is slower
Many peripherals hang off of the South Bridge
Disks may be daisy chained causing bus contention

To deal with these factors and others when using disks:

Read and write pages which are blocks of size 4096 bytes.
Try and read contiguous runs and if writing more than one page write contiguous runs as well.
Batch writes.
Don't read one byte at a time, read several blocks and get the byte that you need.
Employ caching at every layer.
Measure performance in terms of number of disk accesses (eg. Block read and writes).

Back to Hashing

The first adjustment to make is to hash into blocks instead of hashing into individual array buckets. Each block is then a sorted array of entries.

Figure 2. Block File

We could make a fairly straight forward adaption of our separate chained hash table above to this restriction. However, there is a problem: what do we do when the table needs to be expanded? If the table is static then there is not problem, we simply allocate the correct number of blocks right away. But, if we have to expand the table every entry will need to be rehashed. This will cause us to read from every block from our old table (N reads) and write to every block in our new table (2*N writes) -- ouch.

The solution is of course Linear Hashing.

Linear Hashing

How does Linear Hashing compare?

Small mean disk accesses
1. Successful Search
  1. .75 utilization ~ 1.05 disk accesses
  2. .9 utilization ~ 1.35 disk accesses
2. Unsuccessful Search
  1. .75 utilization ~ 1.27 disk accesses
  2. .9 utilization ~ 2.37 disk accesses
3. Insert
  1. .75 utilization ~ 2.62 disk accesses
  2. .9 utilization ~ 3.73 disk accesses
In comparison a B+Tree of reasonable size might need at least 4 disk access for a search. (Of course a B+Tree will can perform range queries but that isn't the point here)
File grows at a linear rate.
Little dynamic re-arrangement
Does not necessarily need address translation.
Simple Algorithm esp. in comparison to B+Trees.

Explanation of the Algorithm

The key insight is to not use all the bits of the hash function "H(.)" all the time. When the table is small we only use as much of the hash function as we need. As the table grows we use more bits. As the table shrinks we use less.

Figure 3. Example

In the figure, n is the number of blocks, i is the number of bits of the hash functions and r is the number of records.

So to find which bucket a key goes to:

bkt_idx = let
  hash = x x x x a_1 a_2 ... a_i (* base 2 expansion of the hash of the key *)
  m = a_1 a_2 ... a_i (* just the first i bits *)
in
  if m < n then
    m
  else
    m - 2^(i-1) (* == 0 a_2 a_3 ... a_i *)
end

In go

func bucket(hash uint) uint {
    m := hash & ((1<<i)-1) // last i bits of hash as
                           // bucket number m
    if m < n {
        return m
    } else {
        return m ^ (1<<(i-1)) // unset the top bit
    }
}

Insertion

Insertion is quite simple now that we know how to get the bucket (assuming we have implemented the appropriate operations on our buckets). First we get the bucket and we put the item into the bucket. If the bucket takes care of chaining on an extra block if it full then the only thing that is left is checking whether or not an expansion (called a split) is needed.

func (self LinearHash) Insert(key Hashable, value []byte) error {
    hash := key.Hash()
    bkt_idx := self.bucket(hash)
    bkt := self.get_bucket(bkt_idx)
    if err := bkt.Put(key,value); err != nil {
        return err
    }
    self.r += 1
    if r > UTILIZATION * self.n * (self.records_per_block) {
        return self.split()
    }
    return nil
}

As I mentioned above, if a bucket is full it should chain out an extra block for itself. This can be handled transparently.

Figure 4. Chaining Example

Splitting

The split mechanism is clever bit of the linear hash algorithm. When the table is too full another block is added to the table:

Figure 5. Split Example Part 1

Note that the bucket we added in the example was

1 a_i == 1 0 == a_1 a_2

There are some keys in the old bucket 0 which is now called 00 which actually belong to bucket 10. So in order to make the addition of the new bucket correct we need to split bucket 00.

Figure 5. Split Example Part 2

In general if we add

1 a_2 a_3 ... a_i

We split

0 a_2 a_3 ... a_i

In code⁵

func (self LinearHash) split() error {
    bkt_idx := self.n % (1 << (self.i - 1))
    bkt_a := self.get_bucket(bkt_idx)
    bkt_b, err := self.allocate()
    if err != nil {
        return err
    }
    self.n += 1
    if n > (1 << i) {
        self.i += 1
    }
    return blk_a.split_into(bkt_b)
    // The split into function is left as
    // an exercise for the reader!
}

Conclusion

Hopefully if you are still with me you have a grasp on what is going on in the Linear Hashing algorithm. The key take aways are:

You can slowly extend how much of the hash function you use.
You don't have to rehash the whole file to add a bucket, just the bucket that collides with the new bucket.

If you want to implement the algorithm I suggest reading the description in the Garcia-Molina book and taking a look at the original paper. You can also take a look at my implementation or my other implementation, an in memory version.

Henderson, T. A. D. (2013) Linear Virtual Hashing. CWRU Hacker Society. Lecture Notes. November, 2013. ↩
Litwin, W. (1980). Linear hashing: a new tool for file and table addressing. In Proceedings of the sixth international conference on Very Large Data Bases - Volume 6 (pp. 212–223). VLDB Endowment. Retrieved from http://dl.acm.org/citation.cfm?id=1286887.1286911 ↩
Garcia-Molina, H., Ullman J. D., and Widom J. (2002) Database Systems: The Complete Book. Prentice Hall. Upper Saddle River, New Jersey. ISBN 0-13-031995-3. Section 13.4.7 "Linear Hash Tables" ↩
Shaffer, C. A. (2007) Hashing Tutorial Virginia Tech Algorithm Research Group. ↩↩
Ok, I am not that mean here is how to do the split_into in this file: https://github.com/timtadh/file-structures/blob/master/linhash/bucket/bucket.go ↩
Henderson, T. A. D. (2013) Linear Hash Implementation ↩

Cryptography and Complexity

2013-11-11T00:00:00-05:00

This is a conversion from a latex paper I wrote. If you want all formatting correct or the bibliography you should read the pdf version.

Cite as:

Henderson, Tim A. D. Cryptography and Complexity. Unpublished. Case Western Reserve University. MATH 408. Spring 2012.

Modern cryptographic systems are built on problems which are assumed to be computationally infeasible. Computational infeasibility means a computation which although computable would take far too many resources to actually compute. Ideally in cryptography one would like to ensure an infeasible computation’s cost is greater than the reward obtained by computing it. At first glance this seems to be an odd notion to base a cryptographic system on. Don’t we want our cryptographic systems to be totally secure? They should be unbreakable! “It may take a long time to break it,” seems like a poor guarantee of security.

However, it is the best guarantee which can exist in either an ideal world (from a mathematical perspective) or the physical world. As we shall see later in the survey, if several widely held assumptions turn out to be false we can not even make the guarantee of computational infeasibility.

Classical Security

In classical cryptographic systems, those known to the academic community prior to the publication of Diffie and Hellman’s paper , security assumptions were based on the results of information theory. This approach is sometimes referred to as information-theoretic and is concerned with whether there exists information in the ciphertext which originated in the plaintext or in the key. We say a system has perfect-secrecy if:

\[\begin{aligned} \forall_{m \in \mathcal{M}} \forall_{c \in \mathcal{C}} \text{ : } Pr[\mathcal{M} = m | \mathcal{C} = c] = Pr[\mathcal{M} = m] \label{perfect-secrecy}\end{aligned}\]

Intuitively this formula says an attacker gains no information about the contents of a message from the ciphertext of the message. Does this mean the attacker knows nothing about the message? Of course not! However, he doesn’t learn anything new about the message by closely examining the ciphertext. Therefore, the ciphertext of the message is essentially useless to an attacker. ¹

However, any system with perfect-secrecy requires the length of the key to be at least as large as the sum of the lengths of all messages encrypted with it. Since the key has to be at least as long as the messages sent such a system is of little value in practical modern situations.²

There are two practical problems with a system with perfect-secrecy the first is “Key Distribution.” Since the sender and receiver must use the same key they must some how secretly agree on a key beforehand. Therefore, there must exist some “second channel” by which the sender and receiver can communicate. The second problem has to do with the length of the key. Since it is as long as the message there seems to be only small utility in the system as the sender and receiver could conceivably securely exchange messages using their secure “second channel” they use for key distribution.

These to problems make the system an unrealistic system for securing (for instance) internet communications. Internet communications do not require the parties to know each other before hand and allow for no secondary secure communications channel to exist. Therefore, some other encryption methodology must be used if one wants to secure communication in this setting.

Modern Security

In modern system we no longer discuss security in terms of whether a system provides perfect-secrecy. Instead, we say if a ciphertext contains information leaked from the plaintext it should be computationally infeasible to extract that information. We can provide this property even in cases where the key is shorter than the message. This property can also be provided in cases where the attacker has access to the encryption key (but not of course to the decryption key).³

In following section [secure-encryption] we shall unpack and rigorously define (using ’s definitions) the definition above. In particular we will look at the definition in the context of symmetric key systems. The difficulties of public key systems will also be briefly presented but without detailed exposition. However, before we get to the fun stuff we will first present complexity theory and one way functions.

A Tour of Computational Complexity Theory

In many ways Computability Theory, and its daughter field Complexity Theory, began with proof of the incompleteness of axiomatic systems in 1931. The proof is a tremendously important result in meta-mathematics stating: no recursively axiomatized mathematical system can be both complete and consistent. Thus, we cannot prove in a particular theory that the same particular theory is consistent. Indeed, if we did construct such a proof it would prove exactly the opposite. Thus, there are some mathematical sentences which are true for which no algorithm can decided on their truth value.

In a similar result, Alan Turing in 1937 proved that for general programs one could not decide whether the programs would halt for all inputs. During the previous year Alonzo Church proved the exact same thing for evaluations of $\lambda$-Calculus expressions. Church and Turing later conjectured that the machine Turing defined (eg. Turing Machines) and Church’s lambda calculus were equivalent. All though this is an unprovable conjecture it is largely accepted today.

Once actual computational machines were produced (as opposed to the abstract machines of Turing and Church), programmers became interested in the notion of the complexity of an algorithm. The complexity of an algorithm is an expression of how much time or space or other resources the algorithm will use. The representation of time and space is abstract and placed in terms of the size of the parameters to the algorithm. Today, we use asymptotic notation to express complexity assertions. The notation was standardized by Don Knuth in 1976 but in wide (although inconsistent) use before then. It was invented by Bachmann in 1894 for use in a different context.

The interest in the complexity of algorithms and work on linguistics (particularly formal language hierarchies) lead to work on classifying the “hardness” computational problems. For instance all of the language classes in the Chomsky hierarchy have hardness results. Type-0 Languages (all recursively enumerable languages) are recognizable but only non-deterministically. While, Type-3 languages (referred to as regular languages) can be recognized in linear time.⁴

This leads to defining complexity classes for problems (as opposed to algorithms). A complexity class typically refers to a bound on the amount time or space needed to solve the problem in the worst case. Thus, complexity classes describe how difficult a problem is to solve in general. The first general results in the theory were obtained in 1965 by Hartmanis and Stearns who defined the meaning computation complexity.

In particular Hartmanis and Stearns modeled their definition using the computational model of an N-Tape Turing Machine. Any computational model could have been used, and today others are used. In particular the authors prove facts about the computability of particular binary strings, $\alpha$ in the paper. They say $\alpha$ is in a complexity class $S_T$ if $T: \mathbb{N} \rightarrow \mathbb{N}$ is a monotone increasing function and there exists a Turing machine $\mathcal{M}$ such that $\mathcal{M}$ computes the $n$th term in $T(n)$ steps.

What does this definition mean intuitively? Think of $T$ as a time function, where time is a function of the number bits generated. A string can belongs to those complexity classes which can compute the string according to the complexity class’s specified time function. Thus, by specifying some general time functions such as ($1$, $n$, $n^2$ …, $2^n$) one can begin classifying bit-strings. The bit-strings correspond to problem solutions. For a simple example consider the bit-string which corresponds to all prime numbers. To compute it, one would need to actually compute exactly those numbers which are prime.

The type of problem Hartmanis and Stearns classified belongs to the class of problems known as Decision Problems. Informally, decision problems are problems to which there is a “yes/no” answer. For instance, deciding whether the first $n$ bits of a string is in a language, $\mathcal{L} \subseteq \{0,1\}^*$, is a decision problem. Complexity classes are more general than just decision problems however, one can construct complexity classes for any type of computational problem, optimizations problems for instance.

The Class NP

Of particular importance to mathematics, computer science and this paper in particular is the complexity class NP (Non-Deterministic Polynomial Time). The class of NP is defined (intuitively) as those problems which have easily verifiable solutions. What does it mean for a solution to be “easily verifiable.” It means given the problem instance and the solution one can check the validity of the solution in $O(n^k)$ where $n$ is parameterized by the problem and $k$ is a constant.

More formally, the class NP is defined in terms of formal languages. Let $\Sigma$ be an alphabet and $\Sigma_{0}$ be $\Sigma - \{*\}$ where $*$ is the empty symbol. Let $\Sigma_{0}^{*}$ be the closure of all finite strings made up of symbols in $\Sigma_{0}$. We define a language, $\mathcal{L}$, as $\mathcal{L} \subseteq \Sigma_{0}^{*}$.

The Class NP⁵

A language, $\mathcal{L}$, belongs to NP if there exists a Deterministic Turing Machine, $\mathcal{M}$, a polynomial, $p(n)$ – such that $p(n)$ defines the complexity class of $\mathcal{M}$⁶ – and on any input $x \in \Sigma_{0}^{*}$:

if $x \in \mathcal{L}$ then there exists a certificate, $y \in \Sigma_{0}^{*} \text{ st. } |y| \leq p(|x|)$, and $\mathcal{M}$ accepts the input string $xy$.

if $x \notin \mathcal{L}$ then for any string, $y \in \Sigma_{0}^{*}$, $\mathcal{M}$ rejects the input string $xy$.

The given definition does not discuss Non-Determinism. To see the role of Non-Determinism consider constructing solutions (certificates) to problem instances (the string x in the definition). If certificates can be chosen and examined non-deterministically then it will take only polynomial time to find a solution. However, if we are testing every possible certificate deterministically it will take $\vert \Sigma_0\vert ^{p(\vert x\vert )}$ examinations, a combinatorial explosion. Thus, problems in NP have solutions which are easy to verify but not necessarily easy to construct.

The Class P

The complexity class P (Polynomial Time) is exactly those problems solvable in deterministic polynomial time. More formally,

The Class P⁷

Let $\Pi$ be a decision problem. Let $L_{\Pi} = \{ x \in \Sigma_0^* | \text{x is an encoding of an instance of } \Pi \}$, that is, $L_{\Pi}$ is the language of $\Pi$. We can then define the class P as:

\[\begin{aligned} P = \{ L \subseteq \Sigma_0^* |& \text{ there is a Deterministic Turing} \text{ Machine, } \mathcal{M} \text{, and a polynomial, } \\ &p(n) \text{, } \text{such that } T_{\mathcal{M}} \leq p(n) \text{ for all } n \geq 1 \} \end{aligned}\]

A language is in P if one can construct a Turing Machine which accepts it (and rejects all non-members) in time less than some polynomial (with respect to the size of the input).

All those problems which belong to P are considered easily solvable, or tractable. While, they are “easy” one should not make the mistake of assuming they are simple. Given a polynomial time algorithm which solves a problem one can easily solve it. However, even if you know a polynomial time algorithm exists for a problem constructing the algorithm may be difficult.

P vs. NP

What is the relationship between P and NP? It is known P is contained in NP (ie. $P \subseteq NP$). However, whether $NP \subseteq P$ is true is one of the greatest open questions in applied mathematics. The class P containment inside of NP is obvious: if we can find a solution in polynomial time it is certainly verifiable in polynomial time. To show NP is contained within P one would need to show every problem in NP can be solved with a polynomial time algorithm.

The methodology for solving P vs NP with the greatest impact relies on the idea of reduction. We say problem, $\Pi_1$ is reducible to another problem, $\Pi_2$, if one can find a mapping from every instance of $\Pi_1$ to equivalent instances of $\Pi_2$ such that the solutions to the constructed instances of $\Pi_2$ correspond to solutions of $\Pi_1$. A reduction is a polynomial reduction if the mapping can be done in polynomial time. A problem, $\Pi_1$, is as “hard” as another problem, $\Pi_2$, if $\Pi_2$ can be reduced to $\Pi_2$. Thus, hardness is a relational notion. Problems are not intrinsically hard, they are hard with respect to other problems.

To prove P contains NP one could prove the hardest problem in NP is also in P. By the definition of hardness above, if a problem, $\Pi_h$, is the hardest problem in NP than every other problem in NP is reducible to $\Pi_h$. Problems which are at least as hard as every problem in NP are know as NP-Hard problems. A problem does not need to be in NP to be NP-Hard. However, if a problem is NP-Hard and it is in NP then it is called an NP-Complete problem.

NP-Complete problems exist and their existence is one of the greatest results in complexity theory. It was proved by Stephen Cook in 1971 who found the first NP-Complete problem. The problem he found is known as SAT (for satisfiability of boolean formulas). He proved any problem solvable in polynomial time by a Nondeterministic Turing Machine can be reduced to finding whether or not a boolean formula is satisfiable. Cook’s result launched a wave of research. The very next year Richard Karp proved 21 other problems were also NP-Complete.

While there have many hundreds of problems proven to be NP-Complete since Cook proved SAT, there have been only fruitless attempts to prove P does or does not contain NP. The leading consensus in the complexity community is P does not contain NP. Furthermore, since it appears work on proving P contains NP is permanently stalled one can safely assume NP-Hard problems are in some platonic sense actually difficult to solve.

Infeasibility

A sharper definition of computational infeasibility can now be given with definitions of Complexity Classes, P, and NP in hand. Recall the opening statement on infeasibility, where we defined an infeasible computation to be one requiring too many resources to actually compute. If one has encrypted a message one would ideally like the ciphertext to be unreadable. If the message is a solution to an NP-Complete problem then the ciphertext could be the problem instance and therefore can only be decrypted by solving the NP-Complete problem. However, assuming NP is not contained in P, the NP-Complete problem will take time proportional to $\vert \Sigma_0^*\vert ^{\vert x\vert }$ (where x is the ciphertext) to solve.

Therefore, a new working definition of an infeasible computation is a “hard” instance of an NP-Hard problem of sufficient size. What is sufficient size? Any size which leads to $\vert \Sigma_0^*\vert ^{\vert x\vert }$ to be so large as to be uncomputable. An example of such as size might be $160$ since trying $2^{160}$ possible solutions is not expected to ever be computable with classical computers in time less than the age of the universe. What is a “hard” instance? A hard instance is one in which there exists no better way to find a solution than trying all possible solutions. Not every instance of a hard problem is hard to solve. Specifying an infeasible computation requires a hard instance is a necessary restriction.

Probabilistic Infeasibility

In the previous section it was assumed all computations were exact. No computation sometimes gave the right answer and sometimes did not. However, with an algorithm which mostly gives right answers could be very useful to the cryptanalyst. Therefore, we briefly turn our attention to probabilistic computations.

Probabilistic Turing Machines

A Probabilistic Turing Machine (PTM) is a Deterministic Turing Machine (DTM) with an extra input tape. The tape is called the “coin flipping tape.” The PTM can read one bit of information at a time from the coin flipping tape. Each bit is assured to be a random bit.⁸ Computation on the machine proceeds as before except at any time a random choice can be made. This allows us to construct algorithms which will “probably” but not necessarily produce the desired answer.

Analyzing the running time of a PTM is a bit different than a DTM. While a DTM’s running time only depends on its program and the initial configuration of the input tape, a PTM also depends on the random bits it reads during the computation. Therefore, the running time of a PTM is a random variable (we denote it as $t_{\mathcal{M}}(x)$). Furthermore, whether a PTM halts or not on a fixed input is also a random variable. A halting PTM is one which halts after a finite number of steps for all inputs and all configurations of the coin tossing tape.

With the definition a halting PTM in hand we are now prepared to reason about its running time. Worst case running time of a PTM, $T_{\mathcal{M}}(n)$, is:

\[\begin{aligned} T_{\mathcal{M}}(n) = \text{max} \{ t \text{ } | \text{ there exists a } x \in \Sigma^{n}_{0} \text{ such that } \text{Pr}[t_{\mathcal{M}}(x) = t] > 0 \}\end{aligned}\]

Informally, this definition states: the worst case running time of a PTM is the maximum running time, $t_{\mathcal{M}}(x)$, for which the machine will run with some probability greater than zero. A polynomial PTM is one in which there exists some positive polynomial, $p(\cdot)$, such that $T_{\mathcal{M}}(n) \le p(n)$ holds.⁹

BPP, Bounded Probability Polynomial Time

A language $\mathcal{L}$ is recognized by a polynomial PTM, $\mathcal{M}$, if:

for every $x \in \mathcal{L}$ it holds that Pr$[\mathcal{M} \text{ accepts } x] \ge \frac{2}{3}$

for every $x \notin \mathcal{L}$ it holds that Pr$[\mathcal{M} \text{ does not accept } x] \ge \frac{2}{3}$

BPP is the class of languages recognized by a polynomial PTM.¹⁰¹¹

The class Bounded Probability Polynomial Time, sometimes called Bounded-Error Probabilistic Polynomial Time, is somewhat analogous to the class P. Computations in BPP are considered feasible computations. The class P is contained within BPP, $P \subseteq BPP$. However, the relationship between NP and BPP has not been established. In practice cryptographers assume $NP \nsubseteq BPP$ which implies $NP \neq P$. All problems unsolvable by a polynomial PTM are considered infeasible, of which NP-Hard problems are a special case. As before, some instances of hard problems may in fact be easy to solve.

Infeasible computations as defined above are nice formalisms but do not seem too useful. To utilize the previous definition one has to answer the following question: Given and instance of a problem is it a hard instance? Unfortunately, we don’t know how to answer this question.¹² As we will see later, if we could easily find hard instances we could construct a simple and secure crypto-system by sampling hard instances. Therefore, cryptographers need better assurances than worst-case assurances; a cryptographer needs to know a typical instance of a problem is hard.

One Way Functions

With a firm grounding in Complexity Theory, we turn our attention to cryptography. First, by capturing the notion of exploitable computational difficulty as epitomized in the one way function. A one way function is a function which is easy to compute but hard to invert. More specifically:

One Way Function [owf]

$\forall_x$ computing $f(x) = y$ is easy to compute.

$\forall_y$ computing $f^{-1}(y)$ such that $f(x) \in f^{-1}(y)$ is hard to compute.

The one way function in definition [owf] is more of a theoretical construct than an actual mathematical construct. Therefore, it uses the notion of easy and hard computations without grounding itself with exact definitions. One can think of this first definition as an abstract, or ideal, definition.

Ignoring for the moment the definitional problems, what use is a one way function to the cryptographer? It turns out one can define secure cryptosystems with one way functions. Such a cryptosystem will be discussed in detail in section [secure-encryption]. For now consider this simple example of the power of the idea:

One day while toiling away, Ian had a flash of insight which would put his mechanical workings to right. A machine danced in his mind, one which would make widgets faster and better than before. So clever his insight he knew no one else would easily come up with the same idea. Thus, he decided not to patent it. Instead, he wrote down his idea and ran it through a one way function producing $y$ his certificate of his idea. He then published $y$ widely, placing it in all the libraries around the country.

Many years passed and Mallory stole Ian’s idea. Mallory being very clever sought to undue Ian and patented the idea. Then, he sued Ian for patent infringement. But, since Ian had a certificate of his invention, $y$, he could prove to the court he had invented and known about the idea long before Mallory had filed for the patent. The court agreed with Ian and invalidated Mallory’s patent.

The challenge in section [secure-encryption] will be transforming the one way function into a workable encryption device. For while a powerful concept, as demonstrated by the story above, it is non-obvious how a crypto-system can be constructed from it. But before crypto-systems, the definition must be tightened. Furthermore, one must be convinced one way functions can be reasonably expected to exist.

Strong One Way Functions

There are two vague terms used in definition [owf], easy and hard computations. Fortunately, we have already defined what an easy computation is: an easy computation is on which can be done in (probabilistic) polynomial time. But what about inversion? What does it mean for a function to be hard to invert? A function, $f$, is hard to invert if every probabilistic polynomial time algorithms will only invert $f$ with negligible probability.

Negligible¹³ [neg]

A function, $\mu : \mathbb{N} \rightarrow \mathbb{R}$, is negligible if for every positive polynomial, $p(\cdot)$, there exists an $N$ such that for all $n > N$,

\[\begin{aligned} \mu(n) < \frac{1}{p(n)} \end{aligned}\]

The definition of negligible is reminiscent of Asymptotic Notation used in the analysis of algorithms. It concerns itself with the behavior of the function, $\mu(n)$, when $n$ grows large. An additional, and useful, feature of the definition is any negligible function remain negligible after multiplication with any polynomial $q(\cdot)$. Therefore, any event which occurs with negligible probability will continue to occur with negligible probability even after polynomial repetitions. Thus, if $f$ is only invertible with polynomial time algorithm, $A$, with negligible probability than no polynomial repetition of $A$ will be likely to invert $f$.

Strong One Way Functions¹⁴ [strong-owf]

A function, $f : \{0,1\}^* \rightarrow \{0,1\}^*$, is strongly one way if it is:

Easy to compute

There exists a (deterministic) polynomial time algorithm A such that on input x algorithm A outputs $f(x)$ (ie. $A(x) = f(x)$).

Hard to invert

For every probabilistic polynomial time algorithm $A'$, every positive polynomial $p(\cdot)$, and all sufficiently large $n$ the probability $A'$ inverts $f$ is negligible. That is:

\[\begin{aligned} \text{Pr}[A'(f(x)) \in f^{-1}(f(x))] < \frac{1}{p(n)} \end{aligned}\]

In the above definition “input x” should be considered as a random variable drawn from a uniform distribution over $\{0,1\}^n$. Thus, the second condition reads: for any random input of size $n$ the probability an arbitrary polynomial time algorithm will find a pre-image is negligible. If such a function could be found or constructed it would offer a strong assurance of computational difficulty.

Weak One Way Functions

While strong one way functions ensure any efficient inversion algorithm has only a negligible likelihood of succeeding; weak one way functions require efficient inversion algorithms will fail with a non-negligible probability.

Weak One Way Functions¹⁵ [weak-owf]

A function, $f : \{0,1\}^* \rightarrow \{0,1\}^*$, is weakly one way if it is:

Easy to compute

There exists a (deterministic) polynomial time algorithm A such that on input x algorithm A outputs $f(x)$ (ie. $A(x) = f(x)$).

Slightly hard to invert

There exists a polynomial $p(\cdot)$ such that for every probabilistic polynomial time algorithm, $A'$, and a sufficiently large $n$’s,

\[\begin{aligned} \text{Pr}[A'(f(x)) \notin f^{-1}(f(x))] > \frac{1}{p(n)} \end{aligned}\]

In definition [strong-owf] the probability that $A'$ could invert $f$ has an upper bound of $p(\cdot)^{-1}$ for every positive polynomial. In definition [weak-owf], there is a single positive polynomial, $p(\cdot)$, such that $p(\cdot)^{-1}$ is a lower bound on the failure of any efficient inversion algorithm. Unlike strong one way functions, weak one way functions are not hard for typical instances. However, they are hard for some percentage of instances.

Amplification of Weak One Way Functions

Since weak functions are hard for a non-negligible percentage of inputs they can be used to construct strong functions. The proof for this bold assertion is given by Goldreich.¹⁶ Since one can convert a weak one way function into a strong one it suffices to find weak ones. While a strongly one way function may yield a more efficient cryptosystem a weak one will still allow a secure system (as discussed in section [secure-encryption]).

Hard Core Predicates

If Alice has a strong one way function $f$, computes $y = f(x)$, and sends $y$ to Bob while Eve eavesdrops what can Eve learn about $x$? Depending on the function $f$ Eve may be able to learn a surprising amount. Since $f$ is hard to invert Eve cannot learn everything about $x$ but she may not need too. Is there some way to quantify which bits of $x$ Eve can learn about and which bits she can’t?

There is! The bits which are hard for a polynomial attacker (like Eve) to learn about are called the “Hard Core” of a one way function. A predicate is a yes/no question, for example: Does $x$ end with a 0? If a yes/no question is hard for Eve to answer it is called a Hard Core Predicate. Since a yes/no question only has 2 possible answers Eve can always guess the answer. Therefore, a predicate is only hard for her to answer if she can’t do better than get it right about half the time. To be precise:

\[\begin{aligned} \text{Pr}[\text{EveGuess\_P}(y) = P(x)] \le \frac{1}{2} + neg(|x|)\end{aligned}\]

where $neg(\vert x\vert )$ is a negligible function (as defined in definition [neg]). This description of Eve trying to guess something about $x$, like whether it starts with 0, leads nicely into a formal definition:

Hard-Core Predicates¹⁷ [hardcore]

A polynomial time computable predicate, $b : \{0,1\}^* \rightarrow \{0,1\}$, is called a hard-core of a function, $f$, if for every probabilistic polynomial time algorithm $A'$, every positive polynomial $p(\cdot)$, and all sufficiently large $|x|$’s,

\[\begin{aligned} \text{Pr}[A'(f(x)) = b(x)] < \frac{1}{2} + \frac{1}{p(|x|)} \end{aligned}\]

Given a hard to invert function, $f$, one knows some of the bits in its input must be hard to predict from the output. How does one know which bits are the hard bits? In general deciding what bits are hard for a function is difficult but one can always construct a Hard-Core Predicate for any strong one way function. Since one can always construct a strong one way function from a weak function this poses no limitation to the framework.

Constructing Hard-Core Predicates

The following result was first proved in 1982 by Yao but we present a simplification due to Goldreich and Levin as presented by Talbot and Welsh.¹⁸ A detailed proof is available as usual in the Goldreich book.¹⁹

Hard-Core Predicates from Strong One Way Functions [con-hard]

Let $f$ be an arbitrary strong one way function. Let $g$ be defined as $g(x, r) = (f(x), r)$, where $|x| = |r|$. Let $r$ be a random bit string. Then define $B(x, r)$ to be a Hard-Core Predicate of $g$ by:

\[\begin{aligned} B(x,r) &= \sum\limits^{|x|}_{i=1} x_i r_i \imod{2} \\ &= \bar{x} \cdot \bar{r} \imod{2} \end{aligned}\]

The theorem states, if $f$ is strongly one way then it will be hard to guess the result of taking an exclusive-or of a random subset of $x$ given $f(x)$ and the subset $r$. If $B(x,r)$ is not a hard-core of $g$ then $f$ is easily invertible. The proof involves constructing an algorithm from the predictor for $B$. For details on the construction once again see Goldreich.

With the result of theorem [con-hard] and the ability to construct strong one way functions from weak one way functions one will always be able to construct a function where at least one predicate on $x$ is hard to compute. If one bit is not enough it turns out hard-core functions are also constructable. However, their specific details are well out of the scope of this paper.

Constructing One Way Functions

It one is going to build a crypto-system based on hard computational problems (specifically strong one way functions) one should have some way of identifying such problems. From a practical perspective there are three number theoretic based problems which are assumed to be one way functions. The first is the discrete log problem: $g^x \equiv y \imod{p}$, second finding square roots mod $N = pq$, and third the “RSA” problem $c \equiv x^e \imod{N}$. While these problems are likely to be used in practice none of them are suspected to be in the class NP-Hard. While, instances of problems in NP-Hard may be efficiently solvable there is good evidence they are not. In contrast these problems are potentially vulnerable to good approximation algorithms.

Thus, an open problem for the aspiring cryptographer to tackle is to suggest a novel one way function. However, serious care needs to be exercised when suggesting such a function. It is not good enough for the function to be difficult in the worst-case it must be difficult in the typical case. Average case complexity analysis relies heavily on the input distribution. Thus, the input distribution must be carefully characterized and uniform sampling techniques must be developed. Without exercising such care the aspiring cryptographer may fall into the trap of defining something which appears secure from a cursory theoretical glance but on close inspection is quite vulnerable.

Secure Encryption

[secure-encryption]

Secure encryption schemes are naturally built on top of strong one way functions with hard-core predicates. However, before the encryption schemes can be defined a formal definition of security must be stated. Until now, our definition has been colloquial: information in the ciphertext should be computationally infeasible to extract. The informal definition is too vague for use in defining an encryption system because the security definition is more important than the cryptographic system itself. A proper definition ensures systems conforming to the definition will be more difficult to attack.

Security Definitions

[sec-def]

Before rigorously defining a modern definition of security let us turn once again to classical security and perfect-secrecy. Recall perfect secrecy says an attackers uncertainty about a message should not be reduced when in possession of a corresponding ciphertext. As noted in the introduction, the obvious criticism of perfect-secrecy is the implied key length. In such a system, the length of the key must be at least as long as the message. Making the definition impractical for most modern uses of cryptography. Therefore, a new definition is indeed necessary.

Polynomial Indistinguishability

The first definition we will consider is polynomial-indistinguishability. Informally, if Alice has two messages, $M_1$ and $M_2$ and she sends Bob a ciphertext, $C$, Eve who has been given both messages and the ciphertext will have no easy way to determine which message it corresponds to. Something is easy for Eve if she can do it in probabilistic polynomial time. Indeed, it is assumed none of our characters can do any computations except easy ones. Formally,

Polynomial Indistinguishability of Encryptions²⁰

An encryption scheme, $(G, E, D)$, where $G$ generates keys, $E$ encrypts messages, and $D$ decrypts messages has indistinguishable encryptions if for every probabilistic polynomial time algorithm, $A'$, every polynomial $p(\cdot)$, all sufficiently large $n$, and every $x,y \in \{0,1\}^{\text{poly}(n)}$ with $|x| = |y|$,

\[\begin{aligned} | \text{Pr}[A'(E_{G(1^n)}(x)) = 1] - \text{Pr}[A'(E_{G(1^n)}(y)) = 1] | < \frac{1}{p(n)} \end{aligned}\]

The above definition was written with a symmetric encryption and decryption keys. However, the public key version only has minor and unimportant complications. The importance of the definition is in the intuition. Eve, the attacker, knows both messages and she has a ciphertext. The only thing she does not know is the key used to create the ciphertext. If the system is polynomially indistinguishable then Eve can only guess which message the ciphertext corresponds to. Since there are two messages she will only get it right half the time. If she can get it right better than half the time then the system is not polynomially indistinguishable.

The security of the definition is perhaps non-obvious but consider the case were Eve can distinguish which message the ciphertext corresponds too. If the system was supposed to have perfect-secrecy then clearly the secrecy would have been violated. Some bit of information would be leaking from the message to the ciphertext. Therefore, what the definition is saying is no information is leaking from the message to the ciphertext which can be extracted in polynomial time.

Semantic Security

The intuitive explanation of polynomial indistinguishability is captured in an alternative definition: semantic-security. A crypto-system is semantically secure if any piece of information Eve can compute given a ciphertext she could just as easily compute without the ciphertext. That is, the ciphertext provides Eve with no advantage for computing any piece of information of interest to her. Formally,

Semantic Security²¹

An encryption scheme, $(G, E, D)$, where $G$ generates keys, $E$ encrypts messages, and $D$ decrypts messages is semantically secure if for every probabilistic polynomial time algorithm, $A$, there exists another probabilistic polynomial time algorithm, $A'$, such that for every message $\mathcal{M}$ of length $n$, every pair of functions with polynomially bounded output $f,h : \{0,1\}^* \rightarrow \{0,1\}^*$, every polynomial $p(\cdot)$, and all sufficiently large $n$,

\[\begin{aligned} \text{Pr}[A(1^n, E_{G_1(1^n)}(\mathcal{M}), h(1^n, \mathcal{M})] = f(1^n,\mathcal{M})] \\ < \text{Pr}[A'(1^n, h(1^n, \mathcal{M})] = f(1^n,\mathcal{M})] + \frac{1}{p(n)} \end{aligned}\]

In the above definition, $f$ represents the information Eve would like to compute. The information Eve wants, $f$, is a function of the message and the length of the message (encoded for technical reasons in unary). The output of $f$ is polynomial however it is not necessary for $f$ to be a computable function. The algorithm $A$ guesses $f$ using the ciphertext, the length of the message, and $h$. The algorithm $A'$ guesses $f$ using only the length of the message and $h$. The function $h$ represents a polynomial amount of a-priori knowledge about the output of $f$.

The definition of semantic security intuitive says the probability Eve can guess $f$ utilizing the ciphertext is at most negligibly greater than guessing $f$ without the ciphertext. The definition places no restrictions on what Eve might be guessing (other than an upper bound on its size). Eve could be guessing whether the message is an order to move troops, or the message is a bank account number; it makes no difference to the definition.

Semantic-security is therefore the complexity theory analog of perfect-secrecy. It provides assurance to the cryptographer that a polynomially bound cryptanalyst will be able to gain no information from the ciphertext. In practice, one only cares about polynomially bound adversaries since exponential adversaries do not exist.

Equivalence of Definitions

In a potentially surprising result it turns out it doesn’t matter which security definition one uses, they imply each other:

Equivalence of Definitions²² [equiv-thm]

An encryption scheme is semantically secure if and only if it has indistinguishable encryptions.

In practice, it is usually far easier to prove a scheme has indistinguishable ciphertexts. However, from a security perspective the property one actually wants is semantic-security. Thus, theorem [equiv-thm] provides the cryptographer with an incredibly useful result.

A Secure Symmetric Key Encryption Scheme

To construct a perfectly secret symmetric key encryption scheme from an information theory perspective one first obtains a large amount of random information. One then takes a random bit for each bit of message and exclusive-ors them together. One now has the perfect cryptographic system. The construction of a semantically secure system is quite similar (in the case of stream ciphers). One takes a bit of random information, referred alternately as the seed of the key, stretches it to create a pseudo-random sequence the same length as the message. The message and the pseudo-random sequence are then xored together. This encryption scheme will clearly be semantically secure if no adversary can distinguish between the pseudo-random sequence and a truly random sequence.

Pseudo-Random Sequence Generators

A pseudo-random bit generator, $G(x)$ is defined as a deterministic polynomial time algorithm taking a bit-string, $x \in \{0,1\}^k$, and outputting a longer string $G(x)$. In other words, the generator stretches the input. For the generator to be pseudo-random in nature, the output must be unpredictable if the input is random. Luckily, we already know how to produce bits which are essentially unguessable by a polynomial adversary. Hard-core predicates by construction cannot be guessed correctly better than half the time.

A Pseudo-Random Generator can be Constructed from any One Way Permutation.²³²⁴ [onebitgen]

Let $f : \{0,1\}^* \rightarrow \{0,1\}^*$ be a one-way function length preserving permutation with a hard core predicate $B : \{0,1\}^* \rightarrow \{0,1\}$ then,

\[\begin{aligned} &G : \{0,1\}^k \rightarrow \{0,1\}^{k+1} \\ &G(x) = (f(x), B(x)) \end{aligned}\]

is a pseudo-random generator.

If $x$ is a random string, and therefore drawn from a uniform distribution over $\{0,1\}^k$, then $f(x)$ is also a random string. Therefore, if there is some test, $T$, which can distinguish $G(x)$ from a random string of length $k+1$ it must be distinguishing the last bit, $B(x)$. Since, it can distinguish $B(x)$ from a random bit then one must be able to guess it significantly better than half the time. However, this contradicts $B(x)$ being a hard-core predicate of $f(x)$. Therefore, $f$ is either not a one-way function or $G(x)$ is a pseudo-random generator.

While, theorem [onebitgen] certainly constructs a pseudo-random number generator it is hardly a useful one. Recall, the issue with perfect secrecy was the key size. If one constructed a stream cipher from using theorem [onebitgen] one would only save 1 bit of key size over a one time pad. Luckily, the following extension also holds:

An $l(k)$ Pseudo-Random Generator²⁵ [gen]

Let $f : \{0,1\}^* \rightarrow \{0,1\}^*$ be a one-way function length preserving permutation with a hard core predicate $B : \{0,1\}^* \rightarrow \{0,1\}$. If $l(\cdot)$ is a positive polynomial then,

\[\begin{aligned} &G : \{0,1\}^k \rightarrow \{0,1\}^{l(k)} \\ &G(x) = (B(x), B(f(x)), B(f^2(x)), ..., B(f^{l(k)-1}(x))) \end{aligned}\]

is a pseudo-random generator.

With the construction in theorem [gen] one can now generate a strong pseudo-random sequence. If $f$ is a strong one way function with a hard-core then no polynomial adversary can discern between the output of the generator above and a truly random string.

A Symmetric Key Stream Cipher²⁶

Setup

Alice chooses a short random key $x \in_{R} \{0,1\}^k$

Key Distribution

Alice secretly shares $x$ with Bob.

Encryption

Alice encrypts an $m$-bit message, $M$, by generating a pseudo-random string:

\[\begin{aligned} G(x) = (B(f(x)), B(f^2(x)), ..., B(f^m(x))) \end{aligned}\]

and forming the cryptogram $C = G(x) \otimes M$

Decryption

Bob creates the same string $G(x)$ an recovers the message via $M = G(x) \otimes C$.

The strength of the cipher relies on the strength of $G(x)$ and the strength of $G(x)$ relies on the underlying properties of the one way function, $f$. The above stream cipher is clearly semantically secure since the ciphertexts are indistinguishable by Eve. If Eve could distinguish the ciphertexts than she could predict $G(x)$. If Eve can predict $G(x)$ than $f$ must not be a one way function.

While the above stream cipher is semantically secure, it is not necessarily the construction one would use in practice. Often, one would instead want to use a block cipher. Luckily, one can also construct block based ciphers from pseudo-random generators. For these an many other complications I refer you to Oded Goldreich’s 2004 book.

Public Key Schemes

I will not discuss the public key schemes in detail. The definitions for security setup in section [sec-def] are implicitly for symmetric key systems. While, the modifications are fairly trivial they should be given proper treatment. In addition the public key systems deserve a thorough explanation. I will settle for some brief remarks.

The RSA cryptographic system does not satisfy the property of polynomial indistinguishability. In particular, if Eve wants to tell whether $C$ corresponds to $M_1$ or to $M_2$ all she has to do is encrypt both messages can compare their ciphertexts. Eve and easily do this since the encryption algorithm in a public key system is public and therefore available to Eve.

The encryption algorithm being publicly available seems to be an insurmountable obstacle at first, but it turns out to be possible to overcome it. In the case of the RSA algorithm one needs to introduce randomness (and thus uncertainty) into the encryption process. One such suggestion Randomized RSA introduces random data into each encryption thus ensuring polynomial indistinguishability. However, Randomized RSA comes with a cost: one must believe a different strong assumption. One must assume RSA has a “large” hard-core of bits in the input. While, this may be a reasonable assumption it is a different assumption and not implied by the usual RSA assumption. For details I once again commend you to Goldreich’s 2004 book.

Concluding Remarks

Basing cryptographic security on computation complexity is a sound practice. It yields systems with strong and extensible security guarantees. However, it also requires strong assumptions. In particular, we must believe in “one way functions.” While, there is good evidence they exist, and several candidate functions appear to work, we do not know they exist. But, until a better formalism comes along complexity theory is secure in its position as the basis of modern cryptography.

See section 4.6.1↩
See pages 2,3↩
See page 3↩
ie. time proportional to the size of the input↩
See page 41 at the bottom.↩
That is $\mathcal{M}$ computes in the $n$th bit of output in $p(n)$ time.↩
See page 23.↩
Since a PTM is a theoretical construction rather than a physical construction we can do away with the nasty realities of life and assume these random bits are actually random! A nice change of pace.↩
See section 4.2↩
Note, any constant greater than $\frac{1}{2}$ can be used here.↩
Definition is a combination of Definition 1.3.4 from and the definition given in Section 4.5 of .↩
From personal discussion with Prof. Harold Connamacher (harold.connamacher@cwru.edu)↩
Definition due to see Def. 1.3.5↩
Definition due to see Def. 2.2.1. Note, I simplified the definition slightly for clarity.↩
Definition due to see Def. 2.2.2↩
See Theorem 2.3.2 for an impractical but demonstrative conversion and Section 2.6 for an efficient conversion in the case of one-way permutations.↩
Definition due to see Def. 2.5.1↩
Theorem 10.8↩
Section 2.5.2↩
Definition due to see Def. 5.2.3, simplified.↩
Definition due to see Def. 5.2.1, simplified.↩
Theorem (and proof) due to Theorem 5.2.5.↩
Theorem and proof due to see theorem 10.9. My proof is a summary of Talbot and Welsh’s main argument↩
A one way permutation is simply a one way function which is a bijection from the domain to the range. The existence of one way functions implies the existence of one way permutations.↩
Theorem due to see Theorem 10.10↩
Definition due to see page 216.↩

Programming Never Gets Easier

2013-11-10T00:00:00-05:00

Another comment on E.W.Dijkstra Archive: The Humble Programmer (EWD 340)

Dijkstra, in the essay, felt that at the current moment in time (1972) the programmer was limited by the tools. That their thought processes were limited by poor languages and poor environment. However, he expressed great hope that this was transitory. Our tools would improve but he warns:

As an aside I would like to insert a warning to those who identify the difficulty of the programming task with the struggle against the inadequacies of our current tools, because they might conclude that, once our tools will be much more adequate, programming will no longer be a problem. Programming will remain very difficult, because once we have freed ourselves from the circumstantial cumbersomeness, we will find ourselves free to tackle the problems that are now well beyond our programming capacity.

I believe he was right. Some might argue are tools have not improved and that is why programming is not better. Indeed they may say it is worse than ever. But, I think our tools have improved. They don't always improve linearly but they are improving in fits and starts, with steps forwards and steps back. No matter how much our tools may have improved the state of programming as a practice has barely advanced at all! We still struggle against our own inadequacies, our own failings. Even when the algorithms are clear, the problem well studied, the programmer rarely gets it right the first time. Programming doesn't get easier as the tools get better. Rather the scope of our ambition climbs. As it climbs the problems get harder and the tried and true solution of yesteryear seems quaint to the practicing programmer of today.

So, we cannot rest on our laurels. We must strive on and construct new tools which ease our current pains, even if inevitably new pains arise. For, in learning, there is suffering.

New Site

2013-11-06T00:00:00-05:00

It has been 6 months maybe more since Posterous shut down and I have just now gotten around to recreating my site. I chose to do this on github pages with pelican. Hosting on github is easy and simple. If github ever stops doing these pages I can easily self host. I have now ported most of my old posts thanks to pandoc. The posts I left behind were mostly terrible so all in all I think the new site is a big improvement. I might even write some more posts.

The Limits of Type Systems

2013-09-03T00:00:00-04:00

E.W.Dijkstra Archive: The Humble Programmer (EWD 340)

Argument three is based on the constructive approach to the problem of program correctness. Today a usual technique is to make a program and then to test it. But: program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence. The only effective way to raise the confidence level of a program significantly is to give a convincing proof of its correctness. But one should not first make the program and then prove its correctness, because then the requirement of providing the proof would only increase the poor programmer’s burden. On the contrary: the programmer should let correctness proof and program grow hand in hand. Argument three is essentially based on the following observation. If one first asks oneself what the structure of a convincing proof would be and, having found this, then constructs a program satisfying this proof’s requirements, then these correctness concerns turn out to be a very effective heuristic guidance. By definition this approach is only applicable when we restrict ourselves to intellectually manageable programs, but it provides us with effective means for finding a satisfactory one among these.

Essentially what Dijkstra is advocating here is what the approach the most advanced type theories are striving for. They aim to allow the programmer to specify in the type system (a proof system) important semantic invariants of the application. The type system is constructed in such a way the program is only well typed if the specified invariants hold. The hope of the type theorists is that with a sufficiently powerful type (proof) system most if the not all properties one cares to prove are in fact provable (for certain programs).

The difficulty for the practicing programmer is most type systems are not nearly powerful enough to specify properties which are actually interesting. This leaves the programmer doing essentially the proof equivalent of book keeping with no benefit. I should distinguish here between dynamically checked types (properties) and statically checked types. Dynamically checked, that is at run time, are always enormously helpful to the programmer because they provide runtime safety. However, statically checked types can be overly burdensome if they require lots of book keeping without sufficiently powerful proofs.

Unfortunately, for a wide variety of statically checked programming languages the available proofs are uninteresting and the burden is high. This is the challenge to the Dijkstra-ist.

We can't automate the programmer.

2013-09-03T00:00:00-04:00

HN Comment

I find this comment thread on how programmers will "automate" our way out of jobs to be especially pertinent as I work on my proposal. A contention I have, as do many others, is the primary practical challenge standing between us and robust systems is complexity. There are several on this thread who believe we have essentially mastered complexity for a broad number important application areas.

I think I could be convinced of this argument as long as those application areas are very narrow. As long as the creators of those applications are content with relatively standardized solutions. The moment there are A) functional requirements not covered by the standard case or B) differentiating requirements (such as aesthetics etc...) the general tool stops working. This is what makes complexity such a beast. If you make a fully generic tool, you have created a programming language. Perhaps a very high level (even graphical language like LabView) but still a language. Such things require trained people to operate.

We will never not need programmers.

Always Pertinent - C a giant leap backwards.

2013-08-25T00:00:00-04:00

From the point of view of software engineering, the rapid spread of C represented a great leap backward. It revealed that the community at large had hardly grasped the true meaning of the term 'high-level language' which became an ill-understood buzzword. What, if anything, was to be 'high-level'? As this issue lies at the core of software engineering, we need to elaborate.

from : Wirth, N. (2008). A Brief History of Software Engineering. IEEE Annals of the History of Computing, 30(3), 32–39. doi:10.1109/MAHC.2008.33

Passmash - The Site Specific Password Munger

2012-01-30T00:00:00-05:00

Passmash is a new commandline password munger. It has been tested to work on Linux with X and on MacOS. It should also work on Windows.

What is a Munger?

A munger takes a password and turns it into another password, "munging" it. In particular passmash takes

A password (supplied interactively at the prompt)
A URL (or other identifier) (supplied as a command line argument)
A secret key (kept at \~/.ssh/passmash.key)

and returns a password. It has the advantages of a password manager without having to worry about syncing a password database. The key file is static, so simply keep a (possibly encrypted) backup of it. If you loose the key file, you will not be able to recover your passwords.

Example Usage

In most circumstances you will want to use the pm command

$ pm myurlhere.com
Password:

This command automatically generates and copies the password to you clipboard. On Linux it uses xclip -selection clipboard, on Mac OS X it uses pbcopy and on Windows it uses clip.

If it is on another operating system (like OpenBSD) it will pretty print the password for easy typing. eg.

$ pm myurlhere.com
## We don't yet support OpenBSD for autoclipboard copying
Password:

5KrUw4pBgC89LGxggXEIFtjM41aPc+/GxH+cumCuTo4
5KrUw - 4pBgC - 89LGx - ggXEI - FtjM4 - 1aPc+ - /GxH+ - cumCu - To4

Technical Details

Passmash uses a SHA256 based HMAC with key strengthening.

def mash(key, url, password):
    h = hmac.new(key, password, sha256)
    h.update(url)
    for i in xrange(250000):
        h.update(h.digest())
    return h.digest()

On my machine (a 2.0 Ghz Core2) it takes around 1 second to derive a password using this function. A more secure version of the same utility could make use of bcrypt or scrypt. However, either would add an external dependency.

This password derivation function should provide strong defense against an attacker who has

A password generated from the function (perhaps obtained from a hacked website).
The algorithm. (eg. they know you use this program to generate your passwords).

And optionally:

The key file
or the "master" password (but not both)

If your "master" password has sufficient entropy then your other passwords generated with the same key should be reasonably secure against a brute force attack.

Happy Munging!

Ternary Search Tries for Fast Flexible String Search : Part 1

2011-06-02T00:00:00-04:00

Searching a large corpus of strings is a problem many applications have to solve, whether the application features autocomplete boxes or full-text search. Efficient methods for conducting such searches are not always readily apparent to the algorithm designer. In this series of articles I will present a data structure known as the Ternary Search Trie (TST) which is designed to assist in solving this problem. For this introductory article I will not discuss algorithms in detail but only provide a high level overview of the structure and algorithmic running time for various operations. In the next article I will detail the process of maintaining the structure with insertions and deletions. The final article will discuss different flexible search algorithms and their implementations.

Symbol Tables: A Short Review

A symbol table is a mapping between a string key and a value which can be any type of object from an integer to a complex nested structure. There are two classic symbol table implementations most programmers are immediately familiar with: Binary Search Trees (BSTs) and hash tables. Both of these structures work by exactly matching the search key to the keys stored in the structure. If there does not exist an exact match then there is a miss. Thus, neither of these structures can serve as useful index for an autocomplete algorithm where only part of the key is known. They may be useful for a full text index, but they will not be as efficient as some of the other structures we will later discuss.

While these simple structures are limited, other symbol table implementations have properties more suited for modification for flexible string search. This series will focus on the Trie category of structures. These structures are more suited for partial match as we will see. But first, why are hash tables not suited for this job? They have such excellent running time characteristics, O(1) lookups! However, one cannot modify the hash table algorithm to effectively serve the purpose of a partial match or a range query. Why? Because hash functions transform strings into numbers (which correspond to buckets in an array). Good hash functions have wide variance in hashing strings and will not hash a similar string or a sub string to the same bucket.

Introducing the Trie

Figure 1. An Example Binary Search Trie

In general a Trie is a special for of a tree. However, instead of comparing entire key at each node during traversal, it only compares parts of keys. The key/value pairs are kept in the leaves (like in a B+ Tree). We will first consider a Binary Search Trie. Like the Binary Search Tree, each node in the Trie has two children, left and right. The left child is defined as the 0'' child and the right as the 1'' child. As a key is inserted, a node is created or visited for each bit in the key. When visiting a node which already exists, the direction to descend is based on the current bit. that is:

let there be a function bit(i, s) which returns the ith bit in the string s.
let depth(r, n) return the depth of the node n in the tree rooted at r
bit(depth(r, n), s) is the bit in the string used to make the decision on which of n's children to visit.

Sedgewick gives the formal definition: "A [binary search] trie is a binary tree that has keys associated with each of its leaves, defined recursively as follows: The trie for an empty set of keys is a null link; the trie for a single key is a leaf containing that key; and the trie for a set of keys of cardinality greater than one is an internal node with left link referring to the trie for the keys whose initial bit is 0 and right link referring to the trie for the keys whose initial bit is 1, with the leading bit considered to be removed for the purpose of constructing the subtrees."¹

A search using this structure is directed by the strings in the database. However, since only one bit is considered at a time in the search for a k-bit string the search will take in the worst case k bit comparisons. This makes for a very tall structure when using string keys, since single characters will be at least 8 bits long in ASCII and much longer in Unicode. Another unfortunate implementation detail is that modern processors typically work more efficiently when accessing bytes or words. Thus, a more efficient structure might consider multiple bits at once.

Multi-way Tries

Figure 2. An Example R-Way Trie

If one considers multiple bits at once one has to increase the fanout (number of children per node) of the tree. Consider figure 2, in which each node has a fanout of 26.² While useful if every node has 26 children, the space to store the pointers becomes wasteful for real data. However, despite the wasted space in comparison to the binary version, searches on the R-Way Trie will perform faster than on the Binary Trie. It will be faster for the CPU to compare the bytes under consideration and there will be fewer comparisons over all. In general, a Binary Trie will require log~2~(N) comparisons to perform a search, and an R-way Trie it will take log~R~(N) comparisons.

However, to produce a usable structure for our purpose (a large in-memory string index) we need to cut down on the space wasted by the extra pointers in each node. The tricky bit is to do this while still maintaining the hard-fought gains in search speed. Simply using a dynamic structure like a hash table in each node to hold the array won't work either because hash tables are slower than an array access and if the hash table becomes overly full it may actually use more space than the array. Thus, a different structure is needed.

Ternary Search Tries

Figure 3. An Example Ternary Search Trie with strings [abc, abs, awe, and].

The Ternary Search Trie helps avoid the unnecessary space needed by a traditional multi-way trie while still maintaining many of its advantages. In a Ternary Search Trie each node contains a character and three pointers. The pointers correspond to the current character under consideration being less than, greater than or equal to the character held by the node. In a sense this structure is like taking the Multi-way Trie and encoding it on to a Binary Search Tree with the keys as current character and the values as another BST corresponding to the next character.

While a Multi-way Trie has about R*N/log~2~(R) pointers, a Ternary Search Trie has R + c*N pointers where c is a small constant, perhaps 3. Consider the graph of their performance:

Figure 4. Links in a Multi-way Trie vs. a Ternary Search Trie

When R is small, 2, 3, 4 Multi-way Tries and Ternary Tries have similar a similar number of pointers, but a low branching factor destroys the advantage of the Multi-way Trie. When R grows to a larger, more reasonable size such as 256, the number of pointers explodes in comparison to the Ternary Trie. Thus, the Ternary formulation will be far more space efficient in the worst case than the Multi-way formulation.

What is the cost for better space efficiency? In the Multi-way Trie we must traverse at most the length of the search key links. In a TST we may need to traverse up to 3 times that many links in the worst case. However, this pathological case is rare. In the average case the situation can be made much better through a few small improvements to the basic structure.

Figure 5. An Improved Ternary Search Trie.

The first improvement, illustrated below in Figure 5, involves collapsing the leaf nodes. Instead of allowing long chains of nodes at the leaves, we collapse them into a single node. This allows the final check to be computed efficiently.

The second improvement, also illustrated in Figure 5, combines the best of the Multi-way Search Trie with the Ternary Search Trie. The root node is an R-Way node like in the Multi-way Search Trie. The rest of the tree is a Ternary Search Trie with the leaf nodes collapsed. In practice these improvements result in an enormous speedup. The theory also supports the practice; according to Sedgewick, these improvements cut the number of comparisons needed in half.³

There is one final improvement to consider not shown the above figure. Similar to the first improvement, it involves collapsing nodes, but instead of collapsing leaf nodes, internal nodes are collapsed. This idea is similar to the Patricia Trie. When a group of strings shares the same contiguous substring, instead of having a node for each character shared, collapse the shared nodes into a single node.

Conclusion and Whats Next

In this post we discussed the theory behind Symbol Tables and the use of Tries as a symbol table implementation. The Trie, and in particular the TST, is an efficient way to implement a symbol table. A good implementation of a TST has comparable performance to a Hash Table.

You can puruse my Go (golang) implementation in my data-structures repository. I also wrote a python implementation.

Sedgewick R. Algorithms. Third Edition. Definition 15.1. http://www.amazon.com/dp/0201314525/ Hereafter: Sedgewick ↩
Note: Usually an R-Way or Multi-Way Trie has a fanout equal to the numbers of characters in the character set or the number of bits in a machine word, half word or byte. So in practice a node in an R-Way Trie might have 256 children or perhaps 2\^16 children. As the fanout (the number of children per node) increases the space efficiency of the Trie decreases. However, the search speed increases. A classic time/space trace off. ↩
Sedgewick Table 15.2 and related text. ↩

How To: Write Self Updating Python Programs Using Pip and Git

2011-05-23T00:00:00-04:00

If you are a pip¹ and virtualenv user you already know how easy it is to install python packages. Unlike the bad old days when I started programming in Python, 9 years ago, it is now easy to add, remove and manage python modules. In fact we can leverage pip to create an update command for a python program, for example and ease of illustration, a shell utility.

Desired Features
Basic Idea
Implementation

Desired Features

Uses a server controlled by the owner (instead of the Python Package Index).
Install an arbitrary version of the program.
Defaults to updating to the newest version of the major release one is tracking.

I often want my programs to update themselves from a specific location. For instance an internal server or perhaps my github account. Fortunately pip already supports such nicities with the -e for the install command.

Additionally, when running a generic update you often want to stay on the same major revision and simply get the bug fixes. However, it is important to provide the option to update to any arbitrary release including tracking the master branch.

Basic Idea

Use Pip and the -e option plus a base URL to automatically update your software. eg. Have you software run pip for the user.

example command:

pip install --upgrade --src="$HOME/.src" -e git+<URL>@<REV>#egg=PACKAGE_NAME

Tracking Major Versions

To track major version updates some care must be taken in setting up the repository. I use branches instead of tags to track major releases. This allows me to push out bug fix updates for every one tracking that release. I tag minor releases to allow users to install a specific version.

Branches

master
stable
r0.1
r0.2
...
rN

Pip Gotcha

When checking out branches using pip you have to supply origin/branchname ex:

pip install --upgrade --src="$HOME/.src" -e git+https://github.com/user/repo.git@origin/branch#egg=PACKAGE_NAME

While when checking out a commit you should not supply origin

pip install --upgrade --src="$HOME/.src" -e git+https://github.com/user/repo.git@COMMIT_ID#egg=PACKAGE_NAME

Why does pip work like this? Because of the commands it executes. For the command:

pip install --upgrade --src="$HOME/.src" -e git+https://github.com/user/repo.git@<VERSION>#egg=PACKAGE_NAME

pip runs

git fetch -q git reset --hard -q <VERSION>

Store the tracked version in the source

To ensure the update command installs the correct updates I put which release to checkout in the source code. This allows me to "release" a version by creating a branch and then changing the RELEASE constant to point the name of the branch.

Implementation

Note: This is example code only, you should modify for security and stability of your enviroment.

Note: I didn't include virtualenv support in this code but it is trivial to add.

from subprocess import check_call as run 
from getopt import getopt, GetoptError 
RELEASE = 'master' # default release 
SRC_DIR = "$HOME/.src" # checkout directory 
UPDATE_CMD = ( # base command 
'pip install --src="%s" --upgrade -e ' 
'git://github.com/timtadh/swork.git@%s#egg=swork' 
)

@command 
def update(args): 
    try: 
        opts, args = getopt(args, 'sr:', ['sudo', 'src=', 'release=', 'commit=']) 
    except GetoptError, err: 
        log(err) 
        usage(error_codes['option'])

    sudo = False 
    src_dir = SRC_DIR 
    release = RELEASE 
    commit = None 
    for opt, arg in opts: 
        if opt in ('-s', '--sudo'): 
            sudo = True 
        elif opt in ('-r', '--release'): 
            release = arg 
        elif opt in ('--src',): 
            src_dir = arg 
        elif opt in ('--commit',): 
            commit = arg

    if release[0].isdigit(): ## Check if it is a version 
        release = 'r' + release 
    release = 'origin/' + release ## assume it is a branch

    if commit is not None: ## if a commit is supplied use that 
        cmd = UPDATE_CMD % (src_dir, commit) 
    else: 
        cmd = UPDATE_CMD % (src_dir, release)

    if sudo: 
        run('sudo %s' % cmd) 
    else: 
        run(cmd)

http://www.pip-installer.org/en/latest/index.html "A Python package installer." ↩

Announcing swork - Simplify your Shell Configuration

2011-05-18T00:00:00-04:00

If you are like me, and if you are reading this you may very well be, you spend an inordinate amount of time juggling inane details, like shell environment variables, while programming. Now there is nothing wrong with setting, exporting, and then unsetting variables, mounting and unmounting FUSE partitions, starting routine backups, and so on but it does get tedious after a while. Eventually, you may have written a host of scripts to solve these various problems. Today I present swork (or start work) a command line utility to help manage these little one off scripts with ease.

Don't Repeat Yourself

A typical pattern seen in scripts, such as virtualenv's activate script, is the storing of old environment variables such that the changes made by the script can be easily undone. Every non-trivial script I write seems to include this detail, and I am tired of it. It is boring, it is simple, and it is abstract-able. So I have abstracted. swork frees you from needing to write this code. When you want to go back the original state of the shell, you simply type:

$ swork restore

As long as you have run swork at some point in the past on the current shell (or rather the current bash process) swork will restore environment of the shell to the state it originally found it.

Writing Configuration Scripts

While, swork saves you the trouble of saving and restoring variables, you still have to write the scripts to run. Fortunately, this is very easy. You simply write a bash script (or any executable) then you add it to the \~/.sworkrc (located conveniently in your home directory).

Example setenv file:

#!/usr/bin/env bash

source env/bin/activate # activate a virtualenv
export SOMEVAR="new value"
export PATH="some/new/stuff":$PATH
export PYTHONPATH="more/new/stuff":$PYTHONPATH

example .sworkrc file:

{
    "project1" : {
        "root":"/path/to/project1/root",
        "start_cmd":"source /path/to/project1/root/then/setenv"
        "teardown_cmd":"echo 'project1 teardown'"
    },
    "project2" : {
        "root":"/path/to/project2/root",
        "start_cmd":"source /path/to/scripts/project2_setenv"
        "teardown_cmd":"echo 'project2 teardown'"
    }
}

Wrapping Up

swork makes it easy for you to manage the environment on you shell allowing you to switch contexts with minimum fuss. It currently implements the minimum functionality to be useful, but is just waiting for your feature request!

check it out on github: https://github.com/timtadh/swork

Grammars, Ambiguity, and Expressibility

2011-02-17T00:00:00-05:00

Last night I gave a talk at CWRU Hacker Society about formal languages. This is the first talk in a series of lectures I will be giving on compilers. Unfortunately, unlike my regular expression talk I did not get a recording of the audio. I may do a write up of exactly what I talked about later. Until then enjoy my "slides."

Interpreting the Free Software Movement as Religion

2010-12-08T00:00:00-05:00

A person should aspire to live an upright life openly with pride, and this means saying “No” to proprietary software.

- RMS

Introduction

The Free Software movement which began in earnest twenty-five years ago has become one of the most quietly influential movements of the Internet age. Today, many social phenomenas occurring in our networked world, such as Wikileaks, can be understood more completely by understanding the Free Software movement. The Free Software movement can be usefully analyzed from many perspectives however, this paper will use the lens of religion. Specifically, the movement will be analyzed from the context of selected writing from its founder, Richard M. Stallman, using the categories defined by Mircea Eliade. Through the use of Eliade's categories one understands Stallman to be demarcating the sacred from the profane in an attempt to return to an archaic past.

Eliade's Terms and Categories

In his 1957 treatise, The Sacred and the Profane , Mircea Eliade creates categories, such as sacred vs profane, and attempts to fit many different religious traditions into the categories. Methodologically this approach has a serious shortcoming, the supporting traditions do not always fit well into the chosen categories. However, since Eliade has defined broadly useful, even if not universally applicable, categories for interpreting religious traditions. The utilization of these categories highlights the religious nature of the Free Software.

Eliade's primary purpose in his treatise is to discuss the experiential demarcation between the sacred and the profane. Eliade defines the sacred in two senses. The first is recursive: the sacred is the opposite of the profane.¹ In this sense one can place objects into two categories, sacred and profane, as long as they don't overlap. Eliade clarifies this slightly through use of Rudolf Otto's term, the wholly other,² indicating the sacred manifests itself as wholly different than the profane. By using Otto's language, Eliade indicates the sacred has an element to the divine.

For the purposes of this paper we will stipulatively take the divine to mean: that which seems to the individual to have a numinous quality. An object which has a numinous quality is one which seems irreducible and the individual thus feels a creature dependence towards.³ Thus, the sacred is the manifestation of the numinous into the corporeal. The profane, on the other hand, is the common, that which seems to be understandable. These stipulative definitions conform to Eliade's requirements: the sacred is opposite of the profane, and the sacred is wholly different than the profane.

In addition to his categories, sacred and profane, Eliade defines two related categories: archaic society and modern society. The man who lives in archaic society, homo religiosus, seeks to exist as much as possible in or around the sacred.⁴ In contrast the man of modern societies, modern man, exists in a desacralized environment.⁵ The modern man depending on his temperament may look back to the religious man either romantically or derisively. Thus, modern and archaic do not indicate, as they traditionally do, an essential ordering or time line. It may be that archaic and modern coexist with each look back to the other as a mythical past while eagerly looking forward to a time when man is more or less religious.

The Free Software Movement

What is the Free Software movement? How can it be understood as religious, using the terms and categories defined by Eliade? The Free Software movement was started in 1984 with the publication of the GNU Manifesto by Richard M. Stallman. Stallman had become disgusted with the unethical nature of software and computer usage and sought a return to an earlier time where users freely shared and modified programs. To enable this return, he set about to create an ecosystem of software which was protected from being made proprietary. In many ways Stallman succeeded: today there is large amount of Free Software available. Every computer user unwittingly uses such software on a daily basis, and companies such as Google and Facebook could not exist without Free Software.

What unethical nature was Stallman disgusted with? The answer lies in the GNU Manifesto where Stallman states, I consider that the golden rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way.⁶ It seems to Stallman he should obey the golden rule. One may guess from the text Stallman would define the golden rule as a principle of reciprocity. Thus, if an individual likes a program and would want another to share it with him he is ethically required to share the program with another as well. Furthermore, an individual must not restrict those he shares software with from further sharing the software, or from modifying the software to suit their needs better. To prevent such sharing and modification would violate his golden rule.⁷

It is important to note Stallman does not believe that all things should be shared alike. He only considers such freedom ethical where there is no harm done to the person who shares by sharing. Stallman states: "Owners say that they suffer harm or economic loss when users copy programs themselves. But the copying has no direct effect on the owner, and it harms no one.⁸ Thus, Stallman believes that copying a program does not harm the original owner, because the owner does not loose the use of the program because a copy is made. Such copies can be made indefinitely. In the same way Stallman defends the right to modify software: whether you run or change a program I wrote affects you directly and me only indirectly. Whether you give a copy to your friend affects you and your friend much more than it affects me. I shouldnt have the power to tell you not to do these things. No one should.⁹ Thus, Stallman has constructed his own ethical system based on how the golden rule seemed to him.

To explain and understand his movement, Stallman constructed a founding narrative. The narrative begins in ancient times when copyright as a concept did not exist. Stallman explains that in those times the roles of authors, copiers, scribes, and commentators were muddled. Everyone who participated in written culture freely copied, improved, and commented on previous works.¹⁰ Stallman holds this copyright-free society up as the exemplar from our historical past for how one should relate to written work.

Stallman continues his narrative by connecting the experiences of early computer programmers (his in particular) to the copyright-free society detailed above. While copyright existed when computing culture began in the 40's and 50's it was not yet universally applied to computer source code. Stallman participated in this society when he joined the MIT Artificial Intelligence Lab in 1971: When I started working at the MIT Articial Intelligence Lab in 1971, I became part of a software-sharing community that had existed for many years. Sharing of software was not limited to our particular community; it is as old as computers, just as sharing of recipes is as old as cooking. But we did it more than most.¹¹ Stallman holds his early experiences in the AI Lab as a second exemplar for the proper orders of society, where software is freely shared, edited, commented, and ported.

However, Stallman's perfect society eventually fell into disrepair. Programmers were asked to sign software licenses and non disclosure agreements when the university purchased new equipment and software. To Stallman these events had a chaogonic¹² quality: This meant that the first step in using a computer was to promise not to help your neighbor. A cooperating community was forbidden. The rule made by the owners of proprietary software was, 'If you share with your neighbor, you are a pirate. If you want any changes, beg us to make them.'¹³ Thus, the ideal of community which had obeyed the golden rule began to unravel. No longer could a programmer freely help his neighbor, no longer could a programmer freely fix bugs, no longer could a programmer port software to new platforms. The programmers were now at the mercy of contracts and legal agreements.

In the depths of this disarray, Stallman experienced a heirophany, Eliade's term for the sacred manifesting itself. It began with the AI Lab being gifted a printer from Xerox. However, despite giving them the printer, Xerox refused to share the code for the driver.¹⁴ Unfortunately, the driver had bugs in it. When Stallman offered to fix the bugs if they gave him their code, they refused. The experience was transformative. Stallman could no longer accept the status quo of license and non-disclosure agreements. He set out to change the world, so he could return to an ideal society where programmers helped their neighbors.¹⁵

Thus, according to Stallman's narrative detailed above, Stallman set out to purge his life of the corrupting influence of proprietary software. Unfortunately, to rid himself of proprietary software he needed to create a new ecosystem of Free Software. So he quit his job at MIT and began working on several Free Software programs. Stallman states: I realized that I was elected to do the job.¹⁶ Thus, he began the GNU project to create a Free operating system and ecosystem of software. Without, such an ecosystem Stallman feels one cannot live an upright life as a programmer.¹⁷

Therefore, the Free Software movement seeks to establish an alternative reality where all software written is Free. Users are free to modify and redistribute software. No one is free to limit another's use of software. This movement can be understood using Eliade's categories of sacred vs profane, and archaic society vs modern society.

Interpretation

The copyright free societies Stallman references in his narrative parallel the Eliadian concept of the archaic society. In both cases these societies are held up as exemplars of what it means to be truly religious, to be a homo religiosus. In Stallman's pre-copyright society, programmers shared with each other freely, they modified programs without hesitation; unwittingly they obeyed his golden rule and helped their neighbors. In modern society programmers no longer share code and modify programs. They are prevented from doing so. Thus, they no longer obey the golden rule. By not obeying the golden rule they have become corrupt.¹⁸

Stallman's narrative fits into Eliade's categories of the modern society vs. the archaic society. Stallman represents himself as a truly religious person living in the modern society seeking to return to the archaic society. His method for returning to the archaic society is to resacralize the world.

To sacralize, one must have a concept of something that is sacred vs. something that is profane. For to sacralize one make the the profane sacred. For Stallman, Free Sofware itself is sacred. Free Software is opposite the profane proprietary software. Proprietary software cannot be shared and it cannot be modified. Free Software can be explicitly shared and modified. Free Software also manifests a numinous quality. Specifically, Free Software is the revelation of the ideal divine society today. In the ideal society all software is Free Software. To have Free Software in present society is to experience the revelation of the ideal. Thus, Free Software is not just sharable and modifiable it also sacred. Free Software is Sacred Software.

Stallman desires to return to his ideal archaic society where programmers were more religious and users could share and modify programs at will. To bring about the return of the archaic society he must resacralize the present profane society. To do so he created the GNU Project to manifest Sacred Software into the present profane space. Thus, Stallman marks off the sacred, Free Software, from the profane, proprietary software, in an attempt to return humanity to the ideal society.

Conclusion

The Free Software movement can be understood as a religious movement using Eliade's terms and categories. Stallman basis his movement on his understanding of the golden rule. He uses his understanding of the rule to construct an ethical system for the production and use of software. Stallman then constructs a narrative to explain how society has moved from a religious ethical past to a profane present. To return society to the ideal past, Stallman attempts resacralize the present society by creating Free Software. Free Software is sacred. By introducing Free Software into present society, society becomes more sacred. The utilization of Eliade's categories clarified the religious aspects of the Free Software movement.

Eliade, M. The Sacred and the Profane: The Nature of Religion trans. Trask, W. Harcourt Inc. New York. 1957. pg 10. Hearafter: Sacred and Profane. ↩
Ibib. pg 2. and Otto, R. The Idea of the Holy trans. Harvey, J. Oxford University Press, New York. 1958. pg 25. Hereafter: The Holy ↩
The Holy pg. 6-7 ↩
Sacred and Profane pg. 12, 15 ↩
Sacred and Profane pg. 17 ↩
Stallman, R. M. Free Software, Free Society: Selected Essays of Richard M. Stallman. GNU Press, Boston, 2002. pg. 34. Hereafter: Free Software. ↩
See Free Software pg. 43 for a discussion of the precise meaning of Free Software. ↩
Free Software pg. 48 ↩
Free Software pg. 49 ↩
Free Software pg. 39 ↩
Free Software pg. 17 ↩
Chaos creating, antonym of Cosmogonic. See Beal, T. K. Religion and its Monsters. Routledge, New York, 2002. ↩
Free Software pg. 18 ↩
A driver is a piece of software which allow the computer to communicate with a piece of hardware. Every piece of hardware has a unique communication protocol, necessitating many different drivers. ↩
Free Software pg. 19 ↩
Free Software pg. 19,20 ↩
Free Software pg. 57 ↩
For an example of Stallman using such language see for instance Free Software pg. 130 ↩

Lessons Learned While Implementing a B+Tree

2010-04-10T00:00:00-04:00

B+Trees are complex disk based trees used to index large amounts of data. They are used in everything from file systems, to relation databases, to new style databases gaining popularity today. Sometimes a domain specific application needs to index a large amount of data, but cannot use a traditional database, or one of the NoSQL databases. In such instances the development team needs to roll their own indices. Here is an introduction to the B+Tree (one of the indexes my team created) and lessons I learned while implementing it.

Introduction to the B+Tree

B+Trees are one of the fundamental index structures used by databases today. This includes new style SQL free databases. The B+Tree popularity stems from their performance approaching optimal performance in terms of disk reads for range queries in a 1 dimensional space. What is a 1 dimensional space when talking about computer data which could be anything (not just numbers)? It is any collection of objects where the user accesses the object using only one attribute at a time.

For example if we have an object which has X, Y, and Z as attributes queries would only take place on X, or Y, or Z, but never on XY, or YZ, or XZ, or XYZ. A collection where multiple attributes are used to access the data elements are known as multidimensional spaces. For these spaces there are many other structures which have better performance than B+Trees.

B+Trees perform particularly well (in comparison to some other indices) when executing range queries. A range query is typically expressed as inequality such as "give me all strings between 'blossom' and 'brunet.'"

When I say their performance is approaching optimal in number of disk reads what do I mean? Why are we not measuring performance in number of instructions executed (like we do when we analyze a binary search)? In memory algorithms and structures like sorted arrays and binary searches are largely bound by the number CPU cycles it takes to execute the algorithm. We usually neglect CPU cache performance and memory locality when analyzing them, arguing these are constant in terms of the asymptotic performance of the algorithm. However, for a disk based structure like B+Trees the time it takes to read (or write) to a disk becomes the dominant term, since disks are extremely slow in comparison to main memory. Therefore for disk structures we analyze their performance in terms of disk reads/writes.

Basic Structure of the B+ Tree

While I will not give a through explanation of the exact structure and properties of B+ Tree (I leave that to algorithm and database textbooks by the likes of Knuth, Sedgewick, and Ullman), I will describe its basic structure.

A B+Tree is best thought of as a key-value store. It is structured as a generalized tree. Instead of having only one key in each node it has N keys in each node, where N is referred to as the degree of the B+Tree. In the B+Tree there are 2 kinds of nodes, interior nodes, and exterior (leaf) nodes. The interior nodes hold keys and pointers to nodes. The exterior nodes hold keys and their associated values. This indicates that the interior nodes have a different (usually higher) degree than the exterior nodes.

The reason the tree is structured this way is because it is rooted in the nature of disk access. Disks to not return 1 byte when you ask for 1 byte instead they return what is called the disk block to which that byte belongs, the operating system then sorts out which byte it is that you need. B+Tree exploit the situation by making their nodes fit exactly into the size of one disk block. Since the degree of the interior nodes is high, this makes the tree extremely wide, which is a good thing since it means fewer disk reads to find the value associated with any one key.

Figure 1. An Example B+Tree

In figure 1 you can see an example B+Tree. For this illustration I neglect showing the values, and have the order of the interior nodes equal to the order of the exterior nodes. In general this will not be the case. One thing to note in this simple example is how the exterior nodes are chained together in order. This is why it is efficient to execute a range query on the B+Tree. One can simply find the first key in the range, and then traverse the leaf nodes until the last key has been found.

Implementing the B+Tree

I made the decision to use TDD (Test Driven Development) for implementing the B+Tree. TDD has a lot of pluses when trying to create a data structure of any kind. When implementing a data structure one typically knows exactly how the structure should function, what it should do, and what it should never do. By writing tests first, you can ensure that when you finish a method, it actually works. This speeds development time especially since you already know how the structure should function. It makes it quicker to find bugs, and to battle test the B+Tree. Since I have released the B+Tree to the rest of my team to use, there has not yet been a bug filled against it.

So knowing that we are using TDD, and knowing what the structure is and how it performs. What is the best way to begin implementing this complex structure? The way I started was to create a general structure called a block file. My block files abstracted the notion of reading and writing blocks (and buffering them). I also created objects to model a block that could contain either keys and pointers, or keys and records (instead of values from here on I will use the term records). Actually my blocks are even more general than that as I intend to reuse them for other disk based index structures like linear hashing in the future.

I also created what I called a ByteSlice. My ByteSlice was an array of bytes of arbitrary length. I use it to represent, keys, records, and pointers; everything in the B+Tree. My ByteSlice implemented a comparator, so it could be sorted, and conversions from integer types of various lengths to the ByteSlice and back again. By implementing this general type my B+Trees can easily deal with any kind of data and perform in exactly the same way.

After the infrastructure was created I began working on my first iteration of the B+Tree. The first iteration was based on the algorithms give by Robert Sedgewick in his excellent book "Algorithms in C++." I managed to get this implementation up, running, and fully tested in a matter of days. However, the version given by Sedgewick which inspired my implementation did not deal gracefully with duplicate keys. Thus, I need to invent my own way of handling duplicate keys.

Approaches to Handling Duplicate Keys in B+Trees

There are several different ways of handling duplicate keys. One way is use an unmodified insert algorithm which allows duplicate keys in blocks but is otherwise unchanged. The issue with a structure such as this is the search algorithm must be modified to take into account several corner cases which arise. For instance one of the invariants of a B+Tree may be violated in this structure. Specifically if there are many duplicate keys, a copy of one of the keys may be in a non-leaf block. However, the key may appear in blocks that which appear logically before the block which is pointed at by the key in the internal block. Thus the search algorithm must be modified to look in the previous blocks to the one suggested by the unmodified search algorithm. This will slow down the common case for search.

There is another issue with this straight forward implementation, if there are many duplicate keys in the index, the index size may be taller than necessary. Consider a situation were for each unique key there are perhaps hundreds of duplicates, the index size will be proportional to the total number of keys in the main file, however, you only need to index an index on the unique keys. One of the files indexed in our program will be indexing has such characteristics to its data. It indexes strings (as the keys) with associated instances where those strings show up in our documents. There can be hundreds to thousands of instances of each unique string.

Therefore the approach I took was to store only the unique keys in the index, and have the duplicates captured in overflow blocks in the main file. An example of such a tree can be seen in figure 2. Consider key 6; there are 5 instances of this key in the tree. The tree is order 3, indicating the keys cannot all fit in one block. To handle this situation an overflow block is chained to the block which is indexed by the tree structure. The overflow block then points to the next relevant block in the tree.

Figure 2. A B+Tree with duplicate keys and overflow blocks.

To create a structure such as this, the insert algorithm had to be modified. Like the previous version these modifications do not come without a cost, in particular the invariant which states all block must be at least half full has been relaxed. This is not true in this B+Tree, some blocks like the one containing key number 7, are not half full. This problem could be partially solved by using key rotations to balance the tree better. However, there are still corner cases where there would be a block which is under-full. One such corner case includes when a key falls between two keys which have overflow blocks. It must then be in a block by itself, since this B+Tree has the invariant which state that if a block is overflowing it can only contain one unique key. In the future we would like to implement key rotations to help partially alleviate the problem of under-full blocks.

The advantage of this approach to B+Trees with duplicate keys is the index size is small no matter how the ratio of duplicate keys to the total number of keys in the file. This property allows our searches to be conducted quicker. Since the overflow blocks are chained into the B+Tree structure we still have the property of being able to fast sequential scans. One consequence is we have defined all queries on our B+Trees to be range queries. This is fine because all of our queries were already range queries. In conclusion we relax the condition that all blocks must be at least half full to gain higher performance during search.

The Lessons Learned

The biggest lessons learned through the journey:

The Value of Test Driven Development The impact TDD had on the development time of the B+Tree vs. other structures in the project cannot be understated. TDD dramatically reduced the time it took to develop the structure (from over a month for some of the other structures to under two weeks for the B+Tree), and has ensured reliability of the structure once it entered production.
The Value of the Iterative Approach By starting simple, testing, and then adding complexity I was able to get a better grasp on the problems posed by the modifications we needed to make to the structure. For instance I before I tried method 2 for duplicate keys, I modeled the data we would be putting into our tree and visualized the resulting structure. I found the structure would perform poorly. However, the same code allowed my to visualize method 2 and see that it would perform well.
Visualizations as Part of Development Writing code allowing you to visualize the structure you are developing can really help you find bugs quicker. The best tool to do this with is graphviz. The pictures used in this blog where generated as part of unit test cases. My building a visualization framework early as part of your unit tests you can further reduce development time. When a bug a appears it can be enormously helpful to visualize the actual structure of the tree at the time the bug manifested.

Conclusions

When the right choice for your project isn't DBMS, but you still need to index large data, don't fear you can write the index structures yourself. By using TDD, iterating, and visualizing as you go you can ensure the index structure you create will perform well, and will never get into an incorrect state. Databases are not a black box, and they are not always the right answer. When required you can create you own system.

Hackthology

Taming the Variants: Multi-Architecture Continuous Testing at Google

Note

Abstract

I. Introduction

A. Contributions

II. Background

A. Glossary of Google Terminology

B. Configuring TAP and Rapid

C. TAP Overview

D. Rapid Overview

III. Configuring CI/CD for the new Architecture

A. Variants

B. Porting Process with Variants

C. Complexity of Implementation

IV. Target Comprehensive Testing

A. Integration with Release

B. Cost Savings

V. Empirical Evaluation

A. Research Questions

B. Note on Statistical Testing

C. Analyzing the Effect on Machine Costs

D. Analyzing the Effects on Project Status Latencies

E. Analyzing Breakage Detection Latency

F. Analyzing the Effects on Release Failure Rate

G. Estimating Engineering Cost Savings

H. Threats to Validity

VI. Results

A. RQ1: Compute Resource Consumption

B. RQ1: Comparison of Project Status Latencies

C. RQ2: Breakage Detection Latency

D. RQ2: Late-Stage Defect Detection Rates

E. RQ3: Engineering Savings

VII. Related Work

VIII. Conclusion

Contributions and Acknowledgments

References

Speculative Testing at Google with Transition Prediction

Note

Abstract

Introduction

The Google Development Environment

TAP Postsubmit

Cost of Breakages

Our Contributions and Findings

Background

Developer Services

Bazel

Forge

Comprehensive Testing Cycles

Ancillary Systems For Build Gardening

Culprit Finding

Culprit Verifier

Autorollback

The Evolution of TAP

The Current Dilemma

Speculative Testing Cycles

Determining Breakage Likelihood

Making Scheduling Decisions

Goal of Finding Test Breakages

Evaluating Performance

Transition Prediction

Features

Labeling

Super Extreme Class Imbalance

Training Configuration

Empirical Evaluation

Evaluation

Research Questions

Measuring Performance (Recall)

Measuring Productivity Outcomes (Latency Improvement)

High Priority Configuration of Training Scheme

Training Split / Window Size

Downsampling and Upweighting

Dataset

Results

Summary of Results

RQ1: What is the performance of Speculative Cycles with Transition Prediction in terms of percentage of novel breakages detected (recall) in comparison to a baseline randomized testing approach?

RQ2: Speculative Cycles compete with full testing cycles for machine resources, what configurations of Speculative Cycles improve productivity outcomes for Google developers?

RQ3:What aspects of the model design most impact performance? Considered aspects include features selected, super-extreme class imbalance corrections, and training environment.