[2025] Pinpointing the Learning Obstacles of an Interactive Theorem Prover

in collaboration with: Andy Zaidman Jesper Cockx

Interactive theorem provers (ITPs) are programming languages which allow users to reason about and verify their programs. Although they promise strong correctness guarantees and expressive type annotations which can act as code summaries, they tend to have a steep learning curve and poor usability. Unfortunately, there is only a vague understanding of the underlying causes for these problems within the research community. To pinpoint the exact usability bottlenecks of ITPs, we conducted an online survey among 41 computer science bachelor students, asking them to reflect on the experience of learning to use the Agda ITP and to list the obstacles they faced during the process. Qualitative analysis of the responses revealed confusion among the participants about the role of ITPs within software development processes as well as design choices and tool deficiencies which do not provide an adequate level of support to ITP users. To make ITPs more accessible to new users, we recommend that ITP designers look beyond the language itself and also consider its wider contexts of tooling, developer environments, and larger software development processes.

ICPC '25

Pre-Print

[2024] How Novices Perceive Interactive Theorem Provers

Interactive theorem provers (ITPs) are known to have a steep learning curve and poor usability. This hinders their spread into commercial software development, wasting their potential to improve software quality. To understand what makes them inaccessible to novices, we conducted an online survey among bachelor students, asking them to list the obstacles they encountered while learning Agda. Analysis of the results revealed design choices and tool deficiencies which do not provide an adequate level of support to beginner nor advanced users. These observations point to one prominent point of improvement: providing a more accessible and sturdy infrastructure for ITP programmers.

ICFP SRC '24
TyDe '24

Extended Abstract TyDe Talk Poster Slides Haskell Interlude

[2023] Bringing Formal Verification into Widespread Programming Language Ecosystems (MSc Thesis)

Formal verification is a powerful tool for ensuring program correctness but is often hard to learn to use and has not yet spread into the commercial world. This thesis focuses on finding an easy-to-use solution to make formal verification available in popular programming language ecosystems. We propose a solution where users can write code in an interactive theorem prover and then transpile it into a more popular programming language. We use Agda2HS as a case study to determine what challenges users find in using such a tool, improve selected features, and then conduct a user study to evaluate the usability. We find that detailed documentation, support for commonly-used features in the target programming language, features that facilitate verification, integration of the tool into the target ecosystem, and user studies are necessary for the accessibility of such a tool.

TU Delft

Repository

[2021] Validating Type Checkers Using Property-Based Testing (BSc Thesis)

Manually testing definitional interpreters and their type checkers is a tedious and error-prone process which can largely benefit from automation. This study evaluates the effectiveness of property-based testing on errors in type checkers. Metrics used include the ability to catch different types of errors as well as the ability to provide a good margin of confidence in the results. We define two languages in Haskell and identify properties which should hold for their type checkers. Using a bottom-up approach to expression generation, we evaluate the effectiveness of property-based testing on test suites of type checkers with various types of errors. The method proves to be effective for both the simple and the complex language and manages to catch all defined error types with at least one property to a sufficient degree of confidence. We conclude that property-based testing is an effective tool to help with manual grading of type checkers for definitional interpreters.

TU Delft

Repository