How to Evaluate Pyrefly in Your Python Project Before Dropping Pytype or Mypy

A practical guide to testing Pyrefly on your own code, measuring cold and warm times, and judging diagnostic quality before migrating.

Software engineering

Google has announced it is replacing its long-standing type checker Pytype with Pyrefly, an open-source tool that originated at Meta. According to Google's open-source blog, the reasons are increased speed, greater accuracy, and lower infrastructure costs.

However, the numbers from someone else's monorepo aren't your numbers. In this guide, we propose a simple method to compare Pyrefly with what you currently use in your own codebase, without relying on third-party benchmarks.

Inline diagnostics are part of what should be compared, in addition to speed.

Why it's worth trying

Pytype analyzed compiled bytecode instead of source code abstract syntax trees. Google explains that this made it increasingly expensive to keep up with new typing PEPs, because bytecode changes between Python versions.

Pyrefly, written in Rust, is about ten times faster than Pytype, according to Google. It reduces incremental rebuild latency by up to 98% and cuts maximum compute cost by over 80%. It also passes approximately 97% of the tests in the Python typing conformance suite. These are Google's internal figures: treat them as hypotheses to verify, not as guarantees.

What you need

• A Python project with type annotations, even if partial.
• An isolated virtual environment to avoid touching project dependencies.
• The checker you currently use (mypy, Pyright, or Pytype) for a fair comparison.
• A test branch and a stopwatch, for example, your shell's time command.

The code and documentation are available in the official repository. Refer to it for installation and current options, as specific commands may vary between versions.

Steps for comparison

1. Create a test branch and a clean virtual environment.
2. Install Pyrefly following the official documentation and verify that it starts.
3. Run your current checker with an empty cache and record the cold time.
4. Repeat the execution without changes to measure the warm time.
5. Do the same with Pyrefly, cold and warm, on exactly the same tree.
6. Modify a single file and measure incremental checking with each tool.
7. Review the diagnostics: how many are real errors, how much is noise, and how clear is the source of the failure.

# Indicative outline; adapt commands to your tool and version
rm -rf .mypy_cache
time <your-current-checker>   # cold
time <your-current-checker>   # warm

What to expect and what to watch out for

Don't expect all tools to agree on the same errors: each interprets some edge cases differently, so a migration might uncover new warnings or make others disappear. In the news comments, there was debate with Ty and mypy; one reader noted that mypy remains very competitive in incremental checking, and another, that Ty was not yet as mature as Pyrefly.

Tip Measure cold and warm times on YOUR codebase, and rate the quality of diagnostics alongside performance. A fast checker that drowns you in noise costs more time than a slower one that points to the actual failure. Meta claims that Pyrefly can check 1.8 million lines per second, which would allow checking on every keystroke; watch it in this video and see if that's noticeable in your editor.

Your own cold and warm measurements are worth more than any external benchmark.

Once you have the results, decide based on criteria that matter to your team: edit wait time, continuous integration cost, and message clarity. There's no universal shortcut; for a small project, the difference might not justify the change.

Opinion · My 12-month prediction

I see that Google has replaced Pytype with Pyrefly, and part of the reason is the cost of maintaining a bytecode-bound analyzer. In the next twelve months, I expect many large teams to try Pyrefly for the compute bill and the editing cycle, more than for saving a few seconds in continuous integration.

I bet that several checkers will coexist depending on the repository size, and that serious comparisons will be made with internal measurements, not headlines.

Related articles from LaRebelión

How to use AI to create a Linux driver from a Windows oneIntegrate DuckDB into Amazon Aurora PostgreSQL with This Technical GuideGuide: Prepare Your Android UI for AI Agents with Jetpack Compose

• Software engineering

Original source: infoq.com

Article generated with AI.larebelion

Kernel · Head of Technology · Spain

«A fast checker that drowns you in noise still costs more than a slow one that points to the actual failure.»