How to Validate a COBOL Migration with AI

Validating a COBOL migration, or a migration of any other legacy system, means verifying that the migrated system preserves the behavior of the original system. One strategy for doing this is to use the original system as the specification: capture how it behaves, repeat the same executions on the migrated system, and analyze every difference.

In systems with millions of lines of code, this is only feasible if you break the problem into more manageable parts and build a risk-based plan. We use AI to work at this scale on legacy code, and we validate the results with the people who know the business.

This challenge is going to become more and more common, because COBOL (like other legacy platforms) is still at the center of many businesses. According to IBM, it processes 95% of ATM transactions and underpins more than 40% of online banking systems. On top of that, an estimated 220 billion lines of COBOL code are still in production. Even many fintechs with modern front-end applications run on a COBOL core behind the scenes.

I've been working in testing for more than 20 years, and I worked on legacy migrations long before generative AI. In this article, I share how we approach this challenge today at Abstracta, with examples from an ongoing project: the cloud migration of a Latin American fintech's core system, written in COBOL and with more than 8 million lines of code.

These are six approaches we use to validate a COBOL migration:

1. Use the original COBOL system as the specification.
2. Understand undocumented code with the help of AI agents.
3. Observe the original system at runtime.
4. Test at different levels of the system, for example, with specific validations for data migration.
5. Define a different strategy for each component based on its risk.
6. Review what AI generates at scale and validate it with the business.

The Example: An 8-Million-Line COBOL Migration¶

The fintech needed to migrate its core system because the version of the platform it runs on was approaching the end of its support period. To avoid operating on technology with no backing in case of future problems, the company decided to move the system to the cloud. The project is planned to last two years and is already past the halfway mark.

Due to the limitations of the language and the underlying platform, the system is poorly modularized and carries three decades of changes and patches. Documentation is scarce or outdated, and many of the people who took part in building it are no longer on the team, so it's common to find portions of code, or entire modules, whose implementation nobody understands.

On top of that, the system processes money movements in a regulated environment, so any change requires a high level of control. What's at stake is business continuity, since everything runs on this core system.

Our role is to support the testing of the migration. The main challenge isn't in development, but in validating that the migrated system keeps the same behavior as the current one and that critical processes keep working correctly.

Why Is Testing the Bottleneck When Migrating COBOL?¶

With all that code in production, the big question is how to migrate it without putting the business at risk.

With AI assistants and agents, rewriting COBOL code in a new architecture takes much less effort than before. The challenge then shifts to testing: validating that the migrated system preserves the behavior of the original and produces the same results.

In February 2026, Anthropic published an article on COBOL modernization with Claude Code and a code modernization guide. After reading it closely, I was left with the impression that it describes a simpler scenario than the ones we find in real systems.

Having a Claude Code license and following that guide may be enough for a system of a few thousand lines. But COBOL systems that have been in production for decades tend to hold business knowledge in enormous volumes of code. At that scale, migration is no longer just a rewriting problem but an engineering one: how to structure the process, how to break it into manageable parts, and how to reduce risk.

More Than 90% of the Project Was Testing¶

More than ten years ago, I worked on one of my first large migration projects. A Latin American supermarket chain, which had started as a small store 25 years earlier, had 10 locations and wanted to open its 11th. But its RPG system used a single-digit store identifier. Changing that field affected the database structure and, from there, processes such as inventory management, money movements, and account balances.

The company had no documentation, no test cases, and no established testing practice, and the people who had built the system had retired, left, or passed away. All that remained was the code and the users. For a year and a half, we interviewed key users to document how the system worked and what behavior they expected from it, rebuilt the specifications, and wrote tests. The code change took just a month and a half. More than 90% of the project was testing.

The project was a success, but that ratio explains why migrations like that one were uncommon. Today, with AI further reducing the effort needed to modify or rewrite code, many migration projects are likely to follow the same pattern: less time spent on development and much more effort devoted to validating that the system keeps working as it should.

Technical Debt and Cognitive Debt¶

Technical debt appears when we solve something in a way that later makes the system harder to maintain or modify: lack of documentation, automated tests, modularization, or other decisions that create future costs.

Cognitive debt, a concept put forward by researcher Margaret-Anne Storey, is a knowledge debt: not understanding how a part of the code works or why it was built that way. I've seen it many times in teams that say, "nobody touches that code because nobody knows why it works." In those cases, even the business rule that originally led to writing it that way may have been lost.

In a COBOL migration, both debts coexist. The challenge isn't only transforming old code, but rebuilding enough knowledge about the system to be able to validate that the new one behaves the same. In a 30-year-old system, cognitive debt usually outweighs technical debt, and that's where AI contributes the most: more than writing new code, it helps build understanding of the old code.

Approach 1: Use the Original COBOL System as the Specification¶

In a COBOL migration, the goal of testing is to verify equivalence: that the migrated system behaves the same as the original. In new development, we look for software that is correct according to what the business expects. In a migration, the legacy system already supports a working business, so its current behavior becomes the specification and the test oracle.

When I talk about legacy code, I like to go back to a 2004 reference book, Working Effectively with Legacy Code, by Michael Feathers. Feathers defines legacy code as code without tests: if something changes, there's no way to validate that the behavior is still what the business needs. For those cases, he proposes the characterization testing approach.

Applied to a COBOL migration, it works like this:

1. Run functional and automated tests that cover as many paths of the original system as possible.
2. Save the output of those executions.
3. Repeat the same executions on the migrated system and save its output.
4. Compare both outputs and analyze every difference, because that's where a migration error may be.

In this approach, it doesn't matter whether the original output is correct. What matters is detecting where the new system behaves differently. In my view, this is the approach to follow in any modernization of this kind.

There's no magic here: what we do is use AI to scale a pattern that Feathers described more than 20 years ago. I like to sum it up like this: AI does the checking, and people do the testing, with the information AI helps them obtain.

To generate that coverage, we work along two complementary tracks, functional and automation, with two perspectives: a static one, on the code, and a dynamic one, on the running system. Approaches 2 and 3 explain each perspective.

Approach 2: Understand COBOL Code Without Documentation¶

When a COBOL system has no up-to-date documentation, the most reliable source becomes the code itself. A friend from college used to say that the best documentation of how a system works is its own source code, because it's the only documentation that's always up to date. In these migrations, it's clear he was right. The difficulty lies in finding what you need within millions of lines, and that's exactly what AI agents make feasible.

An agent that analyzes the entire repository lets you go from the general to the specific and generate different artifacts depending on what you need to understand:

• Architecture and component diagrams, to get a first picture of the system.
• Pseudocode or sequence diagrams, to understand a flow end to end.
• State machines, to follow the lifecycle of a business entity.
• Flowcharts, to understand the logic and detect edge cases.
• Documentation, test cases, and test data derived from the code itself.

That said, I believe the most valuable part isn't the diagrams, but having a knowledge base that the testing team can query in natural language, with business questions such as "How is interest calculated on an overdue loan?"

For automated testing, the same static analysis generates dependency maps and call graphs, from which unit, API, UI, and performance tests are derived. Performance deserves its own attention, because it's another major risk in this type of migration.

A simple way to look at it: the static approach gives you the map, and the dynamic one, which we'll see in Approach 3, confirms that the system still follows it.

How We Do It with Tero¶

At Abstracta, we use Tero, our open-source agent harness for building specialized AI agents that operate with context and governance at scale.

In the latest Abstracta Tech Talk, I showed an agent connected to an open-source repository written in RPG, code I wasn't familiar with, from a system for managing employees and customers.

First, I asked it for an architecture diagram, then to explain what the application does and what its database structure is, including the data types of each attribute. Then I asked how an employee is added: it returned the business rules, the files involved, and the pseudocode for that logic. Since I found the pseudocode complex, I asked it to represent it as a flowchart.

Tero can be deployed in each organization's private cloud or on its own on-prem infrastructure, and it works with the LLM that the organization chooses: models from OpenAI or Anthropic, or even an open-source model installed locally, as one of our clients already does. This way, the system's information stays in a controlled environment.

Through MCP or the APIs available in Tero, agents connect to the tools the team already uses: the code repository, task and documentation managers such as Jira, Confluence, Linear, or Redmine, test management tools such as PractiTest or Zephyr, observability sources (metrics, logs, and traces), and databases. For legacy systems, we also built an MCP that connects to an AS400.

Approach 3: Observe the COBOL System at Runtime¶

There's a lot of talk about the context an AI agent needs to do its job well. There's a lot of talk about the context an AI agent needs to do its job well.need it too, and observability is one of the best ways to give it to them: it lets you see what the original system does behind each action, something the code alone doesn't show, and the screen people use to interact with the system shows even less.

In functional testing, the idea is for the tester to explore the original system alongside a copilot that explains, in natural language, what the backend is doing:

• Which endpoints were called and which records were modified.
• Which jobs were queued and what the trace of each one is.
• Which state changes occurred.
• Which internal errors occurred, even if they aren't shown to the user.

This way, the tester sees logs, traces, and queries without leaving the test, and no longer has to guess what's happening behind the scenes. Together with a partner, we developed a copilot of this kind for an e-banking system, and then applied the same idea in other environments. Behind the scenes, it connects to the logs and observability tools that already existed; what's new is that everything can now be queried in natural language, which opens it up for a wider audience to access (for example, testers, analysts, or PMs with no programming experience).

In automated testing, observability plays a different role: it shows what coverage the tests are achieving and makes it possible to keep improving them.

Can You Have Observability on a Mainframe?¶

Yes, and it's nothing new. About 15 years ago, I came across CA Wily (now DX APM, from Broadcom), one of the first modern observability tools I knew of, which could already connect to mainframes. Its founder later created New Relic (I found that out because he told me himself when I met him at an event in 2016). I remember conversations and working sessions with AS400 administrators and RPG or COBOL developers who showed me how they achieved full traceability, from the user's action to the record that was modified in the database.

More modern platforms, such as Datadog, have focused on cloud and Kubernetes architectures and aren't always as compatible with legacy platforms. Even so, one way or another, you can rebuild the traceability between a user's action and the data it ended up modifying in the database.

Approach 4: Test at Different Levels of the System¶

At the start of the current project with the fintech I mentioned earlier, the first approach seemed simple: read the code, generate unit tests with 100% coverage, run them, and call the job done. But it wasn't that easy.

We analyzed the call tree to test from the leaves upward and mock the dependencies that had already been tested. Several limitations appeared right away. Running the test for a single file takes about a minute, so a basic set for more than 9,000 files would take around 9,000 minutes, and those tests are run many times throughout the project. Unit testing and coverage measurement tools for COBOL are often restricted by licenses or compiler limitations, and each new run requires restoring a database, which takes about 5 hours. In addition, unit tests need well-defined units, and in a monolith like this one, that means first finding its seams (I come back to this in Approach 5).

The Second Approach: Testing from the Entry Points¶

That's why we changed strategy and moved automation up to the level of the DCL scripts, which automate system workflows, and the API wrappers, which are the entry points closest to integration and to the user. This took us from more than 9,000 files to more than 750 entry points. That's still a significant volume, so we prioritized based on usage logs and interviews with key users. One of the first findings was that a high percentage of files hadn't been used in more than a year or had duplicate behavior.

We don't work directly on the COBOL code either. From the code analysis, we generate an intermediate Markdown format that explains each feature: inputs, outputs, variables, possible paths, and edge cases. That format serves two purposes: the LLM uses it to generate the automated tests, and the testing team uses it to validate the behavior with the business owners. The generated tests go through a code review process that combines automated review with a review by one or two people.

This approach has its costs too. We lose some control over code coverage, but we gain alignment with business priorities. And keep in mind that bugs already in production will show up in the characterization tests, because the original system is the reference. It's best to decide in advance what to do with them. Finally, a reminder that applies to any migration: don't forget or underestimate performance.

Characterization Testing at the Data Layer Too¶

As an additional check, we also cover the data layer. The fintech's code has more than 40,000 embedded SQL statements, and the idea is to have automated tests that run those queries against the original database and compare the results with those from the migrated database. A copy of production is used for this. The comparison covers both the data and the response times.

All of this is part of an iterative approach. The goal is to improve the migration tool in early stages: run small tests on its output, detect what doesn't work, and fix it. The final migration will include the full test run, and the idea is to reach that moment with fewer surprises.

Approach 5: Define a Different Strategy for Each Component Based on Its Risk¶

A large COBOL system can't be migrated or tested as a single unit. You have to break it into parts and define a strategy for each one based on its risk. We call these parts slices: they can be vertical business units or technical layers, and each one has its own diagnosis, its own controls, and its own evidence.

What Usually Doesn't Work¶

In many migrations, I see two recurring problems. The first is that the diagnosis is descriptive rather than operational: "legacy" is used as a synonym for "old," a label that measures nothing. Risk is estimated by intuition, and decisions about scope and speed are made based on perception rather than evidence. The second is that execution has no evidence-based gates: migrations are planned as a big bang, there are no clear measurements of test coverage, and rollback procedures are missing.

A Six-Phase Methodology¶

To organize that work, we use a six-phase methodology on each slice:

1. Discovery and baseline: build a verifiable description of the system as it is today.
2. Migration strategy: decide what changes, how, and in what order.
3. Behavior control: build the safety net before changing anything.
4. Implementation: make the change gradually, improve the migration script with early tests, and run the final tests before going to production.
5. Go-live and rollout: expand traffic as evidence builds up, with rollback ready.
6. Stabilization and handover: close the loop with the operations team.

In the strategy phase, three decisions are made for each slice: how to partition it and in what order to proceed; what changes (infrastructure, runtime, architecture, data, vendor, or integrations); and with which transition mode, such as coexistence, strangler, branch by abstraction, or expand-contract. We go deeper into this methodology in the article on legacy modernization with AI.

How to Measure the Risk of Each Slice¶

The diagnosis of each slice is based on eight dimensions that indicate how capable that part of the system is of producing verifiable evidence:

Each dimension is rated on a maturity scale from 0 to 4: nonexistent, initial or isolated, defined, managed, and optimized. Based on that assessment, each slice falls into a risk class:

• Critical: one or more dimensions at level 0. There's no safety net to start migrating.
• High: critical dimensions at level 1, or two support dimensions at level 0. A lot of preparation is needed.
• Medium: critical dimensions at level 2 and no support dimension at level 0. The migration is feasible with targeted reinforcements.
• Low: critical dimensions at level 3 or higher. The migration moves forward with incremental gates.

Here's what it looks like for a fictional system being planned for migration, with the scores for the three critical dimensions (Dimensions 1, 3, and 7):

It's the same system, with four slices that have different risk profiles and a strategy for each one. From there, you can define the deployment strategy and the roadmap.

The Monolith Paradox¶

In a highly interdependent monolith, a paradox appears: can you refactor with confidence, without tests, to make the system testable? To isolate a component and migrate it first, you have to modify the monolith, and to modify it without losing behavior, you need tests that don't exist yet. The key is finding the cracks in the monolith, the points where you can separate a part and move forward little by little.

In my experience, risk-based planning is one of the parts left out when migration is framed in an extremely simplified way as giving the code to an LLM, generating the tests, and running them.

Approach 6: Review What AI Generates at Scale and Validate It with the Business¶

Everything AI generates in a COBOL migration, from a diagram or pseudocode to an automated test, needs a verification scheme designed by the team. Models keep getting better, but they still hallucinate.

Two Teams, Two Levels¶

At the fintech, we apply characterization testing at two levels, and each one has its own team:

• Functional team (end to end): documents the system's behavior, interviews users, validates and updates the documentation, and designs risk-based test cases. What we get from querying the code with an LLM isn't taken at face value: it serves as a starting point for conversations with the people who know the business, who are the ones who confirm whether it makes sense.
• Technical team: analyzes the code and logs, understands the stack's limitations (which automation frameworks exist, how to measure coverage, what observability is available), works with dependency maps, execution trees, data access, and integrations, and generates and validates the automated tests, prioritized according to the system's actual usage.

Reviewing Thousands of Generated Tests¶

Among the tests AI generates, some have no assertions or very weak assertions, others add mocks right on the part you want to test, and others use hardcoded data. With thousands of tests, reviewing them one by one is no longer humanly possible.

I think that's where we need to step up a level: in addition to designing tests, design a scalable scheme that gives us peace of mind about the coverage and quality of the tests we create. For that, I go back to the fundamentals, such as heuristics. A question as simple as "Do all these tests have at least one assertion?" already helps detect some of the ones we need to fix, and from there, you need to dig into what makes a test good in each system. It's a challenge we're still figuring out.

Testing the Agents Too¶

An agent is a piece of software, so it needs tests too. In Tero, we automate them with an LLM-as-a-judge approach. That way, if a faster or cheaper model comes along, you can make the switch and verify that the agent still works as expected.

Where Doesn't AI Save You in a COBOL Migration?¶

AI reduces time and costs in many parts of a COBOL migration, but some challenges still depend on management and on people's judgment. It's best to plan for them from day one:

If an AI tells you your migration is "fully tested," are you really going to hit the deploy button? I don't think we're ready for that today. The decision to turn off the old system and turn on the new one is still a human one, and the more visible the risks and their mitigations are to whoever makes it, he more informed the risk the organization takes on.

The risk of a migration is determined by your ability to verify. If you can't validate behavior in the new environment, you can't govern the migration.

That's why, in my view, the underlying problem in many migrations is epistemological: where the knowledge of the system lives and where we get it from.

About the Author¶

Federico Toledo is co-founder and Chief Quality Officer of Abstracta, holds a PhD in software testing, and has worked in the testing industry for more than 20 years.

He is the author of the first testing book originally written in Spanish, now available for free in both Spanish and English, and he is a community builder: for many years he was part of the organizing team of Testing Uy, he created the Quality Sense podcast and the Quality Sense Conf, and he takes part in initiatives such as TAPIA (Workshop on Testing and Artificial Intelligence) and WOPR (Workshop on Performance and Reliability).

About Abstracta¶

With nearly 2 decades of experience and a global presence, Abstracta is a technology company that helps organizations deliver high-quality software faster by combining AI-powered quality engineering with deep human expertise.

Our expertise spans across industries and complex delivery environments. That’s why we’ve built robust partnerships with industry leaders, such as Microsoft, Datadog, Tricentis, Perforce BlazeMeter, Sauce Labs, and PractiTest.