← Back to Blog

Ten Questions to Ask Before Buying Any AI Mainframe Tool

The market is moving faster than the due diligence. Ask these questions before you sign anything.

The mainframe AI market is moving faster than the due diligence.

Vendors are announcing tools, demos are impressive, and procurement teams are under pressure to modernize. The problem is that a bad AI tool on mainframe does not fail loudly. It fails quietly – producing output that looks correct, passes review, and reaches production carrying problems that surface six months later.

The Ten Questions

1. Does the tool resolve copybooks before analyzing COBOL? Most AI tools analyze source code. COBOL source code without its copybooks is incomplete. If the tool cannot tell you how it handles COPY statements and external data structures, it is working with incomplete information and will not tell you that.
2. Does it understand the execution environment – not just the source? A COBOL program is one layer of a system. The JCL that runs it, the datasets allocated to each DD name, the CICS transaction that calls it, the DB2 plan it is bound to – these determine what the program actually does. Ask the vendor: what does your tool see beyond the source file?
3. How does it handle packed decimal arithmetic? COMP-3 packed decimal is one of the most common sources of subtle errors in COBOL migration and analysis. If the vendor does not immediately know what packed decimal is and how their tool handles it, walk away.
4. Can it trace inter-program dependencies? COBOL applications are rarely single programs. CALL statements, dynamic calls, LINK and XCTL in CICS – a complete picture requires the full dependency graph. Ask for a demonstration on a real multi-program application, not a demo dataset.
5. What is the false positive rate on its findings? Every AI analysis tool produces findings. The question is how many are wrong. A tool that flags ten critical issues when two are real creates more work than it saves. Ask for the false positive rate on a production codebase – not a synthetic benchmark.
6. How does it handle undocumented business logic? The most dangerous code in any mainframe application is the code that works for reasons nobody remembers. Comments are absent, the original developer retired, and the program has been patched over thirty years. Ask the vendor: how does your tool identify and flag undocumented logic? What does it do when it cannot determine intent?
7. What happens when the tool is wrong? It will be wrong. The question is whether you will know. Ask specifically: how does the tool communicate uncertainty? Does it tell you when it is extrapolating versus when it has high confidence? A tool that presents uncertain findings with the same confidence as certain ones is dangerous.
8. Has it been validated on real z/OS production workloads? Demos use curated datasets. Production systems have thirty years of technical debt, naming conventions from five different teams, and edge cases nobody anticipated. Ask for references from organizations running the tool on real production code – not pilot programs on isolated systems.
9. Who on your team understands mainframe well enough to review the output? This is the most important question – and it is about your organization, not the vendor. An AI tool produces output. A qualified human must review and validate that output before it influences any production decision. If you do not have that person, the tool creates risk, not value.
10. What is the exit strategy if the tool fails? Vendor lock-in is a known risk in enterprise software. On mainframe, it is amplified. If the tool produces artifacts – transformed code, documentation, analysis reports – ask who owns those artifacts and what happens if you stop using the tool. Your mainframe systems existed before this vendor. They need to be operable without them.

Related reading: What AI Can and Cannot Do with COBOL · Vibe Coding on Mainframe · We Are Building for the Enterprises That Stay