AI Code Review for QA Engineers

Why production code and test code must be reviewed together

Most code review tools treat production code and test code as separate concerns. Linters check production code for style and potential bugs. Coverage tools measure how much production code is exercised by tests. But neither approach asks the question that matters most to QA engineers: does this codebase have the right tests, written the right way, to catch the failures that will actually occur in production?

A function with 100% line coverage can still harbour critical bugs if the test only exercises the happy path. A test suite that runs in under a minute might achieve speed by mocking everything, testing only that mocks return what they were told to return. An integration test covering a critical workflow might pass in CI but fail under load because it depends on database state that other tests also modify.

Reviewing production code and test code together reveals relationships that neither review catches alone. A complex branching function with only one test case. A critical business rule validated by a test that asserts against a hardcoded value rather than deriving the expected result. An error handling path that is tested in isolation but never exercised through the actual entry point that users encounter. These gaps are architectural, not syntactic, and they require understanding both sides of the codebase simultaneously.

Testability patterns that static analysis cannot evaluate

Testability is a property of production code, not test code. A class that instantiates its own dependencies internally cannot be tested in isolation. A function that reads from the file system, calls an external API, and writes to a database in a single method cannot have its logic tested without either mocking everything or spinning up real infrastructure. A module that relies on global mutable state forces tests to execute in a specific order or risk interference.

These testability problems are invisible to linters because the production code is syntactically valid and functionally correct. The issue is structural: the code works but cannot be verified efficiently. A QA engineer reviewing the code would immediately identify that a critical payment processing function has no seam for injecting a test double for the payment gateway. A linter sees a well-typed function with no unused variables.

Temporal coupling is another testability concern that requires cross-file analysis. When function A must be called before function B, but this ordering is not enforced by the type system, tests must replicate the exact sequence to exercise the code correctly. If the ordering changes during a refactor, tests break in ways that look like test failures rather than production code changes. The root cause is in production code while the symptom appears in test code.

Flaky tests, weak assertions, and test architecture decay

Flaky tests are the most corrosive problem in any test suite. A test that passes 95% of the time trains the team to ignore failures. When a real regression causes that same test to fail, the instinct is to re-run rather than investigate. The causes of flakiness are varied and cross-cutting: timing-dependent assertions, shared database state between tests, network calls to external services, race conditions in asynchronous code, and reliance on system clock values that vary between environments.

Assertion quality is equally important and harder to measure. A test that asserts expect(result).toBeTruthy() when it should assert a specific value provides a false sense of coverage. Snapshot assertions on large JSON responses break on any change regardless of relevance. Asserting on collection length rather than contents will pass even when the wrong items are returned.

Test architecture decay manifests as the ratio between test types shifts over time. Applications often start with a reasonable balance of unit tests for business logic, integration tests for service interactions, and end-to-end tests for critical user flows. As the codebase evolves, developers add tests at whatever level is most convenient rather than most appropriate. The result is a test suite where simple validation logic is tested through full HTTP request cycles, while complex multi-service orchestration has only unit tests that mock every dependency.

Test data management is a related dimension of decay. Hardcoded fixture files created years ago promote brittle tests that pass for the wrong reasons. When test data does not reflect production reality – unicode characters, null fields, timestamps in different time zones – the test suite validates an idealised version of the application rather than the real one.

How VibeRails reviews codebases from a QA perspective

VibeRails performs a full-codebase scan using frontier large language models. Every production file and test file is analysed together – not just the test suite in isolation, but the relationship between what the code does and how it is verified. Configuration files, CI pipeline definitions, and test infrastructure are included in the scan.

For QA-focused reviews specifically, the AI reasons about:

• Testability gaps – production code with hard-wired dependencies that prevent test isolation, functions with no injection seams, modules with global mutable state, and temporal coupling that creates implicit test ordering requirements
• Test coverage quality – high-coverage code with only happy-path tests, critical error handling paths without corresponding test cases, business rules validated against hardcoded expected values, and boundary conditions that are never exercised
• Flaky test patterns – timing-dependent assertions, shared mutable test state, tests that depend on network availability or system clock, non-deterministic ordering in collection assertions, and sleep-based synchronisation in async tests
• Assertion quality – overly broad assertions that pass when they should fail, snapshot tests on volatile data structures, assertions that check existence but not correctness, and missing assertions on error conditions and side effects
• Test architecture balance – simple logic tested through expensive integration paths, complex orchestration covered only by heavily mocked unit tests, missing end-to-end coverage for critical user flows, and test pyramid inversions
• Test data and fixture quality – stale fixture files that no longer reflect production schemas, factory patterns that produce unrealistic data, missing edge-case coverage for unicode, nulls, time zones, and numeric boundaries, and shared test data that creates hidden coupling between test files

Each finding includes the file path, line range, severity, category, and a detailed description explaining why the pattern is problematic and how to address it. Findings are organised into 17 categories so teams can filter and prioritise by area of concern.

Cross-validation for quality judgements

QA concerns often involve judgement rather than clear-cut rules. A heavily mocked test might be the only practical approach for a service that depends on five external APIs. A flaky test might be flaky because the underlying code has a genuine race condition, not because the test is poorly written.

VibeRails supports running reviews with two different AI backends – Claude Code and Codex CLI – in sequence. The first pass discovers potential issues, the second verifies them using a different model architecture. When both models independently flag the same testability gap or assertion weakness, confidence is high. Disagreements highlight pragmatic trade-offs rather than genuine quality risks, reducing noise and letting QA engineers focus their limited time on the findings that genuinely matter.

From findings to a more testable codebase

After triaging findings, VibeRails can dispatch AI agents to implement fixes directly in your local repository. For QA-focused improvements, this typically means introducing dependency injection seams in production code, strengthening weak assertions with specific value checks, replacing timing-dependent test logic with deterministic alternatives, isolating shared test state, updating stale fixtures to match current schemas, and restructuring tests to match the appropriate level of the testing pyramid.

Each fix is generated as a local code change you can inspect, test, and commit or discard. The AI works within the conventions of your existing codebase, matching your project's testing framework, assertion style, and test organisation patterns – whether you use Jest, pytest, RSpec, JUnit, or any other testing framework.

VibeRails runs as a desktop app with a BYOK model – it orchestrates Claude Code or Codex CLI installations you already have. No code is uploaded to VibeRails servers. AI analysis is sent directly to the provider you configured, billed to your existing subscription. Per-developer pricing: $19/month or $299 lifetime, with a free tier of 5 issues per session to evaluate the workflow.