ProgullProgull
The Solution

An accelerator for your L2/L3 team — not a replacement.

Progull is being built as an agentic layer that gives existing mainframe operations teams faster diagnosis, safer remediation and a compounding playbook library. Autonomy is opt-in per abend family, and every action has an audit trail your change board can review.

The problem

Batch abends still run on human reflexes.

Mainframe incident management is mostly people, pagers and PDFs. Tickets sit open while engineers extract logs, pattern-match error codes and decide what to restart.

Manual triage at 3am

Operators page subject-matter experts, hunt logs across SDSF and JES spool, and reassemble context before a single fix can begin.

Tribal knowledge bottlenecks

Recovery depends on a small set of veterans who know the JCL, the DB2 quirks and the historical workarounds.

Recovery measured in hours

Every minute of a stuck overnight batch ripples into SLAs, downstream apps and missed business cut-offs.

Tiered Autonomy Model

Three tiers. Autonomy is earned, per abend family.

No L2 or L3 team hands over the wheel on day one. Every abend family starts at Advisory and only moves forward when the evidence in the audit trail supports it.

Advisory

The agent detects the abend, reasons over it and produces an explainable recommendation. No action is taken on z/OS. This is the default tier for every new customer and every new abend family.

Approval

The agent proposes a specific remediation and waits for a named human to click execute inside ServiceNow. Every action is logged with the reviewer's identity.

Autonomous

The agent detects, decides and acts within policy — reserved for low-risk, high-confidence abend families that have graduated out of the Approval tier with a clean track record.

Hybrid engine

Central orchestrator, domain-specific agents.

A single monolithic 'mainframe AI' doesn't survive contact with a real z/OS estate. Progull is built as a centralized orchestration layer that routes each incident to the right domain agent, then combines the result behind a risk-based human-in-the-loop gate.

Orchestrator

Receives events from the Detection layer, classifies scope, and routes to the right domain agent. Combines partial recommendations into a single proposed action for the approval gate.

JCL Agent

Owns job-control and step-level abend families — S0C7, SB37, U4038. Reasons over JCL, PROCs, dataset state and step return codes.

Db2 Agent

Reasons about SQL-911, deadlocks, buffer-pool contention and SQLCODE-driven abends. Scoped corpus, narrow surface.

CICS Agent

Covers region-level anomalies, transaction abends and connectivity failures across CICS regions.

Every recommendation flows through a risk-based approval gate before any z/OS action. Low-risk actions in Autonomous mode execute directly. Anything above the threshold routes to the Approval tier and waits.

Inside each agent

A four-phase reasoning loop.

Every domain agent implements the same typed contract. Each phase hands off to the next through a signed, audit-logged transition.

Detection

Subscribes to JES spool, SYSLOG and OPERLOG events. Classifies abend codes the moment they appear and opens a ServiceNow incident with first-line context.

Reasoning

Fuses SYSUDUMP, JCL, recent change events, DB2 SQLCODE and historical resolutions into an explainable root-cause narrative.

Remediation

Selects a policy-approved playbook — resubmit step, recycle CICS region, hold downstream job — and stages it for the appropriate autonomy tier.

Closure

Validates recovery, attaches the full reasoning trail and evidence pack to the incident, and closes it in ServiceNow.

Guardrails

Autonomy with an off-switch on every step.

Guardrail

Policy-as-code

Every remediation maps to a versioned, reviewable playbook. Nothing executes outside the approved set.

Guardrail

Human-in-the-loop modes

Run agents in shadow, recommend, or auto-execute — per job class, per environment, per time window.

Guardrail

Full reasoning trace

Inputs, intermediate thoughts and chosen actions are persisted with the incident for audit and learning.

The reasoning loop

Observe → Reason → Act → Verify → Learn.

Every incident traverses the same typed loop. Each transition is logged, signed, and replayable against the original evidence pack.

  1. STEP 01

    Observe

    Subscribe to SYSLOG, JES spool and CICS events. Classify the abend within seconds.

  2. STEP 02

    Reason

    Fuse SYSUDUMP, JCL, SQLCODE and recent change context into an explainable hypothesis.

  3. STEP 03

    Act

    Select a policy-approved playbook. Execute under a named surrogate ID with pre-flight checks.

  4. STEP 04

    Verify

    Confirm RC=00, dataset shape and downstream invariants before declaring recovery.

  5. STEP 05

    Learn

    Promote new patterns into KB candidates. SMEs approve before they become playbooks.

Playbook anatomy

Every remediation is versioned, reviewable code.

A Progull playbook is not a free-form LLM instruction. It is a typed manifest your change board approves once and the agent executes the same way every time.

Trigger conditions

Abend code, job class, LPAR, time window and confidence threshold required to match.

Pre-flight checks

Dataset state, downstream holds, change-freeze windows and dependent jobs verified before any action.

Action set

An ordered list of typed primitives — submit, hold, release, recycle — under a named surrogate ID.

Post-conditions

RC, row counts, dump absence and CICS region health re-checked before declaring recovery.

Failure escalation

If any check fails the playbook stops, opens a Sev-2 worknote and pages the assignment group.

playbook · PB-MF-014APPROVED · v3
name: pb-mf-014-s0c7-payroll
match:
  abend: S0C7
  job_class: PAYROLL
  confidence_min: 0.85
preflight:
  - assert dataset(PAY.OUT).extents_remaining > 2
  - assert job(GLFEED01).state == WAITING
actions:
  - hold     job: GLFEED01
  - submit   job: PAYRUN02
             step: STEP0040
             parm: "CLEAN"
             surrogate_id: PROGULL.PROD.PAY
postcheck:
  - assert step.RC == 0
  - assert dataset(PAY.OUT).row_count > 100000
on_failure:
  escalate: assignment_group=MF-PAYROLL-SRE
Operating modes

Autonomy is a dial, not a switch.

Pick the mode per environment, per job class and per time window. Promote forward only when your operators trust the trail.

TIER 01

Advisory

Default tier. Agents observe and produce a full reasoning trail. Zero action taken on z/OS. Every abend family starts here.

TIER 02

Approval

Agents propose a specific remediation. A named human approves the action inside ServiceNow before it executes.

TIER 03

Autonomous

Reserved for low-risk families that have graduated out of Approval with a clean track record. Agents detect, decide and act within policy.

Failure modes

What happens when the agent is wrong.

Failure handling is designed in. The agent never silently retries; every escalation lands in ServiceNow with the same trail your auditor consumes.

Low-confidence diagnosis

Below the configured threshold, the agent halts at Recommend and pages the assignment group with the evidence pack — never executes.

Pre-flight check fails

The playbook stops on the first failed assertion, records which assertion failed and opens a Sev-2 worknote for human triage.

Post-condition fails

If RC, row counts or region health do not return as expected, the agent does not declare recovery and re-opens the incident with full diff.

Policy not matched

If no approved playbook matches, the agent writes a recommendation only. New playbooks always require a human change request.

Repeated abend pattern

After N occurrences of the same abend in a window, the agent stops auto-resolving and escalates for SME review — the playbook is no longer the answer.

Operator override

Operators can revoke an in-flight action via a single click in ServiceNow. The agent will not retry the same action without a fresh approval.