systematic-debugging-skill

Systematic Debugging

4-phase root cause debugging: understand bugs before fixing.

Skill metadata

Source	Bundled (installed by default)
Path	skills/software-development/systematic-debugging
Version	1.1.0
Author	Hermes Agent (adapted from obra/superpowers)
License	MIT
Platforms	linux, macos, windows
Tags	debugging, troubleshooting, problem-solving, root-cause, investigation
Related skills	test-driven-development, plan, subagent-driven-development

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

Core principle: ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

Violating the letter of this process is violating the spirit of debugging.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven’t completed Phase 1, you cannot propose fixes.

When to Use

Use for ANY technical issue:

• Test failures
• Bugs in production
• Unexpected behavior
• Performance problems
• Build failures
• Integration issues

Use this ESPECIALLY when:

• Time pressure (emergencies tempt guessing)
• “Just a quick fix” seems obvious
• Already tried multiple fixes
• Last fix didn’t work
• Don’t fully understand the problem

NEVER skip this when:

• Problem seems simple (simple bugs have root causes too)
• Time pressure (rushing leads to rework)
• Someone demands an immediate fix (systematic is faster than thrashing)

The Four Phases

Each phase MUST be completed before moving to the next.

---

Phase 1: Root Cause Investigation

Before attempting ANY fixes:

1. Read the error message carefully

• Do NOT skip errors or warnings
• They often contain the exact solution
• Read the full stack trace
• Note line numbers, file paths, error codes
• Action: Use read_file on relevant source files. Use search_files to find error strings in the codebase.

2. Reproduce the issue reliably

• Find the minimal steps to trigger the bug
• Automate reproduction with a test if possible
• If not reproducible, you don’t understand the bug
• Action: Use terminal to run commands, tests, and interact with the system.

3. Check recent changes

• What changed recently? (code, config, data, environment)
• Use git log, git blame, git diff
• Revert changes one by one to isolate the cause
• Action: Use terminal for git commands.

4. Gather evidence

• Logs, metrics, system state, network traffic
• Use print statements, debugger, monitoring tools
• Avoid assumptions; rely on facts
• Action: Use terminal to inspect logs, run tcpdump, strace, etc. Use read_file to examine log files.

5. Trace data flow

• Follow the data from input to output
• Identify where the data becomes incorrect or unexpected
• Draw diagrams if it helps visualize complex systems
• Action: Use read_file to follow code paths. Use search_files to find where variables are defined and modified.

6. Formulate a hypothesis

• Based on evidence, propose a specific root cause
• It must explain ALL observed symptoms
• It must be testable
• Action: Write down your hypothesis clearly.

7. Test the hypothesis

• Design a minimal experiment to confirm or deny your hypothesis
• Change only ONE variable at a time
• If hypothesis is wrong, return to step 4 (gather more evidence)
• Action: Use terminal to run targeted tests or modify code temporarily.

---

Phase 2: Pattern Identification

Once the root cause is understood:

1. Find working examples

• Locate similar code, configurations, or data that work correctly
• Look for established patterns, best practices, or library usage
• Action: Use search_files to find similar code. Use web_search to find documentation or examples.

2. Compare and contrast

• Identify the exact differences between the broken and working examples
• The root cause should align with these differences
• Action: Use read_file to compare files side-by-side.

3. Understand the pattern

• Why does the working example work?
• What is the underlying principle or design choice?
• Action: Use web_search to research the pattern or principle.

---

Phase 3: Solution Hypothesis

With root cause and pattern understood:

1. Propose a fix

• Based on the identified pattern, design a solution that addresses the root cause
• The fix should bring the broken system in line with the working pattern
• Action: Describe the proposed code changes or configuration adjustments.

2. Predict the outcome

• Clearly state what you expect to happen after applying the fix
• How will it resolve the root cause and symptoms?
• Action: Write down your prediction.

3. Test the fix (mentally or with a small experiment)

• Before full implementation, consider edge cases and potential side effects
• Run a quick, isolated test if feasible
• Action: Use terminal for quick tests.

---

Phase 4: Implementation and Verification

Only after a solid solution hypothesis:

1. Write a regression test

• Create an automated test that specifically reproduces the bug
• This test should FAIL before your fix (RED state)
• Action: Use write_file to create a new test file or modify an existing one. Use terminal to run the test.

2. Apply the fix

• Implement the proposed solution
• Ensure it only addresses the root cause, not just symptoms
• Action: Use write_file to modify the source code.

3. Verify the fix

• Run the regression test; it should NOW PASS (GREEN state)
• Run all other relevant tests (unit, integration, end-to-end) to ensure no new bugs were introduced
• Action: Use terminal to run tests.

4. Clean up

• Remove any temporary debugging code, print statements, or test files
• Commit the fix and the new regression test together
• Action: Use delete_file or write_file to clean up. Use terminal for git commit.

5. Reflect and document

• What did you learn?
• Document the bug, its root cause, and the solution
• Consider if architectural changes are needed to prevent similar bugs
• Action: Use write_file to update documentation or create a post-mortem.

---

Anti-Patterns (When to STOP and return to Phase 1)

• “I’ll just try this one thing” (without understanding root cause)
• “Skip the test, I’ll manually verify”
• “It’s probably X, let me fix that”
• “I don’t fully understand but this might work”
• “Pattern says X but I’ll adapt it differently”
• “Here are the main problems: [lists fixes without investigation]”
• Proposing solutions before tracing data flow
• “One more fix attempt” (when already tried 2+)
• Each fix reveals a new problem in a different place

ALL of these mean: STOP. Return to Phase 1.

If 3+ fixes failed: Question the architecture (Phase 4 step 5).

Common Rationalizations

Excuse	Reality
”Issue is simple, don’t need process”	Simple issues have root causes too. Process is fast for simple bugs.
”Emergency, no time for process”	Systematic debugging is FASTER than guess-and-check thrashing.
”Just try this first, then investigate”	First fix sets the pattern. Do it right from the start.
”I’ll write test after confirming fix works”	Untested fixes don’t stick. Test first proves it.
”Multiple fixes at once saves time”	Can’t isolate what worked. Causes new bugs.
”Reference too long, I’ll adapt the pattern”	Partial understanding guarantees bugs. Read it completely.
”I see the problem, let me fix it”	Seeing symptoms ≠ understanding root cause.
”One more fix attempt” (after 2+ failures)	3+ failures = architectural problem. Question the pattern, don’t fix again.

Quick Reference

Phase	Key Activities	Success Criteria
1. Root Cause	Read errors, reproduce, check changes, gather evidence, trace data flow	Understand WHAT and WHY
2. Pattern	Find working examples, compare, identify differences	Know what’s different
3. Hypothesis	Form theory, test minimally, one variable at a time	Confirmed or new hypothesis
4. Implementation	Create regression test, fix root cause, verify	Bug resolved, all tests pass

Hermes Agent Integration

Investigation Tools

Use these Hermes tools during Phase 1:

• search_files — Find error strings, trace function calls, locate patterns
• read_file — Read source code with line numbers for precise analysis
• terminal — Run tests, check git history, reproduce bugs
• web_search/web_extract — Research error messages, library docs

With delegate_task

For complex multi-component debugging, dispatch investigation subagents:

delegate_task(
    goal="Investigate why [specific test/behavior] fails",
    context="""
    Follow systematic-debugging skill:
    1. Read the error message carefully
    2. Reproduce the issue
    3. Trace the data flow to find root cause
    4. Report findings — do NOT fix yet
 
    Error: [paste full error]
    File: [path to failing code]
    Test command: [exact command]
    """,
    toolsets=['terminal', 'file']
)

With test-driven-development

When fixing bugs:

1. Write a test that reproduces the bug (RED)
2. Debug systematically to find root cause
3. Fix the root cause (GREEN)
4. The test proves the fix and prevents regression

Real-World Impact

From debugging sessions:

• Systematic approach: 15-30 minutes to fix
• Random fixes approach: 2-3 hours of thrashing
• First-time fix rate: 95% vs 40%
• New bugs introduced: Near zero vs common

No shortcuts. No guessing. Systematic always wins.