Skip to main content

Testing Guide for Claude Code CLI Automation

This guide provides multiple testing strategies to validate the automation system without merging to trunk.

🧪 Testing Strategies​

  1. Set up test environment variables (replace with your actual values):

    export JIRA_URL="https://superna.atlassian.net"
    export JIRA_EMAIL="your-email@superna.com"
    export JIRA_API_TOKEN="your-actual-token"
    export BITBUCKET_TOKEN="your-actual-token"

  2. Run the local test script:

    ./test-local.sh

  3. Test individual components:

    # Test environment validation
    node scripts/validate-environment.js

    # Test Claude Code CLI
    node scripts/claude-code-executor.js validate
    node scripts/claude-code-executor.js test --debug

    # Test JIRA integration (replace AI-78 with real issue)
    node scripts/jira-integration.js get-issue AI-78
    node scripts/jira-integration.js get-requirements AI-78

    # Test dry-run automation
    node scripts/claude-documentation-automation.js AI-78 --dry-run --debug

Strategy 2: Test Branch Pipeline (Safe - No Trunk Changes)​

  1. Create a test JIRA issue:

    • Create issue in your JIRA (e.g., TEST-123)
    • Add the auto-doc label
    • Add descriptive content for testing

  2. Create test branch locally:

    git checkout -b auto-doc/TEST-123
    git add .
    git commit -m "feat: Add Claude Code CLI automation system"
    git push origin auto-doc/TEST-123

  3. Monitor the pipeline:

    • Go to Bitbucket Pipelines
    • Watch the auto-doc/TEST-123 pipeline execution
    • Review logs in real-time
    • Check artifacts for debug information

  4. Review results:

    • Check if documentation was created in solutions/ folder
    • Review debug logs in artifacts
    • Check JIRA for automation comments

Strategy 3: Feature Branch Testing​

  1. Create a feature branch:

    git checkout -b feature/test-claude-automation
    git add .
    git commit -m "feat: Add Claude Code CLI automation system"
    git push origin feature/test-claude-automation

  2. Create PR to trunk (don't merge yet):

    • This triggers the regular pipeline
    • Tests markdown linting, spell checking
    • Validates the build process
    • Creates preview deployment

  3. Test auto-doc on feature branch:

    # From your feature branch
    git checkout -b auto-doc/TEST-124
    git push origin auto-doc/TEST-124

🔍 Debugging Tools​

1. Debug Logs​

The automation creates detailed debug logs saved as artifacts:

  • tmp/claude_debug_*.log - Contains Claude's analysis and operations

2. Validation Tools​

# Check environment setup
node scripts/validate-environment.js --test-connections

# Validate Claude Code CLI
node scripts/claude-code-executor.js validate

# Test JIRA connectivity
node scripts/jira-integration.js validate

3. Dry Run Mode​

Test without making changes:

node scripts/claude-documentation-automation.js ISSUE-KEY --dry-run --debug

🚨 Common Issues and Solutions​

Issue: "Claude Code CLI not found"​

Solution: Install via npm (requires Node.js 18+):

npm install -g @anthropic-ai/claude-code
claude --version

Issue: "Environment variables missing"​

Solution:

  • For pipeline: Configure Bitbucket Repository Variables
  • For local: Export environment variables or use test script

Issue: "JIRA API authentication failed"​

Solution:

  • Verify JIRA_EMAIL and JIRA_API_TOKEN are correct
  • Check if API token hasn't expired
  • Ensure using correct JIRA_URL

Issue: "Branch doesn't trigger pipeline"​

Solution:

  • Ensure branch name follows auto-doc/ISSUE-KEY format
  • Verify JIRA issue has auto-doc label
  • Check pipeline configuration in bitbucket-pipelines.yml

✅ Testing Checklist​

Pre-Testing Setup​

  • Bitbucket Repository Variables configured
  • Test JIRA issue created with auto-doc label
  • Claude Code CLI available (pipeline installs it)

Local Testing​

  • Environment validation passes
  • Claude Code CLI responds to test prompts
  • JIRA integration retrieves issue details
  • Placement logic generates reasonable paths
  • Dry-run mode completes without errors

Pipeline Testing​

  • Auto-doc branch triggers pipeline
  • Claude Code CLI installs successfully
  • Environment validation passes in pipeline
  • Documentation gets generated in correct location
  • Git commit and PR creation work
  • JIRA gets updated with progress
  • Debug logs are captured in artifacts

Integration Testing​

  • Generated documentation follows existing patterns
  • Sidebar navigation gets updated correctly
  • Markdown syntax is valid
  • File paths follow naming conventions
  • Content quality meets standards

🎯 Recommended Testing Flow​

  1. Start with Local Testing - Validate core functionality
  2. Test Individual Components - Ensure each script works
  3. Create Test Branch - Validate pipeline integration
  4. Review Generated Content - Check documentation quality
  5. Test Edge Cases - Different issue types, complex requirements
  6. Performance Testing - Large documents, multiple files

This approach lets you thoroughly test the system without affecting the main trunk branch.