api_spec_validation

README

API Spec Validation Framework

This framework provides comprehensive testing and validation of OpenAPI specifications against the actual Devtron server implementation. It helps ensure that API specs are accurate, up-to-date, and match the real server behavior.

Features

• Live Server Testing: Tests API specs against a running Devtron server
• Spec-Handler Comparison: Compares OpenAPI specs with REST handler implementations
• Parameter Validation: Validates request parameters, types, and requirements
• Response Validation: Validates response structures and status codes
• Comprehensive Reporting: Generates detailed reports with issues and recommendations
• Example Generation: Enhances specs with realistic examples from actual server responses

Quick Start

Prerequisites

1. Go 1.19 or later
2. A running Devtron server (default: http://localhost:8080)
3. Access to the Devtron codebase

Installation

# Navigate to the validation framework directory
cd tests/api-spec-validation

# Install dependencies
make deps

# Build the validator
make build

Basic Usage

# Run validation against local server
make test

# Run validation against custom server
make test-server SERVER_URL=http://your-server:8080

# Compare specs with REST handlers
make compare

# Run all validations
make all

Detailed Usage

Command Line Options

The validator supports the following command line options:

./bin/validator [options]

Options:
  --server string     Server URL to test against (default "http://localhost:8080")
  --specs string      Directory containing API specs (default "../../../specs")
  --output string     Output directory for reports (default "./reports")
  --token string      Authentication token (optional)
  --verbose           Enable verbose logging

Examples

# Test with authentication
./bin/validator --server=http://localhost:8080 --token=your-auth-token

# Test specific specs directory
./bin/validator --specs=/path/to/specs --output=/path/to/reports

# Verbose output for debugging
./bin/validator --verbose

Framework Architecture

Core Components

1. APISpecValidator: Main validation engine that tests specs against live server
2. SpecComparator: Compares OpenAPI specs with REST handler implementations
3. Test Runner: Orchestrates the validation process and generates reports

Validation Process

1. Spec Loading: Loads all OpenAPI 3.0 specs from the specified directory
2. Endpoint Discovery: Extracts all endpoints and operations from specs
3. Live Testing: Makes actual HTTP requests to the running server
4. Response Validation: Compares actual responses with spec expectations
5. Handler Comparison: Analyzes REST handler implementations for consistency
6. Report Generation: Creates comprehensive validation reports

Validation Types

1. Parameter Validation

• Required vs optional parameters
• Parameter types and formats
• Query, path, and header parameters
• Request body validation

2. Response Validation

• Status code validation
• Response body structure validation
• Content-Type validation
• Error response validation

3. Handler Comparison

• Missing handler implementations
• Parameter mismatches
• Request/response body handling
• Authentication requirements

Report Format

The framework generates detailed reports in Markdown format:

# API Spec Validation Report

Generated: 2024-01-15T10:30:00Z

## Summary
- Total Endpoints: 150
- Passed: 120
- Failed: 20
- Warnings: 10
- Success Rate: 80.00%

## Detailed Results

### ✅ GET /app-store/discover
- **Status**: PASS
- **Duration**: 150ms
- **Spec File**: specs/app-store.yaml
- **Response Code**: 200

### ❌ POST /app-store/install
- **Status**: FAIL
- **Duration**: 200ms
- **Spec File**: specs/app-store.yaml
- **Response Code**: 400

**Issues:**
- **PARAMETER_MISMATCH**: Required parameter 'appId' missing in request
- **RESPONSE_FORMAT_ERROR**: Response is not valid JSON

Configuration

Environment Variables

# Server configuration
DEVRON_SERVER_URL=http://localhost:8080
DEVRON_AUTH_TOKEN=your-token

# Test configuration
VALIDATION_TIMEOUT=30s
MAX_CONCURRENT_REQUESTS=10

Custom Test Data

Create test data files in testdata/ directory:

# testdata/app-store.yaml
test_apps:
  - name: "sample-app-1"
    id: 1
    version: "1.0.0"
  - name: "sample-app-2"
    id: 2
    version: "2.0.0"

Integration with CI/CD

GitHub Actions Example

name: API Spec Validation

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  validate-specs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Go
        uses: actions/setup-go@v3
        with:
          go-version: 1.19
      
      - name: Start Devtron Server
        run: |
          # Start your Devtron server here
          docker-compose up -d
      
      - name: Run API Spec Validation
        run: |
          cd tests/api-spec-validation
          make test
      
      - name: Upload Reports
        uses: actions/upload-artifact@v3
        with:
          name: api-spec-reports
          path: tests/api-spec-validation/reports/

Jenkins Pipeline Example

pipeline {
    agent any
    
    stages {
        stage('Validate API Specs') {
            steps {
                script {
                    // Start Devtron server
                    sh 'docker-compose up -d'
                    
                    // Run validation
                    dir('tests/api-spec-validation') {
                        sh 'make test'
                    }
                    
                    // Archive reports
                    archiveArtifacts artifacts: 'tests/api-spec-validation/reports/**/*'
                }
            }
        }
    }
    
    post {
        always {
            // Cleanup
            sh 'docker-compose down'
        }
    }
}

Troubleshooting

Common Issues

1. Server Connection Failed

   Error: request failed: dial tcp localhost:8080: connect: connection refused
   

   Solution: Ensure Devtron server is running on the specified port

2. Authentication Errors

   Error: 401 Unauthorized
   

   Solution: Provide valid authentication token using --token flag

3. Spec Loading Errors

   Error: Invalid OpenAPI spec
   

   Solution: Validate your OpenAPI 3.0 specs using online validators

4. Timeout Errors

   Error: request failed: context deadline exceeded
   

   Solution: Increase timeout or check server performance

Debug Mode

Enable verbose logging for detailed debugging:

./bin/validator --verbose

This will show:

• Detailed request/response logs
• Spec parsing information
• Handler comparison details
• Performance metrics

Contributing

Adding New Validation Rules

1. Extend the ValidationIssue struct in framework.go
2. Add validation logic in the appropriate validation method
3. Update the report generation to include new issue types
4. Add tests for the new validation rules

Example: Adding Custom Validation

// In framework.go
func (v *APISpecValidator) validateCustomRule(result *ValidationResult, operation *openapi3.Operation) {
    // Your custom validation logic here
    if someCondition {
        result.Issues = append(result.Issues, ValidationIssue{
            Type:     "CUSTOM_RULE_VIOLATION",
            Message:  "Custom validation failed",
            Severity: "ERROR",
        })
    }
}

Best Practices

1. Regular Validation: Run validation tests regularly (daily/weekly)
2. CI/CD Integration: Include validation in your CI/CD pipeline
3. Spec Maintenance: Keep specs updated with code changes
4. Test Data: Use realistic test data for comprehensive validation
5. Documentation: Document any custom validation rules or configurations

Support

For issues and questions:

1. Check the troubleshooting section above
2. Review the generated reports for specific error details
3. Enable verbose logging for debugging
4. Create an issue in the Devtron repository

License

This framework is part of the Devtron project and follows the same license terms.

Documentation

Index

• type APISpecValidator
  • func NewAPISpecValidator(serverURL string, logger *zap.SugaredLogger) *APISpecValidator
  • func (v *APISpecValidator) BuildAndSetCookies(req *http.Request)
  • func (v *APISpecValidator) GenerateReport() string
  • func (v *APISpecValidator) GetResults() []ValidationResult
  • func (v *APISpecValidator) LoadSpecs(specsDir string) error
  • func (v *APISpecValidator) SetAdditionalCookie(name, value string)
  • func (v *APISpecValidator) SetCookiesFromString(cookieString string)
  • func (v *APISpecValidator) SetToken(token string)
  • func (v *APISpecValidator) ValidateAllSpecs() []ValidationResult
• type ComparisonIssue
• type ComparisonResult
• type ExampleData
• type FieldInfo
• type HandlerInfo
• type ParameterInfo
• type RequestBodyInfo
• type ResponseInfo
• type SpecComparator
  • func NewSpecComparator(logger *zap.SugaredLogger) *SpecComparator
  • func (sc *SpecComparator) CompareSpecsWithHandlers(specsDir, handlersDir string) ([]ComparisonResult, error)
• type SpecEnhancer
  • func NewSpecEnhancer(serverURL string, logger *zap.SugaredLogger) *SpecEnhancer
  • func (se *SpecEnhancer) EnhanceSpecs(specsDir, outputDir string) error
  • func (se *SpecEnhancer) SetAuthToken(token string)
• type ValidationIssue
• type ValidationResult

Types

type APISpecValidator

type APISpecValidator struct {
	// contains filtered or unexported fields
}

APISpecValidator handles validation of API specs against actual server implementation

func NewAPISpecValidator

func NewAPISpecValidator(serverURL string, logger *zap.SugaredLogger) *APISpecValidator

NewAPISpecValidator creates a new validator instance

func (*APISpecValidator) BuildAndSetCookies

func (v *APISpecValidator) BuildAndSetCookies(req *http.Request)

BuildAndSetCookies builds and sets the Cookie header similar to the curl request

func (*APISpecValidator) GenerateReport

func (v *APISpecValidator) GenerateReport() string

GenerateReport generates a comprehensive validation report

func (*APISpecValidator) GetResults

func (v *APISpecValidator) GetResults() []ValidationResult

GetResults returns all validation results

func (*APISpecValidator) LoadSpecs

func (v *APISpecValidator) LoadSpecs(specsDir string) error

LoadSpecs loads all OpenAPI specs from the specs directory

func (*APISpecValidator) SetAdditionalCookie

func (v *APISpecValidator) SetAdditionalCookie(name, value string)

SetAdditionalCookie sets an additional cookie for authentication

func (*APISpecValidator) SetCookiesFromString

func (v *APISpecValidator) SetCookiesFromString(cookieString string)

SetCookiesFromString parses a cookie string (like from curl -b flag) and sets the cookies Example: "_ga=GA1.1.654831891.1739442610; _ga_5WWMF8TQVE=GS1.1.1742452726.1.1.1742452747.0.0.0"

func (*APISpecValidator) SetToken

func (v *APISpecValidator) SetToken(token string)

SetToken sets the authentication token for requests

func (*APISpecValidator) ValidateAllSpecs

func (v *APISpecValidator) ValidateAllSpecs() []ValidationResult

ValidateAllSpecs validates all loaded specs against the server

type ComparisonIssue

type ComparisonIssue struct {
	Type    string
	Path    string
	Method  string
	Field   string
	Message string
}

ComparisonIssue represents a specific comparison issue

type ComparisonResult

type ComparisonResult struct {
	SpecFile string
	Issues   []ComparisonIssue
}

ComparisonResult represents the result of comparing a spec with handlers

type ExampleData

type ExampleData struct {
	Request  interface{} `json:"request,omitempty"`
	Response interface{} `json:"response,omitempty"`
	Status   int         `json:"status"`
}

ExampleData represents example data for an endpoint

type FieldInfo

type FieldInfo struct {
	Name     string
	Type     string
	Tag      string
	Required bool
}

FieldInfo represents a field in a struct

type HandlerInfo

type HandlerInfo struct {
	Package     string
	FileName    string
	HandlerName string
	Method      string
	Path        string
	Parameters  []ParameterInfo
	RequestBody *RequestBodyInfo
	Response    *ResponseInfo
}

HandlerInfo represents information about a REST handler

type ParameterInfo

type ParameterInfo struct {
	Name     string
	Type     string
	Location string // "query", "path", "header"
	Required bool
}

ParameterInfo represents a handler parameter

type RequestBodyInfo

type RequestBodyInfo struct {
	Type   string
	Fields []FieldInfo
}

RequestBodyInfo represents request body information

type ResponseInfo

type ResponseInfo struct {
	StatusCode int
	Type       string
	Fields     []FieldInfo
}

ResponseInfo represents response information

type SpecComparator

type SpecComparator struct {
	// contains filtered or unexported fields
}

SpecComparator compares API specs with REST handler implementations

func NewSpecComparator

func NewSpecComparator(logger *zap.SugaredLogger) *SpecComparator

NewSpecComparator creates a new spec comparator

func (*SpecComparator) CompareSpecsWithHandlers

func (sc *SpecComparator) CompareSpecsWithHandlers(specsDir, handlersDir string) ([]ComparisonResult, error)

CompareSpecsWithHandlers compares API specs with REST handler implementations

type SpecEnhancer

type SpecEnhancer struct {
	// contains filtered or unexported fields
}

SpecEnhancer enhances OpenAPI specs with realistic examples

func NewSpecEnhancer

func NewSpecEnhancer(serverURL string, logger *zap.SugaredLogger) *SpecEnhancer

NewSpecEnhancer creates a new spec enhancer

func (*SpecEnhancer) EnhanceSpecs

func (se *SpecEnhancer) EnhanceSpecs(specsDir, outputDir string) error

EnhanceSpecs enhances all specs in the given directory with examples

func (*SpecEnhancer) SetAuthToken

func (se *SpecEnhancer) SetAuthToken(token string)

SetAuthToken sets the authentication token for requests

type ValidationIssue

type ValidationIssue struct {
	Type     string // PARAMETER_MISMATCH, RESPONSE_MISMATCH, AUTH_ERROR, SCHEMA_MISMATCH
	Field    string
	Expected interface{}
	Actual   interface{}
	Message  string
	Severity string // ERROR, WARNING, INFO
}

ValidationIssue represents a specific validation problem

type ValidationResult

type ValidationResult struct {
	Endpoint   string
	Method     string
	Status     string // PASS, FAIL, SKIP
	Issues     []ValidationIssue
	Request    interface{}
	Response   interface{}
	StatusCode int
	Duration   time.Duration
	SpecFile   string // Added for reporting
}

ValidationResult represents the result of validating a single endpoint