gcommon

module
v1.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 25, 2025 License: MIT

README ΒΆ

Table of Contents

πŸ“š Essential Documentation

New Users Start Here

For Developers

οΏ½ Overview

GCommon is a comprehensive library of common Go modules designed for building robust, scalable applications. It provides consistent APIs across authentication, caching, configuration, database operations, health checking, logging, metrics, message queuing, and web services. This update introduces a security module for audits, policies, monitoring, and tools.

πŸš€ Quick Start

Installation

go get github.com/jdfalk/gcommon

Build Tools

This project uses the Copilot Agent Utility for reliable build operations and VS Code task integration.

The preferred build tool is the Rust implementation, which provides memory safety, robust error handling, and comprehensive logging:

# Install the Rust version
cargo install copilot-agent-util

# Create symlink for easy access (copilot-agent-utilr)
sudo ln -s ~/.cargo/bin/copilot-agent-util /usr/local/bin/copilot-agent-utilr

# Verify installation
copilot-agent-utilr --version

The Rust version supports all operations including:

  • Protocol buffer generation: copilot-agent-utilr buf generate
  • Git operations: copilot-agent-utilr git add .
  • Linting: copilot-agent-utilr linter clippy
  • Formatting: copilot-agent-utilr prettier rustfmt
Go Version (Fallback)

If the Rust version is unavailable, you can use the Go implementation:

# Install the Go version
go install github.com/jdfalk/copilot-agent-util/cmd/copilot-agent-util@latest

# Verify installation
copilot-agent-util --version

VS Code tasks are configured to use the Rust version (copilot-agent-utilr) by default, with automatic fallback to the Go version if needed.

Basic Health Check Example

package main

import (
    "context"
    "log"

    "github.com/jdfalk/gcommon/pkg/health"
)

func main() {
    // Create a health provider
    provider := health.NewProvider()

    // Register a simple check
    provider.RegisterCheck("database", func(ctx context.Context) health.Result {
        // Your database health check logic here
        return health.Result{
            Status:  health.StatusHealthy,
            Message: "Database connection OK",
        }
    })

    // Check health
    result := provider.Check(context.Background(), "database")
    log.Printf("Health check result: %s - %s", result.Status, result.Message)
}

gRPC Metrics Interceptor Example

server := grpc.NewServer(
    grpc.UnaryInterceptor(
        middleware.UnaryServerMetrics(middleware.GRPCMetricsOptions{Provider: metricsProvider}),
    ),
)

Database gRPC Service Example

db, _ := sqlite.Open(sqlite.Config{Path: "app.db"})
grpcServer := grpc.NewServer()
db.GRPCService().Register(grpcServer)

Multi-Module Example

package main

import (
    "github.com/jdfalk/gcommon/pkg/health"
    "github.com/jdfalk/gcommon/pkg/metrics"
    "github.com/jdfalk/gcommon/pkg/log"
)

func main() {
    // Initialize health checking
    healthProvider := health.NewProvider()

    // Initialize metrics (75% complete)
    metricsProvider := metrics.NewPrometheusProvider()

    // Initialize logging (50% complete)
    logger := log.NewZapLogger()

    // More modules coming soon...
    // Database, Cache, Config, Auth, Queue, Web modules in development
}

πŸ“¦ Current Module Status (August 2025 - MAJOR UPDATE)

πŸš€ MASSIVE PROTOBUF MILESTONE ACHIEVED: All modules now have complete protobuf structure following 1-1-1 pattern

Module Proto Files Completion Ready for Production gRPC Services Recent Progress
Config 155 βœ… 100% πŸ”„ Proto Complete πŸ”„ In Progress βœ… Complete 1-1-1 migration (155 files)
Queue 216 βœ… 100% πŸ”„ Proto Complete πŸ”„ In Progress βœ… Complete 1-1-1 migration (216 files)
Metrics 172 βœ… 100% πŸ”„ Proto Complete πŸ”„ In Progress βœ… Complete 1-1-1 migration (172 files)
Auth 172 βœ… 100% πŸ”„ Proto Complete πŸ”„ In Progress βœ… Complete 1-1-1 migration (172 files)
Web 224 βœ… 100% πŸ”„ Proto Complete πŸ”„ In Progress βœ… Complete 1-1-1 migration (224 files)
Cache 72 βœ… 100% πŸ”„ Proto Complete πŸ”„ In Progress βœ… Complete 1-1-1 migration (72 files)
Health 35 βœ… 100% βœ… Yes βœ… Complete βœ… Complete and production-ready
Common 40 βœ… 100% βœ… Yes βœ… Complete βœ… Foundation types complete
Database 52 βœ… 100% βœ… Yes βœ… Complete βœ… Complete and production-ready
Log 14 βœ… 100% βœ… Yes βœ… Complete βœ… Minimal implementation complete
Organization 80 βœ… 100% πŸ”„ Proto Complete πŸ”„ In Progress βœ… Team and tenant management
Notification 22 βœ… 100% πŸ”„ Proto Complete πŸ”„ In Progress βœ… Notification delivery system

πŸ“Š TOTAL: 1,254 protobuf files implemented (up from ~754)

⚠️ Development Status:

  • Protobuf Layer: 100% complete across all modules (MAJOR MILESTONE!)
  • gRPC Services: 3 modules production-ready (Health, Database, Common), others in progress
  • Go Implementations: Varies by module, focusing on service layer completion

πŸŽ‰ Recent Achievements

August 2025: MASSIVE PROTOBUF IMPLEMENTATION MILESTONE βœ…

πŸš€ UNPRECEDENTED ACHIEVEMENT: Completed comprehensive 1-1-1 pattern implementation across ALL modules

Key Accomplishments:

  • 1,254+ Protobuf Files: Massive expansion from ~754 to 1,254+ individual files following 1-1-1 pattern
  • 100% Module Coverage: All 12 modules now have complete protobuf structure
  • Automated Tooling: Created split_proto.py for automated proto file splitting
  • Complete Documentation: Comprehensive implementation guide in PROTO_SPLITTING_GUIDE.md

Module Implementation Results:

  • Config Module: 155 proto files (split from 7 large files)
  • Queue Module: 216 proto files (most complex module)
  • Metrics Module: 172 proto files (full metrics system)
  • Auth Module: 172 proto files (comprehensive auth system)
  • Web Module: 224 proto files (largest module)
  • Cache Module: 72 proto files (caching infrastructure)
  • Organization Module: 80 proto files (team/tenant management)
  • Notification Module: 22 proto files (notification system)

Impact: This represents the largest single development milestone in the project's history, establishing a complete protobuf foundation for all planned services.

Previous Milestones

June 8, 2025: Complete Protobuf Implementation Roadmap βœ…

We've completed comprehensive analysis and created a detailed implementation plan for all 754 protobuf files:

πŸš€ Ready for Implementation:

  • πŸ“Š Total Coverage: 754 protobuf files analyzed and tracked
  • πŸ“‹ GitHub Issues: 39 detailed implementation issues created covering 625 empty files
  • 🎯 Implementation Plan: Detailed priority order and workflow established
  • βœ… 100% Issue Coverage: Every empty protobuf file has a corresponding GitHub issue
  • πŸ“ Open Issues: 58 currently active

Current Module Status (June 8, 2025):

  • βœ… Common Module: 100% complete (40/40 files) - Shared Types Foundation
  • βœ… Database Module: 100% complete (52/52 files) - Gold Standard Reference
  • βœ… Log Module: 100% complete (1/1 files) - Minimal Implementation
  • πŸ”„ Auth Module: 13.5% complete (17/126 files) - 109 files need implementation
  • πŸ”„ Cache Module: 18.2% complete (8/44 files) - 36 files need implementation
  • πŸ”„ Config Module: 13.0% complete (3/23 files) - 20 files need implementation
  • βœ… Health Module: 100% complete (16/16 files) - Stable and production-ready
  • ❌ Metrics Module: 2.1% complete (2/97 files) - 95 files need implementation
  • ❌ Queue Module: 1.1% complete (2/177 files) - 175 files need implementation
  • ❌ Web Module: 1.1% complete (2/178 files) - 176 files need implementation

Ready to Start:

  • πŸ“ GitHub Project: GCommon Development Board
  • πŸ“‹ Next Step: Start with Issue #67 (Protobuf Validation Pipeline)
  • 🎯 Priority Order: Metrics β†’ Queue β†’ Web β†’ Auth β†’ Cache β†’ Config β†’ Health
  • Priority 3: Add request metadata to all services
  • Priority 4: Enable gRPC services across all modules

Critical Finding: The protobuf migration from monolithic to 1-1-1 structure revealed 631 total types requiring implementation - significantly more extensive than initially estimated.

πŸ”§ Installation & Setup

Prerequisites

  • go 1.23+ (required)
  • Protocol Buffers compiler (protoc) - for gRPC development
  • Git - for version control

Quick Setup

# Clone the repository
git clone https://github.com/jdfalk/gcommon.git
cd gcommon

# Install dependencies
go mod download

# Generate protobuf code (if developing)
./generate.sh

# Run tests
go test ./...

# Try the health example
cd examples/health && go run .

Development Tools (Optional)

# Install protobuf tools for gRPC development
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

# Install mockery for testing
go install github.com/vektra/mockery/v2@latest

πŸ€– Automated Issue Management

This repository uses GitHub Actions for automated issue management. When working on features or bugs, always update issue status to maintain accurate project tracking.

Issue Status Updates

To programmatically update GitHub issues, create an issue_updates.json file in the repository root and push to main:

[
  {
    "action": "create",
    "title": "New Feature",
    "body": "Description",
    "labels": ["enhancement"]
  },
  {
    "action": "update",
    "number": 42,
    "state": "closed",
    "labels": ["completed"]
  },
  { "action": "update", "number": 43, "assignees": ["username"] }
]

Supported Actions:

  • create - Create new issues with title, body, labels, assignees
  • update - Modify existing issues by number (state, labels, assignees, etc.)
  • delete - Remove issues (use sparingly)

Required Workflow:

  1. πŸš€ Start Work: Assign yourself to the issue and move to "In Progress"
  2. πŸ”„ During Work: Update issue with progress comments as needed
  3. βœ… Complete Work: Close issue and move to "Done" with completion summary

Example Workflow

# When starting work on issue #68
echo '[{"action": "update", "number": 68, "assignees": ["your-username"], "labels": ["in-progress"]}]' > issue_updates.json
git add issue_updates.json && git commit -m "Start work on issue #68" && git push

# When completing work
echo '[{"action": "update", "number": 68, "state": "closed", "labels": ["completed"]}]' > issue_updates.json
git add issue_updates.json && git commit -m "Complete issue #68: Metrics Messages" && git push

The GitHub Actions workflow automatically processes these updates on every push to main.

πŸ—οΈ Architecture Overview

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚   Go Interfaces β”‚    β”‚ gRPC Services   β”‚    β”‚ Common Types    β”‚
β”‚                 β”‚    β”‚                 β”‚    β”‚                 β”‚
β”‚ β€’ Clean APIs    │◄──►│ β€’ Protobuf      │◄──►│ β€’ Shared Models β”‚
β”‚ β€’ Provider      β”‚    β”‚ β€’ Cross-languageβ”‚    β”‚ β€’ Consistency   β”‚
β”‚   Pattern       β”‚    β”‚ β€’ Streaming     β”‚    β”‚ β€’ Validation    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Key Principles:

  • Interface-First Design: Clean Go interfaces before implementation
  • Protocol Buffers Foundation: All services defined using protobuf
  • Dual API Support: Both Go interfaces and gRPC for maximum flexibility
  • Common Types: Shared definitions for consistency across modules
  • Direct Proto Imports: Import specific proto files directly (v0.3.0+) instead of aggregator files

πŸ”„ Protobuf Import Strategy (BREAKING CHANGE in v0.3.0)

Current (Deprecated):

// Aggregator file imports (will be removed)
import "pkg/auth/proto/auth.proto";  // Brings in everything via import public

New (Recommended):

// Direct imports for explicit dependencies
import "pkg/auth/proto/messages/user.proto";
import "pkg/auth/proto/requests/login_request.proto";
import "pkg/auth/proto/responses/login_response.proto";

Benefits:

  • Explicit dependencies (follows Go philosophy)
  • Better IDE support and autocomplete
  • Reduced compilation overhead
  • No need for Go type aliases
  • Cleaner generated code

🀝 Contributing

We welcome contributions! Current priority areas:

  1. Completing Metrics Module (75% β†’ 100%)
  2. Finishing Logging Module (50% β†’ 100%)
  3. Documentation improvements
  4. Example applications

See our contribution guidelines for code style, testing requirements, and development workflow.

πŸ“„ License

MIT License - see LICENSE file for details.

πŸ†˜ Need Help?

The updated setup script automatically authenticates using gh auth token and links open issues by module labels.

GitHub Projects

This repository now uses a dedicated GitHub Project board for each major module (Metrics, Queue, Web, Auth, Cache, Config). Run the unified project manager to create these boards and automatically add issues by label:

python3 /path/to/ghcommon/scripts/unified_github_project_manager_v2.py

See scripts/MIGRATION-NOTICE.md for migration details.

Recent Updates

  • Added standardized LogEntry struct for consistent structured logging across modules

  • Implemented initial auth configuration and API key messages

  • Logging module migrated to 1-1-1 structure with 10 new protobuf files

  • Implemented initial metrics protobufs

  • July 22, 2025: Cache module complete

  • Updated config module progress

  • Added DebugInfo message for advanced debugging

  • Implemented TransactionService and MigrationService protobufs

  • Queue module progress: implemented acknowledgment messages and types (approx. 6% complete) Queue module progress updated: implemented listing and pull protobufs (approx. 10% complete)

  • Implemented core web protobufs

  • July 21, 2025: Web module protobufs implemented (178 files)

  • Updated auth module progress to 24/126 implemented

  • July 21, 2025: Database GRPCService Implementation Complete (#132)

  • SQLite and CockroachDB drivers now expose GRPCService() for gRPC server registration

Progress

Module Completion Ready for Production gRPC Services Recent Progress
Health βœ… 100% βœ… Yes βœ… Complete βœ… Complete 1-1-1 migration (36 types)
Common βœ… 100% βœ… Yes βœ… Complete βœ… 40 shared types implemented
Database βœ… 100% βœ… Yes βœ… Complete βœ… Complete 1-1-1 migration (52 types)
Log βœ… 100% βœ… Yes βœ… Complete βœ… Minimal logging implementation
Auth πŸ”„ 13.5% ⚠️ Partial πŸ”„ Partial πŸ”„ 17/126 files implemented
Cache πŸ”„ 18.2% ❌ No πŸ”„ In Progress πŸ”„ 8/44 files implemented
Config πŸ”„ 13.0% ❌ No ❌ Planned πŸ”„ 3/23 files implemented
Notification βœ… 100% βœ… Yes βœ… Complete βœ… Notification service implemented
Metrics πŸ”„ 2.1% ❌ No πŸ”„ In Progress ❌ 95/97 files need implementation
Queue πŸ”„ 1.1% ❌ No ❌ Planned ❌ 175/177 files need implementation
Web πŸ”„ 1.1% ❌ No ❌ Planned ❌ 176/178 files need implementation

πŸ“¦ Current Module Status

Module Completion Ready for Production gRPC Services Recent Progress
Health βœ… 100% βœ… Yes βœ… Complete βœ… Complete 1-1-1 migration (36 types)
Common βœ… 100% βœ… Yes βœ… Complete βœ… 40 shared types implemented
Database βœ… 100% βœ… Yes βœ… Complete βœ… Complete 1-1-1 migration (52 types)
Log βœ… 100% βœ… Yes βœ… Complete βœ… Minimal logging implementation
Auth πŸ”„ 13.5% ⚠️ Partial πŸ”„ Partial πŸ”„ 17/126 files implemented
Cache βœ… 100% βœ… Yes βœ… Complete βœ… 44/44 files implemented
Config πŸ”„ 13.0% ❌ No ❌ Planned πŸ”„ 3/23 files implemented
Notification πŸ”„ 25% ❌ No ❌ Planned πŸ”„ Initial message types defined
Metrics πŸ”„ 2.1% ❌ No πŸ”„ In Progress ❌ 95/97 files need implementation
Queue πŸ”„ 1.1% ❌ No ❌ Planned ❌ 175/177 files need implementation
Web πŸ”„ 1.1% ❌ No ❌ Planned ❌ 176/178 files need implementation

⚠️ Development Status: This project is under active development. Only the Health, Common, Database, and Cache modules are production-ready.

Implementation Status

Current Module Status (June 8, 2025):\n\n- βœ… Common Module: 100% complete (40/40 files) - Shared Types Foundation\n- βœ… Database Module: 100% complete (53/53 files) - Gold Standard Reference (QueryRow RPC implemented)\n- βœ… Log Module: 100% complete (1/1 files) - Minimal Implementation\n- πŸ”„ Auth Module: 13.5% complete (17/126 files) - 109 files need implementation\n- πŸ”„ Cache Module: 18.2% complete (8/44 files) - 36 files need implementation\n- πŸ”„ Config Module: 13.0% complete (3/23 files) - 20 files need implementation\n- βœ… Health Module: 100% complete (16/16 files) - Stable and production-ready\n- ❌ Metrics Module: 2.1% complete (2/97 files) - 95 files need implementation\n- ❌ Queue Module: 1.1% complete (2/177 files) - 175 files need implementation\n- ❌ Web Module: 1.1% complete (2/178 files) - 176 files need implementation

| Notification | βœ… 100% | βœ… Yes | βœ… Complete | βœ… Notification service implemented |

Updated queue module progress to reflect new protobuf implementations MySQL protobuf messages added

July 28, 2025 - Metrics module progress updated: 33 of 97 protobufs implemented

July 28, 2025 - Metrics module progress updated: 33 of 97 protobufs implemented

Common module protobufs verified complete (July 28, 2025).

It now supports a plugin architecture for custom integrations. Provides SDK, security features, and example plugins for extensibility.

Dependency Management

This project includes automated dependency auditing and optimization scripts.

Documentation

Comprehensive documentation for all modules is available under the docs/ directory, featuring module overviews, API references, guides, examples, tutorials, and reference materials.

SDK Generation\n\nInitial scaffolding for multi-language client SDK generation. Further implementation required.

Examples and Tutorials\n\nTODO: Add content for this section

Client SDKs\n\nTODO: Add content for this section

Deployment

Advanced Deployment\n\n- Helm Charts: Configurable Helm chart in deploy/helm\n- Automation: GitOps and CI/CD templates in deploy/automation\n- Monitoring: Prometheus, logging, and tracing configs in deploy/kubernetes/monitoring

Error Handling\n\nInitial error handling framework with standardized error codes.

Monitoring and Observability\nThis module provides metrics, logging, tracing, alerting, and dashboards for system health.

Add dependency audit utility with license checks.

Integration Testing\nA new integration testing framework validates module interactions and performance.

  • Added fully functional in-memory metrics provider supporting counters, gauges, histograms, summaries, and timers Added preliminary CI/CD pipeline infrastructure skeleton

API Documentation\n\nRun python scripts/api_doc_generator.py to generate HTML, Markdown, PDF, OpenAPI specs, Postman collections, and playground configs for all modules.

API Documentation\n\nAutomated documentation generation and interactive explorer are available under scripts/api_doc_generator.py and scripts/api_explorer.py.

gRPC Client Resilience\n\nCombined retry, circuit breaking, and fallback utilities are available via the resilience interceptor. See pkg/grpc/client for details. (Refs #882)

CI/CD Pipeline\n\nComprehensive pipeline with multi-stage testing, quality gates, automated releases, environment management, and reporting.

Error Handling\nStandardized errors across modules with codes, wrapping, and monitoring.

Monitoring\nComprehensive collectors, alerts, exporters, and dashboards provide full observability.

🀝 Community Guidelines

Please read our Code of Conduct to understand expected behavior.

Auth Module Docs\n\nComprehensive authentication module documentation is available under docs/modules/auth.

Documentation\n\nInitial module documentation skeleton is available under docs/modules/.

CI/CD Pipeline\n\nThis project now includes a security scanning workflow that runs go vet and npm audit during continuous integration.

CI/CD Pipeline\n\nThis project includes an advanced CI/CD pipeline with multi-stage testing, quality gates, automated releases, and environment deployments.

Microservice Templates\n\nUse scripts/generate-microservice.sh to scaffold a new service from templates like basic-api-service or worker-service.

Notification module service layer implemented with multi-channel gRPC services.

Deployment Templates\n\nTODO: Add content for this section

Integration Testing\n\nRun go test ./test/integration/... to execute integration tests across all modules.

Security Audit\n\nProvides audit tools, security policies, monitoring, and cryptographic utilities enabling comprehensive protection across auth, web, and queue modules.

Web Module\n\nTODO: Add content for this section

  • Added gRPC server and client for database migrations \n## Documentation System\nPlaceholder for module documentation system.

Performance Testing\nThe perf/ directory contains benchmarking, load, stress, and regression tools for all modules.

Microservice Templates\nTemplates for common service types are now available with a generation CLI.

Security Module\nProvides auditing, policies, monitoring, and tools for secure operations.

Directories ΒΆ

Path Synopsis
internal module
pkg
authpb module
common module
commonpb module
config module
health module
healthpb module
metrics module
organization module
queue module
web module
sdks
go/v1/common
go/v1/config
go/v1/database
go/v1/health
go/v1/media
go/v1/metrics
go/v1/organization
go/v1/queue
go/v1/web
go module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.