From c1bbea09fea5af6586bc6791a6298a557ad50f47 Mon Sep 17 00:00:00 2001 From: bpmcdevitt Date: Thu, 10 Jul 2025 16:23:36 -0500 Subject: [PATCH] update README --- README.md | 118 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 112 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 590fe1a..d4f1de5 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ An advanced automated platform that processes comprehensive CVE data and generat ### Intelligence Generation - **Enhanced SIGMA Rules**: Creates rules using real exploit indicators from curated PoCs +- **AI-Powered Rule Generation**: Multi-provider LLM integration (OpenAI, Anthropic, Ollama) +- **Local LLM Processing**: Built-in Ollama container for offline AI rule generation - **Quality Tiers**: Excellent, Good, Fair, Poor, Very Poor classification system - **Smart Template Selection**: AI-driven template matching based on PoC characteristics - **Advanced Indicator Extraction**: Processes, files, network, registry, and command patterns @@ -29,6 +31,7 @@ An advanced automated platform that processes comprehensive CVE data and generat - **Frontend**: React with Tailwind CSS - **Database**: PostgreSQL - **Cache**: Redis (optional) +- **LLM Engine**: Ollama (local models) with multi-provider support - **Containerization**: Docker & Docker Compose ## Quick Start @@ -70,6 +73,7 @@ docker-compose up -d --build - Frontend: http://localhost:3000 - Backend API: http://localhost:8000 - API Documentation: http://localhost:8000/docs + - Ollama API: http://localhost:11434 ### First Run @@ -99,6 +103,7 @@ curl -X POST http://localhost:8000/api/fetch-cves ### API Endpoints +#### Core Endpoints - `GET /api/cves` - List all CVEs - `GET /api/cves/{cve_id}` - Get specific CVE details - `GET /api/sigma-rules` - List all SIGMA rules @@ -106,15 +111,28 @@ curl -X POST http://localhost:8000/api/fetch-cves - `POST /api/fetch-cves` - Manually trigger CVE fetch - `GET /api/stats` - Get application statistics +#### LLM-Enhanced Endpoints +- `POST /api/llm-enhanced-rules` - Generate SIGMA rules using LLM AI analysis +- `GET /api/llm-status` - Check LLM API availability and configuration +- `POST /api/llm-switch` - Switch between LLM providers and models + ## Configuration ### Environment Variables +#### Core Configuration - `DATABASE_URL`: PostgreSQL connection string - `NVD_API_KEY`: Optional NVD API key for higher rate limits (5→50 requests/30s) - `GITHUB_TOKEN`: Optional GitHub personal access token for exploit analysis - `REACT_APP_API_URL`: Backend API URL for frontend +#### LLM Configuration +- `LLM_PROVIDER`: LLM provider selection (openai, anthropic, ollama) - **Default: ollama** +- `LLM_MODEL`: Model selection (provider-specific) - **Default: llama3.2** +- `OLLAMA_BASE_URL`: Ollama service URL - **Default: http://ollama:11434** +- `OPENAI_API_KEY`: Optional OpenAI API key for GPT models +- `ANTHROPIC_API_KEY`: Optional Anthropic API key for Claude models + ### GitHub Integration (Optional) For enhanced SIGMA rule generation with exploit analysis: @@ -141,8 +159,9 @@ Additional templates can be added to the database via the `rule_templates` table ## SIGMA Rule Generation Logic -The enhanced rule generation process: +The enhanced rule generation process supports multiple generation methods: +### Traditional Rule Generation 1. **CVE Analysis**: Analyzes CVE description and affected products 2. **GitHub Exploit Search**: Searches GitHub for exploit code using multiple query strategies 3. **Code Analysis**: Extracts specific indicators from exploit code: @@ -156,13 +175,25 @@ The enhanced rule generation process: 6. **MITRE ATT&CK Mapping**: Maps to specific MITRE ATT&CK techniques 7. **Confidence Scoring**: Higher confidence for exploit-based rules +### AI-Enhanced Rule Generation +1. **LLM Provider Selection**: Chooses between OpenAI, Anthropic, or local Ollama +2. **Contextual Analysis**: LLM analyzes CVE description and PoC code +3. **Intelligent Rule Creation**: AI generates sophisticated SIGMA rules with: + - Proper YAML syntax and structure + - Advanced detection logic + - Contextual field selection + - Relevant MITRE ATT&CK mappings +4. **Automatic Validation**: Generated rules are validated for syntax compliance +5. **Fallback Mechanism**: Falls back to template-based generation if LLM fails + ### Rule Quality Levels - **Basic Rules**: Generated from CVE description only - **Exploit-Based Rules**: Enhanced with GitHub exploit analysis (marked with 🔍) +- **AI-Enhanced Rules**: Generated using LLM analysis of PoC code (marked with 🤖) - **Confidence Ratings**: - - **High**: CVSS ≥9.0 + exploit analysis - - **Medium**: CVSS ≥7.0 or exploit analysis + - **High**: CVSS ≥9.0 + exploit analysis + AI enhancement + - **Medium**: CVSS ≥7.0 or exploit analysis or AI enhancement - **Low**: Basic CVE description only ### Template Matching @@ -172,8 +203,9 @@ The enhanced rule generation process: - **Network Connection**: Exploit demonstrates network communications - **File Modification**: Exploit involves file system operations -### Example Enhanced Rule +### Example Enhanced Rules +**Traditional Exploit-Based Rule:** ```yaml title: CVE-2025-1234 Exploit-Based Detection description: Detection for CVE-2025-1234 remote code execution [Enhanced with GitHub exploit analysis] @@ -191,13 +223,38 @@ detection: level: high ``` +**AI-Enhanced Rule (Generated by Ollama):** +```yaml +title: CVE-2025-1234 AI-Enhanced Detection +description: Detection for CVE-2025-1234 remote code execution [AI-Enhanced with PoC analysis] +tags: + - attack.t1059.001 + - attack.t1071.001 + - cve-2025-1234 + - ai.enhanced +detection: + selection_process: + Image|endswith: '\powershell.exe' + CommandLine|contains: + - '-EncodedCommand' + - 'bypass' + - 'downloadstring' + selection_network: + Initiated: true + DestinationPort: + - 443 + - 80 + condition: selection_process and selection_network +level: high +``` + ## Development ### Local Development -1. Start the database: +1. Start the database and services: ```bash -docker-compose up -d db redis +docker-compose up -d db redis ollama ``` 2. Run the backend: @@ -214,6 +271,34 @@ npm install npm start ``` +### Testing Ollama Integration + +To test the local LLM functionality: + +1. **Check Ollama Status**: +```bash +curl http://localhost:11434/api/tags +``` + +2. **Test LLM API Status**: +```bash +curl http://localhost:8000/api/llm-status +``` + +3. **Generate AI-Enhanced Rule**: +```bash +curl -X POST http://localhost:8000/api/llm-enhanced-rules \ + -H "Content-Type: application/json" \ + -d '{"cve_id": "CVE-2025-1234", "poc_content": "example exploit code"}' +``` + +4. **Switch LLM Provider**: +```bash +curl -X POST http://localhost:8000/api/llm-switch \ + -H "Content-Type: application/json" \ + -d '{"provider": "ollama", "model": "llama3.2"}' +``` + ### Database Migration The application automatically creates tables on startup. For manual schema changes: @@ -298,6 +383,27 @@ For enhanced exploit-based SIGMA rules: Without a GitHub token: Basic rules only With a GitHub token: Enhanced rules with exploit analysis (🔍 Exploit-Based) +**LLM API Keys (Optional)** +For AI-enhanced SIGMA rule generation: + +**Local Ollama (Recommended - No API Key Required)** +- Ollama runs locally in Docker container +- No external API dependencies +- Models downloaded automatically on first use +- Default model: llama3.2 (configurable) + +**OpenAI API (Optional)** +1. Visit: https://platform.openai.com/api-keys +2. Create API key +3. Add to your `.env` file: `OPENAI_API_KEY=your_key_here` +4. Set `LLM_PROVIDER=openai` in `.env` + +**Anthropic API (Optional)** +1. Visit: https://console.anthropic.com/ +2. Create API key +3. Add to your `.env` file: `ANTHROPIC_API_KEY=your_key_here` +4. Set `LLM_PROVIDER=anthropic` in `.env` + ### Rate Limits Without an API key, NVD limits requests to 5 per 30 seconds. With an API key, the limit increases to 50 per 30 seconds.