update README

This commit is contained in:
Brendan McDevitt 2025-07-10 16:23:36 -05:00
parent 696a1a3462
commit c1bbea09fe

118
README.md
View file

@ -12,6 +12,8 @@ An advanced automated platform that processes comprehensive CVE data and generat
### Intelligence Generation ### Intelligence Generation
- **Enhanced SIGMA Rules**: Creates rules using real exploit indicators from curated PoCs - **Enhanced SIGMA Rules**: Creates rules using real exploit indicators from curated PoCs
- **AI-Powered Rule Generation**: Multi-provider LLM integration (OpenAI, Anthropic, Ollama)
- **Local LLM Processing**: Built-in Ollama container for offline AI rule generation
- **Quality Tiers**: Excellent, Good, Fair, Poor, Very Poor classification system - **Quality Tiers**: Excellent, Good, Fair, Poor, Very Poor classification system
- **Smart Template Selection**: AI-driven template matching based on PoC characteristics - **Smart Template Selection**: AI-driven template matching based on PoC characteristics
- **Advanced Indicator Extraction**: Processes, files, network, registry, and command patterns - **Advanced Indicator Extraction**: Processes, files, network, registry, and command patterns
@ -29,6 +31,7 @@ An advanced automated platform that processes comprehensive CVE data and generat
- **Frontend**: React with Tailwind CSS - **Frontend**: React with Tailwind CSS
- **Database**: PostgreSQL - **Database**: PostgreSQL
- **Cache**: Redis (optional) - **Cache**: Redis (optional)
- **LLM Engine**: Ollama (local models) with multi-provider support
- **Containerization**: Docker & Docker Compose - **Containerization**: Docker & Docker Compose
## Quick Start ## Quick Start
@ -70,6 +73,7 @@ docker-compose up -d --build
- Frontend: http://localhost:3000 - Frontend: http://localhost:3000
- Backend API: http://localhost:8000 - Backend API: http://localhost:8000
- API Documentation: http://localhost:8000/docs - API Documentation: http://localhost:8000/docs
- Ollama API: http://localhost:11434
### First Run ### First Run
@ -99,6 +103,7 @@ curl -X POST http://localhost:8000/api/fetch-cves
### API Endpoints ### API Endpoints
#### Core Endpoints
- `GET /api/cves` - List all CVEs - `GET /api/cves` - List all CVEs
- `GET /api/cves/{cve_id}` - Get specific CVE details - `GET /api/cves/{cve_id}` - Get specific CVE details
- `GET /api/sigma-rules` - List all SIGMA rules - `GET /api/sigma-rules` - List all SIGMA rules
@ -106,15 +111,28 @@ curl -X POST http://localhost:8000/api/fetch-cves
- `POST /api/fetch-cves` - Manually trigger CVE fetch - `POST /api/fetch-cves` - Manually trigger CVE fetch
- `GET /api/stats` - Get application statistics - `GET /api/stats` - Get application statistics
#### LLM-Enhanced Endpoints
- `POST /api/llm-enhanced-rules` - Generate SIGMA rules using LLM AI analysis
- `GET /api/llm-status` - Check LLM API availability and configuration
- `POST /api/llm-switch` - Switch between LLM providers and models
## Configuration ## Configuration
### Environment Variables ### Environment Variables
#### Core Configuration
- `DATABASE_URL`: PostgreSQL connection string - `DATABASE_URL`: PostgreSQL connection string
- `NVD_API_KEY`: Optional NVD API key for higher rate limits (5→50 requests/30s) - `NVD_API_KEY`: Optional NVD API key for higher rate limits (5→50 requests/30s)
- `GITHUB_TOKEN`: Optional GitHub personal access token for exploit analysis - `GITHUB_TOKEN`: Optional GitHub personal access token for exploit analysis
- `REACT_APP_API_URL`: Backend API URL for frontend - `REACT_APP_API_URL`: Backend API URL for frontend
#### LLM Configuration
- `LLM_PROVIDER`: LLM provider selection (openai, anthropic, ollama) - **Default: ollama**
- `LLM_MODEL`: Model selection (provider-specific) - **Default: llama3.2**
- `OLLAMA_BASE_URL`: Ollama service URL - **Default: http://ollama:11434**
- `OPENAI_API_KEY`: Optional OpenAI API key for GPT models
- `ANTHROPIC_API_KEY`: Optional Anthropic API key for Claude models
### GitHub Integration (Optional) ### GitHub Integration (Optional)
For enhanced SIGMA rule generation with exploit analysis: For enhanced SIGMA rule generation with exploit analysis:
@ -141,8 +159,9 @@ Additional templates can be added to the database via the `rule_templates` table
## SIGMA Rule Generation Logic ## SIGMA Rule Generation Logic
The enhanced rule generation process: The enhanced rule generation process supports multiple generation methods:
### Traditional Rule Generation
1. **CVE Analysis**: Analyzes CVE description and affected products 1. **CVE Analysis**: Analyzes CVE description and affected products
2. **GitHub Exploit Search**: Searches GitHub for exploit code using multiple query strategies 2. **GitHub Exploit Search**: Searches GitHub for exploit code using multiple query strategies
3. **Code Analysis**: Extracts specific indicators from exploit code: 3. **Code Analysis**: Extracts specific indicators from exploit code:
@ -156,13 +175,25 @@ The enhanced rule generation process:
6. **MITRE ATT&CK Mapping**: Maps to specific MITRE ATT&CK techniques 6. **MITRE ATT&CK Mapping**: Maps to specific MITRE ATT&CK techniques
7. **Confidence Scoring**: Higher confidence for exploit-based rules 7. **Confidence Scoring**: Higher confidence for exploit-based rules
### AI-Enhanced Rule Generation
1. **LLM Provider Selection**: Chooses between OpenAI, Anthropic, or local Ollama
2. **Contextual Analysis**: LLM analyzes CVE description and PoC code
3. **Intelligent Rule Creation**: AI generates sophisticated SIGMA rules with:
- Proper YAML syntax and structure
- Advanced detection logic
- Contextual field selection
- Relevant MITRE ATT&CK mappings
4. **Automatic Validation**: Generated rules are validated for syntax compliance
5. **Fallback Mechanism**: Falls back to template-based generation if LLM fails
### Rule Quality Levels ### Rule Quality Levels
- **Basic Rules**: Generated from CVE description only - **Basic Rules**: Generated from CVE description only
- **Exploit-Based Rules**: Enhanced with GitHub exploit analysis (marked with 🔍) - **Exploit-Based Rules**: Enhanced with GitHub exploit analysis (marked with 🔍)
- **AI-Enhanced Rules**: Generated using LLM analysis of PoC code (marked with 🤖)
- **Confidence Ratings**: - **Confidence Ratings**:
- **High**: CVSS ≥9.0 + exploit analysis - **High**: CVSS ≥9.0 + exploit analysis + AI enhancement
- **Medium**: CVSS ≥7.0 or exploit analysis - **Medium**: CVSS ≥7.0 or exploit analysis or AI enhancement
- **Low**: Basic CVE description only - **Low**: Basic CVE description only
### Template Matching ### Template Matching
@ -172,8 +203,9 @@ The enhanced rule generation process:
- **Network Connection**: Exploit demonstrates network communications - **Network Connection**: Exploit demonstrates network communications
- **File Modification**: Exploit involves file system operations - **File Modification**: Exploit involves file system operations
### Example Enhanced Rule ### Example Enhanced Rules
**Traditional Exploit-Based Rule:**
```yaml ```yaml
title: CVE-2025-1234 Exploit-Based Detection title: CVE-2025-1234 Exploit-Based Detection
description: Detection for CVE-2025-1234 remote code execution [Enhanced with GitHub exploit analysis] description: Detection for CVE-2025-1234 remote code execution [Enhanced with GitHub exploit analysis]
@ -191,13 +223,38 @@ detection:
level: high level: high
``` ```
**AI-Enhanced Rule (Generated by Ollama):**
```yaml
title: CVE-2025-1234 AI-Enhanced Detection
description: Detection for CVE-2025-1234 remote code execution [AI-Enhanced with PoC analysis]
tags:
- attack.t1059.001
- attack.t1071.001
- cve-2025-1234
- ai.enhanced
detection:
selection_process:
Image|endswith: '\powershell.exe'
CommandLine|contains:
- '-EncodedCommand'
- 'bypass'
- 'downloadstring'
selection_network:
Initiated: true
DestinationPort:
- 443
- 80
condition: selection_process and selection_network
level: high
```
## Development ## Development
### Local Development ### Local Development
1. Start the database: 1. Start the database and services:
```bash ```bash
docker-compose up -d db redis docker-compose up -d db redis ollama
``` ```
2. Run the backend: 2. Run the backend:
@ -214,6 +271,34 @@ npm install
npm start npm start
``` ```
### Testing Ollama Integration
To test the local LLM functionality:
1. **Check Ollama Status**:
```bash
curl http://localhost:11434/api/tags
```
2. **Test LLM API Status**:
```bash
curl http://localhost:8000/api/llm-status
```
3. **Generate AI-Enhanced Rule**:
```bash
curl -X POST http://localhost:8000/api/llm-enhanced-rules \
-H "Content-Type: application/json" \
-d '{"cve_id": "CVE-2025-1234", "poc_content": "example exploit code"}'
```
4. **Switch LLM Provider**:
```bash
curl -X POST http://localhost:8000/api/llm-switch \
-H "Content-Type: application/json" \
-d '{"provider": "ollama", "model": "llama3.2"}'
```
### Database Migration ### Database Migration
The application automatically creates tables on startup. For manual schema changes: The application automatically creates tables on startup. For manual schema changes:
@ -298,6 +383,27 @@ For enhanced exploit-based SIGMA rules:
Without a GitHub token: Basic rules only Without a GitHub token: Basic rules only
With a GitHub token: Enhanced rules with exploit analysis (🔍 Exploit-Based) With a GitHub token: Enhanced rules with exploit analysis (🔍 Exploit-Based)
**LLM API Keys (Optional)**
For AI-enhanced SIGMA rule generation:
**Local Ollama (Recommended - No API Key Required)**
- Ollama runs locally in Docker container
- No external API dependencies
- Models downloaded automatically on first use
- Default model: llama3.2 (configurable)
**OpenAI API (Optional)**
1. Visit: https://platform.openai.com/api-keys
2. Create API key
3. Add to your `.env` file: `OPENAI_API_KEY=your_key_here`
4. Set `LLM_PROVIDER=openai` in `.env`
**Anthropic API (Optional)**
1. Visit: https://console.anthropic.com/
2. Create API key
3. Add to your `.env` file: `ANTHROPIC_API_KEY=your_key_here`
4. Set `LLM_PROVIDER=anthropic` in `.env`
### Rate Limits ### Rate Limits
Without an API key, NVD limits requests to 5 per 30 seconds. With an API key, the limit increases to 50 per 30 seconds. Without an API key, NVD limits requests to 5 per 30 seconds. With an API key, the limit increases to 50 per 30 seconds.