update README
This commit is contained in:
parent
696a1a3462
commit
c1bbea09fe
1 changed files with 112 additions and 6 deletions
118
README.md
118
README.md
|
@ -12,6 +12,8 @@ An advanced automated platform that processes comprehensive CVE data and generat
|
||||||
|
|
||||||
### Intelligence Generation
|
### Intelligence Generation
|
||||||
- **Enhanced SIGMA Rules**: Creates rules using real exploit indicators from curated PoCs
|
- **Enhanced SIGMA Rules**: Creates rules using real exploit indicators from curated PoCs
|
||||||
|
- **AI-Powered Rule Generation**: Multi-provider LLM integration (OpenAI, Anthropic, Ollama)
|
||||||
|
- **Local LLM Processing**: Built-in Ollama container for offline AI rule generation
|
||||||
- **Quality Tiers**: Excellent, Good, Fair, Poor, Very Poor classification system
|
- **Quality Tiers**: Excellent, Good, Fair, Poor, Very Poor classification system
|
||||||
- **Smart Template Selection**: AI-driven template matching based on PoC characteristics
|
- **Smart Template Selection**: AI-driven template matching based on PoC characteristics
|
||||||
- **Advanced Indicator Extraction**: Processes, files, network, registry, and command patterns
|
- **Advanced Indicator Extraction**: Processes, files, network, registry, and command patterns
|
||||||
|
@ -29,6 +31,7 @@ An advanced automated platform that processes comprehensive CVE data and generat
|
||||||
- **Frontend**: React with Tailwind CSS
|
- **Frontend**: React with Tailwind CSS
|
||||||
- **Database**: PostgreSQL
|
- **Database**: PostgreSQL
|
||||||
- **Cache**: Redis (optional)
|
- **Cache**: Redis (optional)
|
||||||
|
- **LLM Engine**: Ollama (local models) with multi-provider support
|
||||||
- **Containerization**: Docker & Docker Compose
|
- **Containerization**: Docker & Docker Compose
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
@ -70,6 +73,7 @@ docker-compose up -d --build
|
||||||
- Frontend: http://localhost:3000
|
- Frontend: http://localhost:3000
|
||||||
- Backend API: http://localhost:8000
|
- Backend API: http://localhost:8000
|
||||||
- API Documentation: http://localhost:8000/docs
|
- API Documentation: http://localhost:8000/docs
|
||||||
|
- Ollama API: http://localhost:11434
|
||||||
|
|
||||||
### First Run
|
### First Run
|
||||||
|
|
||||||
|
@ -99,6 +103,7 @@ curl -X POST http://localhost:8000/api/fetch-cves
|
||||||
|
|
||||||
### API Endpoints
|
### API Endpoints
|
||||||
|
|
||||||
|
#### Core Endpoints
|
||||||
- `GET /api/cves` - List all CVEs
|
- `GET /api/cves` - List all CVEs
|
||||||
- `GET /api/cves/{cve_id}` - Get specific CVE details
|
- `GET /api/cves/{cve_id}` - Get specific CVE details
|
||||||
- `GET /api/sigma-rules` - List all SIGMA rules
|
- `GET /api/sigma-rules` - List all SIGMA rules
|
||||||
|
@ -106,15 +111,28 @@ curl -X POST http://localhost:8000/api/fetch-cves
|
||||||
- `POST /api/fetch-cves` - Manually trigger CVE fetch
|
- `POST /api/fetch-cves` - Manually trigger CVE fetch
|
||||||
- `GET /api/stats` - Get application statistics
|
- `GET /api/stats` - Get application statistics
|
||||||
|
|
||||||
|
#### LLM-Enhanced Endpoints
|
||||||
|
- `POST /api/llm-enhanced-rules` - Generate SIGMA rules using LLM AI analysis
|
||||||
|
- `GET /api/llm-status` - Check LLM API availability and configuration
|
||||||
|
- `POST /api/llm-switch` - Switch between LLM providers and models
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
### Environment Variables
|
### Environment Variables
|
||||||
|
|
||||||
|
#### Core Configuration
|
||||||
- `DATABASE_URL`: PostgreSQL connection string
|
- `DATABASE_URL`: PostgreSQL connection string
|
||||||
- `NVD_API_KEY`: Optional NVD API key for higher rate limits (5→50 requests/30s)
|
- `NVD_API_KEY`: Optional NVD API key for higher rate limits (5→50 requests/30s)
|
||||||
- `GITHUB_TOKEN`: Optional GitHub personal access token for exploit analysis
|
- `GITHUB_TOKEN`: Optional GitHub personal access token for exploit analysis
|
||||||
- `REACT_APP_API_URL`: Backend API URL for frontend
|
- `REACT_APP_API_URL`: Backend API URL for frontend
|
||||||
|
|
||||||
|
#### LLM Configuration
|
||||||
|
- `LLM_PROVIDER`: LLM provider selection (openai, anthropic, ollama) - **Default: ollama**
|
||||||
|
- `LLM_MODEL`: Model selection (provider-specific) - **Default: llama3.2**
|
||||||
|
- `OLLAMA_BASE_URL`: Ollama service URL - **Default: http://ollama:11434**
|
||||||
|
- `OPENAI_API_KEY`: Optional OpenAI API key for GPT models
|
||||||
|
- `ANTHROPIC_API_KEY`: Optional Anthropic API key for Claude models
|
||||||
|
|
||||||
### GitHub Integration (Optional)
|
### GitHub Integration (Optional)
|
||||||
|
|
||||||
For enhanced SIGMA rule generation with exploit analysis:
|
For enhanced SIGMA rule generation with exploit analysis:
|
||||||
|
@ -141,8 +159,9 @@ Additional templates can be added to the database via the `rule_templates` table
|
||||||
|
|
||||||
## SIGMA Rule Generation Logic
|
## SIGMA Rule Generation Logic
|
||||||
|
|
||||||
The enhanced rule generation process:
|
The enhanced rule generation process supports multiple generation methods:
|
||||||
|
|
||||||
|
### Traditional Rule Generation
|
||||||
1. **CVE Analysis**: Analyzes CVE description and affected products
|
1. **CVE Analysis**: Analyzes CVE description and affected products
|
||||||
2. **GitHub Exploit Search**: Searches GitHub for exploit code using multiple query strategies
|
2. **GitHub Exploit Search**: Searches GitHub for exploit code using multiple query strategies
|
||||||
3. **Code Analysis**: Extracts specific indicators from exploit code:
|
3. **Code Analysis**: Extracts specific indicators from exploit code:
|
||||||
|
@ -156,13 +175,25 @@ The enhanced rule generation process:
|
||||||
6. **MITRE ATT&CK Mapping**: Maps to specific MITRE ATT&CK techniques
|
6. **MITRE ATT&CK Mapping**: Maps to specific MITRE ATT&CK techniques
|
||||||
7. **Confidence Scoring**: Higher confidence for exploit-based rules
|
7. **Confidence Scoring**: Higher confidence for exploit-based rules
|
||||||
|
|
||||||
|
### AI-Enhanced Rule Generation
|
||||||
|
1. **LLM Provider Selection**: Chooses between OpenAI, Anthropic, or local Ollama
|
||||||
|
2. **Contextual Analysis**: LLM analyzes CVE description and PoC code
|
||||||
|
3. **Intelligent Rule Creation**: AI generates sophisticated SIGMA rules with:
|
||||||
|
- Proper YAML syntax and structure
|
||||||
|
- Advanced detection logic
|
||||||
|
- Contextual field selection
|
||||||
|
- Relevant MITRE ATT&CK mappings
|
||||||
|
4. **Automatic Validation**: Generated rules are validated for syntax compliance
|
||||||
|
5. **Fallback Mechanism**: Falls back to template-based generation if LLM fails
|
||||||
|
|
||||||
### Rule Quality Levels
|
### Rule Quality Levels
|
||||||
|
|
||||||
- **Basic Rules**: Generated from CVE description only
|
- **Basic Rules**: Generated from CVE description only
|
||||||
- **Exploit-Based Rules**: Enhanced with GitHub exploit analysis (marked with 🔍)
|
- **Exploit-Based Rules**: Enhanced with GitHub exploit analysis (marked with 🔍)
|
||||||
|
- **AI-Enhanced Rules**: Generated using LLM analysis of PoC code (marked with 🤖)
|
||||||
- **Confidence Ratings**:
|
- **Confidence Ratings**:
|
||||||
- **High**: CVSS ≥9.0 + exploit analysis
|
- **High**: CVSS ≥9.0 + exploit analysis + AI enhancement
|
||||||
- **Medium**: CVSS ≥7.0 or exploit analysis
|
- **Medium**: CVSS ≥7.0 or exploit analysis or AI enhancement
|
||||||
- **Low**: Basic CVE description only
|
- **Low**: Basic CVE description only
|
||||||
|
|
||||||
### Template Matching
|
### Template Matching
|
||||||
|
@ -172,8 +203,9 @@ The enhanced rule generation process:
|
||||||
- **Network Connection**: Exploit demonstrates network communications
|
- **Network Connection**: Exploit demonstrates network communications
|
||||||
- **File Modification**: Exploit involves file system operations
|
- **File Modification**: Exploit involves file system operations
|
||||||
|
|
||||||
### Example Enhanced Rule
|
### Example Enhanced Rules
|
||||||
|
|
||||||
|
**Traditional Exploit-Based Rule:**
|
||||||
```yaml
|
```yaml
|
||||||
title: CVE-2025-1234 Exploit-Based Detection
|
title: CVE-2025-1234 Exploit-Based Detection
|
||||||
description: Detection for CVE-2025-1234 remote code execution [Enhanced with GitHub exploit analysis]
|
description: Detection for CVE-2025-1234 remote code execution [Enhanced with GitHub exploit analysis]
|
||||||
|
@ -191,13 +223,38 @@ detection:
|
||||||
level: high
|
level: high
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**AI-Enhanced Rule (Generated by Ollama):**
|
||||||
|
```yaml
|
||||||
|
title: CVE-2025-1234 AI-Enhanced Detection
|
||||||
|
description: Detection for CVE-2025-1234 remote code execution [AI-Enhanced with PoC analysis]
|
||||||
|
tags:
|
||||||
|
- attack.t1059.001
|
||||||
|
- attack.t1071.001
|
||||||
|
- cve-2025-1234
|
||||||
|
- ai.enhanced
|
||||||
|
detection:
|
||||||
|
selection_process:
|
||||||
|
Image|endswith: '\powershell.exe'
|
||||||
|
CommandLine|contains:
|
||||||
|
- '-EncodedCommand'
|
||||||
|
- 'bypass'
|
||||||
|
- 'downloadstring'
|
||||||
|
selection_network:
|
||||||
|
Initiated: true
|
||||||
|
DestinationPort:
|
||||||
|
- 443
|
||||||
|
- 80
|
||||||
|
condition: selection_process and selection_network
|
||||||
|
level: high
|
||||||
|
```
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
### Local Development
|
### Local Development
|
||||||
|
|
||||||
1. Start the database:
|
1. Start the database and services:
|
||||||
```bash
|
```bash
|
||||||
docker-compose up -d db redis
|
docker-compose up -d db redis ollama
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Run the backend:
|
2. Run the backend:
|
||||||
|
@ -214,6 +271,34 @@ npm install
|
||||||
npm start
|
npm start
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Testing Ollama Integration
|
||||||
|
|
||||||
|
To test the local LLM functionality:
|
||||||
|
|
||||||
|
1. **Check Ollama Status**:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:11434/api/tags
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Test LLM API Status**:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8000/api/llm-status
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Generate AI-Enhanced Rule**:
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/llm-enhanced-rules \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"cve_id": "CVE-2025-1234", "poc_content": "example exploit code"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Switch LLM Provider**:
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/llm-switch \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"provider": "ollama", "model": "llama3.2"}'
|
||||||
|
```
|
||||||
|
|
||||||
### Database Migration
|
### Database Migration
|
||||||
|
|
||||||
The application automatically creates tables on startup. For manual schema changes:
|
The application automatically creates tables on startup. For manual schema changes:
|
||||||
|
@ -298,6 +383,27 @@ For enhanced exploit-based SIGMA rules:
|
||||||
Without a GitHub token: Basic rules only
|
Without a GitHub token: Basic rules only
|
||||||
With a GitHub token: Enhanced rules with exploit analysis (🔍 Exploit-Based)
|
With a GitHub token: Enhanced rules with exploit analysis (🔍 Exploit-Based)
|
||||||
|
|
||||||
|
**LLM API Keys (Optional)**
|
||||||
|
For AI-enhanced SIGMA rule generation:
|
||||||
|
|
||||||
|
**Local Ollama (Recommended - No API Key Required)**
|
||||||
|
- Ollama runs locally in Docker container
|
||||||
|
- No external API dependencies
|
||||||
|
- Models downloaded automatically on first use
|
||||||
|
- Default model: llama3.2 (configurable)
|
||||||
|
|
||||||
|
**OpenAI API (Optional)**
|
||||||
|
1. Visit: https://platform.openai.com/api-keys
|
||||||
|
2. Create API key
|
||||||
|
3. Add to your `.env` file: `OPENAI_API_KEY=your_key_here`
|
||||||
|
4. Set `LLM_PROVIDER=openai` in `.env`
|
||||||
|
|
||||||
|
**Anthropic API (Optional)**
|
||||||
|
1. Visit: https://console.anthropic.com/
|
||||||
|
2. Create API key
|
||||||
|
3. Add to your `.env` file: `ANTHROPIC_API_KEY=your_key_here`
|
||||||
|
4. Set `LLM_PROVIDER=anthropic` in `.env`
|
||||||
|
|
||||||
### Rate Limits
|
### Rate Limits
|
||||||
|
|
||||||
Without an API key, NVD limits requests to 5 per 30 seconds. With an API key, the limit increases to 50 per 30 seconds.
|
Without an API key, NVD limits requests to 5 per 30 seconds. With an API key, the limit increases to 50 per 30 seconds.
|
||||||
|
|
Loading…
Add table
Reference in a new issue