bpmcdevitt/auto_sigma_rule_generator
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
auto_sigma_rule_generator/backend
History
bpmcdevitt 9bde1395bf Optimize performance and migrate to Celery-based scheduling 
This commit introduces major performance improvements and migrates from custom job scheduling to Celery Beat for better reliability and scalability.

### 🚀 Performance Optimizations

**CVE2CAPEC Client Performance (Fixed startup blocking)**
- Implement lazy loading with 24-hour cache for CVE2CAPEC mappings
- Add background task for CVE2CAPEC sync (data_sync_tasks.sync_cve2capec)
- Remove blocking data fetch during client initialization
- API endpoint: POST /api/sync-cve2capec

**ExploitDB Client Performance (Fixed webapp request blocking)**
- Implement global file index cache to prevent rebuilding on every request
- Add lazy loading with 24-hour cache expiry for 46K+ exploit index
- Background task for index building (data_sync_tasks.build_exploitdb_index)
- API endpoint: POST /api/build-exploitdb-index

### 🔄 Celery Migration & Scheduling

**Celery Beat Integration**
- Migrate from custom job scheduler to Celery Beat for reliability
- Remove 'finetuned' LLM provider (logic moved to ollama container)
- Optimized daily workflow with proper timing and dependencies

**New Celery Tasks Structure**
- tasks/bulk_tasks.py - NVD bulk processing and SIGMA generation
- tasks/data_sync_tasks.py - All data synchronization tasks
- tasks/maintenance_tasks.py - System maintenance and cleanup
- tasks/sigma_tasks.py - SIGMA rule generation tasks

**Daily Schedule (Optimized)**
```
1:00 AM  → Weekly cleanup (Sundays)
1:30 AM  → Daily result cleanup
2:00 AM  → NVD incremental update
3:00 AM  → CISA KEV sync
3:15 AM  → Nomi-sec PoC sync
3:30 AM  → GitHub PoC sync
3:45 AM  → ExploitDB sync
4:00 AM  → CVE2CAPEC MITRE ATT&CK sync
4:15 AM  → ExploitDB index rebuild
5:00 AM  → Reference content sync
8:00 AM  → SIGMA rule generation
9:00 AM  → LLM-enhanced SIGMA generation
Every 15min → Health checks
```

### 🐳 Docker & Infrastructure

**Enhanced Docker Setup**
- Ollama setup with integrated SIGMA model creation (setup_ollama_with_sigma.py)
- Initial database population check and trigger (initial_setup.py)
- Proper service dependencies and health checks
- Remove manual post-rebuild script requirements

**Service Architecture**
- Celery worker with 4-queue system (default, bulk_processing, sigma_generation, data_sync)
- Flower monitoring dashboard (localhost:5555)
- Redis as message broker and result backend

### 🎯 API Improvements

**Background Task Endpoints**
- GitHub PoC sync now uses Celery (was blocking backend)
- All sync operations return task IDs and monitoring URLs
- Consistent error handling and progress tracking

**New Endpoints**
- POST /api/sync-cve2capec - CVE2CAPEC mapping sync
- POST /api/build-exploitdb-index - ExploitDB index rebuild

### 📁 Cleanup

**Removed Files**
- fix_sigma_model.sh (replaced by setup_ollama_with_sigma.py)
- Various test_* and debug_* files no longer needed
- Old training scripts related to removed 'finetuned' provider
- Utility scripts replaced by Docker services

### 🔧 Configuration

**Key Files Added/Modified**
- backend/celery_config.py - Complete Celery configuration
- backend/initial_setup.py - First-boot database population
- backend/setup_ollama_with_sigma.py - Integrated Ollama setup
- CLAUDE.md - Project documentation and development guide

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-17 18:58:47 -05:00
..
tasks Optimize performance and migrate to Celery-based scheduling 2025-07-17 18:58:47 -05:00
templates added git submodule for more exploits. added template dir for base yaml templates for sigma rules 2025-07-09 11:58:29 -05:00
bulk_seeder.py add kev support, exploitDB mirror support 2025-07-10 16:19:43 -05:00
celery_config.py Optimize performance and migrate to Celery-based scheduling 2025-07-17 18:58:47 -05:00
cisa_kev_client.py add reference data gathering 2025-07-10 17:30:12 -05:00
claude_client.py add claude client + generic llm client using langchain 2025-07-09 18:02:45 -05:00
cve2capec_client.py Optimize performance and migrate to Celery-based scheduling 2025-07-17 18:58:47 -05:00
delete_sigma_rules.py script to clear old sigma rules and starting to tweak system prompt to send to llm for rule generation 2025-07-11 19:20:03 -05:00
Dockerfile init commit. main app + frontend/backend 2025-07-08 08:34:28 -05:00
enhanced_sigma_generator.py only use our LLM for help with generating detection: portion of SIGMA rule. enhance poc analyzer program python indicators 2025-07-16 13:02:11 -05:00
exploitdb_client_local.py Optimize performance and migrate to Celery-based scheduling 2025-07-17 18:58:47 -05:00
initial_setup.py Optimize performance and migrate to Celery-based scheduling 2025-07-17 18:58:47 -05:00
initialize_templates.py added git submodule for more exploits. added template dir for base yaml templates for sigma rules 2025-07-09 11:58:29 -05:00
job_executors.py script to clear old sigma rules and starting to tweak system prompt to send to llm for rule generation 2025-07-11 19:20:03 -05:00
job_scheduler.py add job scheduler 2025-07-11 09:16:57 -05:00
llm_client.py Optimize performance and migrate to Celery-based scheduling 2025-07-17 18:58:47 -05:00
main.py Optimize performance and migrate to Celery-based scheduling 2025-07-17 18:58:47 -05:00
mcdevitt_poc_client.py added git submodule for more exploits. added template dir for base yaml templates for sigma rules 2025-07-09 11:58:29 -05:00
nomi_sec_client.py script to clear old sigma rules and starting to tweak system prompt to send to llm for rule generation 2025-07-11 19:20:03 -05:00
nvd_bulk_processor.py more updates for bulk 2025-07-08 17:50:01 -05:00
poc_analyzer.py only use our LLM for help with generating detection: portion of SIGMA rule. enhance poc analyzer program python indicators 2025-07-16 13:02:11 -05:00
reference_client.py add ollama to docker-compose for local model testing 2025-07-10 21:32:15 -05:00
requirements.txt add job scheduler 2025-07-11 09:16:57 -05:00
scheduler_config.yaml script to clear old sigma rules and starting to tweak system prompt to send to llm for rule generation 2025-07-11 19:20:03 -05:00
setup_ollama.py add ollama to docker-compose for local model testing 2025-07-10 21:32:15 -05:00
setup_ollama_with_sigma.py Optimize performance and migrate to Celery-based scheduling 2025-07-17 18:58:47 -05:00
test_enhanced_generation.py add templates to enhanced sigma generator 2025-07-09 07:22:51 -05:00