49963338d3
Add Celery dependencies and enhance bulk seeder
...
- Add Celery, Flower, and related dependencies to requirements.txt
- Update bulk_seeder.py with progress callback support for Celery integration
- Clean up finetuned model dependencies (now served through Ollama)
- Update setup_ollama.py for enhanced configuration
🤖 Generated with [Claude Code](https://claude.ai/code )
Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-17 18:59:07 -05:00
9bde1395bf
Optimize performance and migrate to Celery-based scheduling
...
This commit introduces major performance improvements and migrates from custom job scheduling to Celery Beat for better reliability and scalability.
### 🚀 Performance Optimizations
**CVE2CAPEC Client Performance (Fixed startup blocking)**
- Implement lazy loading with 24-hour cache for CVE2CAPEC mappings
- Add background task for CVE2CAPEC sync (data_sync_tasks.sync_cve2capec)
- Remove blocking data fetch during client initialization
- API endpoint: POST /api/sync-cve2capec
**ExploitDB Client Performance (Fixed webapp request blocking)**
- Implement global file index cache to prevent rebuilding on every request
- Add lazy loading with 24-hour cache expiry for 46K+ exploit index
- Background task for index building (data_sync_tasks.build_exploitdb_index)
- API endpoint: POST /api/build-exploitdb-index
### 🔄 Celery Migration & Scheduling
**Celery Beat Integration**
- Migrate from custom job scheduler to Celery Beat for reliability
- Remove 'finetuned' LLM provider (logic moved to ollama container)
- Optimized daily workflow with proper timing and dependencies
**New Celery Tasks Structure**
- tasks/bulk_tasks.py - NVD bulk processing and SIGMA generation
- tasks/data_sync_tasks.py - All data synchronization tasks
- tasks/maintenance_tasks.py - System maintenance and cleanup
- tasks/sigma_tasks.py - SIGMA rule generation tasks
**Daily Schedule (Optimized)**
```
1:00 AM → Weekly cleanup (Sundays)
1:30 AM → Daily result cleanup
2:00 AM → NVD incremental update
3:00 AM → CISA KEV sync
3:15 AM → Nomi-sec PoC sync
3:30 AM → GitHub PoC sync
3:45 AM → ExploitDB sync
4:00 AM → CVE2CAPEC MITRE ATT&CK sync
4:15 AM → ExploitDB index rebuild
5:00 AM → Reference content sync
8:00 AM → SIGMA rule generation
9:00 AM → LLM-enhanced SIGMA generation
Every 15min → Health checks
```
### 🐳 Docker & Infrastructure
**Enhanced Docker Setup**
- Ollama setup with integrated SIGMA model creation (setup_ollama_with_sigma.py)
- Initial database population check and trigger (initial_setup.py)
- Proper service dependencies and health checks
- Remove manual post-rebuild script requirements
**Service Architecture**
- Celery worker with 4-queue system (default, bulk_processing, sigma_generation, data_sync)
- Flower monitoring dashboard (localhost:5555)
- Redis as message broker and result backend
### 🎯 API Improvements
**Background Task Endpoints**
- GitHub PoC sync now uses Celery (was blocking backend)
- All sync operations return task IDs and monitoring URLs
- Consistent error handling and progress tracking
**New Endpoints**
- POST /api/sync-cve2capec - CVE2CAPEC mapping sync
- POST /api/build-exploitdb-index - ExploitDB index rebuild
### 📁 Cleanup
**Removed Files**
- fix_sigma_model.sh (replaced by setup_ollama_with_sigma.py)
- Various test_* and debug_* files no longer needed
- Old training scripts related to removed 'finetuned' provider
- Utility scripts replaced by Docker services
### 🔧 Configuration
**Key Files Added/Modified**
- backend/celery_config.py - Complete Celery configuration
- backend/initial_setup.py - First-boot database population
- backend/setup_ollama_with_sigma.py - Integrated Ollama setup
- CLAUDE.md - Project documentation and development guide
🤖 Generated with [Claude Code](https://claude.ai/code )
Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-17 18:58:47 -05:00
54db665711
only use our LLM for help with generating detection: portion of SIGMA rule. enhance poc analyzer program python indicators
2025-07-16 13:02:11 -05:00
cf57944c7f
add poc analyzer code
2025-07-16 10:15:55 -05:00
06c4ed74b8
add cve2capec client to map mitre attack data to cves
2025-07-14 15:48:10 -05:00
d38edff1cd
script to clear old sigma rules and starting to tweak system prompt to send to llm for rule generation
2025-07-11 19:20:03 -05:00
d17f961b9d
add job scheduler
2025-07-11 09:16:57 -05:00
08d6e33bbc
add ollama to docker-compose for local model testing
2025-07-10 21:32:15 -05:00
3c120462ac
add reference data gathering
2025-07-10 17:30:12 -05:00
696a1a3462
add kev support, exploitDB mirror support
2025-07-10 16:19:43 -05:00
20b3a63c78
add claude client + generic llm client using langchain
2025-07-09 18:02:45 -05:00
e4a3cc6cb9
make nvd sync all cves, fix interpolation for templates
2025-07-09 12:42:18 -05:00
455a46c88f
added git submodule for more exploits. added template dir for base yaml templates for sigma rules
2025-07-09 11:58:29 -05:00
cfaad8b359
add templates to enhanced sigma generator
2025-07-09 07:22:51 -05:00
790e4bd91f
more updates for bulk
2025-07-08 17:50:01 -05:00
5a9ae34996
Adding in rule generation from github exploits
2025-07-08 10:20:54 -05:00
cc825fdb86
updated backend code fixed bad UUID error
2025-07-08 09:45:53 -05:00
e331f1763d
fix build errors
2025-07-08 09:10:25 -05:00
967886ef49
init commit. main app + frontend/backend
2025-07-08 08:34:28 -05:00
