bpmcdevitt/data_importer

Fork 0

This is a rails/postgres application that will serve json data from the following data sources: - Cves - Cpes - CNA security advisories - GHSA Github security advisories - Github repositories that track public exploits for cves. - A list of github usernames github API data.

Find a file

Brendan McDevitt 5b859c6986 make query ILIKE instead of LIKE for case insensitive searches like google or Google		2022-04-07 18:48:24 -05:00
app	make query ILIKE instead of LIKE for case insensitive searches like google or Google	2022-04-07 18:48:24 -05:00
bin	make json rendering for cpes and cves	2022-04-04 13:18:03 -05:00
config	add a route to search via orgname	2022-04-07 18:26:42 -05:00
db	add support for CNA info	2022-04-07 18:15:21 -05:00
lib	add support for CNA info	2022-04-07 18:15:21 -05:00
log	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
public	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
storage	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
test	make json rendering for cpes and cves	2022-04-04 13:18:03 -05:00
tmp	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
vendor	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
.gitignore	add support for trickest poc cve github repo	2022-04-07 01:51:44 -05:00
.ruby-version	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
config.ru	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
docker-compose.yml	add base support for trickest poc cves to github	2022-04-06 22:31:52 -05:00
Dockerfile	added files for importing cves from cvelist repo to database	2022-04-01 13:36:20 -05:00
entrypoint.sh	docker-compose and postgres basic rails app skeleton	2022-03-30 22:52:39 -05:00
Gemfile	add base support for trickest poc cves to github	2022-04-06 22:31:52 -05:00
Gemfile.lock	add base support for trickest poc cves to github	2022-04-06 22:31:52 -05:00
package.json	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
Rakefile	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
README.md	add support for CNA info	2022-04-07 18:15:21 -05:00

README.md

data_importer

This is a rails/postgres application that will serve json data from the following data sources:

Cves
Cpes
CNA Security Advisories
Github repositories that track public exploits for Cves.

Check the HTTP API section below for specific endpoints that can be queried via http.

Supported data models:

Cve data from cve_list github repo.
Cpe data from nvd 2.2 format.
Cna data from mitre.
GithubPoc data from nomi-sec github repo.
InthewildCveExploit data from inthewild.io exploited feed.
TrickestPocCve data from trickest github repo.
CvemonCve data from ARPSyndicate github repo.

Initial Setup

Build container

docker-compose build

Database creation and seeding initial data

docker-compose run web rake db:create
docker-compose run web rake db:migrate
docker-compose run web rake db:seed

Launch Pry console

docker-compose run web rails console

HTTP API

For now unauthenticated api over localhost:3000 until I put in some basic token auth. All response data is json rendered.

Cves

  get "/cves", to: "cves#index"
  get "/cves/:cve_id", to: "cves#show"
  get "/cves/years/:year", to: "cves#show_year"

Cpes

  get "/cpes", to: "cpes#index"
  get "/cpes/:id", to: "cpes#show"

Cnas

  get "/cnas", to: "cnas#index"
  get "/cnas/:id", to: "cnas#show"
  get "/cnas/cna/:cna_id", to: "cnas#show_for_cna"

GithubPocs

  get "/github_pocs", to: "github_pocs#index"
  get "/github_pocs/:id", to: "github_pocs#show"
  get "/github_pocs/cve/:cve_id", to: "github_pocs#show_for_cve"
  get "/github_pocs/years/:year", to: "github_pocs#show_year"

InthewildCveExploits

  get "/inthewild_cve_exploits", to: "inthewild_cve_exploits#index"
  get "/inthewild_cve_exploits/:cve_id", to: "inthewild_cve_exploits#show"

TrickestPocCves

  get "/trickest_poc_cves", to: "trickest_poc_cves#index"
  get "/trickest_poc_cves/:id", to: "trickest_poc_cves#show"
  get "/trickest_poc_cves/cve/:cve_id", to: "trickest_poc_cves#show_for_cve"
  get "/trickest_poc_cves/years/:year", to: "trickest_poc_cves#show_year"

CvemonCves

  get "/cvemon_cves", to: "cvemon_cves#index"
  get "/cvemon_cves/:id", to: "cvemon_cves#show"
  get "/cvemon_cves/cve/:cve_id", to: "cvemon_cves#show_for_cve"
  get "/cvemon_cves/years/:year", to: "cvemon_cves#show_year"