bpmcdevitt/data_importer

Fork 0

This is a rails/postgres application that will serve json data from the following data sources: - Cves - Cpes - CNA security advisories - GHSA Github security advisories - Github repositories that track public exploits for cves. - A list of github usernames github API data.

Find a file

Brendan McDevitt db6afbfd59 now we pull github security advisories		2022-04-11 18:45:02 -05:00
app	now we pull github security advisories	2022-04-11 18:45:02 -05:00
bin	make an importers dir in lib dir	2022-04-09 23:08:48 -05:00
config	now we pull github security advisories	2022-04-11 18:45:02 -05:00
db	now we pull github security advisories	2022-04-11 18:45:02 -05:00
lib	now we pull github security advisories	2022-04-11 18:45:02 -05:00
log	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
public	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
storage	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
test	make json rendering for cpes and cves	2022-04-04 13:18:03 -05:00
tmp	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
.gitignore	added github api client with graphql	2022-04-09 15:49:47 -05:00
.ruby-version	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
config.ru	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
docker-compose.yml	added github api client with graphql	2022-04-09 15:49:47 -05:00
Dockerfile	added files for importing cves from cvelist repo to database	2022-04-01 13:36:20 -05:00
entrypoint.sh	docker-compose and postgres basic rails app skeleton	2022-03-30 22:52:39 -05:00
Gemfile	now we pull github security advisories	2022-04-11 18:45:02 -05:00
Gemfile.lock	now we pull github security advisories	2022-04-11 18:45:02 -05:00
package.json	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
Rakefile	init commit rails new data_importer	2022-03-30 22:12:56 -05:00
README.md	now we pull github security advisories	2022-04-11 18:45:02 -05:00

README.md

data_importer

This is a rails/postgres application that will serve json data from the following data sources:

Cves
Cpes
CNA security advisories
GHSA Github security advisories
Github repositories that track public exploits for cves.

Check the HTTP API section below for specific endpoints that can be queried via http.

Supported data models:

Cve data from cve_list github repo.
Cpe data from nvd 2.2 format.
Cna data from mitre.
GithubPoc data from nomi-sec github repo.
GithubAdvisories data from github_advisories_database github repo.
InthewildCveExploit data from inthewild.io exploited feed.
TrickestPocCve data from trickest github repo.
CvemonCve data from ARPSyndicate github repo.

Initial Setup

Build container

docker-compose build

Database creation and seeding initial data

docker-compose run web rake db:create
docker-compose run web rake db:migrate
docker-compose run web rake db:seed

Launch Pry console

docker-compose run web rails console

HTTP API

For now unauthenticated api over localhost:3000 until I put in some basic token auth. All response data is json rendered.

Cves

  get "/cves", to: "cves#index"
  get "/cves/:cve_id", to: "cves#show"
  get "/cves/years/:year", to: "cves#show_year"

Cpes

  get "/cpes", to: "cpes#index"
  get "/cpes/:id", to: "cpes#show"

Cnas

  get "/cnas", to: "cnas#index"
  get "/cnas/:id", to: "cnas#show"
  get "/cnas/cna/:cna_id", to: "cnas#show_for_cna"

GithubAdvisories

  get "/github_advisories", to: "github_advisories#index"
  get "/github_advisories/:ghsa_id", to: "github_advisories#show"

GithubPocs

  get "/github_pocs", to: "github_pocs#index"
  get "/github_pocs/:id", to: "github_pocs#show"
  get "/github_pocs/cve/:cve_id", to: "github_pocs#show_for_cve"
  get "/github_pocs/years/:year", to: "github_pocs#show_year"

InthewildCveExploits

  get "/inthewild_cve_exploits", to: "inthewild_cve_exploits#index"
  get "/inthewild_cve_exploits/:cve_id", to: "inthewild_cve_exploits#show"

TrickestPocCves

  get "/trickest_poc_cves", to: "trickest_poc_cves#index"
  get "/trickest_poc_cves/:id", to: "trickest_poc_cves#show"
  get "/trickest_poc_cves/cve/:cve_id", to: "trickest_poc_cves#show_for_cve"
  get "/trickest_poc_cves/years/:year", to: "trickest_poc_cves#show_year"

CvemonCves

  get "/cvemon_cves", to: "cvemon_cves#index"
  get "/cvemon_cves/:id", to: "cvemon_cves#show"
  get "/cvemon_cves/cve/:cve_id", to: "cvemon_cves#show_for_cve"
  get "/cvemon_cves/years/:year", to: "cvemon_cves#show_year"