Compare commits
No commits in common. "2f16d95d50ccda5c57a37103cd780a7e1056fabc" and "f0fc98b6d7165977e8659181c5cf98f9ab586f6f" have entirely different histories.
2f16d95d50
...
f0fc98b6d7
3 changed files with 35 additions and 198 deletions
101
README.md
101
README.md
|
@ -1,15 +1,12 @@
|
||||||
# GitHub Searcher
|
# GitHub Searcher
|
||||||
|
|
||||||
GitHub Searcher is a command-line tool to search GitHub repositories and users for PoC exploits and CVEs. It supports various search types and can output results in JSON format.
|
GitHub Searcher is a command-line tool to search for PoC exploits for CVEs and security vulnerabilities in open-source software on GitHub. It allows you to search repositories, users, and repository names, and optionally fetch open issues and repository contents.
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
- Search GitHub repositories by query.
|
- Search GitHub repositories, users, and repository names.
|
||||||
- Search GitHub users by query.
|
- Fetch open issues of a repository.
|
||||||
- Search within repository names.
|
- Fetch contents of a repository.
|
||||||
- Retrieve open issues and pull requests for repositories.
|
|
||||||
- Retrieve file contents for repositories.
|
|
||||||
- Output results in JSON format.
|
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
|
@ -19,100 +16,24 @@ GitHub Searcher is a command-line tool to search GitHub repositories and users f
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
1. Clone the repository:
|
1. Clone the repository:
|
||||||
```sh
|
```bash
|
||||||
git clone https://github.com/yourusername/github_searcher.git
|
git clone https://github.com/yourusername/github_searcher.git
|
||||||
cd github_searcher
|
cd github_searcher
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Install the required dependencies:
|
2. Install the required dependencies:
|
||||||
```sh
|
```bash
|
||||||
pip install PyGithub
|
pip install PyGithub
|
||||||
```
|
```
|
||||||
|
|
||||||
3. Set the `GITHUB_ACCESS_TOKEN` environment variable with your GitHub access token:
|
3. Set your GitHub access token as an environment variable:
|
||||||
```sh
|
```bash
|
||||||
export GITHUB_ACCESS_TOKEN=your_access_token
|
export GITHUB_ACCESS_TOKEN=your_access_token
|
||||||
```
|
```
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
```sh
|
Run the script with the required arguments:
|
||||||
python github_searcher.py --query QUERY --search_type SEARCH_TYPE [--get_file_contents] [--get_open_issues] [--get_open_pull_requests] [--json]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Arguments
|
```bash
|
||||||
|
python3 github_searcher.py --query=<query> --search_type=<search_type> [--get_open_issues] [--get_file_contents]
|
||||||
- `--query`: The search query (required).
|
|
||||||
- `--search_type`: The type of search to perform (required). Choices are:
|
|
||||||
- `repo`: Search repositories.
|
|
||||||
- `users`: Search users.
|
|
||||||
- `in-repo-name`: Search within repository names.
|
|
||||||
- `--get_file_contents`: Get the contents of repository results (optional).
|
|
||||||
- `--get_open_issues`: Get the open issues of repository results (optional).
|
|
||||||
- `--get_open_pull_requests`: Get the open pull requests of repository results (optional).
|
|
||||||
- `--json`: Output the results in JSON format (optional).
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
1. Search repositories with a query:
|
|
||||||
```sh
|
|
||||||
python github_searcher.py --query "machine learning" --search_type repo
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Search users with a query:
|
|
||||||
```sh
|
|
||||||
python github_searcher.py --query "john doe" --search_type users
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Search within repository names:
|
|
||||||
```sh
|
|
||||||
python github_searcher.py --query "tensorflow" --search_type in-repo-name
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Get open issues and pull requests for repositories:
|
|
||||||
```sh
|
|
||||||
python github_searcher.py --query "machine learning" --search_type repo --get_open_issues --get_open_pull_requests
|
|
||||||
```
|
|
||||||
|
|
||||||
5. Output results in JSON format:
|
|
||||||
```sh
|
|
||||||
python github_searcher.py --query "machine learning" --search_type repo --json
|
|
||||||
```
|
|
||||||
|
|
||||||
6. Use Docker Compose to search for a specific CVE-ID and output results to a file:
|
|
||||||
```sh
|
|
||||||
docker-compose run --rm app python3 /usr/src/app/github_searcher.py --query=CVE-2024-5932 --search_type=in-repo-name --get_open_issues --get_open_pull_requests --get_file_contents --json > cve-2024-5932
|
|
||||||
```
|
|
||||||
|
|
||||||
7. Search for a specific repository by name and get open issues:
|
|
||||||
```sh
|
|
||||||
python github_searcher.py --query "torvalds/linux" --search_type in-repo-name --get_open_issues
|
|
||||||
```
|
|
||||||
|
|
||||||
8. Search for a specific repository by name and get open pull requests:
|
|
||||||
```sh
|
|
||||||
python github_searcher.py --query "apple/swift" --search_type by-repo-name --get_open_pull_requests
|
|
||||||
```
|
|
||||||
|
|
||||||
9. Search for a specific repository by name and get file contents:
|
|
||||||
```sh
|
|
||||||
python github_searcher.py --query "microsoft/vscode" --search_type by-repo-name --get_file_contents
|
|
||||||
```
|
|
||||||
|
|
||||||
10. Search for a specific repository by name and output results in JSON format:
|
|
||||||
```sh
|
|
||||||
python github_searcher.py --query "facebook/react" --search_type by-repo-name --json
|
|
||||||
```
|
|
||||||
|
|
||||||
## Output
|
|
||||||
|
|
||||||
The output will include the following details based on the flags provided:
|
|
||||||
|
|
||||||
- **Repository URL**: The URL of the repository.
|
|
||||||
- **Open Issues**: A list of open issues with their titles and URLs.
|
|
||||||
- **Open Pull Requests**: A list of open pull requests with their titles and URLs.
|
|
||||||
- **File Contents**: A list of file contents with their names, SHAs, sizes, encodings, and URLs.
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
This project is licensed under the MIT License.
|
|
|
@ -1,4 +1,4 @@
|
||||||
services:
|
services:
|
||||||
github_searcher:
|
app:
|
||||||
build: .
|
build: .
|
||||||
env_file: ".env"
|
env_file: ".env"
|
||||||
|
|
|
@ -1,13 +1,11 @@
|
||||||
import os
|
import os
|
||||||
import argparse
|
import argparse
|
||||||
import logging
|
import logging
|
||||||
import json
|
|
||||||
from github import Github, Auth, GithubException
|
from github import Github, Auth, GithubException
|
||||||
|
|
||||||
# Configure logging
|
# Configure logging
|
||||||
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
|
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
|
||||||
|
|
||||||
# authentication
|
|
||||||
def auth():
|
def auth():
|
||||||
access_token = os.getenv("GITHUB_ACCESS_TOKEN")
|
access_token = os.getenv("GITHUB_ACCESS_TOKEN")
|
||||||
if not access_token:
|
if not access_token:
|
||||||
|
@ -16,43 +14,6 @@ def auth():
|
||||||
g = Github(auth=auth)
|
g = Github(auth=auth)
|
||||||
return g
|
return g
|
||||||
|
|
||||||
"""
|
|
||||||
This class is used to search GitHub repositories to gather information about repos and users.
|
|
||||||
- search_repo: Search for repositories based on the query.
|
|
||||||
- search_users: Search for users based on the query.
|
|
||||||
- search_in_repo_name: Search for repositories based on the query in the repository name.
|
|
||||||
- search_by_repo_name: Search for repositories based on the query in the repository name.
|
|
||||||
- get_repo_open_issues: Get the open issues of a repository.
|
|
||||||
- get_repo_stars: Get the number of stars of a repository.
|
|
||||||
- get_repo_open_pull_requests: Get the open pull requests of a repository.
|
|
||||||
- get_repo_contents: Get the contents of a repository.
|
|
||||||
- get_result: Get the search result.
|
|
||||||
- query: The search query.
|
|
||||||
- result: The search result.
|
|
||||||
- g: The GitHub object.
|
|
||||||
|
|
||||||
Example usage:
|
|
||||||
# Searching for a CVE-ID in repositories:
|
|
||||||
searcher = GithubSearcher("CVE-2021-1234") # Initialize the searcher with the query
|
|
||||||
searcher.search_repo() # Search for repositories based on the query. This would search github for repositories with the query "CVE-2021-1234"
|
|
||||||
result = searcher.get_result() # Get the search result
|
|
||||||
|
|
||||||
# Searching for a specific repo by name:
|
|
||||||
searcher = GithubSearcher("php/php-src") # Initialize the searcher with the query
|
|
||||||
searcher.search_by_repo_name() # Search for repositories based on the query in the repository name. This would search github for the repository "php/php-src"
|
|
||||||
result = searcher.get_result() # Get the search result
|
|
||||||
|
|
||||||
# Getting open issues of a repository:
|
|
||||||
repo = result[0] # Get the first repository from the search result
|
|
||||||
open_issues = searcher.get_repo_open_issues(repo) # Get the open issues of the repository
|
|
||||||
for issue in open_issues:
|
|
||||||
print(issue.title) # Print the title of the issue
|
|
||||||
|
|
||||||
# Getting the number of stars of a repository:
|
|
||||||
repo = result[0] # Get the first repository from the search result
|
|
||||||
stars = searcher.get_repo_stars(repo) # Get the number of stars of the repository
|
|
||||||
|
|
||||||
"""
|
|
||||||
class GithubSearcher():
|
class GithubSearcher():
|
||||||
def __init__(self, query):
|
def __init__(self, query):
|
||||||
self.g = auth()
|
self.g = auth()
|
||||||
|
@ -73,44 +34,30 @@ class GithubSearcher():
|
||||||
logging.error(f"Error searching users: {e}")
|
logging.error(f"Error searching users: {e}")
|
||||||
self.result = None
|
self.result = None
|
||||||
|
|
||||||
def search_in_repo_name(self):
|
def search_in_name(self):
|
||||||
try:
|
try:
|
||||||
self.result = self.g.search_repositories('in:name ' + self.query)
|
self.result = self.g.search_repositories('in:name ' + self.query)
|
||||||
except GithubException as e:
|
except GithubException as e:
|
||||||
logging.error(f"Error searching in name: {e}")
|
logging.error(f"Error searching in name: {e}")
|
||||||
self.result = None
|
self.result = None
|
||||||
|
|
||||||
def search_by_repo_name(self):
|
|
||||||
try:
|
|
||||||
self.result = self.g.search_repositories(f'repo:{self.query}')
|
|
||||||
except GithubException as e:
|
|
||||||
logging.error(f"Error searching by repo name: {e}")
|
|
||||||
self.result = None
|
|
||||||
|
|
||||||
def get_repo_open_issues(self, repo):
|
def get_repo_open_issues(self, repo):
|
||||||
try:
|
try:
|
||||||
return repo.get_issues(state='open')
|
issues = repo.get_issues(state='open')
|
||||||
|
for issue in issues:
|
||||||
|
print(issue)
|
||||||
|
return issues
|
||||||
|
|
||||||
except GithubException as e:
|
except GithubException as e:
|
||||||
logging.error(f"Error getting open issues: {e}")
|
logging.error(f"Error getting open issues: {e}")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
def get_repo_stars(self, repo):
|
|
||||||
try:
|
|
||||||
return repo.stargazers_count
|
|
||||||
except GithubException as e:
|
|
||||||
logging.error(f"Error getting stars: {e}")
|
|
||||||
return None
|
|
||||||
|
|
||||||
def get_repo_open_pull_requests(self, repo):
|
|
||||||
try:
|
|
||||||
return repo.get_pulls(state='open')
|
|
||||||
except GithubException as e:
|
|
||||||
logging.error(f"Error getting open pull requests: {e}")
|
|
||||||
return None
|
|
||||||
|
|
||||||
def get_repo_contents(self, repo):
|
def get_repo_contents(self, repo):
|
||||||
try:
|
try:
|
||||||
return repo.get_contents("")
|
contents = repo.get_contents("")
|
||||||
|
for content_file in contents:
|
||||||
|
print(content_file)
|
||||||
|
return contents
|
||||||
except GithubException as e:
|
except GithubException as e:
|
||||||
logging.error(f"Error getting repository contents: {e}")
|
logging.error(f"Error getting repository contents: {e}")
|
||||||
return None
|
return None
|
||||||
|
@ -121,12 +68,9 @@ class GithubSearcher():
|
||||||
def main():
|
def main():
|
||||||
parser = argparse.ArgumentParser(description="Search GitHub repositories and users for PoC exploits and CVEs.")
|
parser = argparse.ArgumentParser(description="Search GitHub repositories and users for PoC exploits and CVEs.")
|
||||||
parser.add_argument("--query", type=str, required=True, help="The search query.")
|
parser.add_argument("--query", type=str, required=True, help="The search query.")
|
||||||
parser.add_argument("--search_type", type=str, required=True, choices=["repo", "users", "in-repo-name", "by-repo-name"], help="The type of search to perform: 'repo', 'users', 'in-repo-name', or 'by-repo-name'.")
|
parser.add_argument("--search_type", type=str, required=True, choices=["repo", "users", "name"], help="The type of search to perform: 'repo', 'users', or 'name'.")
|
||||||
parser.add_argument("--get_file_contents", action="store_true", help="Get the contents of repo results.")
|
parser.add_argument("--get_file_contents", action="store_true", help="Get the contents of a repository.")
|
||||||
parser.add_argument("--get_open_issues", action="store_true", help="Get the open issues of repo results.")
|
parser.add_argument("--get_open_issues", action="store_true", help="Get the open issues of a repository.")
|
||||||
parser.add_argument("--get_open_pull_requests", action="store_true", help="Get the open pull requests of repo results.")
|
|
||||||
parser.add_argument("--json", action="store_true", help="Output results in JSON format.")
|
|
||||||
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
searcher = GithubSearcher(args.query)
|
searcher = GithubSearcher(args.query)
|
||||||
|
@ -135,57 +79,29 @@ def main():
|
||||||
searcher.search_repo()
|
searcher.search_repo()
|
||||||
elif args.search_type == "users":
|
elif args.search_type == "users":
|
||||||
searcher.search_users()
|
searcher.search_users()
|
||||||
elif args.search_type == "in-repo-name":
|
elif args.search_type == "name":
|
||||||
searcher.search_in_repo_name()
|
searcher.search_in_name()
|
||||||
elif args.search_type == "by-repo-name":
|
|
||||||
searcher.search_by_repo_name()
|
|
||||||
|
|
||||||
result = searcher.get_result()
|
result = searcher.get_result()
|
||||||
if result is None:
|
if result is None:
|
||||||
print("No results found.")
|
print("No results found.")
|
||||||
return
|
return
|
||||||
|
|
||||||
output = []
|
|
||||||
for item in result:
|
for item in result:
|
||||||
if args.search_type == "users":
|
if args.search_type == "users":
|
||||||
user_repos = item.get_repos()
|
user_repos = item.get_repos()
|
||||||
|
|
||||||
for repo in user_repos:
|
for repo in user_repos:
|
||||||
repo_info = {"repo_url": repo.html_url, "repo_name": repo.name, "repo_description": repo.description}
|
print(repo.html_url)
|
||||||
output.append(repo_info)
|
|
||||||
else:
|
else:
|
||||||
repo_info = {"repo_url": item.html_url, "repo_name": item.name, "repo_description": item.description}
|
print(item.html_url)
|
||||||
|
|
||||||
if args.get_open_issues:
|
if args.get_open_issues:
|
||||||
open_issues = searcher.get_repo_open_issues(item)
|
searcher.get_repo_open_issues(item)
|
||||||
if open_issues:
|
print("\n")
|
||||||
repo_info["open_issues"] = [{"title": issue.title, "url": issue.html_url} for issue in open_issues]
|
|
||||||
repo_info['total_open_issues'] = item.open_issues_count
|
|
||||||
if args.get_open_pull_requests:
|
|
||||||
open_prs = searcher.get_repo_open_pull_requests(item)
|
|
||||||
if open_prs:
|
|
||||||
repo_info["open_pull_requests"] = [{"title": pr.title, "url": pr.html_url} for pr in open_prs]
|
|
||||||
repo_info["total_open_pull_requests"] = item.open_pulls_count
|
|
||||||
if args.get_file_contents:
|
if args.get_file_contents:
|
||||||
contents = searcher.get_repo_contents(item)
|
searcher.get_repo_contents(item)
|
||||||
if contents:
|
print("\n")
|
||||||
repo_info["contents"] = [
|
|
||||||
{
|
|
||||||
"name": content_file.name,
|
|
||||||
"sha": content_file.sha,
|
|
||||||
"size": content_file.size,
|
|
||||||
"encoding": content_file.encoding,
|
|
||||||
"html_url": content_file.html_url
|
|
||||||
} for content_file in contents
|
|
||||||
]
|
|
||||||
|
|
||||||
output.append(repo_info)
|
|
||||||
|
|
||||||
if args.json:
|
|
||||||
print(json.dumps(output, indent=4))
|
|
||||||
else:
|
|
||||||
for item in output:
|
|
||||||
print(item)
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
main()
|
Loading…
Add table
Reference in a new issue