Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
384e16b
Grab the html using Beutifulsoup
altafshaikh Jul 10, 2020
0e86518
grab the subdomain column of cert.sh
altafshaikh Jul 10, 2020
e44e043
clean the data(subdoamin)--removed-><td>,<br>,<br/></td>
altafshaikh Jul 10, 2020
9d7d6a0
getting all domains list
altafshaikh Jul 10, 2020
a4d6b7c
extracted all unique domains
altafshaikh Jul 10, 2020
82d2970
writing subdomain into text files wrt domain name
altafshaikh Jul 10, 2020
7bf1c29
tested for-> google.com and realpython.com
altafshaikh Jul 10, 2020
6507b0b
added requirements.txt
altafshaikh Jul 13, 2020
caa31f7
added .gitignore
altafshaikh Jul 13, 2020
81d8fb5
removed output
altafshaikh Jul 13, 2020
380de13
script renamed
altafshaikh Jul 13, 2020
25688d2
setup censys
altafshaikh Jul 13, 2020
79535c2
censys subdomain finder function implemeted
altafshaikh Jul 13, 2020
f827196
implemente save subdomains to file fucntion
altafshaikh Jul 13, 2020
7fe651e
Censys Scrapping complete
altafshaikh Jul 13, 2020
4b6b484
shodan integration failed
altafshaikh Jul 14, 2020
4c2d127
hacker target working
altafshaikh Jul 20, 2020
13d2303
added hacker target
altafshaikh Jul 20, 2020
0003c3b
added ThreatCrowd added
altafshaikh Jul 20, 2020
07188ab
tried some website didn't work
altafshaikh Jul 20, 2020
327f134
enhanced the code
altafshaikh Jul 21, 2020
3843374
renamed the script
altafshaikh Jul 21, 2020
e3fbc41
error handling
altafshaikh Jul 21, 2020
07b7028
added sample results
altafshaikh Jul 21, 2020
fb03f42
formated
altafshaikh Jul 21, 2020
7d98eb7
Update README.md
altafshaikh Jul 21, 2020
096559b
Update README.md
altafshaikh Jul 21, 2020
06a7dab
added youtube demo link
altafshaikh Jul 21, 2020
5a3afc9
username update
altafshaikh Sep 23, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
env/*
set_env_variables.txt
outputs/*
envvariable.sh
98 changes: 97 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,97 @@
# Tools
# X Scrappy

## The Subdomain Finder

X Scrappy is a python based tool. which is used to find and collect sub-domains for a given domain from various websites.

## Supported Services:
The tool is collecting data from the following services:
* [CRT.sh](https://crt.sh/)
* [Censys](https://censys.io/)
* [HackerTarget](https://hackertarget.com/)
* [ThreatCrowd](https://www.threatcrowd.org/)

## Features
* Find Subdomains from various services
* Filter Out Duplicates sub-domains
* Stores all the unique subdomains into a file inside the folder with the name of the domain.

## Demo
[![Tool Working Demo](https://img.youtube.com/vi/4thNDNLRat0/0.jpg)](https://www.youtube.com/watch?v=4thNDNLRat0)

## Dependencies
`python3`

## Requirements

|Package|Version|
|---|---|
|requests|2.22.0|
|beautifulsoup4|4.8.2|
|censys|0.0.8|
|shodan|1.11.1|
|termcolor|1.1.0|

## Installation

1. Normal installation:
```
$ git clone https://github.com/altaf99/Tools.git
$ python3 -m pip install -r requirements.txt
```

2. Preferably install in a virtualenv:
```
Clone the repo into your machine
$ git clone https://github.com/altaf99/Tools.git

Create a virtual environment,
$ virtualenv -p /usr/bin/python3.8 myvenv

Now, at last, we just need to activate it, using the command
$ source myvenv/bin/activate

Now you are in a Python virtual environment

You can deactivate using
$ deactivate

and now install all the requirements
$ python3 -m pip install -r requirements.txt

```

## Setup
the tool requires an API key for Censys,
visit [Censys API](https://censys.io/api) and Sign Up to get the below API credentials.
```
CENSYS_API_ID
CENSYS_API_SECRET
```
## Setting environment variable :
use the below command to set the environment variable, assign the value to the variables with your credentials, and enter the command in the terminal.
```
export CENSYS_API_ID="YOUR_ID"
export CENSYS_API_SECRET="YOUR_SECRET"
```

## Usage:
```
python3 x_scrappy.py domain
```

## Example:
```
python3 x_scrappy.py google.com
```
## Credits:

### Developed by
[Altaf Shaikh](https://github.com/altafshaikh)<br>
[Email](mailto:iamaltafshaikh@gmail.com)

### Special Thanks
The tool is made as a part of the STTP Program by [We Are plymouths](https://github.com/wap-plymouths)<br>
Thank You So much for conduction such a wonderful 21 days program :100:

Also thanks to [Harsh Bothra](https://github.com/harsh-bothra) sir for helping me to build the tool :)
6 changes: 6 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
requests==2.22.0
beautifulsoup4==4.8.2
censys==0.0.8
shodan==1.11.1
termcolor==1.1.0

Loading