api connection Failed to resolve '2.intelx.io'

[Michael-ccy]

Using pip3, I installed the API package and wrote a test.py script to simulate part of the functionality of python intelx.py in the SDK. However, when using a loop to call test.py for file downloads, I encountered two types of errors. One is the inability to resolve /intelligent/search, and the other, more common, error is /file/read?type=0&systemid=9e93df53-b7e4-439c-8c8c-fffac6eaf77f&bucket=leaks.public.general Both seem to result in the error "Failed to resolve '2.intelx.io' ([Errno -2] Name or service not known)."

I would like to ask whether this is purely a network issue or if it is related to query frequency limits or the subscription plan. Thank you.

ERROR: Traceback (most recent call last):                                                                                                                                                                           
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn                                                                                               
    sock = connection.create_connection(                                                                                                                                                                           
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                           
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):                                                                                                                                         
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                              
  File "/usr/lib/python3.12/socket.py", line 976, in getaddrinfo                                                                                                                                                   
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):                                                                                                                                        
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                         
socket.gaierror: [Errno -2] Name or service not known                                                                                                                                                              
                                                                                                                                       
The above exception was the direct cause of the following exception:                                                                   
                                                                                                                                       
Traceback (most recent call last):                                                                                                     
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen                 
    response = self._make_request(                                                                                                     
               ^^^^^^^^^^^^^^^^^^^                                                                                                     
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e                                                    
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)                                      
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()                                                                                                                     
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/connection.py", line 704, in connect
    self.sock = sock = self._new_conn()                                                                                                                                                                                                                                        
                       ^^^^^^^^^^^^^^^^                            
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/connection.py", line 205, in _new_conn
    raise NameResolutionError(self.host, self, e) from e                                                                               
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7f43f00a67b0>: Failed to resolve '2.intelx.io' ([Errno -2] Name or service not known)
                                                                   
The above exception was the direct cause of the following exception:                                                                   
                                                                   
Traceback (most recent call last):                                 
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(                                           
           ^^^^^^^^^^^^^                                           
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(                                   
              ^^^^^^^^^^^^^^^^^^                                   
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]                                                      
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='2.intelx.io', port=443): Max retries exceeded with url: /intelligent/search (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f43f00a67b0>: Failed to resolve '2.intelx.io' ([Errno 
-2] Name or service not known)"))                                  

During handling of the above exception, another exception occurred:                                                                    

Traceback (most recent call last):                                 
  File "/user/Desktop/tools/intX/test.py", line 279, in <module>                                                                       
    data = search(                                                 
           ^^^^^^^                                                 
  File "/user/Desktop/tools/intX/test.py", line 13, in search                                                                          
    s = ix.search(args.search, maxresults, buckets, timeout, datefrom, dateto, sort, media, terminate)                                 
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                 
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/intelxapi.py", line 549, in search                               
    search_id = self.INTEL_SEARCH(term, maxresults, buckets, timeout, datefrom, dateto, sort, media, terminate)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/intelxapi.py", line 275, in INTEL_SEARCH
    r = requests.post(self.API_ROOT + '/intelligent/search', headers=h, json=p)                                                        
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                        
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/requests/api.py", line 115, in post                              
    return request("post", url, data=data, json=json, **kwargs)                                                                        
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                        
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)                                                                           
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                           
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)                          
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                          
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)                            
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                            
  File "/user/Desktop/tools/intX/pipxapi/lib/python3.12/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)                      
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='2.intelx.io', port=443): Max retries exceeded with url: /intelligent/search (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f43f00a67b0>: Failed to resolve '2.intelx.io' ([Err
no -2] Name or service not known)")) 

ix.FILE_READ()
error HTTPSConnectionPool(host='2.intelx.io', port=443): Max retries exceeded with url: /file/read?type=0&systemid=9e93df53-b7e4 -439c-8c8c-fffac6eaf77f&bucket=leaks.public.general (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7fd9e3ef31a0>: Failed to resolve '2.intelx.io' ([Errno -2] Name or service not known)"))

[cmlh]

I can resolve 2.intelx.io right now so I'd assume @Michael-ccy network connection dropped out therefore resulting in https://urllib3.readthedocs.io/en/stable/reference/urllib3.exceptions.html#urllib3.exceptions.NameResolutionError

Can @Michael-ccy send me their script?

[Michael-ccy]

When performing batch queries, not every query and download encounter the issue of being unable to resolve 2.intelx.io. Currently, it appears occasionally and causes the program to terminate.

test.py

from intelxapi import intelx
import os,sys
import re,csv
import argparse
from datetime import datetime

def format_list(content):
    content = content.replace(" ", "")
    return content.split(",")

def search(ix, query, maxresults=100, buckets=[], timeout=5, datefrom="", dateto="", sort=4, media=0, terminate=[]):
    s = ix.search(args.search, maxresults, buckets, timeout, datefrom, dateto, sort, media, terminate)
    return s

def sanitize_filename(filename):
    sanitized = re.sub(r'[^\w]+', '_', filename)
    sanitized = sanitized.strip('_')
    return sanitized

def process_csv(PATH, input_file, domain_str):
    file_size = os.path.getsize(input_file)
    if file_size < 23:
        return

    internal_network_file = os.path.join(PATH, "01_"+domain_str.replace('.','_')+"_internal.csv")
    domain_file           = os.path.join(PATH, "02_"+domain_str.replace('.','_')+"_urls.csv")
    others_file           = os.path.join(PATH, "03_"+domain_str.replace('.','_')+"_other_urls.csv")

    internal_network_pattern = re.compile(
        r"^(http:\/\/|https:\/\/)?(192\.168\.\d{1,3}\.\d{1,3}|10\.\d{1,3}\.\d{1,3}\.\d{1,3}|172\.(1[6-9]|2\d|3[0-1])\.\d{1,3}\.\d{1,3}|127\.0\.0\.1)"

    )

    # for unique use
    internal_network = set()
    domain_urls      = set()
    others           = set()
    
    with open(input_file, newline='', encoding='utf-8') as csvfile:
        reader = csv.DictReader(csvfile)
        for row in reader:
            url = row["URL"]
            if internal_network_pattern.match(url):
                internal_network.add((row["URL"], row["USERNAME"], row["PASSWORD"]))
            elif domain_str in url:
                domain_urls.add((row["URL"], row["USERNAME"], row["PASSWORD"]))
            else:
                others.add((row["URL"], row["USERNAME"], row["PASSWORD"]))


    def write_to_file(output_file, data):
 
        sorted_data = sorted(data, key=lambda x: x[0])  
        with open(output_file, mode='w', newline='', encoding='utf-8') as csvfile:
            writer = csv.writer(csvfile)
            writer.writerow(["URL", "USERNAME", "PASSWORD"]) 
            writer.writerows(sorted_data)

    write_to_file(internal_network_file, internal_network)
    write_to_file(domain_file, domain_urls)
    write_to_file(others_file, others)

    print(f"Output save hrer：\n- {internal_network_file}\n- {domain_file}\n- {others_file}")



def extract_url_user_pass(data, csv_writer):           
                                                                                      
    filter_url = r'http[s]?://[^\s]+'                               
    patterns = [                                                                      
                r'URL:\s*(http[^\s]+)\s*USER:\s*([^\s]+)\s*PASS:\s*(\S+)',           
                r'(http[^\s]+):([^:]+):(\S+)',                                         
                r'(http[^\|]+)\|([^|]+)\|(\S+)',                                       
                r'(http[^\s|]+)[|:]([^|:]+)[|:]([^|:]+)'                    
    ]          
                                           
    for idx, item in enumerate(data):
        
        url_match = re.search(filter_url, item)
        if url_match and (idx+2 < len(data)) and ('USER:' in data[idx+1]) and ('PASS:' in data[idx+2]):
            url = url_match.group()  
            user = data[idx + 1].split("USER:")[1].strip()      
            password = data[idx + 2].split("PASS:")[1].strip()  
            csv_writer.writerow([url, user, password])

        else:
            def fix_url(url):
                url = url.rstrip(',')
                if not url.startswith('http'):
                    url = 'https://' + url
                return url
            for pattern in patterns:       
                match = re.search(pattern, item)
                if match:
                    url_origin, user_origin, password = match.groups()
                    if ('http' in user_origin) or ('http' in password):
                        continue
                    if (' ' in user_origin):
                        url = url_origin+user_origin.split(' ')[0]
                        user = user_origin.split(' ')[1]
                        csv_writer.writerow([url, user, password])
                    else:
                        url, user, password = match.groups()                              
                        csv_writer.writerow([url, user, password])                      

          

def extract_credentials_and_urls(input_dir, output_file, domain_url):
    url_pattern = r'((http|https)://(?:[a-zA-Z0-9.-]+|\d{1,3}(?:\.\d{1,3}){3})(:\d+)?(/[\w.-]*)?)'
    with open(output_file, 'w', newline="", encoding="utf-8") as outfile:
        writer = csv.writer(outfile)
        writer.writerow(["URL","USERNAME","PASSWORD"])
        for root, dirs, files in os.walk(input_dir):
            for file in files:
                file_path = os.path.join(root, file)
                with open(file_path, 'r', encoding='utf-8', errors='ignore') as infile:
                    content = infile.read()
                    url_matches = re.findall(url_pattern, content)                    
                    for url in url_matches:
                        url_str = url[0]
                        if domain_url in url_str and len(url_str) < 250:
                            lines = content.splitlines()
                            for idx, line in enumerate(lines):
                                if url_str in line:
                                    start_idx = max(0, idx-2)
                                    end_idx = min(len(lines), idx+3) # +3 contain itself and for more 2 lines
                                    context_lines = lines[start_idx:end_idx]
                                    extract_url_user_pass(context_lines, writer)



if __name__ == '__main__':
    
    BOLD = '\033[1m'
    END = '\033[0m'
	banner = r'''
	{}     _____      _       ___   __
		|_   _|    | |     | \ \ / /
		| | _ __ | |_ ___| |\ V /
		| || '_ \| __/ _ \ |/   \
		_| || | | | ||  __/ / /^\ \
		\___/_| |_|\__\___|_\/   \/

		a command line client
           for intelx.io         {}

	'''.format(BOLD, END)
    print(banner)
    parser = argparse.ArgumentParser(
        description="Command line interface for https://intelx.io",
        epilog="Usage: iris -search 'riseup.net' -buckets 'pastes, darknet'"
    )

    parser.add_argument('-apikey', help="set the api key via command line")
    parser.add_argument('-search', help="search query")
    parser.add_argument('-buckets', help="set which buckets to search")
    parser.add_argument('-limit', help="set the amount of results to show")
    parser.add_argument('-timeout', help="set the timeout value")
    parser.add_argument('-datefrom', help="begin search starting from state")
    parser.add_argument('-dateto', help="begin search ending from date")
    parser.add_argument('-media', help="set the media value")
    parser.add_argument('-download', help="download the specified item specified by its ID")
    parser.add_argument('--view', help="show full contents of search results", action="store_true")
    parser.add_argument('-name', help="set the filename to save the item as")
    parser.add_argument('-keyword', help="data key word with case insensitive ex.login, user, pass, etc.")
    args = parser.parse_args()
    

    ix = intelx('myapikeyhere')
    buckets_set = [
            "darknet.i2p",
            "darknet.tor",
            "documents.public.scihub",
            "dumpster",
            "dumpster.web.1",
            "dumpster.web.ssn",
            "leaks.logs",
            "leaks.private.general", 
            "leaks.public.general", 
            "leaks.public.wikileaks",
            "pastes",
            "patent.us",
            "usenet",
            "web.gov.ru",
            "web.public.af",
            "web.public.ams",
            "web.public.aq",
            "web.public.business",
            "web.public.cee",
            "web.public.cn",
            "web.public.com",	
            "web.public.de",
            "web.public.eu",
            "web.public.gov",
            "web.public.info",
            "web.public.kp",
            "web.public.mea",
            "web.public.misc",	
            "web.public.net",
            "web.public.nord",	
            "web.public.oc",
            "web.public.org",
            "web.public.peer",	
            "web.public.ru",
            "web.public.social",	
            "web.public.tech",
            "web.public.we",
            "whois"
    ]

    default_buckets = [
            "leaks.logs",
            "leaks.private.general", 
            "leaks.public.general", 
            "leaks.public.wikileaks"
            ]
    class CustomHelpFormatter(argparse.HelpFormatter):
      def format_help(self):
        help_text = super().format_help()
        bucket_info = "\nAvailable buckets:\n"
        bucket_info += "\n".join(f"  - {bucket}" for bucket in bucket_set)
        return help_text + bucket_info
    
    media_type = 24 # Text file

    if args.search:

        #The maxresults field tells how many results to query maximum per bucket
        maxresults = 100
        buckets = []
        timeout = 5
        datefrom = ""
        dateto = ""
        sort = 4
        media = 0
        terminate = []
        textview = 0
        DIR_PATH = os.path.join('./data', args.search.replace('.','_'))
        if not os.path.exists(DIR_PATH):
            os.mkdir(DIR_PATH)

        if args.limit:
            maxresults = int(args.limit)
        if args.buckets:
            buckets = format_list(args.buckets)
        else:
            print(f"The '-buckets' argument is required .\n\nAvailable buckets:\n" +
                 "\n".join(f"  - {bucket}" for bucket in buckets_set))
            print(f" Using default buckets:{default_buckets}")
            buckets = default_buckets
        if args.timeout:
            timeout = int(args.timeout)
        if args.datefrom:
            datefrom = args.datefrom
        if args.dateto:
            dateto = args.dateto
            sort = 2  # sort by date
        if args.media:
            media = int(args.media)
        if args.keyword:
            pattern = format_list(args.keyword.lower())
        if args.name:
            output_file = DIR_PATH + str(args.name)+'.txt'
        else:
            output_file = DIR_PATH + '_final.txt'
            

        data = search(
            ix,
            args.search,
            maxresults=maxresults,
            buckets=buckets,
            timeout=timeout,
            datefrom=datefrom,
            dateto=dateto,
            #sort=sort,
            media=media
            #terminate=terminate
        )
        for i, result in enumerate(data['records'], start=1):
            downloaded_files = set()
            if args.keyword:
                for key in pattern:
                    if key in result['name'].lower():
                        output_filename = os.path.join(DIR_PATH,sanitize_filename(result['name'])+'.txt')
                        if output_filename not in downloaded_files:
                            print("=" * 50)
                            print(f"{'File Downloader'.center(50)}")
                            print(f"{output_filename.center(50)}")
                            print("=" * 50)
                            ix.FILE_READ(result['systemid'], textview, result['bucket'], output_filename)
                            downloaded_files.add(output_filename)

                

            else:
                print("=" * 50)
                print(f"{'No keyword param, so search all'.center(50)}")
                print("=" * 50)
                downloaded_files = set()
                output_filename = os.path.join(DIR_PATH,sanitize_filename(result['name'])+'.txt')
                if output_filename not in downloaded_files:
                    print("=" * 50)
                    print(f"{'File Downloader'.center(50)}")
                    print(f"{output_filename.center(50)}")
                    print("=" * 50)
                    ix.FILE_READ(result['systemid'], textview, result['bucket'], output_filename)
                    downloaded_files.add(output_filename)
                    



        # parser data
        print("=" * 50)
        print(f"{'Processing credential and url'.center(50)}")
        print("=" * 50)
        extract_credentials_and_urls(DIR_PATH, output_file, args.search)

        file_size = os.path.getsize(output_file)
        if file_size > 30: #file with onely title url,user,password is 23 bytes
            process_csv(DIR_PATH, output_file, args.search)
            current_time =datetime.now().strftime("%Y-%m-%d %H:%M:%S")
            print(f"\n{('Finished ' + current_time).center(50)}")
        else:
            current_time =datetime.now().strftime("%Y-%m-%d %H:%M:%S")
            print(f"\n{('No user pass info ' + current_time).center(50)}")
        os.remove(output_file)

loop.py

#!/usr/bin/python
import subprocess
import time


keywords = "login,user,pass,browswer,edge,chrome,rar,history,cache"
limit = 200


with open("/path/to/domain/domain_list.txt", "r") as file:
    domains = [line.strip() for line in file if line.strip()]

def run_intelx(domain):
    try:

        command = [
            "python", "test.py",
            "-search", domain,
            "-keyword", keywords,
            "-limit", str(limit)
        ]


        print(f"command: {' '.join(command)}")
        result = subprocess.run(command, capture_output=True, text=True)


        print(f"Output: {result.stdout}")
        if result.stderr:
            print(f"error: {result.stderr}")

    except Exception as e:
        print(f"execute {domain} error: {e}")


for domain in domains:
    print(domain)
    run_intelx(domain)
    time.sleep(3)

I tried modifying the lib/python3.12/site-packages/intelxapi.py file starting from line 144 to prevent the program from terminating, but I feel that this is not the correct solution. That's why I am seeking advice here. Thank you.

145         h = {'x-key': self.API_KEY, 'User-Agent': self.USER_AGENT}                                   
146         try:                                                                                         
147             r = requests.get(f"{self.API_ROOT}/file/read?type={type}&systemid={id}&bucket={bucket}", headers=h, stream=True)                                                                                   
148             r.raise_for_status()                                                                     
149             if not r.content:                   
150                 print(f"{filename} downlaod error")                                                  
151                 return True                     
152                                                                                                      
153             with open(f"{filename}", "wb") as f:
154                 f.write(r.content)                                                                   
155                 f.close()                                                                            
156             return True                                                                              
157         except requests.exceptions.RequestException as e:                                            
158             print(f"Download {filename} error {e}")                                                  
159             return True                                                                              
160         except Exception as e :                                                                      
161             print(f"unknown error {e}")                                                              
162             return True                        

The program output will be as follows:

==================================================                                                                                                                                                                 
                 File Downloader                                                                                                                                                                                   
./data/test_domain_for_download/MetaMask_125_70_163_84_rar_Chrome_Default_History_txt.txt                                                                                                                           
==================================================                                                                                                                                                                 
==================================================                                                                                                                                                                 
                 File Downloader                                                                                                                                                                                   
./data/test_domain_for_download/101_64_34_117_2023_09_11_04_23_24_rar_cookie_list_txt.txt                                                                                                                           
==================================================                                                                                                                                                                 
==================================================                                                                                                                                                                 
                 File Downloader                                                                                                                                                                                   
./data/test_domain_for_download/101_64_34_117_2023_09_11_04_23_24_rar_cookies_Microsoft_Edge_Default_txt.txt                                                                                                        
==================================================                                                                                                                                                                 
Download ./data/test_domain_for_download/101_64_34_117_2023_09_11_04_23_24_rar_cookies_Microsoft_Edge_Default_txt.txt error HTTPSConnectionPool(host='2.intelx.io', port=443): Max retries exceeded with url: /file/
read?type=0&systemid=34dfa0df-b40b-43de-8597-6bdf013d2ff8&bucket=leaks.logs (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f28dbf96600>: Failed to resolve '2.intelx.io' ([Errno 
-2] Name or service not known)"))                                                                                                                                                                                  
==================================================                                                                                                                                                                 
                 File Downloader                  
./data/test_domain_for_download/101_64_34_117_2023_09_11_04_23_24_rar_cookies_Google_Chrome_Default_txt.txt                                                                                                         
==================================================                                                                                                                                                                 
==================================================                                                                                                                                                                 
                 File Downloader                  
./data/test_domain_for_download/2023_08_29_17_43_yxba0e_rar__AllCookies_list_txt.txt                      
==================================================                                                                                                                                                                 
==================================================                                                                                                                                                                 
                 File Downloader                  
./data/test_domain_for_download/125_69_9_34_rar_Chrome_Profile_1_Cookies_txt.txt                          
==================================================          

[cmlh]

I believe the solution would be to cache dns.resolver that is reused by requests

Can @Michael-ccy issue a Pull Request of #571 (comment) please?