Add rudimentary keyword search for CCPA policies using HTML file

[objorkman]

url = 'https://docs.github.com/en/github/site-policy/github-privacy-statement'  # Not used!
result = polipy.get_policy(url, html_file='policy.html')  # policy.html contains file to check

result.save(output_dir='.')

Output is currently printed and stored as a string

[nsamarin]

Looks good! I would do it this way so you can save the extracted keywords separately, as a separate extractor:

def extract(extractor, **kwargs):
    content = None
    if extractor == 'text':
        content = extract_text(**kwargs)
    elif extractor == 'keywords' and 'html_file' in kwargs and kwargs['html_file'] is not None:
        content = extract_ccpa_info(kwargs['html_file'])
    return content

def extract_text(url_type, url=None, dynamic_source=None, static_source=None, **kwargs):
    if url_type is None or url_type in ['html', 'other']:
        content = extract_html(dynamic_source, url)
    elif url_type == 'pdf':
        content = extract_pdf(static_source)
    elif url_type == 'plain':
        content = dynamic_source
    else:
        content = dynamic_source
    return content

Maybe take a look and see if that achieves the same functionality? (apart from not saving it under the "text" extractor).

[nsamarin]

Don't forget to remove the print statements at some later point.

[nsamarin]

Is this import needed here?

[nsamarin]

Yeah, I think this is fine for now. Maybe ultimately we could merge the self.htm_file better into the Polipy object, as that's basically the exact same attribute as the self.source['dynamic_html'] after the scraping part. But otherwise it's good to have an option of either providing an URL (and then scraping) or the already-scraped HTML to the constructor.