ProSpector

SaaS Lead generation software, commissioned by Mediavest AS.
Data source: brreg.no  nbsp;nbsp extraction targets: gulesider.no, 1881.no, proff.no and gooogle.com.   

Copyright Stamement

Please note that this repository is temporarly set to public due to job-applications, this is a private source-code to a product for sale. You do not have premission to use this code for private use or financial gain. As defined by the Copyright act 1968; no content from this repository may be reproduced, transmitted or copied without the ecpress written permission of A.Borgerød.
Application made by Aleksander Borgerød, all rights reserved to (C) 2022 A.Borgerød ENK.

Project Summary

• Job Completed: October 2022 [renovated; november 2022]
• Job type: B2B SaaS development
• Job categogy: Data Extraction, Lead Generation, Database server, Cloud Computing, API integration.
• Language: Python, Dart
• Style: OOP

Rundown:

[ENG] The software's tasks:

So what we want to achieve is that you can retrieve lists of companies that have both paid advertisements in the catalogs and have a vacant, deficient or no profile on Google.

• Step 1: Gulesider 1881, or proff.no after --> Companies with paid paid

• Step 2: Scrape google maps for --> Which does not have an owner confirmed (Listed as "do you own this business")

• Step 2 addition: Must also scrape for companies that have an "insufficient profile" (has confirmed the owner without content)

  The software structure, simply explained:

  The shop             : App (frontend)  : /App
  The back-office      : App (backend)   : /fast_api_server/*
  The delivery truck   : Api             : /fast_api_server/main.py
  The Warehouse        : Cloud storage   : [AWS](https://aws.amazon.com/)
  The Manifacturer     : Data Extractor  : /backend
  

[NO] Programvarens oppgaver:

Så det vi vil få til er at man kan hente ut lister over bedrifter som har både betalt annonsering hos katalogene og har ledig, mangelfull eller ingen profil på Google.

• Steg 1: Gulesider 1881, eller proff.no etter --> Bedrifter med betalt oppføring

• Steg 2: Skrape google maps etter --> Som ikke har en eier bekreftet (Står oppført som "eier du denne bedriften")

• Steg 2 tillegg: Skal også skrape etter bedrifter som har "mangelfull profil" ( har bekreftet eier men ingen innhold)

  The software structure, simply explained:

  Butikken : App (frontend) : /App Bakrommet : App (backend) : /fast_api_server/* varebil : Api : /fast_api_server/main.py Lageret : Cloud storage : AWS Fabrikken : Data Extractor : /backend

Notes About Porject:

What is good and what is bad?

• [bad] virtual_env; I am very new to venv, so it's a mess and is probably the least useful part of the project.
• [bad] user_backend/fast_api_server; The overall structure is too complicated and is hard to use. SQL in dev_backend is much more easier to understand.
• [bad] ProSpector; Still contains some dead code, unnessasary files and backups, will be removed.
• [bad] App/prospector_app; Some implementations are very crude, e.g.: dark-light mode
• [good] App/prospector_app; Overall clear file structure and visually pleasing UI
• [good] dev_backend; is the most refined part, and runs fast
• [good] ProSpector; Has an overall clear file structure

Personal notes

• Learn how to properly use virtual env
  • it seems like if done properly; python_installer
• Lean how to gitignore

Project Map

PreView

GLOSSARY

Term	Definition
2C4E	Too crude for Efficiency

TODO

[Todo-list last updated: 28.11.2022]

Current Progress for todo tasks:

Task 2. PhoneNumber-Implemention for GoogleExtractor

No.	Progress
1	added tlf to input_table [tested]
2	added tlf to Proff Extractor [untested]
3	added tlf to Gulesider Extractor [tested]
4	added tlf to 1881 Extractor [untested]
5	implemented tlf to SQL code

Todo List:

Overall:

• fix .gitignore
• fix issues with & cleanup all env files
• Make gtihub repo private and give installer access key (or something like that)
• rename /Mediavest_Scraper_bot/ to /ProSpector/
• add to .bat add uvicorn to path
• add a python checker in setup.exe
• Fix Login button error [high prio] (ROOT BUG)
• Fix loading screen bug [low prio] (Unknown)
• fix Expanded - Stack bug [high prio] (ROOT BUG)
• fix plain_background error [medium prio] (RESIDUAL BUG)
• Add installer
• Implement update-function
• Add function that starts fastAPI server upon app-launch
• change update function to change "current version" to 1.X+1 after the installation was sucsessfull.
• make sure that server-terminal opens either in background / minimized or behind UI
• ...

Frontend:

• add phone_verification to your app
• BIG ISSUE: CallListWidget{} when changing to darkmode/lightmode the "Ringestatus" checkboxes resets.
• Fix darkmode Issue on app Startup [note: see small bugs]
• Make notes-state presistant [note: a small bug with loading]
• Complete translation to Norwegian
• Write instructions
• Add: "press enter" function to LoginWidget{}
• Finish "remember me" knapp
• fix text borders in instructions
• [ Low Prio ] Fix border Issue with popups
• [ Low Prio ] Increase min.windowsize
• [ Low Prio ] Change fontweight for bodytext in "renew list"
• [ Low Prio ] Make "Instructions from Supervisor" section in "Instructions"
  • Add section editable for supervisors & other privilages
  • Add Column "supervisor" in user-db
• [ Low Prio ] Clean up files (but first verify functionality, then backup)
• [ Low Prio ] Maybe add splash screen / loading screen
• [ Low Prio ] Make instructions look Nicer
• ...

Frontend (small bugs):

• Loading issue with presistant state [notes_page.dart]
• Laggy transition of light/darkmode [menu_widget.dart]
• Acryllic allways opens in darkmode [menu_widget.dart]
• Maximize window, often needs 2 clicks [window_title_bar.dart] (I think)
• Overflow in About in smallwindow [about_widget.dart / main.dart]
• ...

API Server:

• Make api server
• Finish change-password api
• Finish/fix "view call list" / "renew list" button.
• add Exeption handling for more precise exceptions, irt. signup-errors (all types of signup-errors are returned as 500 "Internal server error" )
• reorganize fast_api_server; remove all the unnessasary folders.
• re-Enable send EMAIL functionality
• Consider merging ./backend/SQL/ & ./fast_api_server/ ==> Will merge? [ ] yes or [x] no
• ...

Extractor:

• Finish brreg
• Finsih proff
• Finish 1881
• Finish gulesider
• Finish google
• Add function that detects googles "search-suggestion", parse & extract
• make phone_verification-backend
• integrate phone_verification-backend to fast-api-server
• make sure if you need an if statement to restrict to one account pr phone number
• Investigate "wrong org_num"-bug for some of the prospects, see bugs[1]
  • add tag on input_list from BRREG: "if Organisasjonsform == "Norskregistrert utenlandsk foretak": check & confirm org_num"
• Renovation of code in postgres.py --> Status Quo: 2C4E
• In GoogleExtractor; Include location data in google search
• Include Extraction of Phone numbers
• make file that fetches location data from brreg_table
• ...

Extractor Optimization:

Use practises from "clean code" to optimize and convert code to OOP..

• Renovation of GulesiderExtractor
• Renovation of ProffExtractor
• Renovation of _1881Extractor
• Renovation of GoogleExtractor

Backend:

• Overview: manage user progress (call list)
• Manage user login
• Password hasher
• ...

Database:

• Remove prospects in "call_list" that has [true, true, true]
• add phone_numbers to your user database

AWS:

• uplaod db to aws
• Test if RDS works without postgres.
• ...

Notes from Client:

Det eneste vi ser etter i et prospekt er om de har:
-  Betalt oppføring på: Proff, 1881 eller Gulesider
-  Ingen, ikke-bekreftet eller mangelfull Google-profil 
	
"De andre kriteriene er ikke viktig for da det tyder på at de har behov for markedsføring da de har kjøpt oppføring hos katalogene."

utelukke:
- mediebyråer & reklamebyråer
•	Mediebyråer , Markedsføringskonsulenter  , Medie- og kommunikasjonsrådgivning , Internettdesign og programmering , Reklamebyrå