wsp2j

Web Scrapping Project 2 for Java (wsp2j). Automated locally hosted periodic website monitoring and logging using JSoup.

Input Files

target.list - Input Target List in CSV

• Required Fileds: 3 (Target Classification, Target ID, Target URL)
• Header: Optional (Required Header Format: CLASSIFICATION,xxx,yyy)

wsp2j.frequency - Scheduler Timings File (optional; you will be asked to provide timings in the console if the file is not present)

• Required Format HH mm SS, hours minutes seconds

targets.obj - Targets State Space (optional; autogenerated;)

• Serialized vector of targets generated everytime the targets are refreshed.

Output Files

wsp2j-monitoring-YYYYmmDD-HHmmSS.txt - wsp2j Monitor Log (autogenerated)

Operation Modes

Keys l - Input List File, o - Input State Space, t - Input Timings File, - - Any, x - Not Found

• --t : Timings loaded from input file
• --x : Timings loaded from console prompt
• lx- : Create and Start Mode: Creates target vector database and initiates timing scheduler
• xo- : Resume Mode: Loads target vector database from file and initiates timing scheduler
• lo- : Update and Resume Mode: Loads and updates the vector database from files and initiates timing scheduler

Operational Details

• Initialization Sequence

  1. Scan targets and load the database into memory
  2. Load required SSH certificates (into local JDK keystore) for relevant domains (skipped if all certificates are up-to-date)
  3. Scan and load timings for the scheduler sequence
  4. Scan and load the required plugins

• Run Sequence

  1. Schedule single-threaded executor service to refresh target hashcodes.
  2. Trigger plugin hook if hashcode change is registered.

• Error Handling

  Not present; you are on your own.

Copyright (C) 2024, Prajval K (@prajvalk) & Chlorine Pentoxide (@ChlorinePentoxide)

MIT LICENSE