PHP-SecurityServicesScraper is a proof of concept project aimed at solving data collection, centralization and correlation issues that can occur from deploying numerous, indepedent security tools throughout an environment.
This application uses the Laravel framework. Data collection is performed through scheduled artisan commands. Each command is basically either a web crawler or an API client, depending on whether or not the target application offers any services for accessing its data.
The commands authenticate to their target application and query for data. When data collection and normalization is complete the commands ship their data to an instance of Kafka. Each command sends data to one or more Kafka topics, unique to that particular data. Kafka, acting as an event queue, serves the data to a Logstash consumer (logstash-input-kafka) which then upserts the data to an Elasticsearch cluster.