Structure scrape

[kennethjor]

Depends on #483, #489

Scrape /universe/structures/ and /universe/structures/{structure_id}/ at regular intervals. This will require a server-side login, for the second endpoint at least.

• IDs from public ID endpoint
• Get structure information
• Create file
• Upload file
• Check market presence
• IDs from contracts
• IDs from markets
• IDs from sovereignty
• Structure IDs from past public structure ID scrapes
• Filter market structure type IDs from dogma for market module: https://everef.net/types/35892 - depends on Can fit data in reference data #489
• Import backfill
• Run initial big scrape
• Save archive
• Delete old structures
• Document scrape
• Submit PR for updated structure type IDs for Fuzzwork - Updated structure type IDs with markets fuzzysteve/FuzzMarket#11

[kennethjor]

The basic structures endpoint doesn't require a login. A scrape is online now: data.everef.net/structures/

[kennethjor]

This should scrape from the list of public structures, but should also do its own discovery from other sources:

• Public contracts, including end location from courier contracts
• Buy orders returned by the main orders endpoint

To support #482, it should also use the /markets/structures/{structure_id}/ endpoint to populate a has_public_market: true|false entry for each structure. This should be false if the endpoint returns a 403, but true if it returns anything else, even an empty array. It should also to back off checking the market orders for some time for anything returning a 403. The maintainer of EVE Tycoon says they don't check for another 12 hours.

This scrape will also need to run on its own server so as not to throttle other scrapes, as it will generate a fair few 403s.

[kennethjor]

Steve Ronuken's script for updating citadels: https://github.com/fuzzysteve/FuzzMarket/blob/master/scripts/citadelgetter.py

[kennethjor]

Structure Scrape

Draft notes

• Discover structure IDs from various sources:
  • Previous structure scrape file
  • ESI public structure IDs endpoint
  • Market orders - set latest timestamp in last_seen_in_markets
  • Public contracts (location and end location) - set latest timestamp in last_seen_in_contracts
  • Sovereignty structures - set is_sovereignty_structure and last_seen_sovereignty_structure
• Fetch public structure IDs and for each set is_public_structure = true and last_seen_public_id = now
• Scrape structure information
  • If is_gettable_structure == true || last_structure_get is more than 12 hours ago
    • Scrape information
      • If returned, set is_gettable_structure = true and last_structure_get = now
      • Else, set is_gettable_structure = false
• Scrape structure markets
  • If has_markets == true || last_seen_in_markets is more than 12 hours ago
    • Scrape markets
      • If returned, set has_markets = true and last_seen_in_markets = now
      • Else, set has_markets = false
• Clean old structures
  • If the min(last_seen_public_id, last_seen_in_markets, last_seen_in_contracts, last_market_scrape) is more than 30 days ago, remove the structure completely

Meta information

• is_public_structure - true if the latest structure scrape saw the structure on the ESI public structure IDs endpoint
• is_sovereignty_structure - true if the latest structure scrape saw the structure on the ESI sovereignty structure endpoint
• is_gettable_structure - true if the latest structure scrape was able to fetch structure information
• has_markets - true if the latest structure scrape was able to fetch structure markets
• last_seen_public_id - timestamp of when the structure ID was last seen on the ESI public structure ID endpoint
• last_seen_sovereignty_structure - timestamp of when the structure ID was last seen on the ESI sovereignty structure endpoint
• last_structure_get - timestamp of when structure information was last accessible
• last_market_available - timestamp of when the structure markets were last accessible on the ESI market structure endpoint
• last_seen_in_markets - timestamp of when the structure ID was last seen in a market orders scrape
• last_seen_in_contracts - timestamp of when the structure ID was last seen in a public contracts scrape

[kennethjor]

Some backfill provided by Sir SmashAlot:

• CorporationOfficeStructures.csv corporationOfficeStructures.csv
• PublicStructureIDs.csv

[kennethjor]

Depends on #489 for market structure lookup. Lower level structures like Raitaru, Astrahus, and Athanor cannot fit markets. The dogma is correct.

[kennethjor]

Backfill file from Adam4Eve: https://static.adam4eve.eu/IDs/playerStructure_IDs.csv (mirror)