Turning raw Polymarket data into AI/ML-ready datasets using automated multi-agent pipelines.
This project demonstrates how to:
- Collect raw market data from the Polymarket GAMMA API, where no public dataset previously existed.
- Transform it into a machine-learning-ready dataset with financial features.
- Use AI agents to automatically discover and extract relevant external information about each market.
- Enable downstream ML tasks such as sentiment analysis, market analysis, and decision optimization.
A custom pipeline gathers comprehensive raw market data, including:
- General market information
- Hourly market price data
- Market volume and liquidity
- Daily trades
- Number of holders and their positions
- Daily order books
From this, additional financial scoring features are engineered for each market.
➡️ The complete dataset is available on Kaggle.
A Search Strategist Agent (powered by crewAI) takes as input a market_question and market_description, then:
- Generates optimized queries using Tavily Search.
- Scores results by source credibility, direct relevance, timeline, and data quality.
- Ranks and outputs the Top 10 high-quality URLs for each market.
📌 Config details:
- Agent:
search_strategist - Task:
url_discovery_task - Tool:
tavily_search.py
╭──────────────────────────────────────────────────────────────────────────── 🔧 Agent Tool Execution ─────────────────────────────────────────────────────────────────────────────╮
│ │
│ Agent: Search Strategist and URL Analyst │
│ │
│ Thought: I have a comprehensive plan to analyze the market question regarding the 2024 Taiwanese presidential election. I've generated a list of targeted search queries to │
│ gather relevant information. Now, I need to execute these queries using the Tavily Search Tool to collect the URLs and their content for analysis. │
│ │
│ Using Tool: Tavily Search Tool │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────── Tool Input ───────────────────────────────────────────────────────────────────────────────────╮
│ │
│ "{\"queries\": [{\"query\": \"2024 Taiwan election polls Hou Yu-ih Ko Wen-je\"}, {\"query\": \"Taiwan presidential election Hou Yu-ih vs Ko Wen-je\"}, ... ]}" │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────── Tool Output ───────────────────────────────────────────────────────────────────────────────────╮
│ │
│ { │
│ "results": [ │
│ { "url": "https://apnews.com/article/taiwan-election-hou-talks...", "title": "Taiwan presidential hopeful Hou promises...", "score": 0.98524 }, │
│ { "url": "https://international.thenewslens.com/feature/2024-taiwan-election...", "title": "Ko Overtakes Lai in Presidential Race...", "score": 0.98205 }, │
│ ... │
│ ] │
│ } │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
A Structured Extractor Agent takes the top URLs from the Search Strategist and:
- Uses the WebsiteSearchTool + Google’s Gemini (via RAG) to extract structured insights.
- Returns a rich JSON schema per article, including:
- Headline, date, author, summary, main text
- Named entities, key events
- Market mentions, causal statements, Wikipedia edit info
📌 Config details:
- Agent:
structured_extractor - Task:
extract_structured_articles_task - Tool:
rag_extract.py
╭────────────────────────────────────────────────────────────────────────────── 🔧 Agent Tool Execution ──────────────────────────────────────────────────────────────────────────────╮
│ │
│ Agent: Structured Data Extractor │
│ │
│ Thought: I will attempt to extract the information from the second URL, as the first one returned a forbidden error. I will use the tool to search for the relevant information │
│ │
│ Using Tool: Search in a specific website │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭──────────────────────────────────────────────────────────────────────────────────── Tool Input ─────────────────────────────────────────────────────────────────────────────────────╮
│ "{\"search_query\": \"2024 Taiwanese Presidential Election Hou Yu-ih vs. Ko Wen-je\", \"website\": \"https://www.csis.org/...\"}" │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭──────────────────────────────────────────────────────────────────────────────────── Tool Output ────────────────────────────────────────────────────────────────────────────────────╮
│ Relevant Content: Taiwan’s 2024 Presidential Election ... A three-way race between Lai (DPP), Hou Yu-ih (KMT), and Ko Wen-je (TPP). │
│ Hou has little diplomatic experience; election framed as “war vs peace.” │
│ ... │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
🚀 Crew: crew
├── 📋 Task: url_discovery_task → ✅ Completed using Tavily Search Tool
└── 📋 Task: extract_structured_articles_task → Executing (used Search in a specific website)
╭───────────────────────────────────────────────────────────────────────────── ✅ Agent Final Answer ────────────────────────────────────────────────────────────────────────────────╮
│ Taiwan’s 2024 Elections: Results and Implications │
│ - William Lai (DPP) won with ~40% of the vote. │
│ - Legislative Yuan split: DPP 51, KMT 52, TPP 8 → no majority. │
│ - Next four years: challenges in cross-Strait relations, U.S. support needed, domestic divisions persist. │
│ ... │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
- Add sentiment analysis module for enriched market datasets.
- Build predictive ML models for price forecasting and strategy optimization.
- Extend pipeline to other market APIs beyond Polymarket.