Integrate Apify E-Commerce Scraper for real-time product data

ecomm-demo/.env.example

@@ -0,0 +1,3 @@
+# Apify API Token
+# Get your token at https://console.apify.com/account#/integrations
+APIFY_TOKEN=your_apify_token_here

ecomm-demo/.gitignore

@@ -31,7 +31,11 @@ yarn-error.log*
 .pnpm-debug.log*
 
 # env files (can opt-in for committing if needed)
-.env*
+.env
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
 
 # vercel
 .vercel
@@ -42,5 +46,4 @@ next-env.d.ts
 
 # internal
 task-manager.md
-PRD.md
-README.md
+PRD.md

ecomm-demo/README.md

@@ -0,0 +1,156 @@
+# E-Commerce Demo with Apify Integration
+
+This Next.js application demonstrates how to integrate the Apify E-Commerce Scraping Tool to scrape real-time product data from e-commerce websites.
+
+## Features
+
+- 🔍 **Real-time Product Search**: Search for products using keywords
+- 🚀 **Apify Integration**: Powered by Apify's E-Commerce Scraping Tool
+- 📊 **Multiple Views**: View products as statistics, table, or cards
+- 💰 **Dynamic Pricing**: Automatically calculates average prices
+- 🎨 **Modern UI**: Built with Next.js, Tailwind CSS, and shadcn/ui
+
+## Prerequisites
+
+- Node.js 20 or higher
+- An Apify account (free tier available)
+- Apify API token
+
+## Setup Instructions
+
+### 1. Get Your Apify API Token
+
+1. Go to [Apify Console](https://console.apify.com/account#/integrations)
+2. Create a free account if you don't have one
+3. Navigate to Settings → Integrations
+4. Copy your API token
+
+### 2. Configure Environment Variables
+
+1. Copy the example environment file:
+   ```bash
+   cp .env.example .env.local
+   ```
+
+2. Edit `.env.local` and add your Apify token:
+   ```env
+   APIFY_TOKEN=your_actual_token_here
+   ```
+
+### 3. Install Dependencies
+
+```bash
+npm install
+```
+
+### 4. Run the Application
+
+For development:
+```bash
+npm run dev
+```
+
+For production:
+```bash
+npm run build
+npm start
+```
+
+Open [http://localhost:3000](http://localhost:3000) in your browser.
+
+## How to Use
+
+1. **Enter a Search Query**: Type a product keyword (e.g., "laptop", "headphones") into the search bar
+2. **Click Submit**: The app will call the Apify E-Commerce Scraping Tool to scrape products from Amazon
+3. **View Results**: Products will be displayed in three sections:
+   - **Statistics Cards**: Total products, average price, and data source
+   - **Product Table**: Detailed tabular view with images, titles, prices, descriptions, and URLs
+   - **Product Cards**: Visual card layout for browsing products
+
+## Architecture
+
+### Components
+
+- **`app/page.tsx`**: Main page with search functionality and state management
+- **`app/api/scrape/route.ts`**: API route that calls the Apify Actor (server-side only)
+- **`components/SearchBar.tsx`**: Search input component
+- **`components/StatsCards.tsx`**: Statistics display component
+- **`components/ProductTable.tsx`**: Tabular product view
+- **`components/ProductCards.tsx`**: Card-based product view
+
+### Data Flow
+
+1. User enters search query → `SearchBar` component
+2. Client sends POST request → `/api/scrape` API route
+3. API route calls Apify Actor → `apify/e-commerce-scraping-tool`
+4. Actor scrapes products from Amazon → Returns dataset
+5. API route transforms data → Returns to client
+6. Client updates state → Components re-render with new data
+
+### Security
+
+- **APIFY_TOKEN** is stored server-side only in environment variables
+- API route handles all Apify Actor calls, keeping credentials secure
+- Client never has direct access to the Apify token
+
+## Configuration
+
+### Scraping Settings
+
+In `app/api/scrape/route.ts`, you can customize:
+
+- **Marketplaces**: Currently set to Amazon US, but supports many others
+- **Max Results**: Limited to 20 products for demo (adjustable)
+- **Scrape Mode**: Set to "AUTO" (can be "BROWSER" or "HTTP")
+
+### Supported Marketplaces
+
+The E-Commerce Scraping Tool supports many marketplaces including:
+- Amazon (all regions)
+- Walmart
+- eBay
+- Alibaba
+- IKEA
+- And many more
+
+To change the marketplace, edit the `marketplaces` array in the API route.
+
+## Cost Information
+
+The E-Commerce Scraping Tool uses Apify's pay-per-event pricing model:
+- **Actor Start**: ~$0.0001
+- **Product Listing**: ~$0.0005 per page
+- **Product Details**: ~$0.006 per product
+
+Apify offers a free tier with monthly credits. See [Apify Pricing](https://apify.com/pricing) for details.
+
+## Troubleshooting
+
+### "APIFY_TOKEN not configured" Error
+
+Make sure you have:
+1. Created a `.env.local` file
+2. Added your `APIFY_TOKEN`
+3. Restarted the development server
+
+### No Products Found
+
+Try:
+- Using different search keywords
+- Checking your Apify account quota
+- Viewing the Apify Console run details (link provided after search)
+
+### Image Not Loading
+
+The app is configured to allow images from common e-commerce domains. If you see broken images, check the `next.config.ts` file and add the required domain.
+
+## Learn More
+
+- [Apify Documentation](https://docs.apify.com)
+- [E-Commerce Scraping Tool](https://apify.com/apify/e-commerce-scraping-tool)
+- [Next.js Documentation](https://nextjs.org/docs)
+- [Apify JavaScript SDK](https://docs.apify.com/sdk/js/)
+
+## License
+
+MIT

ecomm-demo/app/api/scrape/route.ts

@@ -0,0 +1,66 @@
+import { NextRequest, NextResponse } from "next/server";
+import { ApifyClient } from "apify-client";
+
+// Initialize the ApifyClient with API token from environment variables
+const client = new ApifyClient({
+  token: process.env.APIFY_TOKEN,
+});
+
+export async function POST(request: NextRequest) {
+  try {
+    const { query } = await request.json();
+
+    if (!query || typeof query !== "string") {
+      return NextResponse.json(
+        { error: "Query parameter is required" },
+        { status: 400 }
+      );
+    }
+
+    if (!process.env.APIFY_TOKEN) {
+      return NextResponse.json(
+        { 
+          error: "APIFY_TOKEN not configured. Please set up your Apify API token.",
+          details: "Get your token at https://console.apify.com/account#/integrations"
+        },
+        { status: 500 }
+      );
+    }
+
+    // Call the E-commerce Scraping Tool Actor with keyword search
+    const run = await client.actor("apify/e-commerce-scraping-tool").call({
+      keyword: query,
+      marketplaces: ["www.amazon.com"], // Default to Amazon for demo
+      maxProductResults: 20, // Limit to 20 products for demo
+      scrapeMode: "AUTO",
+    });
+
+    // Fetch the results from the Actor's default dataset
+    const { items } = await client.dataset(run.defaultDatasetId).listItems();
+
+    // Transform the data to match our Product type
+    const products = items.map((item: Record<string, unknown>) => ({
+      url: item.url || "",
+      title: item.name || "Unknown Product",
+      image: item.image || "",
+      description: item.description || "",
+      price: (item.offers as Record<string, unknown>)?.price || 0,
+    }));
+
+    return NextResponse.json({ 
+      products,
+      runId: run.id,
+      runUrl: `https://console.apify.com/actors/runs/${run.id}`
+    });
+  } catch (error) {
+    const errorMessage = error instanceof Error ? error.message : "Unknown error occurred";
+    console.error("Error calling Apify Actor:", error);
+    return NextResponse.json(
+      { 
+        error: "Failed to scrape products",
+        details: errorMessage
+      },
+      { status: 500 }
+    );
+  }
+}

ecomm-demo/app/page.tsx

@@ -1,12 +1,65 @@
-import { products } from "@/data/products";
+"use client";
+
+import { useState } from "react";
+import { products as mockProducts } from "@/data/products";
 import { StatsCards } from "@/components/StatsCards";
 import { ProductTable } from "@/components/ProductTable";
 import { ProductCards } from "@/components/ProductCards";
 import { SearchBar } from "@/components/SearchBar";
 import { Separator } from "@/components/ui/separator";
 import Image from "next/image";
+import type { Product, ScraperResponse, ScraperError } from "@/lib/types";
 
 export default function Home() {
+  const [products, setProducts] = useState<Product[]>(mockProducts);
+  const [loading, setLoading] = useState(false);
+  const [error, setError] = useState<string | null>(null);
+  const [dataSource, setDataSource] = useState<"Mock" | "Apify">("Mock");
+  const [runUrl, setRunUrl] = useState<string | null>(null);
+
+  const handleSearch = async (query: string) => {
+    if (!query.trim()) {
+      return;
+    }
+
+    setLoading(true);
+    setError(null);
+    setRunUrl(null);
+
+    try {
+      const response = await fetch("/api/scrape", {
+        method: "POST",
+        headers: {
+          "Content-Type": "application/json",
+        },
+        body: JSON.stringify({ query }),
+      });
+
+      if (!response.ok) {
+        const errorData: ScraperError = await response.json();
+        throw new Error(errorData.details || errorData.error || "Failed to fetch products");
+      }
+
+      const data: ScraperResponse = await response.json();
+
+      if (data.products && data.products.length > 0) {
+        setProducts(data.products);
+        setDataSource("Apify");
+        if (data.runUrl) {
+          setRunUrl(data.runUrl);
+        }
+      } else {
+        setError("No products found for this search query. Try a different keyword.");
+      }
+    } catch (err) {
+      const errorMessage = err instanceof Error ? err.message : "An error occurred while fetching products";
+      setError(errorMessage);
+      console.error("Search error:", err);
+    } finally {
+      setLoading(false);
+    }
+  };
+
   return (
     <div className="min-h-screen bg-background">
       {/* Header */}
@@ -37,19 +90,54 @@ export default function Home() {
             Product Catalog Demo
           </h1>
           <p className="text-lg text-muted-foreground max-w-2xl mb-6">
-            Showcase of e-commerce product data ready for Apify integration. 
-            This demo displays scraped product information in multiple formats.
+            Search for products using Apify&apos;s E-Commerce Scraping Tool.
+            Enter a product keyword below to scrape real-time data from Amazon.
           </p>
 
           {/* Search Bar */}
           <div className="mt-8">
-            <SearchBar />
+            <SearchBar onSearch={handleSearch} />
           </div>
+
+          {/* Loading State */}
+          {loading && (
+            <div className="mt-4 p-4 bg-blue-50 dark:bg-blue-950 border border-blue-200 dark:border-blue-800 rounded-lg">
+              <p className="text-blue-800 dark:text-blue-200">
+                🔄 Scraping products... This may take a few moments.
+              </p>
+            </div>
+          )}
+
+          {/* Error State */}
+          {error && (
+            <div className="mt-4 p-4 bg-red-50 dark:bg-red-950 border border-red-200 dark:border-red-800 rounded-lg">
+              <p className="text-red-800 dark:text-red-200">
+                ⚠️ {error}
+              </p>
+            </div>
+          )}
+
+          {/* Run URL Link */}
+          {runUrl && (
+            <div className="mt-4 p-4 bg-green-50 dark:bg-green-950 border border-green-200 dark:border-green-800 rounded-lg">
+              <p className="text-green-800 dark:text-green-200">
+                ✓ Data scraped successfully!{" "}
+                <a 
+                  href={runUrl} 
+                  target="_blank" 
+                  rel="noopener noreferrer"
+                  className="underline font-medium hover:text-green-900 dark:hover:text-green-100"
+                >
+                  View run details in Apify Console
+                </a>
+              </p>
+            </div>
+          )}
         </div>
 
         {/* Stats Section */}
         <section className="mb-12">
-          <StatsCards productCount={products.length} />
+          <StatsCards products={products} dataSource={dataSource} />
         </section>
 
         <Separator className="my-12" />

ecomm-demo/components/ProductCards.tsx

@@ -10,11 +10,6 @@ interface ProductCardsProps {
 }
 
 export function ProductCards({ products }: ProductCardsProps) {
-  const truncate = (text: string, maxLength: number) => {
-    if (text.length <= maxLength) return text;
-    return text.substring(0, maxLength) + "...";
-  };
-
   return (
     <div>
       <div className="mb-6">