Automation/bot auto pr

.deployignore

@@ -0,0 +1,28 @@
+# Files and directories to exclude from deploy
+.git
+.gitignore
+.github/
+.env
+*.env
+venv/
+.venv/
+env/
+__pycache__/
+*.pyc
+local_markets_db/
+local_db*
+*.sqlite
+*.db
+logs/
+*.log
+*.pem
+*.key
+node_modules/
+coverage/
+dist/
+build/
+.idea/
+.vscode/
+*.egg-info/
+
+# Add any other local artifacts you don't want deployed

.github/automation/README_AUTOMATION.md

@@ -0,0 +1,31 @@
+Automations‑Guide (semi‑automatic workflow)
+
+Kurz:
+- Workflow `.github/workflows/bot-auto-pr.yml` erstellt automatisch einen Draft‑PR (`automation/bot-auto-pr`) nachdem Lint/Tests gelaufen sind.
+- Du reviewst den PR manuell und mergest nach Freigabe.
+
+Empfohlene Einstellungen:
+1) Branch‑Protection auf `main`
+   - Require pull request reviews before merging (1 reviewer)
+   - Require status checks to pass (CI)
+   - Include administrators (optional)
+
+2) Secrets / Tokens
+- Der Workflow nutzt das automatisch bereitgestellte `GITHUB_TOKEN` — kein PAT nötig für PR‑Erstellung.
+- Falls du später Aktionen brauchst, die externen Zugriff benötigen (z.B. Deployment), erstelle einen separaten PAT mit minimalen Scopes und lege ihn in Repository → Settings → Secrets → Actions.
+  Empfohlene minimale Scopes für Deploy (wenn nötig):
+  - repo (only if pushing tags/branches required)
+  - workflow (if triggering workflows)
+  - Weitere Scopes nur bei Bedarf.
+
+3) Review‑Prozess
+- PR wird als Entwurf erstellt. Prüfe Änderungen lokal oder in GitHub UI, führe Tests aus und merge erst nach Review.
+
+4) Sicherheit
+- Niemals Tokens in Code oder Issue‑Vorlagen einchecken.
+- Revoke/drehe Tokens sofort, falls sie versehentlich veröffentlicht wurden.
+
+Wenn du möchtest, kann ich:
+- eine einfache Deploy‑Action (nur beim Merge) anlegen, die nach Merge automatisch in ein staging Verzeichnis deployed (benötigt Secret).  
+- oder zusätzliche PR‑Templates/labels für automatisierte PRs erstellen.
+

.github/automation/README_AUTOMATION_EXT.md

@@ -0,0 +1,40 @@
+Ergänzung: Deploy‑Secrets und Schnellstart
+
+Kurz: Diese Anleitung erklärt, wie du die optionalen Deploy‑Secrets sicher anlegst, damit der Deploy‑Workflow nach Merge funktioniert.
+
+1) SSH‑Key erzeugen (lokal)
+   ssh-keygen -t rsa -b 4096 -C "deploy@yourhost" -f ~/.ssh/agents_deploy_key
+   - Public:  ~/.ssh/agents_deploy_key.pub
+   - Private: ~/.ssh/agents_deploy_key
+
+2) Public key auf Zielserver installieren
+   - Melde dich auf dem Zielserver als Deploy‑User an und füge die Public key zu `~/.ssh/authorized_keys`.
+   - Setze korrekte Rechte: `chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys`
+
+3) Secrets in GitHub setzen
+   Repository → Settings → Secrets and variables → Actions → New repository secret
+   Erstelle folgende Secrets (Namen exakt verwenden):
+   - DEPLOY_HOST    (z. B. example.com)
+   - DEPLOY_USER    (z. B. deployuser)
+   - DEPLOY_PORT    (optional, wenn nicht gesetzt: 22)
+   - DEPLOY_TARGET  (Zielpfad, z. B. /var/www/agents)
+   - DEPLOY_SSH_KEY (Inhalt der privaten Key‑Datei `~/.ssh/agents_deploy_key` — ganze Datei einfügen)
+
+4) Sicherheitshinweise
+   - Verwende einen dedizierten Deploy‑User mit minimalen Rechten.
+   - Lege keine Secrets in Repo‑Dateien oder Chatnachrichten ab.
+   - Rotiere oder widerrufe Keys sofort, falls sie kompromittiert wurden.
+
+5) Testen
+   - Merge oder push auf `main` (oder simuliere lokal). Deploy‑Job läuft nur, wenn die oben genannten Secrets vorhanden sind.
+
+Optional: Ich kann ein kurzes Shell‑Testscript hinzufügen, das vor dem ersten Merge die SSH‑Verbindung prüft (ssh -i KEY -p PORT USER@HOST echo ok). Soll ich das zusätzlich anlegen? (ja/nein)
+
+Hinweis zum Testscript:
+- Datei: `scripts/test_deploy_ssh.sh` (bereits vorhanden im Repo)
+- Beispielaufruf:
+  - `./scripts/test_deploy_ssh.sh -h example.com -u deployuser -k ~/.ssh/agents_deploy_key -p 22 -t /var/www/agents`
+- Das Script prüft SSH‑Login und optionales SCP eines kleinen Testfiles in das angegebene Zielverzeichnis.
+
+Führe das Script lokal aus, bevor du Secrets in GitHub setzt, um sicherzustellen, dass der Deploy‑User korrekt konfiguriert ist.
+

.github/workflows/bot-auto-pr.yml

@@ -0,0 +1,87 @@
+name: Bot — Auto PRs (semi-automatic)
+
+on:
+  push:
+    branches:
+      - 'bot/**'
+  schedule:
+    - cron: '0 2 * * *' # daily at 02:00 UTC
+  workflow_dispatch:
+
+jobs:
+  test-and-lint:
+    name: Test & Lint
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies (if any)
+        run: |
+          python -m pip install --upgrade pip
+          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+          pip install black==23.12.0 flake8 pytest || true
+
+      - name: Run black (check)
+        run: |
+          if command -v black >/dev/null 2>&1; then
+            black --check . || true
+          fi
+
+      - name: Run flake8
+        run: |
+          if command -v flake8 >/dev/null 2>&1; then
+            flake8 || true
+          fi
+
+      - name: Run tests (pytest)
+        run: |
+          if command -v pytest >/dev/null 2>&1; then
+            pytest -q || true
+          fi
+      # flake8 already run above in "Run flake8" step; avoid duplicate execution.
+
+  create-pr:
+    name: Create Pull Request
+    runs-on: ubuntu-latest
+    needs: test-and-lint
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Configure git for auto-commits
+        run: |
+          git config user.name "automation-bot"
+          git config user.email "automation-bot@users.noreply.github.com"
+
+      - name: Autoformat with black (if installed)
+        run: |
+          if python -c "import importlib.util,sys; sys.exit(0 if importlib.util.find_spec('black') else 1)"; then
+            python -m black . || true
+            git add -A
+            git diff --quiet --cached || (git commit -m "style: autoformat with black" || true)
+          fi
+
+      # This action will create a PR from branch `automation/bot-auto-pr` when changes exist.
+      - name: Create Pull Request (draft)
+        uses: peter-evans/create-pull-request@v5
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          commit-message: 'chore(ci): automated fixes / checks'
+          branch: automation/bot-auto-pr
+          title: "chore(ci): automated bot PR"
+          body: |
+            This pull request was created automatically by the repository automation workflow.
+            It contains automated lint/test fixes or CI suggestions. Please review before merging.
+          labels: automated
+          draft: true
+

.github/workflows/deploy-on-merge.yml

@@ -0,0 +1,89 @@
+name: Deploy on Merge (optional)
+
+# This workflow performs an optional SSH/SCP deploy when code is merged to `main`.
+# It only runs the deploy step if the required secrets are provided in the repository:
+# - DEPLOY_HOST
+# - DEPLOY_USER
+# - DEPLOY_SSH_KEY
+# - DEPLOY_TARGET
+#
+# To enable: add the above secrets in GitHub → Settings → Secrets → Actions.
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  build:
+    name: Build / Tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+
+      - name: Install deps (if any)
+        run: |
+          python -m pip install --upgrade pip
+          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+
+      - name: Run quick tests (if pytest)
+        run: |
+          if command -v pytest >/dev/null 2>&1; then pytest -q || true; fi
+
+  deploy:
+    name: Deploy (conditional)
+    runs-on: ubuntu-latest
+    needs: build
+    env:
+      HAS_DEPLOY_SECRETS: ${{ secrets.DEPLOY_HOST != '' && secrets.DEPLOY_SSH_KEY != '' }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Prepare SSH key
+        if: env.HAS_DEPLOY_SECRETS == 'true'
+        run: |
+          mkdir -p ~/.ssh
+          echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/id_rsa
+          chmod 600 ~/.ssh/id_rsa
+
+      - name: Ensure known_hosts (optional)
+        if: env.HAS_DEPLOY_SECRETS == 'true'
+        run: |
+          ssh-keyscan -p "${{ secrets.DEPLOY_PORT }}" -H "${{ secrets.DEPLOY_HOST }}" >> ~/.ssh/known_hosts || true
+
+      - name: Copy files to target via scp
+        if: env.HAS_DEPLOY_SECRETS == 'true'
+        run: |
+          TARGET="${{ secrets.DEPLOY_TARGET }}"
+          HOST="${{ secrets.DEPLOY_HOST }}"
+          USER="${{ secrets.DEPLOY_USER }}"
+          PORT="${{ secrets.DEPLOY_PORT }}"
+          if [ -z "$PORT" ]; then PORT=22; fi
+          echo "Deploying to $USER@$HOST:$TARGET (port $PORT) using rsync with .deployignore"
+          # Prefer using .deployignore in repo root to control excludes.
+          if [ -f .deployignore ]; then
+            rsync -az --delete --exclude-from='.deployignore' -e "ssh -p $PORT -o StrictHostKeyChecking=yes" . "$USER@$HOST:$TARGET"
+          else
+            echo ".deployignore not found, using conservative inline excludes"
+            rsync -az --delete \
+              --exclude='.git' \
+              --exclude='.env' \
+              --exclude='*.env' \
+              --exclude='venv/' \
+              --exclude='.venv/' \
+              --exclude='env/' \
+              --exclude='__pycache__/' \
+              --exclude='*.pyc' \
+              --exclude='local_markets_db/' \
+              --exclude='*.sqlite' \
+              --exclude='*.db' \
+              --exclude='logs/' \
+              -e "ssh -p $PORT -o StrictHostKeyChecking=yes" . "$USER@$HOST:$TARGET"
+          fi

.github/workflows/hourly-summary.yml

@@ -0,0 +1,59 @@
+name: Hourly Automation Summary
+
+on:
+  schedule:
+    - cron: '0 * * * *' # hourly at minute 0
+  workflow_dispatch:
+
+jobs:
+  summary:
+    name: Post hourly summary for automation PR
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install jq
+        run: |
+          sudo apt-get update -y
+          sudo apt-get install -y jq
+
+      - name: Build summary and post comment to PR
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          BRANCH: automation/bot-auto-pr
+          API: https://api.github.com/repos/${{ github.repository }}
+        run: |
+          set -euo pipefail
+          owner_repo="${GITHUB_REPOSITORY}"
+          owner="${owner_repo%%/*}"
+
+          echo "Looking up open PR for head ${owner}:${BRANCH}..."
+          prs=$(curl -sS -H "Authorization: token ${GITHUB_TOKEN}" "${API}/pulls?state=open&head=${owner}:${BRANCH}")
+          pr_number=$(echo "$prs" | jq -r '.[0].number // empty')
+          if [ -z "$pr_number" ]; then
+            echo "No open PR found for branch ${BRANCH}. Nothing to post."
+            exit 0
+          fi
+
+          echo "Found PR #${pr_number} — gathering workflow runs..."
+          runs=$(curl -sS -H "Authorization: token ${GITHUB_TOKEN}" "${API}/actions/runs?branch=${BRANCH}&per_page=50")
+          total_runs=$(echo "$runs" | jq '.workflow_runs | length')
+          in_progress=$(echo "$runs" | jq '[.workflow_runs[] | select(.status=="in_progress")] | length')
+          completed=$(echo "$runs" | jq '[.workflow_runs[] | select(.status=="completed")] | length')
+          failed=$(echo "$runs" | jq '[.workflow_runs[] | select(.conclusion=="failure")] | length')
+          success=$(echo "$runs" | jq '[.workflow_runs[] | select(.conclusion=="success")] | length')
+          pending=$(echo "$runs" | jq '[.workflow_runs[] | select(.status=="queued")] | length')
+
+          latest_codeql=$(echo "$runs" | jq -r '.workflow_runs[] | select(.name=="CodeQL") | .conclusion' | head -n1 || echo "none")
+
+          now=$(date -u +"%Y-%m-%d %H:%M UTC")
+          body="Hourly Automation Summary — ${now}\n\n"
+          body+="PR: #${pr_number}\n"
+          body+="Branch: ${BRANCH}\n\n"
+          body+="Workflow runs (recent 50): total=${total_runs}, completed=${completed}, in_progress=${in_progress}, queued=${pending}\n"
+          body+="Success: ${success}, Failed: ${failed}\n\n"
+          body+="Latest CodeQL conclusion: ${latest_codeql}\n\n"
+          body+="_This comment was posted automatically by the repository automation workflow._"
+
+          echo "Posting summary comment to PR #${pr_number}..."
+          post=$(jq -n --arg body "$body" '{body: $body}')
+          curl -sS -X POST -H "Authorization: token ${GITHUB_TOKEN}" -H "Content-Type: application/json" "${API}/issues/${pr_number}/comments" -d "$post" >/dev/null
+          echo "Posted."