This repository contains a complete GitOps specification for deploying a graph visualization system on Red Hat OpenShift Service on AWS (ROSA) using Argo CD. The stack includes:
- Neo4j Database: Graph database for storing and querying graph data
- Frontend Visualization: Neovis.js-based web interface for graph visualization
- OAuth Security: OpenShift OAuth Proxy for authentication
- ETL Components: Automated data ingestion from JIRA and GitHub APIs
- GitOps Management: Argo CD for declarative deployment and management
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Argo CD │───▶│ Graph Stack │───▶│ Neo4j │
│ Application │ │ Deployment │ │ Database │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│ ▲
▼ │
┌──────────────────┐ │
│ Frontend + │ │
│ OAuth Proxy │ │
└──────────────────┘ │
│ │
▼ │
┌──────────────────┐ │
│ OpenShift │ │
│ Route │ │
└──────────────────┘ │
│
┌─────────────────┐ ┌──────────────────┐ │
│ JIRA API │───▶│ JIRA ETL Job │──────────────┤
│ (Issues) │ │ (CronJob) │ │
└─────────────────┘ └──────────────────┘ │
│
┌─────────────────┐ ┌──────────────────┐ │
│ GitHub API │───▶│ GitHub ETL Job │──────────────┘
│ (Issues/PRs) │ │ (CronJob) │
└─────────────────┘ └──────────────────┘
graph-visualisation/
├── applications/
│ ├── graph-stack.yaml # Argo CD Application for main stack
│ ├── etl-stack.yaml # Argo CD Application for ETL jobs
│ └── company-integration-stack.yaml # Argo CD Application for company-specific JIRA-GitHub integration
├── company-specific/ # Company-specific JIRA-GitHub integration (separate for security)
│ ├── schema/
│ │ └── jira-github-integration-schema.cypher
│ ├── etl/
│ │ ├── jira-github-integration/
│ │ └── shared/
│ ├── applications/
│ └── README.md
├── neo4j/
│ ├── kustomization.yaml # Kustomize configuration
│ └── values.yaml # Helm values for Neo4j
├── frontend/
│ ├── Dockerfile # NGINX container serving Neovis.js HTML
│ ├── index.html # Visualization app (Neovis, etc.)
│ ├── deployment.yaml # Frontend + OAuth Proxy
│ ├── service.yaml # Kubernetes Service
│ └── route.yaml # OpenShift Route
├── etl/
│ ├── shared/
│ │ └── configmap.yaml # Shared ETL configuration and Cypher queries
│ ├── jira/
│ │ ├── cronjob.yaml # JIRA ETL CronJob
│ │ ├── Dockerfile # JIRA ETL container
│ │ ├── requirements.txt # Python dependencies
│ │ └── jira_etl.py # JIRA ETL script
│ └── github/
│ ├── cronjob.yaml # GitHub ETL CronJob
│ ├── Dockerfile # GitHub ETL container
│ ├── requirements.txt # Python dependencies
│ └── github_etl.py # GitHub ETL script
└── README.md # This file
- ROSA Cluster: Version 4.12+ with admin access
- Machine Pool: At least 3 worker nodes with 4 vCPU, 16GB RAM each
- Storage: Default storage class configured (gp3-csi recommended)
- Networking: Cluster must have internet access for container registry pulls
Install the following tools on your local machine:
# Install OpenShift CLI
curl -O https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz
tar -xzf openshift-client-linux.tar.gz
sudo mv oc kubectl /usr/local/bin/
# Install Podman (for container builds)
sudo dnf install -y podman
# Install Git
sudo dnf install -y git
# Install jq (for JSON processing)
sudo dnf install -y jq- Quay.io Account: Create account at https://quay.io
- Registry Credentials: Generate robot account or use personal credentials
- Repository Access: Create repositories for your containers
- JIRA Cloud Instance: Access to Atlassian JIRA Cloud
- API Token: Generate from https://id.atlassian.com/manage-profile/security/api-tokens
- User Email: Email address associated with JIRA account
- Project Access: Read permissions for target JIRA projects
- GitHub Account: Personal or organization account
- Personal Access Token: Generate from GitHub Settings > Developer settings > Personal access tokens
- Required Scopes:
repo,read:org,read:user - Organization Access: Read permissions for target repositories
If Argo CD is not already installed on your ROSA cluster:
First, you need to connect to your ROSA cluster. There are several ways to do this:
Option A: Using ROSA CLI (Recommended)
# Install ROSA CLI if not already installed
curl -Ls https://mirror.openshift.com/pub/openshift-v4/clients/rosa/latest/rosa-linux.tar.gz | tar xz
sudo mv rosa /usr/local/bin/rosa
# Login to your Red Hat account
rosa login
# List your clusters
rosa list clusters
# Get login command for your cluster
rosa describe cluster YOUR-CLUSTER-NAME --output json | jq -r '.api.url'
# Login using the cluster API URL
oc login https://api.YOUR-CLUSTER-NAME.RANDOM-STRING.p1.openshiftapps.com:6443Option B: Using OpenShift Console
# 1. Go to https://console.redhat.com/openshift
# 2. Select your ROSA cluster
# 3. Click "Open Console"
# 4. In the OpenShift console, click your username (top right)
# 5. Select "Copy login command"
# 6. Click "Display Token"
# 7. Copy and run the oc login command
# Example of what you'll copy:
oc login --token=sha256~EXAMPLE-TOKEN --server=https://api.YOUR-CLUSTER.p1.openshiftapps.com:6443Option C: Using Cluster Credentials
# If you have cluster admin credentials
oc login -u kubeadmin -p YOUR-KUBEADMIN-PASSWORD https://api.YOUR-CLUSTER.p1.openshiftapps.com:6443# Verify you're connected to the right cluster
oc whoami
oc cluster-info
# Check your permissions
oc auth can-i create namespace
oc auth can-i create deployment
# List existing projects
oc get projects# Create Argo CD namespace
oc create namespace argocd
# Install Argo CD operator
oc apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Create required secrets that are missing from the default installation
# Create Redis secret (required for ArgoCD components)
oc create secret generic argocd-redis --from-literal=auth="" -n argocd
# Generate and add server secret key to argocd-secret
SECRET_KEY=$(openssl rand -base64 32)
oc patch secret argocd-secret -n argocd --type merge -p="{\"data\":{\"server.secretkey\":\"$(echo -n $SECRET_KEY | base64)\"}}"
# Configure ArgoCD server to run in insecure mode (required for OpenShift routes)
oc patch configmap argocd-cmd-params-cm -n argocd --type merge -p='{"data":{"server.insecure":"true"}}'
# Add dex configuration to argocd-cm ConfigMap
oc patch configmap argocd-cm -n argocd --type merge -p='{"data":{"dex.config":"issuer: https://argocd-dex-server.argocd.svc.cluster.local:5556/dex\nstorage:\n type: memory\nweb:\n http: 0.0.0.0:5556\nlogger:\n level: \"debug\"\n format: text\nconnectors:\n- type: oidc\n id: oidc\n name: OpenShift\n config:\n issuer: https://kubernetes.default.svc.cluster.local\n clientID: system:serviceaccount:argocd:argocd-dex-server\n clientSecret: \"\"\n requestedScopes: [\"openid\", \"profile\", \"email\", \"groups\"]\n requestedIDTokenClaims: {\"groups\": {\"essential\": true}}\nstaticClients:\n- id: argo-cd-cli\n name: \"Argo CD CLI\"\n public: true\n- id: argo-cd\n name: \"Argo CD\"\n secret: \"$2a$10$mivhwttXM0VwgbPLQxcZJOa.ClzGraLqXtx5Mq8gLjHA3wTTILjjK\"\n redirectURIs:\n - https://argocd-server/auth/callback"}}'
# Wait for Argo CD to be ready (this may take a few minutes)
oc wait --for=condition=available --timeout=600s deployment/argocd-server -n argocd
# Restart ArgoCD deployments to pick up the new configuration
oc rollout restart deployment/argocd-dex-server -n argocd
oc rollout restart deployment/argocd-server -n argocd
oc rollout restart deployment/argocd-repo-server -n argocd
oc rollout restart statefulset/argocd-application-controller -n argocd
# Wait for all components to be running
echo "Waiting for ArgoCD components to be ready..."
oc wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-dex-server -n argocd --timeout=300s
oc wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server -n argocd --timeout=300s
oc wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-repo-server -n argocd --timeout=300s
oc wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-application-controller -n argocd --timeout=300s
# Create Argo CD route (using HTTP port since server runs in insecure mode)
oc create route edge argocd-server --service=argocd-server --port=http --insecure-policy=Redirect -n argocd
# Get Argo CD admin password
ARGOCD_PASSWORD=$(oc get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d)
# Get the Argo CD URL
ARGOCD_URL="https://$(oc get route argocd-server -n argocd -o jsonpath='{.spec.host}')"
echo "Argo CD URL: $ARGOCD_URL"
echo "Username: admin"
echo "Password: $ARGOCD_PASSWORD"# Get the Argo CD URL and admin password
ARGOCD_URL="https://$(oc get route argocd-server -n argocd -o jsonpath='{.spec.host}')"
ARGOCD_PASSWORD=$(oc get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d)
echo "Argo CD URL: $ARGOCD_URL"
echo "Username: admin"
echo "Password: $ARGOCD_PASSWORD"
# Open in browser (macOS)
open "$ARGOCD_URL"
# Open in browser (Linux)
xdg-open "$ARGOCD_URL"# Fork this repository to your GitHub account
# Then clone your fork
git clone https://github.com/YOUR-USERNAME/graph-visualisation.git
cd graph-visualisationUpdate the repository URLs in the Argo CD applications:
# Update graph-stack application
sed -i 's|https://github.com/myorg/graph-visualisation|https://github.com/YOUR-USERNAME/graph-visualisation|g' applications/graph-stack.yaml
# Update etl-stack application
sed -i 's|https://github.com/myorg/graph-visualisation|https://github.com/YOUR-USERNAME/graph-visualisation|g' applications/etl-stack.yaml
# Commit changes
git add applications/
git commit -m "Update repository URLs for deployment"
git push origin main# Login to your ROSA cluster
oc login --token=YOUR-TOKEN --server=https://api.YOUR-CLUSTER.YOUR-REGION.aroapp.io:6443
# Verify cluster access
oc whoami
oc get nodes# Create the graph-system namespace
oc new-project graph-system
# Set as current project
oc project graph-system# Create registry secret for Quay.io
oc create secret docker-registry quay-registry-secret \
--docker-server=quay.io \
--docker-username=YOUR-QUAY-USERNAME \
--docker-password=YOUR-QUAY-PASSWORD \
--docker-email=YOUR-EMAIL \
-n graph-system
# Link secret to default service account
oc secrets link default quay-registry-secret --for=pull -n graph-system
oc secrets link builder quay-registry-secret -n graph-system# Create JIRA credentials secret
oc create secret generic jira-credentials \
--from-literal=url=https://YOUR-DOMAIN.atlassian.net \
--from-literal=username=your-email@company.com \
--from-literal=api-token=YOUR-JIRA-API-TOKEN \
-n graph-system
# Create GitHub credentials secret
oc create secret generic github-credentials \
--from-literal=token=YOUR-GITHUB-PERSONAL-ACCESS-TOKEN \
-n graph-system
# Create Neo4j auth secret (change password from default)
oc create secret generic neo4j-auth \
--from-literal=password=YOUR-SECURE-NEO4J-PASSWORD \
-n graph-systemUpdate the ETL configuration with your specific settings:
# Update JIRA projects in cronjob
sed -i 's/AAP,PLATFORM,DEVOPS/YOUR-PROJECT-1,YOUR-PROJECT-2,YOUR-PROJECT-3/g' etl/jira/cronjob.yaml
# Update GitHub organization and repositories
sed -i 's/myorg/YOUR-GITHUB-ORG/g' etl/github/cronjob.yaml
sed -i 's/repo1,repo2,repo3/YOUR-REPO-1,YOUR-REPO-2,YOUR-REPO-3/g' etl/github/cronjob.yaml
# Update container registry references
find . -name "*.yaml" -exec sed -i 's/quay.io\/myorg/quay.io\/YOUR-QUAY-USERNAME/g' {} \;
# Update Neo4j password in values.yaml
sed -i 's/changeme/YOUR-SECURE-NEO4J-PASSWORD/g' neo4j/values.yaml
# Commit configuration changes
git add .
git commit -m "Update configuration for deployment"
git push origin main# Login to Quay.io
podman login quay.io
# Build and push frontend container
cd frontend
podman build -t quay.io/YOUR-QUAY-USERNAME/graph-frontend:latest .
podman push quay.io/YOUR-QUAY-USERNAME/graph-frontend:latest
# Build and push JIRA ETL container
cd ../etl/jira
podman build -t quay.io/YOUR-QUAY-USERNAME/jira-etl:latest .
podman push quay.io/YOUR-QUAY-USERNAME/jira-etl:latest
# Build and push GitHub ETL container
cd ../github
podman build -t quay.io/YOUR-QUAY-USERNAME/github-etl:latest .
podman push quay.io/YOUR-QUAY-USERNAME/github-etl:latest
cd ../..# Get the Argo CD server URL for OAuth redirect
ARGOCD_URL=$(oc get route argocd-server -n argocd -o jsonpath='{.spec.host}')
# Create OAuth client for the frontend
oc create oauthclient graph-frontend \
--redirect-uri=https://graph-frontend-graph-system.apps.YOUR-CLUSTER.YOUR-REGION.aroapp.io/oauth2/callback \
--secret=$(openssl rand -base64 32)
# Get the OAuth client secret
OAUTH_SECRET=$(oc get oauthclient graph-frontend -o jsonpath='{.secret}')
# Generate cookie secret
COOKIE_SECRET=$(openssl rand -base64 32)
# Create OAuth proxy secrets
oc create secret generic oauth-proxy-secrets \
--from-literal=client-secret=$OAUTH_SECRET \
--from-literal=cookie-secret=$COOKIE_SECRET \
-n graph-system# Apply the Argo CD applications
oc apply -f applications/graph-stack.yaml
oc apply -f applications/etl-stack.yaml
# For company-specific JIRA-GitHub integration (optional)
# First configure company-specific secrets (see company-specific/README.md)
oc apply -f applications/company-integration-stack.yaml
# Verify applications are created
oc get applications -n argocdIf you want to deploy the advanced JIRA-GitHub integration for strategy-to-implementation traceability:
# Create company-specific secrets
oc create secret generic jira-github-integration-secrets \
--from-literal=jira-server=https://your-jira-instance.atlassian.net \
--from-literal=jira-username=your-jira-username \
--from-literal=jira-token=your-jira-api-token \
--from-literal=github-token=your-github-personal-access-token \
-n graphserver
# Build and push the company-specific ETL container
cd company-specific/etl/jira-github-integration
podman build -t quay.io/YOUR-QUAY-USERNAME/jira-github-integration-etl:latest .
podman push quay.io/YOUR-QUAY-USERNAME/jira-github-integration-etl:latest
# Update the image reference in the CronJob
sed -i 's|jira-github-integration-etl:latest|quay.io/YOUR-QUAY-USERNAME/jira-github-integration-etl:latest|g' cronjob.yaml
# Deploy the company integration (ArgoCD will handle dependencies automatically)
oc apply -f ../../applications/company-integration-stack.yaml
cd ../../..Deployment Order & Dependencies: The system uses ArgoCD sync waves to ensure proper deployment order:
- Wave 1: Neo4j database, ConfigMaps, Secrets
- Wave 2: Health checks, schema setup, basic ETL jobs
- Wave 3: Company-specific advanced ETL
Health Checks: The deployment includes automatic health checks that:
- Verify Neo4j connectivity before starting ETL
- Apply the JIRA-GitHub integration schema automatically
- Validate APOC plugin availability
- Ensure database is ready before data loading
Advanced Features:
- Strategy-to-implementation traceability between JIRA and GitHub
- Hierarchical filtering (only open items + closed items with open dependencies)
- Cross-reference detection and technology tracking
- Gap analysis for strategic planning
- Automatic schema management and health validation
See company-specific/README.md for detailed configuration and usage.
# Get Argo CD admin password
ARGOCD_PASSWORD=$(oc get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d)
# Login to Argo CD CLI (optional)
argocd login $ARGOCD_URL --username admin --password $ARGOCD_PASSWORD --insecure
# Add repository to Argo CD (via UI or CLI)
argocd repo add https://github.com/YOUR-USERNAME/graph-visualisation.git# Check application sync status
oc get applications -n argocd
# Monitor pods in graph-system namespace
oc get pods -n graph-system -w
# Check application logs
oc logs -f deployment/graph-frontend -n graph-system
# Verify Neo4j is running
oc get pods -l app=neo4j -n graph-system
# Check ETL job status
oc get cronjobs -n graph-system
oc get jobs -n graph-system# Get the frontend route URL
oc get route graph-frontend -n graph-system -o jsonpath='{.spec.host}'
# Open in browser
echo "Access the application at: https://$(oc get route graph-frontend -n graph-system -o jsonpath='{.spec.host}')"To enable automatic deployments when you push changes to your repository:
- Get Argo CD Webhook URL:
ARGOCD_URL=$(oc get route argocd-server -n argocd -o jsonpath='{.spec.host}')
echo "Webhook URL: https://$ARGOCD_URL/api/webhook"-
Configure GitHub Webhook:
- Go to your GitHub repository settings
- Navigate to "Webhooks" section
- Click "Add webhook"
- Set Payload URL:
https://YOUR-ARGOCD-URL/api/webhook - Set Content type:
application/json - Set Secret: (optional, for security)
- Select events:
PushandPull request - Click "Add webhook"
-
Configure Argo CD Application for Auto-sync:
# Enable auto-sync for applications
oc patch application graph-stack -n argocd --type='merge' -p='{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}'
oc patch application etl-stack -n argocd --type='merge' -p='{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}'# Port forward to Neo4j
oc port-forward svc/neo4j 7474:7474 7687:7687 -n graph-system
# Access Neo4j browser at http://localhost:7474
# Login with: neo4j / YOUR-SECURE-NEO4J-PASSWORD# Manually trigger JIRA ETL job
oc create job jira-etl-manual --from=cronjob/jira-etl -n graph-system
# Manually trigger GitHub ETL job
oc create job github-etl-manual --from=cronjob/github-etl -n graph-system
# Check job logs
oc logs job/jira-etl-manual -n graph-system
oc logs job/github-etl-manual -n graph-system// Check JIRA data
MATCH (i:Issue) RETURN count(i) as jira_issues;
// Check GitHub data
MATCH (g:GitHubIssue) RETURN count(g) as github_issues;
// Check relationships
MATCH (u:User)-[r]->(i) RETURN type(r), count(r);If you see ArgoCD Redis pods failing with SCC violations like:
pods "argocd-redis-6496f58877-" is forbidden: unable to validate against any security context constraint:
[provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2:
.initContainers[0].runAsUser: Invalid value: 999: must be in the ranges: [1000800000, 1000809999]]
Root Cause: ArgoCD Redis deployment tries to run as user ID 999, but OpenShift's restricted SCC requires user IDs in the range 1000800000-1000809999.
Solution:
# 1. Remove the hardcoded runAsUser from Redis deployment
oc patch deployment argocd-redis -n argocd --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/securityContext/runAsUser"}]'
# 2. Check if Redis secret has empty password (common issue)
oc get secret argocd-redis -n argocd -o jsonpath="{.data.auth}" | base64 -d
# 3. If password is empty, generate and set a new one
oc patch secret argocd-redis -n argocd --type='json' -p='[{"op": "replace", "path": "/data/auth", "value": "'$(openssl rand -base64 32 | base64 -w 0)'"}]'
# 4. Restart the Redis deployment
oc rollout restart deployment/argocd-redis -n argocd
# 5. Verify the rollout completed successfully
oc rollout status deployment/argocd-redis -n argocd
# 6. Check that all ArgoCD components are running
oc get pods -n argocdVerification:
# All ArgoCD pods should be in Running state
oc get pods -n argocd | grep -E "(argocd-redis|argocd-server|argocd-application-controller)"
# Check ArgoCD application topology - Redis should show as healthy
# Access ArgoCD UI and verify no warnings on Redis componentTo retrieve your ArgoCD admin credentials:
# Get ArgoCD admin password
oc get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d
# Get ArgoCD server URL
oc get route argocd-server -n argocd -o jsonpath="{.spec.host}"
# Display login information
echo "ArgoCD URL: https://$(oc get route argocd-server -n argocd -o jsonpath='{.spec.host}')"
echo "Username: admin"
echo "Password: $(oc get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d)"-
Pod ImagePullBackOff:
- Verify container registry credentials
- Check image names and tags
- Ensure images are pushed to registry
-
OAuth Authentication Fails:
- Verify OAuth client configuration
- Check redirect URI matches route
- Validate secrets are created correctly
-
Neo4j Connection Issues:
- Check Neo4j pod logs
- Verify service names and ports
- Confirm password matches across configurations
-
ETL Jobs Failing:
- Check API credentials and permissions
- Verify network connectivity to external APIs
- Review job logs for specific errors
-
Dependency/Health Check Issues:
- Check ArgoCD sync waves are deploying in order
- Verify Neo4j health check job completed successfully
- Ensure schema setup job ran without errors
- Check if APOC plugin is available in Neo4j
# Check ArgoCD sync wave order
oc get applications -n argocd -o custom-columns=NAME:.metadata.name,SYNC-WAVE:.metadata.annotations.'argocd\.argoproj\.io/sync-wave',STATUS:.status.sync.status
# Check health check job status
oc get jobs -n graphserver -l app=neo4j-health-check
oc logs job/neo4j-health-check -n graphserver
# Check schema setup job status
oc get jobs -n graphserver -l app=jira-github-schema-setup
oc logs job/jira-github-schema-setup -n graphserver
# Verify Neo4j is ready for connections
oc exec -it deployment/neo4j -n graphserver -- cypher-shell -u neo4j -p YOUR-PASSWORD "CALL db.ping();"
# Check if company-specific ETL is waiting for dependencies
oc describe cronjob jira-github-integration-etl -n graphserver# Check all resources in namespace
oc get all -n graph-system
# Describe problematic pods
oc describe pod POD-NAME -n graph-system
# Check events
oc get events -n graph-system --sort-by='.lastTimestamp'
# Check Argo CD application details
oc describe application graph-stack -n argocd
# Force sync Argo CD application
oc patch application graph-stack -n argocd --type='merge' -p='{"operation":{"sync":{"revision":"HEAD"}}}'- Secrets Management: All sensitive data is stored in OpenShift Secrets
- Network Policies: Consider implementing network policies for additional security
- RBAC: Configure appropriate role-based access controls
- TLS: All external traffic uses TLS termination
- Container Security: All containers run as non-root users
- API Rate Limits: ETL jobs include retry logic and respect API rate limits
- Increase CPU/memory in
neo4j/values.yaml - Configure persistent volume size based on data requirements
- Consider Neo4j clustering for high availability
- Adjust CronJob schedules based on data volume
- Implement parallel processing for large datasets
- Monitor API rate limits and adjust accordingly
- Increase replica count in
frontend/deployment.yaml - Configure horizontal pod autoscaling
- Optimize Cypher queries for better performance
For issues and questions:
- Check the Argo CD application status and logs
- Review pod logs for specific error messages
- Verify all prerequisites are met
- Ensure API credentials have proper permissions
- Check network connectivity to external services
This configuration is provided as-is for educational and development purposes.