DataMesh Pro

Connect · Build · Run · Observe · Heal: end to end.

An intelligent data-pipeline orchestrator that lets a team connect data sources, build transformation pipelines visually, run them with live status, and observe the result: quality, schema drift, statistical anomalies, lineage, and blast radius, while a rules engine self-heals common failures and an AI assistant explains the rest. Built as a real, shipped-quality product: every feature is backed by a typed API, a relational schema with migrations, and an automated test suite (157 backend + 15 frontend, all passing).

The problem

Data teams spend most of their time babysitting pipelines that break silently. Schema drift, quality regressions, and anomalies get caught days late, usually by a downstream dashboard going weird. Traditional orchestrators like Airflow require constant babysitting, lack intelligent error recovery, and force engineers to write the same retry / alert / explain logic from scratch on every project.

How it works

An end-to-end pipeline platform with eleven first-class features: PostgreSQL / file / REST connectors with encrypted-at-rest secrets, a React Flow visual pipeline builder validated as a DAG client- and server-side, a synchronous topological executor with per-node metrics + structured logs + cooperative cancel, a 0-100 explainable data-quality score, schema-snapshot drift detection, scikit-learn IsolationForest anomaly detection, a self-healing rules engine (retry / auto-cast / quarantine / alert), table-level lineage with severity-ranked blast-radius impact analysis, multi-channel alerting (email + Slack + webhook) with dedup + quiet hours + escalation, and an Anthropic-grounded AI assistant that explains failures and suggests fixes.

What it does

  • Three connector types: PostgreSQL, file, REST, with type-specific Pydantic validation, Fernet-encrypted secrets, masked-on-read, live test-connection
  • Visual pipeline builder on a React Flow canvas: drag source / transform / quality / destination nodes, validated as a DAG (Kahn's algorithm) on both ends
  • Synchronous topological executor with per-node rows-in/out + duration + status, structured logs, retry-from-start AND retry-from-failed (reuses upstream outputs), cooperative cancel
  • Deterministic 0-100 data-quality score with an explainable breakdown: null rate, rule pass rate, duplicate rate, type conformance: the details payload shows exactly why a dataset scored what it did
  • Schema-drift detection: snapshot diffing classifies changes as added / removed / type-changed columns, surfaces on the lineage graph
  • Anomaly detection via scikit-learn IsolationForest over run history (row counts, durations, quality): abstains when too few samples to be meaningful
  • Self-healing rules engine: retry / auto-cast / quarantine / skip-and-continue / alert, with unsafe cast falling back to quarantine and a fully auditable healing history
  • Table-level lineage graph stitched from pipeline node configs across raw → staging → mart chains, plus blast-radius BFS that ranks impacted assets by severity (graph distance × fan-out × SLA protection)
  • Multi-channel alerting (email + Slack + webhook) with severity routing, dedup-within-window, quiet hours (critical alerts always break through), and escalation of unacked alerts
  • Anthropic-grounded AI assistant: explains failed runs, suggests fixes, describes pipelines in plain language; system prompt forbids invention; graceful degradation when no key configured
  • Home dashboard with KPIs + sparklines + pipeline-health cards + live anomalies/alerts feed; command palette (Cmd-K), keyboard shortcuts, first-run onboarding, dark mode

At a glance

Backend tests
157 passing
Connectors
Postgres · File · REST
Channels
Email · Slack · Webhook

Notes from the build

157 backend pytest cases passing + 15 frontend Vitest cases + strict TypeScript project build all clean. Built under a strict anti-hallucination methodology: only real maintained libraries with pinned versions, never an invented function or endpoint, every feature run and its actual output verified before being called done. Verification is performed against the running application: live API calls, real database queries, browser screenshots, not against assumptions. Seeded demo data is always prefixed so it can never be confused with real data.

Built with

Python 3.11+ · FastAPI · Uvicorn · Pydantic v2 · SQLAlchemy 2.x · Alembic · SQLite (dev) · PostgreSQL (prod) · scikit-learn (IsolationForest) · Anthropic Messages API · psycopg · httpx · cryptography (Fernet) · React 19 + TypeScript + Vite 8 · Tailwind v4 · Radix + shadcn · TanStack Query v5 · Zustand · React Flow · Recharts