This 3D scene needs WebGL. The legend below lists what it shows.
Revanth Posina
Data Engineer (AI/ML) at Microsoft.
I build the pipelines, forecasts and agents that power finance analytics at scale.
Data Engineer (AI/ML) @ Microsoft
Pacific time
About
I turn messy operational data into numbers people plan on, and I build the agents that explain them.
Since 2020 I've built data systems across gaming telemetry, healthcare claims and finance analytics. The domains differ, the work doesn't: get the data in reliably, model it so the numbers hold up, and make it easy for people to ask why.
I care most about the parts that make data trustworthy: clear contracts between teams, quality checks, lineage, and agents with guardrails and evals. Forecasts and anomaly detection are only useful when people believe the data underneath.
I studied data science at Indiana University Bloomington. Away from the keyboard I like to walk, watch Anime and Formula 1.
Work and projects
A distributed analytical query engine built from first principles, to show the planning, shuffles, scheduling and fault tolerance that Spark, Trino and DuckDB normally hide.
What it does
- A coordinator parses SQL into logical and physical plans, applies optimizer rules and schedules a distributed execution DAG across worker nodes.
- Operators for scans, filters, projections, aggregations, sorts and hash joins. Broadcast or shuffle hash join, chosen from table statistics and configurable thresholds.
- A custom shuffle layer with hash and range partitioning, bounded buffers, backpressure and spill to disk when memory runs short.
- Its own catalog of partitions, row counts and min/max stats, for partition pruning, predicate pushdown and projection pruning.
- Heartbeats and task state on the coordinator. When a worker dies, its unfinished partitions are reassigned and retried. Failure-injection tests kill workers mid-scan, mid-shuffle and mid-aggregation and check the results stay correct.
What I'm measuring
TPC-H style
workloads from a few GB to 100+ GB
Each optimization
benchmarked against a baseline
Bytes and shuffle
scanned and moved per query
Recovery
retries and correctness under kills
Not built to compete with production databases. It's here to show the systems fundamentals underneath them. Benchmarks publish with the repo.
RustPythonParquetArrowDocker ComposeKubernetesPrometheusGrafana
repo coming soon
A streaming lab that detects, diagnoses and recovers from data and infrastructure failures, benchmarked on a ladder from 100M up to 1.2B events an hour.
What it does
- Load generators drive a multi-broker Kafka cluster with configurable event sizes, key skew, bursts and schema versions.
- Spark Structured Streaming or Flink validates, enriches and aggregates into Iceberg or Delta. Bad events go to a replayable dead-letter queue.
- A reliability control plane runs detect, diagnose, decide, remediate, validate, escalate. Safe actions go through a bounded policy engine; risky ones, like breaking schema changes, get quarantined and escalated.
- A chaos framework injects broker loss, hot partitions, skew, memory pressure, slow sinks, and late or duplicate events, then records time to detect and recover, loss and duplicate rates.
- An optional AI reliability agent writes root-cause hypotheses but cannot run infrastructure commands. Every action passes through deterministic policy.
What I'm measuring
100M to 1.2B
events an hour, benchmark ladder
p50 / p95 / p99
end-to-end latency
MTTD and MTTR
per failure scenario
Cost per 1B
events processed
These are targets, not results yet. Numbers go up when the benchmarks run.
KafkaSchema RegistrySpark or FlinkIceberg or DeltaPrometheusGrafanaKubernetes
repo coming soon
The data systems behind model development: reproducible, versioned training datasets and an evaluation platform with regression gates. Working name TrainForge.
What it does
- Ingests documents, app events, structured records, model outputs and human feedback into an immutable raw lakehouse layer.
- Distributed preprocessing: schema validation, normalization, language detection, PII filtering, quality scoring, tokenization, and semantic dedup with MinHash and LSH.
- A Dataset Registry versions every dataset with its source snapshots, transforms, filters, tokenizer and schema versions, and lineage, so any experiment can be reproduced exactly.
- Point-in-time snapshots, incremental processing, and safe backfills when a quality rule changes.
- Eval workers score models, prompts and retrieval strategies on accuracy, similarity, assertions, LLM-as-judge, latency, tokens and cost. Regression gates block promotion below thresholds, and a contamination check catches train and eval overlap.
What it enforces
MinHash + LSH
semantic dedup
Versioned
datasets with full lineage
Regression gates
quality, latency, cost
Contamination
train vs eval overlap checks
In design and early build. Drift monitoring and freshness metrics land in Prometheus and Grafana.
Spark or RayIceberg or DeltaKafkaPostgreSQLKubernetesPrometheusGrafana
repo coming soon
Natural-language analytics over a governed warehouse: a text-to-SQL agent grounded in a semantic layer of metric definitions and schema metadata.
What it does
- Grounded in a semantic layer of metric definitions and schema metadata rather than raw tables.
- Generated SQL is parsed and validated before execution, runs under read-only credentials with row limits, and self-corrects on errors.
- Evaluated on a gold question set against a schema-only baseline: execution accuracy, p95 latency and cost per query.
Eval
Running
execution accuracy on a gold set
Running
schema-only baseline
Running
p95 latency
Running
cost per query
Built on public TPC-H data and separate from my Microsoft work. Eval results go up here once the run finishes.
PythonLangGraphPostgreSQLTPC-H
repo coming soon
Explainable risk scoring on about 430K CDC BRFSS 2023 survey responses, deployed as a low-latency SageMaker endpoint.
What it does
- Mapped 350+ survey fields through the BRFSS codebook down to 33 curated features, with log and z-score transforms on skewed fields and statistical ranking using Cohen's d, t-tests and chi-squared.
- Compared XGBoost against LightGBM and logistic regression, tuned it with 150+ Optuna trials, and used SHAP for global and per-person explanations.
- Checked robustness by retraining without the dominant prior-diagnosis feature.
- Deployed on a SageMaker endpoint at under 150 ms p95, with CloudWatch latency alarms, MLflow tracking and an Airflow DAG from data prep through deploy.
- A Streamlit app for CSV upload, live scoring and SHAP views.
Results
~430K
survey responses
33
curated features
under 150 ms
p95 inference latency
150+
Optuna trials
XGBoostOptunaSHAPSageMakerCloudWatchMLflowAirflowStreamlit
A serverless AWS pipeline that pulls Spotify track, artist and album data every day and makes it queryable in Athena.
What it does
- A Lambda function pulls from the Spotify API on a daily CloudWatch Events schedule and lands raw data in S3.
- S3 events trigger a transform Lambda that cleans the data, checks the schema and writes to a transformed zone.
- Glue Data Catalog for schemas and Athena for SQL. Snowflake via Snowpipe is the planned next step.
Results
Daily
scheduled extract
2
Lambda stages
Serverless
nothing to keep running
PythonAWS LambdaS3GlueAthenaCloudWatch
An n8n agent that checks my calendar, the weather and air quality, then emails me the best trail for the day, or a reason to skip it.
What it does
- Five tools: checkCalendar, getWeather, getAirQuality (AirNow PM2.5), getHikeList from a trail sheet, and sendEmail.
- Recommends a trail only when the weather and air quality are good, and picks one that fits my time, elevation and shade preferences.
- Otherwise it sends a heads-up with the reason, so every decision is explainable.
Results
5
agent tools
PM2.5
air quality gate
n8nOpenAIGoogle CalendarGoogle SheetsGmailAirNow
A 3D open-world platformer scene inspired by the classics, built in Unity and C#.
What it does
- A 3D scene that takes the classic platformer idea to an open-world scale.
- Player interactions and character animations, with TextMeshPro for in-game text.
- A gameplay video lives in the repo README. The goal is a fully interactive open world with multiple levels.
UnityC#TextMeshProAnimation
Experience
MS Jan 2025 – Present
Microsoftcurrent
Building the data foundation, forecasting and LLM agents behind finance analytics.
P9 2025
Project990
Turned IRS Form 990 filings into clean datasets and a search service for analysts.
BI May 2023 – Dec 2024
Bloom Insurance
Worked on improving Partner Provider plans and claims data in legacy SSIS to AWS migrations.
EN Jun 2020 – Jul 2022
Ivy Comptech (Entain)
Built real-time and batch data pipelines for gaming analytics and set data contracts across producer teams.
Education and more
IU MS, Data Science, Indiana University Bloomington 2022 – 2024
KL BTech, Electronics and Communication, KL University 2016 – 2020
AWS Certified Solutions Architect, Associate certification
Fusion of visible and infrared images, Springer publication
Stack
Languages
7What I write every day, plus what I use for systems and game work.
PythonSQLPySparkT-SQLRustC#Java
Processing and streaming
5Batch and streaming engines.
SparkDatabricksKafkaKinesisSpark Structured Streaming
Lakehouse and storage
6Open table formats, columnar files and object storage.
Microsoft FabricDelta LakeIcebergS3ParquetArrow
Warehousing and modeling
7Where curated data lives, and how it is shaped.
SnowflakeRedshiftdbtKimball + SCD2Glue CatalogPostgreSQLAthena
Orchestration and quality
6Scheduling, data contracts, tests and CI/CD.
AirflowGreat ExpectationsAvro + Schema RegistryAzure DevOps CI/CDStep Functionsn8n
ML and MLOps
8Models from features to monitored endpoints.
ForecastingAnomaly detectionXGBoostscikit-learnSHAPOptunaMLflowSageMaker
GenAI and agents
8Retrieval and agents, with guardrails and evals.
MCPLangChainLangGraphRAGFAISSFastAPIOpenAI APILLM evals
Infra and observability
7Running it, and knowing when it breaks.
TerraformDockerKubernetesAWS LambdaCloudWatchPrometheusGrafana
Off the clock
Fig. 2, placeholder terrain
Also into
HikingFormula 1AnimeSpace documentariesGame dev, see Rex_Runner in Work
Working on forecasting, streaming or LLM systems? Let's talk.
Email is the fastest way to reach me.