Omar Chenine

AI systems

Decisions, tradeoffs, and failures.

Seven system cards. Each one names the flow, the decision that shaped it, the tradeoff it cost, and the failure that still needs a write up. The Mermaid diagram loads only when you ask for it.

RAG

Retrieval is evaluated before the generator is allowed to speak.

  1. Query
  2. Retriever
  3. Top k chunks
  4. Reranker
  5. Cited answer
Decision
Score retrieval first, then add generation. The retriever is the part a reviewer can measure without trusting a model's prose.
Tradeoff
A retrieval only gate slows the build and exposes weak chunks early. That is the point. A fluent answer hides a bad retriever.
nDCG@10Pending

Failure found and fixed

Pending: Add one real retrieval failure and the fix

System diagram

AI agents

Tool use with typed inputs, explicit failure paths, and a trace a reviewer can read.

  1. Intent
  2. Planner
  3. Tool call
  4. Validation
  5. Response or fallback
Decision
Every tool call returns a typed object and every step is logged. A free form agent loop is harder to evaluate and harder to debug.
Tradeoff
Typed tools and explicit branches take longer to add. They make the failure surface smaller and the transcript readable.
Task successPending

Failure found and fixed

Pending: Add one real tool call failure and the fix

System diagram

Voice agents

A latency budget by stage, with a fallback for every timeout.

  1. Speech in
  2. Transcript
  3. Model
  4. Tool
  5. Speech out
Decision
Set a per stage latency budget before writing prompts. A voice turn that sounds smart after two seconds still feels broken.
Tradeoff
Short model calls and narrow tools reduce quality on complex requests. The fallback path has to be good enough to carry the turn.
p95 latencyPending

Failure found and fixed

Pending: Add one real latency or turn taking failure and the fix

System diagram

Computer vision

Image input is treated as a signal problem, not a classification demo.

  1. Image
  2. Clean
  3. Extract trace
  4. Intervals
  5. Review flag
Decision
A low confidence result returns a review flag instead of a confident label. The interface has to make uncertainty actionable.
Tradeoff
The flag reduces automation and adds work for a reviewer. It also prevents the system from pretending a noisy trace is a diagnosis.
Digitization errorPending

Failure found and fixed

Pending: Add one real image or trace failure and the fix

System diagram

Signals

ECG images become intervals and a review decision, with calibration stated up front.

  1. Grid calibration
  2. Trace
  3. R peaks
  4. PR and QRS
  5. Rhythm
Decision
Make grid calibration explicit. Axis and interval numbers are meaningless without a stated scale and lead assumption.
Tradeoff
Explicit calibration limits the app to images that meet the assumption. It keeps the output honest.
PR interval errorPending

Failure found and fixed

Pending: Add one real calibration or signal failure and the fix

System diagram

Classic ML

Baselines, fold safe preprocessing, and a stack that has to beat a single model.

  1. Raw data
  2. Fold safe prep
  3. Models
  4. Out of fold
  5. Stack
Decision
Fit every transform inside the fold and report the best single model next to the stack. A stack that cannot beat one model is not a result.
Tradeoff
Repeated cross validation and nested preprocessing cost time. They also catch the optimistic scores that make a portfolio look good and a model fail.
RMSE, log targetPending

Failure found and fixed

Pending: Add one real leakage or stacking failure and the fix

System diagram

LLM evaluation

A retrieval metric, a task set, and a failure taxonomy before a model is called good.

  1. Task set
  2. Run
  3. Score
  4. Failure review
  5. Decision
Decision
Use a small, frozen task set with a baseline. A moving eval set turns every run into a new story.
Tradeoff
A frozen set goes stale. It has to be versioned and refreshed on a schedule, not edited after each run.
Task successPending

Failure found and fixed

Pending: Add one real eval failure and the fix

System diagram

The rule behind all seven

A result without a baseline is a story. Every card therefore carries a metric and a pending state until the number exists, and every case study keeps its evaluation design visible. That is slower to write and much harder to fake.