Building an AI Design Review Assistant – Week 1: Understanding LLM APIs

This week’s goal was to understand what actually happens when an application talks to an LLM.

What does an LLM request look like? What does the response contain? Beyond the generated text, what metadata does the API return that would be useful when building an application?

To answer those questions, I built a small Python script that sends a sample software design document to an LLM and prints the model, latency, token usage, and response.

The first version

I created a small Python script that sends a sample software design document to the model and prints the response. Since this project is about building an AI Design Review Assistant, I used a deliberately incomplete order service design.

Sample design document

Order Service Design

The service receives order creation requests through a REST API.
It stores orders in PostgreSQL.
After saving the order, it publishes an OrderCreated event to Kafka.
Other services consume the event for inventory, payment, and fulfillment.

Current design:
- No retry strategy
- No dead-letter queue
- No idempotency key
- No explicit timeout policy
- No monitoring or alerting described
- PostgreSQL is the main source of truth

The prompt was intentionally simple.

You are a Principal Software Engineer reviewing a backend system design.

Review the design below. List strengths, risks, and gaps across reliability,
scalability, security, observability, API design, data consistency, and operations.

The first response honestly exceeded my expectations. The model didn’t just point out the obvious gaps like retries and monitoring, it also discussed topics such as the transactional outbox pattern, Kafka partitioning, API idempotency, and distributed tracing.

Along with the review, I also logged a few basic metrics.

Model: gpt-4.1-mini
Prompt length: 707 characters
Latency: 14.7 seconds

At this point, I had a simple script working. The response was perfectly readable for a human, but not particularly useful for an application.

If I wanted to build a frontend, highlight only critical findings, filter issues by category, or store the review for later analysis, parsing several pages of free-form text wasn’t the right approach.

From paragraphs to structured data

The next step was to make the response predictable.

Instead of asking the model to generate a free-form review, I updated the application to use structured outputs so the response always follows a predefined schema.

{
  "confidence": 0.9,
  "summary": "...",
  "needs_human_review": true,
  "findings": [
    {
      "category": "reliability",
      "priority": "critical",
      "summary": "No retry strategy..."
    }
  ]
}

Even though the review itself is largely the same, the structured response is much easier for an application to work with. Instead of parsing paragraphs and headings, the application now receives typed data that can be validated, filtered, and eventually displayed in a UI.

As part of that refactoring, I also extracted the OpenAI integration into a reusable LLMClient. The review script is now responsible only for building the prompt and displaying the results, while the client handles communication with the model, logging latency and token usage, retries, validation, and response parsing.

Running the updated version produced much more than just a design review.

Model: gpt-4.1-mini-2025-04-14
Validation: direct
Latency: 5953 ms
Input tokens: 442
Output tokens: 295
Estimated cost: $0.000649

One thing I wanted from the beginning was visibility into the LLM call itself. Logging latency, token usage, request IDs, and other metadata may not seem important for a small script, but I expect they’ll become much more valuable as I add RAG, document ingestion, and larger prompts in the coming weeks.

What’s next?

For now, everything still runs from a single Python script. The next step is to turn it into an actual backend service using FastAPI. The goal is to expose a /review endpoint, separate the API layer from the review logic, and start structuring the project like a production backend service rather than a standalone script.

Build notes

Development tools

• ChatGPT – Helped me create a weekly learning plan, explain concepts
• Cursor – Used as my coding assistant while implementing the project as a pair programmer.

What I built

• Initial Python script to review a software design document using an LLM.
• Reusable LLMClient to encapsulate OpenAI interactions.
• Structured JSON responses using Pydantic models.
• Basic request logging (latency, token usage, request IDs, and estimated cost).
• Response validation before returning results.

GitHub snapshot

• design_review_llm.py — Entry point and sample design review script
• llm_client.py — Reusable OpenAI client with retries, logging, token usage, and validation
• models.py — Pydantic models for structured outputs