Building an AI Design Review Assistant – Part 2: Script to FastAPI Service

In Part 1, I built a Python script that could review a design document and return a structured JSON response. The natural next step was to to expose this functionality as an API which can be called from other applications.

With the initial version, almost everything lived inside a single script. With HTTP API in picture, I wanted a cleaner separation of responsibilities. Following the same principles I’d use when building any backend service, I extracted the core review logic into a ReviewService.

ReviewService is responsible for:

building the prompt,
calling the LLM,
validating the response, and
returning a structured result.

FastAPI simply accepts an HTTP request and delegates the work to the service layer.

The first endpoint

With that structure in place, exposing the review engine over HTTP became surprisingly small.

@app.post("/review")
def review(request: ReviewRequest):
    return review_design(request.design_doc)

The endpoint has only responsibility is translating the Http request into a call to the service layer. The review logic would be abstracted from the Interface layer.

Calling the API

Once the endpoint was wired up, testing it was as simple as sending an HTTP request with a design document.

Request

curl -X POST http://127.0.0.1:8000/review \
  -H "Content-Type: application/json" \
  -d '{
    "design_doc": "Order service uses PostgreSQL and Kafka. No retries, no DLQ, no idempotency."
  }'

The API accepts a design document as input.

Response

The response is more than just the review itself.

{
  "review": {
    "confidence": 0.9,
    "summary": "The design leverages PostgreSQL and Kafka but lacks retries, dead-letter queues, and idempotency...",
    "needs_human_review": true,
    "findings": [
      {
        "category": "reliability",
        "priority": "critical",
        "summary": "Absence of retries, dead-letter queue, and idempotency increases the risk of message loss."
      },
      {
        "category": "data_consistency",
        "priority": "critical",
        "summary": "High risk of duplicate or lost messages."
      }
    ]
  },
  "metadata": {
    "model": "gpt-4.1-mini-2025-04-14",
    "latency_ms": 9502.67,
    "token_usage": {
      "input_tokens": 479,
      "output_tokens": 375,
      "total_tokens": 854
    },
    "estimated_cost_usd": 0.001604
  }
}

(I’ve trimmed the response here for readability.)

The metadata block also turned out to be very useful. Besides the review itself, every request returns information about the model used, latency, token usage and estimated cost.

Cost is the first thing I looked at. This gives me visibility into how much money I m burning :). it’s an easy way to understand how expensive different prompts are. But as the project grows, I expect latency and token usage to become equally valuable for understanding where optimizations are needed.

The latency is still huge similar to the python script which is around 10s for this request

While reading through the OpenAI documentation, I noticed the stream=true option. I wasn’t sure whether it would reduce the latency, but it seemed like a good opportunity to understand how streaming works in practice.

Streaming endpoint

To explore that, I added a second endpoint: /review/stream.

This endpoint calls the OpenAI Responses API with stream=true.

When my application talks to OpenAI, the SDK emits events such as:

response.output_text.delta
response.completed

Rather than exposing OpenAI’s event types directly, ReviewService translates them into events that make sense for clients consuming my API.

Calling the streaming endpoint

The request body is the same as the synchronous endpoint:

curl -N -X POST http://127.0.0.1:8000/review/stream \
  -H "Content-Type: application/json" \
  -d '{
    "design_doc": "Order service uses PostgreSQL and Kafka. No retries, no DLQ, no idempotency.",
    "use_retrieval": false
  }'

The response uses Server-Sent Events and contains two main event types.

As the model generates text, the endpoint sends a series of delta events:

event: delta
data: {"text": "{\""}

event: delta
data: {"text": "confidence"}

...
event: done
data: { ... validated review ... }

The client receives multiple delta events while generation is in progress, followed by a final done event containing the validated review and metadata.

At this stage of the project, I haven’t built a frontend. My primary client is still curl running in a terminal, so I’m not actually consuming the stream as it arrives.

Because of that, streaming doesn’t provide much immediate value today. The final validated response is still what I care about.

Looking back

The review logic itself changed very little from Week 1.

Most of the work went into turning a prototype into a service: separating responsibilities and designing a API

As next step I’ll shift my focus from exposing the service to making it smarter. So far, every review has relied entirely on the model’s built-in knowledge. I’ll start exploring Retrieval-Augmented Generation (RAG) so the assistant can incorporate external engineering documents and produce reviews grounded in project-specific context.

Engineering Notes