Automating Session QA with LLMs at Artfills

At Artfills, mentors were reviewing live sessions manually against course rubrics. It took time, and two people reviewing the same session didn't always agree. The goal wasn't to replace mentors — it was to give ops a consistent first pass so humans could focus on the cases that actually needed attention.

We used OpenAI's Assistants API with rubrics baked into the prompts. Session QA returns structured JSON — scores, flags, short evidence notes — not a free-form paragraph. Learner data for recommendations flows through PostgreSQL and DynamoDB, with retries and deduplication on the pipeline side. GraphQL APIs expose results to admin dashboards. If the model returns something malformed, validation catches it before it lands in production data.

The check I use for this kind of feature: can someone on the ops team verify the output quickly? If not, it's not ready to ship. Rubrics, schema validation, and retry logic aren't flashy, but they're what made the team comfortable relying on it week to week.