← Questions

Load daily LLM chat logs into a warehouse idempotently

0
CohereSystem designLast asked Sep 2026

Every day one JSONL file of LLM conversations lands in object storage; each line is a conversation with an id, a user id, its messages and metadata. Design the job that loads it into a data warehouse for analysis.

Cover the table schema (conversations, messages), validating lines and where bad ones go, removing duplicates within and across days, making a rerun of the same day safe (idempotent loads), and how the pipeline would grow if the logs became a stream.

Your canvas
Draw your answer hereBoxes, arrows and notes, saved for you to come back to.Sign in to start drawing