Learning from 6,000+ Prompts to My AI App Without Reading the Messages: Privacy-First Topic Modeling in Python

If you ship an AI product to real users, conversation logs are your best product signal, but they can contain family stories, health details, and other sensitive content. The naive approach is to read a sample of conversations for user insights, but that does not scale and creates real privacy risk.

This talk walks through the Python topic-modeling pipeline I built for CantoAI, an AI Cantonese conversation partner. The dataset is 400+ users, 1,200+ conversations, and 6,000+ logged turns. CantoAI is the case study, but this pipeline can be applied to any AI product with sensitive user inputs: customer support agents, copilots, and more.

With this pipeline, I do not qualitatively review individual user turns to label topics. Instead, this is done algorithmically: clustering runs on embeddings computed from user text in an offline script.

Outputs use aggregates and keyword-derived labels, not verbatim quotes to keep anonymity. I generate a public treemap that shows category names, topic labels, counts, and percentages. These topic labels come from BERTopic keywords.

I'll cover the full pipeline:

• BERTopic with the multilingual paraphrase-multilingual-MiniLM-L12-v2 embedding model for Cantonese-English code-switching
• HDBSCAN clustering with min_cluster_size=10 set deliberately as a privacy threshold, so a topic only forms if at least 10 user turns cluster together in the embedding space. Smaller or sparse groups are labeled noise and excluded from visualizations.
• UMAP dimensionality reduction tuned separately for clustering and visualization
• Regex-based PII filtering and aggregated outputs so visualizations never surface verbatim quotes. I use aggregation-only pandas outputs using functions like:

topic_summary = (
    df.groupby(["topic", "scenario"], dropna=False)
    .agg(
        turns=("query", "size"),
        users=("user_id", "nunique"),
    )
    .reset_index()
)

This pattern is the discipline I use throughout the analysis: every chart and table starts from groupby + agg, never from printing or plotting raw query rows.

• Plotly for the final interactive visualization: a topic map showing what heritage learners actually talk about with my AI.

Attendees will leave with the code patterns and a concrete checklist they can apply to their own AI products immediately.

Related reading: my Substack series introducing this topic (part 1, part 2) on the original 2,800-conversation analysis; Anthropic's Clio paper as related work at million-conversation scale.