Clustering Unstructured Text with LLM Embeddings and HDBSCAN - MachineLearningMastery.com
This guide shows how to turn raw documents into semantic embeddings using a pre-trained sentence-transformers model, reduce their dimensionality with UMAP, and then apply HDBSCAN to uncover topic clusters without labels. It walks through data loading, embedding generation, dimensionality reduction, clustering, and sampling of cluster contents in Python.
- Pulling 150 posts from three newsgroup categories and encoding them with all-MiniLM-L6-v2 gave 384-dim embeddings in seconds.
- UMAP compressed those vectors from 384 to 5 dimensions to make density-based clustering feasible.
- HDBSCAN (min cluster size 8, min_samples 3) found just two clusters (101 and 49 docs) with zero points labeled noise, though sci.space and sci.med content blurred together in one cluster while autos formed a distinct second group.
- Adjusting HDBSCAN's cluster size and sample thresholds would change the number of clusters detected, and plotting all pairwise UMAP dimensions offers a visual check on whether clusters match intuitive topics.
You start by pulling 150 unlabeled posts from three 20-newsgroups categories—sci.space, sci.med and rec.autos—stripping out headers, footers and quotes so only raw text remains. After filtering out very short posts, you load the “all-MiniLM-L6-v2” model from sentence-transformers and encode each document into a 384-dimensional vector. Generating embeddings runs in seconds with a progress bar showing the workload.
Next you tackle dimensionality. High-dim vectors don’t work well for density clustering, so you apply UMAP with n_neighbors=15, n_components=5 and min_dist=0.0, shrinking each embedding down to five numbers. That step preserves local density patterns but makes the data easier to scan. The resulting shape goes from (150, 384) to (150, 5).
For grouping you pick HDBSCAN with a minimum cluster size of eight and min_samples=3. Feeding the reduced embeddings into HDBSCAN produces two clusters: one with 101 docs and the other with 49. No points end up labeled as noise. You then peek inside each group by printing three sample texts per cluster. Topic 0 mixes space-science and medical discussion (“blurring the distinction between real disease caused by Candida albicans…”), while Topic 1 clearly centers on auto performance (“diamond star cars…put out 190 hp in the turbo models…”).
The article points out that tweaking HDBSCAN’s hyperparameters—cluster size, sample threshold—can change how many groups you see. For a visual sanity check, you can plot all pairwise combinations of the five UMAP dimensions using matplotlib and seaborn. That scatterplot matrix helps you spot dense regions that HDBSCAN picked up and verify whether clusters line up with your intuitive topic boundaries.