Clustering

Standalone clustering functions. Each one is also exposed as a Dataset method of the same name. They all take a set of numeric feature_cols and return (new_df, figures), where new_df is the input dataframe with an added cluster column.

KMeans

Partitions the data into k clusters by minimizing within-cluster variance, with an automatic search over k via the silhouette score when k isn’t given explicitly.

RepLikCompare.clustering.compute_cluster_kmean.compute_cluster_kmean(df, feature_cols=None, max_cluster=20, random_state=0, n_init=10, max_iter=300, ncluster=None, silhouette_plot=True, context='notebook', figsize=(10, 5))[source]

Cluster the data using the KMeans algorithm.

Parameters:

  • df (pandas.DataFrame) – Data to cluster. If feature_cols is given, only those columns are used to fit KMeans (the returned dataframe keeps ALL of df’s original columns, plus cluster) – this is what you want when df also carries metadata columns (sim_name, replica, frame, …) that must not be fed to the clustering algorithm itself. If feature_cols is None (default, backward-compatible behavior), df is used as-is – every column must be numeric, or KMeans.fit will raise.

  • feature_cols (list of str, optional) – Column names in df to actually cluster on. Strongly recommended whenever df has metadata columns beyond the raw numeric features. Default None (use all of df).

  • max_cluster (int) – Maximum number of clusters tested when ncluster is None.

  • random_state (int) – Random seed.

  • ncluster (int or None) – If provided, uses this number of clusters directly (skips the silhouette search entirely).

  • silhouette_plot (bool) – If True (and ncluster is None, i.e. a search is actually performed), compute and return the silhouette-vs-k plot. Default True.

  • context (str, optional) – Seaborn plotting context for the silhouette plot. Default “notebook”.

  • figsize (tuple, optional) – Figure size for the silhouette plot. Default (10, 5).

Returns:

  • new_df (pandas.DataFrame) – df (all original columns preserved) with a ‘cluster’ column added (categorical, 1..k).

  • figures (dict) – {“silhouette”: matplotlib.figure.Figure} if a silhouette search was performed and silhouette_plot=True; otherwise {} (e.g. when ncluster was given directly, or silhouette_plot=False). Always a dict – never None, never silently dropped – so the plot is always retrievable as an object for further tweaking, e.g. figures[“silhouette”].savefig(…).

Note

This function no longer produces a generic 2D scatter of the cluster assignment (it used to plot df.iloc[:, 0] vs df.iloc[:, 1] unconditionally, which is meaningless once df has more than two feature columns, or the “wrong” two). To visualize the resulting clusters, merge the cluster column back into your full dataframe and use plot_scatter (or Dataset.plot_scatter) with whichever two columns you actually want on the axes, e.g.:

new_df, figures = compute_cluster_kmean(df, feature_cols=[...], ncluster=4)
plot_scatter(new_df, x="PC1", y="PC2", sim_name_col="cluster")

Gaussian Mixture Model

Soft, probabilistic clustering that fits a mixture of Gaussians to the data; the number of components can be selected automatically via AIC or BIC.

RepLikCompare.clustering.compute_cluster_GMM.compute_cluster_GMM(df, feature_cols=None, max_cluster=20, min_cluster=1, random_state=0, n_init=1, max_iter=100, covariance_type='full', tol=0.001, reg_covar=1e-06, init_params='kmeans', ncluster=None, criterion='bic', ic_plot=True, context='notebook', figsize=(10, 5))[source]

Cluster the data using a Gaussian Mixture Model (GMM).

Parameters:

  • df (pandas.DataFrame) – Data to cluster. If feature_cols is given, only those columns are used to fit the GMM (the returned dataframe keeps ALL of df’s original columns, plus cluster) – use this whenever df also carries metadata columns (sim_name, replica, frame, …) that must not be fed to the model itself. If feature_cols is None (default, backward-compatible), df is used as-is – every column must be numeric.

  • feature_cols (list of str, optional) – Column names in df to actually cluster on. Default None (use all of df).

  • max_cluster (int) – Maximum number of components (clusters) tested when ncluster is None.

  • min_cluster (int) – Minimum number of components (clusters) tested when ncluster is None.

  • random_state (int) – Random seed.

  • n_init (int) – Number of initializations performed for each number of components.

  • max_iter (int) – Maximum number of EM iterations.

  • covariance_type (str) – Covariance type, one of “full”, “tied”, “diag”, “spherical”.

  • tol (float) – Convergence threshold for the EM algorithm.

  • reg_covar (float) – Non-negative regularization added to the diagonal of covariance matrices, to avoid singular covariance matrices.

  • init_params (str) – Method used to initialize the weights, means and covariances, one of “kmeans”, “k-means++”, “random”, “random_from_data”.

  • ncluster (int or None) – If provided, uses this number of components directly (skips the AIC/BIC search entirely).

  • criterion (str) – Criterion used to select the best number of components when ncluster is None, one of “aic” or “bic”. The default is “bic”.

  • ic_plot (bool) – If True (and ncluster is None), compute and return the AIC/BIC vs number-of-components plot. Default True.

  • context (str, optional) – Seaborn plotting context for the AIC/BIC plot. Default “notebook”.

  • figsize (tuple, optional) – Figure size for the AIC/BIC plot. Default (10, 5).

Returns:

  • new_df (pandas.DataFrame) – df (all original columns preserved) with a ‘cluster’ column added (categorical, 1..k).

  • figures (dict) – {“aic_bic”: matplotlib.figure.Figure} if a search was performed and ic_plot=True; otherwise {}. Always a dict – never None, never silently dropped – so the plot is always retrievable as an object for further tweaking, e.g. figures[“aic_bic”].savefig(…).

Note

This function no longer produces a generic 2D scatter of the cluster assignment (it used to plot df.iloc[:, 0] vs df.iloc[:, 1] unconditionally). To visualize the resulting clusters, merge the cluster column back into your full dataframe and use plot_scatter (or Dataset.plot_scatter) with whichever two columns you actually want on the axes.

DBSCAN

Density-based clustering that finds clusters of arbitrary shape and flags low-density points as noise, without needing to choose a number of clusters in advance.

RepLikCompare.clustering.compute_cluster_dbscan.compute_cluster_dbscan(df, feature_cols=None, eps=0.5, min_samples=5)[source]

Cluster the data using the DBSCAN algorithm.

Parameters:

  • df (pandas.DataFrame) – Data to cluster. If feature_cols is given, only those columns are used to fit DBSCAN (the returned dataframe keeps ALL of df’s original columns, plus cluster) – use this whenever df also carries metadata columns (sim_name, replica, frame, …) that must not be fed to the clustering itself. If feature_cols is None (default, backward-compatible), df is used as-is – every column must be numeric.

  • feature_cols (list of str, optional) – Column names in df to actually cluster on. Default None (use all of df).

  • eps (float, optional) – The maximum distance between two samples for one to be considered in the neighborhood of the other. Default 0.5.

  • min_samples (int, optional) – The number of samples in a neighborhood for a point to be considered as a core point. Default 5.

Returns:

  • new_df (pandas.DataFrame) – df (all original columns preserved) with a ‘cluster’ column added: real clusters are renumbered 1..k by decreasing population, and noise/unclustered points are labeled -1 – the standard scikit-learn/DBSCAN convention, and what plot_cluster_timeline/assign_cluster_representative expect by default (noise_labels=(-1,) / noise_label=-1).

  • figures (dict) – Always {} – DBSCAN has no natural model-selection curve to plot (no k to search over). Kept in the return signature for API consistency with the other compute_cluster_* functions (new_df, figures = compute_cluster_dbscan(…)). To visualize the resulting clusters, merge the cluster column back into your full dataframe and use plot_scatter (or Dataset.plot_scatter).

HDBSCAN

Hierarchical variant of DBSCAN that adapts to clusters of varying density and only requires a minimum cluster size, rather than a fixed neighborhood radius (eps).

RepLikCompare.clustering.compute_cluster_hdbscan.compute_cluster_hdbscan(df, feature_cols=None, min_cluster_size=50, min_samples=50)[source]

Cluster the data using the HDBSCAN algorithm.

Parameters:

  • df (pandas.DataFrame) – Data to cluster. If feature_cols is given, only those columns are used to fit HDBSCAN (the returned dataframe keeps ALL of df’s original columns, plus cluster) – use this whenever df also carries metadata columns (sim_name, replica, frame, …) that must not be fed to the clustering itself. If feature_cols is None (default, backward-compatible), df is used as-is – every column must be numeric.

  • feature_cols (list of str, optional) – Column names in df to actually cluster on. Default None (use all of df).

  • min_cluster_size (int, optional) – Minimum cluster size. Default 50.

  • min_samples (int, optional) – Minimum number of samples. Default 50.

Returns:

  • new_df (pandas.DataFrame) – df (all original columns preserved) with a ‘cluster’ column added: real clusters are renumbered 1..k by decreasing population, and noise/unclustered points are labeled -1 – the standard scikit-learn/HDBSCAN convention, and what plot_cluster_timeline/assign_cluster_representative expect by default (noise_labels=(-1,) / noise_label=-1).

    Earlier versions of this function remapped noise to 0 internally and then dropped that category via .remove_categories(…), which silently turned it into NaN instead – inconsistent with the -1 convention documented (and used) elsewhere in this package. Noise is now kept as an explicit -1, not NaN.

  • figures (dict) – Always {} – HDBSCAN has no natural model-selection curve to plot (no k to search over). Kept in the return signature for API consistency with the other compute_cluster_* functions (new_df, figures = compute_cluster_hdbscan(…)). To visualize the resulting clusters, merge the cluster column back into your full dataframe and use plot_scatter (or Dataset.plot_scatter).

Hierarchical (agglomerative) clustering

Builds a dendrogram by iteratively merging the closest clusters, then cuts it at a chosen number of clusters; useful when the nested/ hierarchical relationship between clusters matters, not just the final partition.

RepLikCompare.clustering.hierarchical_clustering.hierarchical_clustering(df, feature_cols=None, max_cluster=20, min_cluster=2, ncluster=None, method='ward', metric='euclidean', max_data=None, silhouette_plot=True, dendrogram_plot=True, truncate_mode='lastp', p=30, context='notebook', figsize=(10, 5))[source]

Cluster the data using Agglomerative Hierarchical Clustering.

Parameters:

  • df (pandas.DataFrame) – Data to cluster. If feature_cols is given, only those columns are used to compute the linkage matrix (the returned dataframe keeps ALL of df’s original columns, plus cluster) – use this whenever df also carries metadata columns (sim_name, replica, frame, …) that must not be fed to the clustering itself. If feature_cols is None (default, backward-compatible), df is used as-is – every column must be numeric.

  • feature_cols (list of str, optional) – Column names in df to actually cluster on. Default None (use all of df).

  • max_cluster (int) – Range of cluster counts tested when ncluster is None.

  • min_cluster (int) – Range of cluster counts tested when ncluster is None.

  • ncluster (int or None) – If provided, uses this number of clusters directly (skips the silhouette search).

  • method (str) – Passed to scipy.cluster.hierarchy.linkage.

  • metric (str) – Passed to scipy.cluster.hierarchy.linkage.

  • max_data (int or None, optional) – If df has more rows than this, it is randomly subsampled first (linkage is O(n^2) memory / O(n^3) time – this keeps it tractable on large trajectories). Default None (no subsampling).

  • silhouette_plot (bool) – If True (and ncluster is None), compute and return the silhouette-vs-k plot. Default True.

  • dendrogram_plot (bool) – If True, compute and return the dendrogram plot. Default True.

  • truncate_mode (optional) – Passed to scipy.cluster.hierarchy.dendrogram.

  • p (optional) – Passed to scipy.cluster.hierarchy.dendrogram.

  • context (str, optional) – Seaborn plotting context. Default “notebook”.

  • figsize (tuple, optional) – Figure size for each plot. Default (10, 5).

Returns:

  • new_df (pandas.DataFrame) – df (all original columns preserved, possibly subsampled if max_data triggered) with a ‘cluster’ column added.

  • figures (dict) – Any of {“silhouette”: fig, “dendrogram”: fig}, depending on which plots were requested/applicable – e.g. {} if both silhouette_plot=False and dendrogram_plot=False, or just {“dendrogram”: fig} if ncluster was given directly (no silhouette search performed). Always a dict – never None, never silently dropped.

Note

This function no longer produces a generic 2D scatter of the cluster assignment (it used to plot df.iloc[:, 0] vs df.iloc[:, 1] unconditionally, silently discarding the actual silhouette/dendrogram figures in the process – a real bug in the previous version). To visualize the resulting clusters, merge the cluster column back into your full dataframe and use plot_scatter (or Dataset.plot_scatter) with whichever two columns you actually want on the axes.

Cluster representatives

Given an already-computed cluster column, picks one representative frame per cluster – the point closest to the cluster’s centroid or medoid – so downstream analysis (e.g. visual inspection) can work with a single frame per state instead of the whole cluster.

RepLikCompare.clustering.assign_cluster_representative.assign_cluster_representative(df, cluster_col, feature_cols, group_cols=None, method='medoid', noise_label=-1, include_distance=True)[source]

For each cluster, find the single row (frame) that best represents it, and return those rows as a small lookup table.

Parameters:

  • df (pandas.DataFrame) – Long-format dataframe, one row per frame, already containing a cluster assignment column (e.g. the output of compute_cluster_hdbscan/compute_cluster_kmean/…).

  • cluster_col (str) – Column with the integer cluster label per frame.

  • feature_cols (str or list[str]) – Columns defining the space the cluster was computed in (e.g. the same features/PCA components fed into the clustering method). The representative is chosen by distance in THIS space, so pass the same features used for clustering, not arbitrary other columns.

  • group_cols (str, list[str] or None, default None) – If given (e.g. [sim_name_col], or [sim_name_col, replica_col]), representatives are chosen independently within each group AND within each cluster label – use this if the same cluster_col integer can mean different things in different groups, or if you specifically want one representative per (group, cluster) pair rather than one per cluster label globally. Leave as None for the common case of a single global clustering across the whole dataset.

  • method ({"centroid", "medoid"}, default "medoid") – “centroid”: representative = the row closest (Euclidean, in feature_cols space) to the mean of the cluster. Fast (O(n)), and the standard choice. “medoid”: representative = the row with the smallest average distance to every OTHER row in the cluster. More robust to skewed/non-convex clusters, but O(n^2) per cluster – avoid for very large clusters (thousands of frames).

  • noise_label (int or None, default -1) – Cluster label treated as unclustered/noise (the HDBSCAN/DBSCAN convention) and excluded from the output. Pass None if your clustering method has no noise sentinel (e.g. KMeans/GMM/ hierarchical).

  • include_distance (bool, default True) – If True, include the representative’s distance to the centroid/medoid reference in the output (useful as a rough “how tight is this cluster” sanity check – large distance means even the best representative is a poor stand-in for the cluster).

Returns:

pandas.DataFrame – One row per (group, cluster) [or just per cluster, if group_cols is None], containing: the group_cols (if any), cluster_col, frame_index (the original df.index value of the representative row – use this to look the frame back up, e.g. df.loc[frame_index]), cluster_size, and, if include_distance=True, distance_to_reference.

Notebook example