Healthcare Price Transparency

// The Question

U.S. hospital pricing is famously opaque: the same procedure can be billed at wildly different rates depending on the hospital, the payer, and the ZIP code. Federal transparency rules forced hospitals to publish their prices — producing a mountain of messy data that few people have actually analyzed. This project asks: what actually drives hospital price variability, and can we segment the market in a way that's useful for pricing strategy and payer negotiations?

// The Data

The Kaggle Transparency in Hospital Prices dataset: roughly 300 million price records (27.4 GB) plus metadata on 1,800+ hospitals. Too large for pandas in memory, so the pipeline starts with Dask — converting CSV to Parquet across 40 partitions before any modeling. After cleaning and feature engineering, the modeling matrix covers 227,708 records across 12,255 columns, spanning 1,562 hospitals in 1,085 cities and 44 states.

// The Approach

• Classification — Bernoulli and Gaussian Naïve Bayes, a depth-5 decision tree, and class-balanced linear/RBF SVMs predicting hospital pricing tiers
• Clustering — K-Means (k=4) with PCA, HDBSCAN density clustering, and agglomerative clustering with dendrogram analysis to map market structure
• Association rules — Apriori mining (MLxtend) with bootstrap resampling and cross-validation stability filtering, so only patterns that survive resampling are reported

// Results

Bernoulli Naïve Bayes classifies pricing tiers at 0.869 accuracy and 0.957 ROC-AUC against a ~0.554 majority-class baseline. The decision tree is close behind (ROC-AUC 0.867, minority-class F1 0.762, 5-fold CV ROC-AUC 0.863 ± 0.009), and the RBF SVM reaches ~0.778 accuracy. K-Means resolves four coherent market segments (silhouette 0.451): value-oriented small facilities, two premium standard tiers, and large enterprise systems. Of 156 association rules found via cross-validation, 72 (46.2%) were stable in every fold.

• No single feature dominates price — procedure code, payer, geography, and facility size each contribute weak signals, which is exactly why multi-variable models are needed
• Median outpatient price ~$550 vs. ~$228 inpatient; implantable devices top the price range (code C1776, median ~$2,095)
• Location-anchored pricing patterns concentrate in Kentucky (Louisville, Lexington, Paducah)

// A Note on Honesty

An earlier version of this pipeline reported 0.927 accuracy. Auditing the code revealed a silent chi-squared feature-selection fallback that inflated the number. The corrected figure — 0.869 — is the one reported, and the bug is documented in the repo. Knowing what your model is actually doing matters more than a prettier headline metric.