Lost In Translation - Understanding nuance in SAS® to Python code conversion for Hospital Star Ratings

Gabe Dumbrille | Oct. 31, 2022

An increasing number of organizations are converting longstanding SAS® code into open-source languages such as Python. There are many reasons organizations are migrating from SAS® to open source. Regardless of the rationale, each conversion project will be challenged to “find the right words” so-to-speak to translate SAS® into the chosen open-source language. Doing this effectively involves leaning on the experience of your team, researching documentation, and experimenting through trial and error.

During a recent conversion of the CMS Hospital Compare SAS® package into Python, a team of SAS and Python developers unearthed a very specific difference in how SAS® and Python handled the same operation.

Specifically, consider the following SAS® code below:

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.


	/* k-means*/
	proc fastclus data=Summary_score2 maxc=5 converge=0 maxiter=1000 seed=s33;************;
	var summary_score;

	odsoutput ClusterCenters=seeds2;
	odsoutput ConvergenceStatus=cstatus;
	run;
	proc print data=cstatus;title"QA: FASTCLUS convergence"; run;

	*Step2. using results from Step 1 as initial seeds, run K-means to complete convergence with 'strict=1' added;
	*to avoid potential outliers effect on clustering;
	proc fastclus data=Summary_score2 maxc=5out=clusters converge=0 maxiter=1000 seed=seeds2 strict=1;************;
	var summary_score;

	odsoutput ClusterCenters=Cluster_mean;
	odsoutput ConvergenceStatus=cstatus2;
	run;
	proc print data=cstatus2;title"QA: FASTCLUS convergence"; run;

Here proc fastclus is used to implement the well know k-means clustering algorithm to assign Overall Star Ratings to each hospital. In the first call of proc fastclus, SAS outputs the initial cluster assignment for each observation and the distance from cluster center. In “Step 2” proc fastclus is called a second time and precipitated an in-depth analysis of the last argument: strict=1. The experienced SAS® developers on the team researched the documentation, and leaning on prior work, were able to determine that the “strict=1” was essentially a parameter to remove any outliers in the clustering, which were more than 1 standard deviation away from the minimum distance in the clusters produced by the first run of proc fastclus.

The Python conversion leaned on the well-known scikit-learn machine learning library (sklearn). The K-means function from sklearn is the closest and best Python analog to the SAS® proc fastclus.

Taking a glance at the function’s arguments, it’s apparent there is no one-to-one “strict” argument comparison in the K-means function. In converting this SAS® code to Python, the first attempt yielded very similar results between the two. However, differences in the output between the SAS® and Python implementation remained.

With this understanding in place, and considering the analysis of the SAS® code regarding “strict=1”, the Python implementation was adjusted with the following lines of code:


	kmeans1=KMeans(
	n_clusters=5,
	init=s33.reshape(-1,1),
	n_init=1,
	tol=0,
	max_iter=1000
	).fit(s2['summary_score'].values.reshape(s2['summary_score'].shape[0],1))

	s2['star'] =kmeans1.labels_.astype('float')

	for i in range(s2['grp'].nunique()):
	s2['cluster'+str(i+1)] =kmeans1.cluster_centers_[i][0]
	s2['dist'+str(i+1)] =euclidean_distances(s2['cluster'+str(i+1)]
	.values.reshape(s2['cluster'+str(i+1)].shape[0],1),
	s2['summary_score'].values.reshape(s2['summary_score'].shape[0],1))[0]

	s2['min_dist'] =s2[['dist1','dist2','dist3','dist4', 'dist5']].min(axis=1)
	outliers=pd.DataFrame(s2.loc[s2['min_dist'] >=1])
	drop_idx=s2.loc[s2['min_dist'] >=1].index

	s2=s2.drop(drop_idx)
	s2=s2.drop(['dist1','dist2','dist3','dist4', 'dist5']
	+ ['cluster'+str(i+1) for i in range(s2['grp'].nunique())], axis=1)
	outliers=outliers.drop(['dist1','dist2','dist3','dist4', 'dist5']
	+ ['cluster'+str(i+1) for i in range(s2['grp'].nunique())], axis=1)

	kmeans2=KMeans(
	n_clusters=5,
	init=kmeans1.cluster_centers_,
	n_init=1,
	tol=0,
	max_iter=1000
	).fit(s2['summary_score'].values.reshape(s2['summary_score'].shape[0],1))

After identifying and dropping “outliers” exceeding 1 standard deviation, there was one last tweak to make in the Python implementation. There is a parameter in the sklearn K-Means function called “tol”. This parameter specifies the stopping point or tolerance, for the clustering algorithm. Through experimentation, it was found that adjusting the “tol” argument from its default value to 0 allowed the sklearn K-means to match the SAS® proc fastclus output exactly. See the final Python K-means implementation below:


	s2['min_dist'] =s2[['dist1','dist2','dist3','dist4', 'dist5']].min(axis=1)
	outliers=pd.DataFrame(s2.loc[s2['min_dist'] >=1])
	drop_idx=s2.loc[s2['min_dist'] >=1].index

By matching the equivalent parameters between the sklearn K-means and the SAS® proc fastclus, the few lines of Python code added to account for the effect of "strict=1" were able to complete the translation of SAS® to Python to match perfectly.

Code conversion is a complex and nuanced endeavor. Being able to leverage experience and documentation, as well as experimentation, go a long way in gaining the understanding needed to convert code successfully. Our team was equipped with diverse and complimentary skillsets which fostered the collaboration needed to solve this tricky problem.

We hope you enjoyed reading about our experience in code conversion and that our findings may help guide or inspire your own code conversions.