In the glut of Python data analysis tools, I'm sometimes embarrassed by my lack of comfort with Python for analysis. Static types, Java/Scaladoc, and slick IDEs in concert with compilers provide a guides that I haven't been able to replace in Python. Additionally, the problem of dynamic types seems to exacerbate problems with library interoperability. With Anaconda and Jupyter, though, I can share some quick notes on getting started.
Here are some notes on surveying some admittedly canned data to classify malignant/benign tumors. The Web is littered with examples of using sklearn to classify iris species using feature dimensions, so I thought I would share some notes exploring one of the other datasets included with scikit-learn, the Breast Cancer Wisconsin (Diagnostic) Data Set. I've also decided to use Python 3 to take advantage of comprehensions and because that's what the Python community uses where I work.
The notebook below illustrates how to load demo data (loading csv is simple, too), convert the scikit-learn matrix to a DataFrame if you want to use Pandas for analysis, and applies linear and logistic regression to classify tumors as malignant or benign.
%matplotlib inline import numpy as np import matplotlib.pyplot as plt from sklearn.linear_model import LinearRegression import pylab as pl import pandas as pd from sklearn import datasets # demo numpy matrix to Pandas DataFrame bc = datasets.load_breast_cancer() pbc = pd.DataFrame(data=bc.data,columns=bc.feature_names) pbc.describe()
|mean radius||mean texture||mean perimeter||mean area||mean smoothness||mean compactness||mean concavity||mean concave points||mean symmetry||mean fractal dimension||...||worst radius||worst texture||worst perimeter||worst area||worst smoothness||worst compactness||worst concavity||worst concave points||worst symmetry||worst fractal dimension|
8 rows × 30 columns
from math import sqrt from sklearn.linear_model import LinearRegression from sklearn.linear_model import LogisticRegression # Plot training-set size versus classifier accuracy. def make_test_train(test_count): n = bc.target.size trainX = bc.data[0:test_count,:] trainY = bc.target[0:test_count] testX = bc.data[n//2:n,:] testY = bc.target[n//2:n] return trainX, trainY, testX, testY def eval_lin(trainX, trainY, testX, testY): regr = LinearRegression() regr.fit(trainX, trainY) y = regr.predict(testX) err = ((y.T > 0.5) - testY) correct = [x == 0 for x in err] return sum(correct) / err.size, np.std(correct) / sqrt(err.size) def eval_log(trainX, trainY, testX, testY): regr = LogisticRegression() regr.fit(trainX, trainY) correct = (regr.predict(testX) - testY) == 0 return sum(correct) / testY.size, np.std(correct) / sqrt(correct.size) def lin_log_cmp(n): trainX, trainY, testX, testY = make_test_train(n) # min 20 lin_acc, lin_stderr = eval_lin(trainX, trainY, testX, testY) log_acc, log_stderr = eval_log(trainX, trainY, testX, testY) return lin_acc, log_acc xs = range(20,280,20) lin_log_acc = [lin_log_cmp(x) for x in xs] pl.figure() lin_lin, = pl.plot(xs, [y for y in lin_log_acc], label = 'linear') log_lin, = pl.plot(xs, [y for y in lin_log_acc], label = 'logistic') pl.legend(handles = [lin_lin, log_lin]) pl.xlabel('training size from ' + str(bc.target.size)) pl.ylabel('accuracy');
Incidentally, I used the iPython nbconvert to paste the notebook here.
Caveats: Without types, it's pretty easy to make mistakes in manipulating the raw data. Python and numpy scalar, array, and matrix arithmetic operators are gracious in accepting parameters, so you might get a surprise or two if you're not careful. That combined with operating with black-box analysis tools gives me some skepticism of any conclusions, but it's a start, and the investment was cheap.
Other Plotting Tools: Seaborn.pairplot generates some slick scatter plot and histograms that will help identify outliers, describe ranges, and demonstrated redundancy in the data dimensions. I tried removing some of obviously redundant data columns, resulting in no quality change in logistic classification and less than statistically significant reduction linear classification.
Linear or Logistic? It surprises me that logistic regression proved inferior classification to linear, but economists frequently use linear regression to model 0/1 variables. Paul von Hippel has a post comparing relative advantages of linear versus logistic regression. As a student, I had trouble both with application of logistic regression and conveying my travails to a thesis adviser. I wish I had read more commentary comparing the two 20 years ago.