网站关键词设置网站建设素材收集通知
两个最主要的处理缺失值的方法是：
❏ 删除缺少值的行；
❏ 填充缺失值；
我们首先将serum_insulin的中的字段值0替换为None，可以看到缺失值的数量为374个；
print(pima['serum_insulin'].isnull().sum())
pima['serum_insulin'] = pima['serum_insulin'].map(lambda x:x if x != 0 else None)
print(pima['serum_insulin'].isnull().sum())
# 0
# 374
替换所有的缺失字段，可以看到不同字段缺失值的情况是不一样的；
columns = ['serum_insulin', 'bmi', 'plasma_glucose_concentration','diastolic_blood_pressure', 'triceps_thickness']
for c in columns:pima[c].replace([0], [None], inplace=True)print(pima.isnull().sum())
# times_pregnant 0
# plasma_glucose_concentration 5
# diastolic_blood_pressure 35
# triceps_thickness 227
# serum_insulin 374
# bmi 11
# pedigree_function 0
# age 0
# onset_diabetes 0
# dtype: int64
可以看到此时describe不会针对有缺失值的列进行计算
print(pima.describe())
# times_pregnant pedigree_function age onset_diabetes
# count 768.000000 768.000000 768.000000 768.000000
# mean 3.845052 0.471876 33.240885 0.348958
# std 3.369578 0.331329 11.760232 0.476951
# min 0.000000 0.078000 21.000000 0.000000
# 25% 1.000000 0.243750 24.000000 0.000000
# 50% 3.000000 0.372500 29.000000 0.000000
# 75% 6.000000 0.626250 41.000000 1.000000
# max 17.000000 2.420000 81.000000 1.000000
我们可以自己手动计算均值
# print(pima['plasma_glucose_concentration'].mean(), pima['plasma_glucose_concentration'].std())# 121.6867627785059 30.53564107280403
处理缺失数据最简单的方式就是丢弃数据行，我们使用dropna方法进行处理，可以看到将近丢弃一半的数据；从机器学习的角度考虑，尽管数据都有值、很干净，但是我们没有利用尽可能多的数据，忽略了一半以上的观察值。
pima_dropped = pima.dropna()
rows = pima.shape[0]
rows_dropped = pima_dropped.shape[0]
num_rows_lost = round(100*(rows-rows_dropped)/rows)
print('lost {}% rows'.format(num_rows_lost))
# lost 49% rows
通过以下我们可以看到糖尿病的患病概率影响并不是很大；
print(pima['onset_diabetes'].value_counts(normalize=True))
print(pima_dropped['onset_diabetes'].value_counts(normalize=True))
# onset_diabetes
# 0 0.651042
# 1 0.348958
# Name: proportion, dtype: float64
# onset_diabetes
# 0 0.668367
# 1 0.331633
# Name: proportion, dtype: float64
通过以下可以看到各个字段的均值处理前后的大小
pima_mean = pima.mean()
pima_dropped_mean = pima_dropped.mean()
print(pima_mean)
print(pima_dropped_mean)
# times_pregnant 3.845052
# plasma_glucose_concentration 121.686763
# diastolic_blood_pressure 72.405184
# triceps_thickness 29.15342
# serum_insulin 155.548223
# bmi 32.457464
# pedigree_function 0.471876
# age 33.240885
# onset_diabetes 0.348958
# dtype: object# times_pregnant 3.30102
# plasma_glucose_concentration 122.627551
# diastolic_blood_pressure 70.663265
# triceps_thickness 29.145408
# serum_insulin 156.056122
# bmi 33.086224
# pedigree_function 0.523046
# age 30.864796
# onset_diabetes 0.331633
# dtype: object
可以看到进行数据处理之后，每个字段的变化率
mean_percent = (pima_dropped_mean - pima_mean) / pima_mean
print(mean_percent)
# times_pregnant -0.141489
# plasma_glucose_concentration 0.007731
# diastolic_blood_pressure -0.024058
# triceps_thickness -0.000275
# serum_insulin 0.003265
# bmi 0.019372
# pedigree_function 0.108439
# age -0.071481
# onset_diabetes -0.04965
# dtype: object
通过饼图查看各个字段的百分比变化；
ax = mean_percent.plot(kind='bar', title='% change in average column values')
ax.set_ylabel('% change')
plt.show()
可以看到，times_pregnant（怀孕次数）的均值在删除缺失值后下降了14%，变化很大！pedigree_function（糖尿病血系功能）也上升了11%，也是个飞跃。可以看到，删除行（观察值）会严重影响数据的形状，所以应该保留尽可能多的数据。
使用处理过的数据训练scikit-learn的K最近邻（KNN，k-nearest neighbor）分类模型，可以看到最好的邻居数是7个，此时KNN模型的准确率是74.5%；
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCVX_dropped = pima_dropped.drop('onset_diabetes', axis= 1)
print('learning from {} rows'.format(X_dropped.shape[0]))
y_dropped = pima_dropped['onset_diabetes']knn_para = {'n_neighbors':[1,2,3,4,5,6,7]}
knn = KNeighborsClassifier()
grid = GridSearchCV(knn, knn_para)
grid.fit(X_dropped, y_dropped)
print(grid.best_score_, grid.best_params_)# learning from 392 rows
# 0.7348263550795197 {'n_neighbors': 7}