Experiments with using a different classifier for scoring answers than logistic regression (on pipeline phase0). LogisticRegression baseline (B1): MRR 0.480 +-0.028 GradientBoosting baseline (B2): MRR 0.498 +-0.021 GradientBoosting baseline (B3): MRR 0.519 +-0.027 key: cross-validation mean/S.D. MRR LogisticRegression experiments: ./answer-train-logistic.py (fit_intercept=False) ***B1 // (mean) PERANS acc/prec/rcl/F2 = 0.888/0.122/0.477/0.301, @70 prec/rcl/F2 = 0.190/0.314/0.278, PERQ avail 1.000, any good = [0.598] MRR 0.480 // (S.D.) PERANS acc/prec/rcl/F2 = 0.006/0.007/0.020/0.013, @70 prec/rcl/F2 = 0.010/0.019/0.014, PERQ avail 0.000, any good = [0.039] MRR 0.028 // training took 155 seconds // (full) PERANS acc/prec/rcl/F2 = 0.896/0.147/0.557/0.358, @70 prec/rcl/F2 = 0.245/0.357/0.327, PERQ avail 1.000, any good = [0.644] MRR 0.531 ./answer-train-logistic.py "exclude=['.simpleScore']" // (mean) PERANS acc/prec/rcl/F2 = 0.886/0.119/0.479/0.299, @70 prec/rcl/F2 = 0.187/0.314/0.276, PERQ avail 1.000, any good = [0.586] MRR 0.464 // (S.D.) PERANS acc/prec/rcl/F2 = 0.005/0.007/0.020/0.013, @70 prec/rcl/F2 = 0.010/0.019/0.014, PERQ avail 0.000, any good = [0.031] MRR 0.022 // training took 90 seconds // (full) PERANS acc/prec/rcl/F2 = 0.893/0.144/0.556/0.353, @70 prec/rcl/F2 = 0.244/0.365/0.332, PERQ avail 1.000, any good = [0.637] MRR 0.507 GradientBoosting max_depth experiments: GradientBoostingClassifier-n_estimators=200,max_depth=2,min_samples_split=10,min_samples_leaf=5,max_features='sqrt'.pkl // (mean) PERANS acc/prec/rcl/F2 = 0.918/0.151/0.402/0.301, @70 prec/rcl/F2 = 0.274/0.212/0.222, PERQ avail 1.000, any good = [0.614] MRR 0.493 // (S.D.) PERANS acc/prec/rcl/F2 = 0.007/0.011/0.017/0.009, @70 prec/rcl/F2 = 0.020/0.009/0.007, PERQ avail 0.000, any good = [0.036] MRR 0.026 // training took 54 seconds // (full) PERANS acc/prec/rcl/F2 = 0.918/0.185/0.553/0.396, @70 prec/rcl/F2 = 0.347/0.308/0.315, PERQ avail 1.000, any good = [0.667] MRR 0.533 GradientBoostingClassifier-n_estimators=200,max_depth=2,min_samples_split=10,min_samples_leaf=5.pkl // (mean) PERANS acc/prec/rcl/F2 = 0.922/0.157/0.390/0.300, @70 prec/rcl/F2 = 0.283/0.205/0.217, PERQ avail 1.000, any good = [0.622] MRR 0.499 // (S.D.) PERANS acc/prec/rcl/F2 = 0.006/0.010/0.017/0.009, @70 prec/rcl/F2 = 0.018/0.013/0.011, PERQ avail 0.000, any good = [0.032] MRR 0.019 // training took 402 seconds // (full) PERANS acc/prec/rcl/F2 = 0.919/0.193/0.584/0.416, @70 prec/rcl/F2 = 0.362/0.326/0.333, PERQ avail 1.000, any good = [0.701] MRR 0.563 GradientBoostingClassifier-n_estimators=200,max_depth=3,min_samples_split=10,min_samples_leaf=5,max_features='sqrt'.pkl // (mean) PERANS acc/prec/rcl/F2 = 0.929/0.172/0.386/0.308, @70 prec/rcl/F2 = 0.300/0.203/0.217, PERQ avail 1.000, any good = [0.630] MRR 0.510 // (S.D.) PERANS acc/prec/rcl/F2 = 0.006/0.012/0.015/0.008, @70 prec/rcl/F2 = 0.016/0.010/0.009, PERQ avail 0.000, any good = [0.026] MRR 0.022 // training took 83 seconds // (full) PERANS acc/prec/rcl/F2 = 0.924/0.216/0.635/0.457, @70 prec/rcl/F2 = 0.394/0.392/0.392, PERQ avail 1.000, any good = [0.717] MRR 0.598 max_depth=3 TODO GradientBoostingClassifier-n_estimators=200,max_depth=4,min_samples_split=10,min_samples_leaf=5,max_features='sqrt'.pkl // (mean) PERANS acc/prec/rcl/F2 = 0.940/0.193/0.344/0.297, @70 prec/rcl/F2 = 0.331/0.186/0.203, PERQ avail 1.000, any good = [0.643] MRR 0.519 // (S.D.) PERANS acc/prec/rcl/F2 = 0.004/0.009/0.020/0.012, @70 prec/rcl/F2 = 0.020/0.011/0.011, PERQ avail 0.000, any good = [0.034] MRR 0.024 // training took 120 seconds // (full) PERANS acc/prec/rcl/F2 = 0.932/0.251/0.710/0.520, @70 prec/rcl/F2 = 0.454/0.462/0.461, PERQ avail 1.000, any good = [0.809] MRR 0.699 GradientBoostingClassifier-n_estimators=200,max_depth=4,min_samples_split=10,min_samples_leaf=5.pkl // (mean) PERANS acc/prec/rcl/F2 = 0.945/0.202/0.310/0.280, @70 prec/rcl/F2 = 0.328/0.164/0.182, PERQ avail 1.000, any good = [0.633] MRR 0.507 // (S.D.) PERANS acc/prec/rcl/F2 = 0.005/0.015/0.020/0.010, @70 prec/rcl/F2 = 0.021/0.016/0.016, PERQ avail 0.000, any good = [0.033] MRR 0.023 // training took 802 seconds // (full) PERANS acc/prec/rcl/F2 = 0.940/0.287/0.747/0.566, @70 prec/rcl/F2 = 0.489/0.513/0.508, PERQ avail 1.000, any good = [0.852] MRR 0.737 GradientBoostingClassifier-n_estimators=200,max_depth=5,min_samples_split=10,min_samples_leaf=5,max_features='sqrt'.pkl // (mean) PERANS acc/prec/rcl/F2 = 0.952/0.230/0.283/0.270, @70 prec/rcl/F2 = 0.373/0.142/0.162, PERQ avail 1.000, any good = [0.637] MRR 0.516 // (S.D.) PERANS acc/prec/rcl/F2 = 0.004/0.016/0.014/0.009, @70 prec/rcl/F2 = 0.022/0.012/0.013, PERQ avail 0.000, any good = [0.030] MRR 0.024 // training took 83 seconds // (full) PERANS acc/prec/rcl/F2 = 0.942/0.301/0.780/0.591, @70 prec/rcl/F2 = 0.523/0.555/0.548, PERQ avail 1.000, any good = [0.871] MRR 0.774 max_depth=5 TODO GradientBoosting n_estimators experiments: GradientBoostingClassifier-n_estimators=200,max_depth=2,min_samples_split=10,min_samples_leaf=5,max_features='sqrt',exclude=['\!.*'].pkl ***B2 // (mean) PERANS acc/prec/rcl/F2 = 0.918/0.152/0.404/0.302, @70 prec/rcl/F2 = 0.274/0.212/0.222, PERQ avail 1.000, any good = [0.622] MRR 0.498 // (S.D.) PERANS acc/prec/rcl/F2 = 0.006/0.011/0.018/0.010, @70 prec/rcl/F2 = 0.020/0.013/0.012, PERQ avail 0.000, any good = [0.031] MRR 0.021 // training took 51 seconds // (full) PERANS acc/prec/rcl/F2 = 0.918/0.184/0.547/0.392, @70 prec/rcl/F2 = 0.346/0.310/0.317, PERQ avail 1.000, any good = [0.676] MRR 0.538 GradientBoostingClassifier-n_estimators=100,max_depth=2,min_samples_split=10,min_samples_leaf=5,max_features='sqrt',exclude=['\!.*'].pkl // (mean) PERANS acc/prec/rcl/F2 = 0.921/0.151/0.380/0.291, @70 prec/rcl/F2 = 0.284/0.176/0.190, PERQ avail 1.000, any good = [0.595] MRR 0.477 // (S.D.) PERANS acc/prec/rcl/F2 = 0.003/0.006/0.023/0.011, @70 prec/rcl/F2 = 0.028/0.017/0.016, PERQ avail 0.000, any good = [0.030] MRR 0.027 // training took 32 seconds // (full) PERANS acc/prec/rcl/F2 = 0.920/0.172/0.475/0.352, @70 prec/rcl/F2 = 0.334/0.249/0.262, PERQ avail 1.000, any good = [0.640] MRR 0.512 GradientBoostingClassifier-n_estimators=200,max_depth=3,min_samples_split=10,min_samples_leaf=5,max_features='sqrt'.pkl // (mean) PERANS acc/prec/rcl/F2 = 0.929/0.172/0.386/0.308, @70 prec/rcl/F2 = 0.300/0.203/0.217, PERQ avail 1.000, any good = [0.630] MRR 0.510 // (S.D.) PERANS acc/prec/rcl/F2 = 0.006/0.012/0.015/0.008, @70 prec/rcl/F2 = 0.016/0.010/0.009, PERQ avail 0.000, any good = [0.026] MRR 0.022 // training took 83 seconds // (full) PERANS acc/prec/rcl/F2 = 0.924/0.216/0.635/0.457, @70 prec/rcl/F2 = 0.394/0.392/0.392, PERQ avail 1.000, any good = [0.717] MRR 0.598 GradientBoostingClassifier-n_estimators=400,max_depth=3,min_samples_split=10,min_samples_leaf=5,max_features='sqrt'.pkl // (mean) PERANS acc/prec/rcl/F2 = 0.937/0.179/0.346/0.291, @70 prec/rcl/F2 = 0.302/0.197/0.211, PERQ avail 1.000, any good = [0.624] MRR 0.508 // (S.D.) PERANS acc/prec/rcl/F2 = 0.004/0.015/0.020/0.016, @70 prec/rcl/F2 = 0.028/0.014/0.013, PERQ avail 0.000, any good = [0.018] MRR 0.015 // training took 108 seconds // (full) PERANS acc/prec/rcl/F2 = 0.927/0.238/0.711/0.509, @70 prec/rcl/F2 = 0.424/0.473/0.462, PERQ avail 1.000, any good = [0.789] MRR 0.670 hypothesis: there is a "saturation level" after which adding depth or estimators doesn't help GradientBoostingClassifier minimums experiment: GradientBoostingClassifier-n_estimators=200,max_depth=2,min_samples_split=10,min_samples_leaf=5,max_features='sqrt',exclude=['.simpleScore','\!.*'].pkl // (mean) PERANS acc/prec/rcl/F2 = 0.918/0.151/0.400/0.300, @70 prec/rcl/F2 = 0.273/0.202/0.213, PERQ avail 1.000, any good = [0.602] MRR 0.483 // (S.D.) PERANS acc/prec/rcl/F2 = 0.006/0.009/0.020/0.008, @70 prec/rcl/F2 = 0.019/0.014/0.013, PERQ avail 0.000, any good = [0.036] MRR 0.024 // training took 41 seconds // (full) PERANS acc/prec/rcl/F2 = 0.920/0.187/0.545/0.394, @70 prec/rcl/F2 = 0.354/0.311/0.319, PERQ avail 1.000, any good = [0.667] MRR 0.539 GradientBoostingClassifier-n_estimators=200,max_depth=3,min_samples_split=10,min_samples_leaf=5,max_features='sqrt',exclude=['\!.*'].pkl // (mean) PERANS acc/prec/rcl/F2 = 0.929/0.173/0.385/0.308, @70 prec/rcl/F2 = 0.296/0.201/0.214, PERQ avail 1.000, any good = [0.637] MRR 0.510 // (S.D.) PERANS acc/prec/rcl/F2 = 0.005/0.011/0.021/0.009, @70 prec/rcl/F2 = 0.025/0.017/0.016, PERQ avail 0.000, any good = [0.023] MRR 0.014 // training took 58 seconds // (full) PERANS acc/prec/rcl/F2 = 0.923/0.214/0.639/0.457, @70 prec/rcl/F2 = 0.389/0.388/0.388, PERQ avail 1.000, any good = [0.732] MRR 0.603 GradientBoostingClassifier-n_estimators=200,max_depth=2,min_samples_split=10,min_samples_leaf=5,exclude=['\!.*'].pkl // (mean) PERANS acc/prec/rcl/F2 = 0.922/0.158/0.390/0.301, @70 prec/rcl/F2 = 0.281/0.203/0.215, PERQ avail 1.000, any good = [0.625] MRR 0.493 // (S.D.) PERANS acc/prec/rcl/F2 = 0.007/0.011/0.017/0.008, @70 prec/rcl/F2 = 0.016/0.012/0.010, PERQ avail 0.000, any good = [0.029] MRR 0.019 // training took 315 seconds // (full) PERANS acc/prec/rcl/F2 = 0.919/0.193/0.584/0.416, @70 prec/rcl/F2 = 0.362/0.326/0.333, PERQ avail 1.000, any good = [0.701] MRR 0.563 GradientBoostingClassifier-n_estimators=200,max_depth=3.pkl ***B3 // (mean) PERANS acc/prec/rcl/F2 = 0.935/0.181/0.354/0.296, @70 prec/rcl/F2 = 0.318/0.188/0.204, PERQ avail 1.000, any good = [0.642] MRR 0.519 // (S.D.) PERANS acc/prec/rcl/F2 = 0.005/0.019/0.019/0.014, @70 prec/rcl/F2 = 0.031/0.017/0.016, PERQ avail 0.000, any good = [0.033] MRR 0.027 // training took 690 seconds // (full) PERANS acc/prec/rcl/F2 = 0.927/0.232/0.671/0.486, @70 prec/rcl/F2 = 0.413/0.423/0.421, PERQ avail 1.000, any good = [0.763] MRR 0.636 GradientBoostingClassifier-n_estimators=200,max_depth=3,max_features='sqrt'.pkl // (mean) PERANS acc/prec/rcl/F2 = 0.928/0.171/0.386/0.308, @70 prec/rcl/F2 = 0.297/0.204/0.217, PERQ avail 1.000, any good = [0.632] MRR 0.511 // (S.D.) PERANS acc/prec/rcl/F2 = 0.006/0.012/0.017/0.008, @70 prec/rcl/F2 = 0.024/0.009/0.007, PERQ avail 0.000, any good = [0.022] MRR 0.022 // training took 85 seconds // (full) PERANS acc/prec/rcl/F2 = 0.923/0.216/0.642/0.460, @70 prec/rcl/F2 = 0.394/0.388/0.389, PERQ avail 1.000, any good = [0.722] MRR 0.610 GradientBoostingClassifier-n_estimators=200,max_depth=2,min_samples_split=10,min_samples_leaf=5,max_features='sqrt',base_class_ratio=1.0%4,exclude=['\!.*'].pkl // (mean) PERANS acc/prec/rcl/F2 = 0.955/0.238/0.263/0.257, @70 prec/rcl/F2 = 0.388/0.118/0.137, PERQ avail 1.000, any good = [0.622] MRR 0.495 // (S.D.) PERANS acc/prec/rcl/F2 = 0.003/0.017/0.016/0.012, @70 prec/rcl/F2 = 0.036/0.011/0.012, PERQ avail 0.000, any good = [0.022] MRR 0.019 // training took 40 seconds // (full) PERANS acc/prec/rcl/F2 = 0.958/0.303/0.377/0.360, @70 prec/rcl/F2 = 0.501/0.188/0.215, PERQ avail 1.000, any good = [0.693] MRR 0.543