Ensemble of Narrow DNN Chains

Our paper available at: “Ensemble of Narrow DNN Chains” (my Machine Learning course essay at Oxford).

Our code is publicly available at https://github.com/vtu81/ENDC.

We propose the Ensemble of Narrow DNN Chains (ENDC) framework:

1. first train such narrow DNN chains that perform well on one-vs-all binary classification tasks,
2. then aggregate them together by voting to predict for the multiclassification task.

Our ensemble framework could:

• utilize the abstract interpretability of DNNs,
• outperform traditional ML significantly on CIFAR-10,
• while being 2-4 orders of magnitude smaller than normal DNN and 6+ times smaller than traditional ML models,
• furthermore compatible with full parallelism in both the training and deployment stage.

Our empirical study shows that a narrow DNN chain could learn binary classifications well. Moreover, our experiments on three MNIST, Fashion-MNIST, CIFAR-10 confirm the potential power of ENDC. Compared with traditional ML models, ENDC, with the smallest parameter number, could achieve similar accuracy on MNIST and Fashion-MNIST, and significantly better accuracy on CIFAR-10.

Results

Overall Accuracy

• Each binary classifier’s parameter number is even smaller than the input entry (130 < 28x28 for MNIST and Fashion-MNIST, 493 < 3x32x32 for CIFAR-10)!

Comparison

We compare ENDC with traditional ML models:

• Logistic Regression (LR)
• Support Vector Classifier (SVC)

and normal DNNs. Their results are referenced from internet, see our paper for sources and details.

MNIST

Fashion-MNIST

CIFAR-10

Per-class Accuracy