Notes on Vickers’ Decision Curve Analysis

Robin Chauhan, Pathway Intelligence
April 18, 2022
Vancouver BC Canada

Decision Curve Analysis [1] enables clinical decisions that account for model error, cost / benefit of interventions, and risk preferences. With DCA, net benefit of models can be compared directly. This talk builds on our previous talk “On the Evaluation of Binary Classifiers”.

This brief tutorial includes contributions taken from Dr. Andrew Vickers and Dr. Karandeep Singh, though is not reviewed/endorsed by them. Errata welcome, I apologise for any mistakes, they are entirely due to the deck author (Robin).

[1] Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006 Nov-Dec;26(6):565-74.

 

On the Evaluation of Binary Classifiers

Robin Chauhan, Pathway Intelligence
Jan 13, 2022
Vancouver BC Canada

A look at fundamental evaluation methods for binary classifiers.

 

Empirical Analysis of LSTM Performance

Early share of this work in progress. How does LSTM learning scale on smaller problems?

Robin Chauhan, Pathway Intelligence
Oct 16, 2019
Vancouver BC Canada

This work explores how performance of Recurring Neural Networks scale with various parameters, to build intuition and understanding of the Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) layers.

 

OpenAI GPT and Friends

A brief look at the new generation of models that are dominating Natural Language Processing (NLP). Focusing on GPT but also including ULMFiT, ElMO, BERT, and GPT2. Packed full of details and insight from the original papers, secondary sources, plus some original content.

Robin Chauhan , Pathway Intelligence
May 15, 2019
Simon Fraser University VentureLabs in Harbourfront Center, Vancouver BC Canada

A brief but detailed talk on the most recent Natural Language Processing (NLP) models based on unsupervised pre-training, primarily GPT, BERT, and GPT-2. A mix of material from papers, secondary sources, and my own experience with running and customizing GPT-2.

 

Traditional Machine Learning vs Deep Learning

I took a close look at a recent predictive modelling competition on Kaggle.com. The challenge was to predict gocery re-orders on Instacart.com, based on historical data and the particular user's history. I focused on two submissions, ranked #2 and #3 on the leader board. These two solutions yielded nearly identical scores, using completely different approaches.

Robin Chauhan , Pathway Intelligence
Nov 3, 2017
Simon Fraser University VentureLabs in Harbourfront Center, Vancouver BC Canada

I presented my analysis of a Kaggle competition held in 2017, for predicting Instacart e-commerce reorders. This competition was of particular interest for a few reasons. Two of the top submissions provided a perfect case study in comparing traditional ML, using hand-designed features, with deep learning, which involved ensembles of models tasked with predicting different aspects of the result (including somewhat exotic models).