Speaker: Dr  Sevvandi Kandanaarachchi -RMIT

Evaluating algorithm portfolios using Item Response Theory

How do you evaluate a portfolio of algorithms? Suppose we have the results for a set of algorithms on a given set of problems. We can find which algorithm performs best for each problem and find the algorithm that performs best on the greatest number of problems. But, there is a limitation to this approach. We are only looking at the overall best! Suppose a certain algorithm gives the best performance on hard problems, but not on easy problems. We would miss this algorithm by using the “overall best” approach.

Item Response Theory (IRT) is used to design, analyse and score test questions/questionnaires that measure hidden qualities such as stress proneness, political inclinations, or verbal/mathematical ability. It is a methodology used in educational psychometrics. Participants take tests and IRT is used to determine the ability of participants and discrimination and difficulty of test questions. We use a novel mapping of the traditional IRT framework modified to the algorithm evaluation domain. Using this new mapping, we elicit a richer suite of characteristics including stability and anomalousness that describe important aspects of algorithm performance. We find the strengths and weaknesses of algorithms in the problem space. Using the algorithm strengths and weaknesses we construct a smaller portfolio of algorithms that gives a good performance.

Bio: Sevvandi is an applied mathematician working on data science/analytics related topics such as anomaly detection, meta-learning and streaming data. She likes working on real-world problems, especially ones that are motivated by industry. She came to machine/statistical learning from a pure mathematics background.

WED 04 MAY 4PM – 5PM AEST

ZOOM MEETING ID: 873 1557 5255; PASSWORD: 778635