Home Events OPTIMA Seminar Series 04 May 2022


May 04 2022


4:00 pm - 5:00 pm



OPTIMA Seminar Series 04 May 2022

Speaker: Dr  Sevvandi Kandanaarachchi -RMIT

Evaluating algorithm portfolios using Item Response Theory

How do you evaluate a portfolio of algorithms? Suppose we have the results for a set of algorithms on a given set of problems. We can find which algorithm performs best for each problem and find the algorithm that performs best on the greatest number of problems. But, there is a limitation to this approach. We are only looking at the overall best! Suppose a certain algorithm gives the best performance on hard problems, but not on easy problems. We would miss this algorithm by using the “overall best” approach.

Item Response Theory (IRT) is used to design, analyse and score test questions/questionnaires that measure hidden qualities such as stress proneness, political inclinations, or verbal/mathematical ability. It is a methodology used in educational psychometrics. Participants take tests and IRT is used to determine the ability of participants and discrimination and difficulty of test questions. We use a novel mapping of the traditional IRT framework modified to the algorithm evaluation domain. Using this new mapping, we elicit a richer suite of characteristics including stability and anomalousness that describe important aspects of algorithm performance. We find the strengths and weaknesses of algorithms in the problem space. Using the algorithm strengths and weaknesses we construct a smaller portfolio of algorithms that gives a good performance.

Bio: Sevvandi is an applied mathematician working on data science/analytics related topics such as anomaly detection, meta-learning and streaming data. She likes working on real-world problems, especially ones that are motivated by industry. She came to machine/statistical learning from a pure mathematics background.


ZOOM MEETING ID: 873 1557 5255; PASSWORD: 778635

More Info


The event is finished.


May 04 2022

Advancing an industry-ready optimisation toolkit, while training a new generation of industry practitioners and over 120 young researchers, who will vanguard a highly skilled workforce of change agents for industrial transformation.

Monash University
Clayton, Victoria, 3080

University of Melbourne
Parkville, Victoria, 3010

© 2021 ARC Industrial Transformation Training Centre in Optimisation Technologies, Integrated Methodologies and Applications (OPTIMA)