New report identifies degrees of unease with how automated decision-making is used in New Zealand

News | 2021-04-16 | By: The Digital Council

New report identifies degrees of unease with how automated decision-making is used in New Zealand

Research conducted by the Digital Council for Aotearoa identifies that some New Zealanders are uncomfortable with how automated decision-making systems are used in our society, particularly by government. The report, ‘Towards trustworthy and trusted automated decision-making in Aotearoa’, also notes that New Zealanders have an appreciation for the way it can be used to improve people’s lives. 

Automated decision-making is when some aspects of decision-making processes (eg, visa applications, media consumption, recruitment, youth support, parole decisions, surgical waiting lists) are carried out or informed by computer algorithms.

Concerns about bias and discrimination

“People appreciate that automated decision-making is useful for processing data at speed and at scale, and as an ‘assistant’ to people,” says Digital Council Chairperson, Mitchell Pham. “But people worry that systems, programmers and decision-makers can introduce bias into these.” 

As one workshop participant said, “Algorithms are only as good as the people who designed them. Machine learning might help with that, but right now most of the algorithms are people-designed, so people’s individual biases and so on can come into play on that.”

More transparency and better communication needed

“The government and private companies hold significant datasets about the lives of individuals, whānau and communities,” says Mr Pham. “Communities need assurance and transparency about the data that feeds algorithms – what algorithms are doing, their purpose, who is making the decisions that affect people’s lives, and what will be done with the data they use and collect.”

There was a general consensus among participants about transparency and how people should be “... able to see the impact of the algorithm: so we know what decisions it’s making, what impacts those decisions have, who is designing it and making the decisions about design, how often is getting things wrong.”

Better involvement and representation needed

“People who use automated decision-making tools are making sizable moral and ethical decisions that positively or negatively impact people’s lives,” says Mr Pham. “Those people need to be a good reflection of the lived experience of the citizens they’re there to serve.” 

A workshop participant commented that, “My hunch is that the health professionals and systems creating these algorithms don’t have disabled people’s views in them. And what those people see as quality of life is quite often different to what we think of as quality of life.”

“My biggest concern,” said another participant, “is that the algorithms involved with identifying these kids are probably created by white middle-aged men, which therefore marginalises indigenous people and values. So my concern is the bias of the programmer will come through in the algorithm. I think the idea is quite cool and can see the intended benefits — but I’d feel more comfortable if indigenous people are involved in the processes of creating these algorithms.”

What it would take for people have more trust in algorithms and those using them

“People are not passive mathematical problems to be fixed by remote,” says Mr Pham. “The answer isn’t hidden in an algorithm. The government needs to be looking at relational responses to addressing public unease.” 

“Communities have the right and necessity to be involved in creating algorithms that are used on their people. This is a key part of honouring Te Tiriti o Waitangi, and was heard by the Digital Council from Māori, Pasifika, migrant, disability and youth communities alike.”

Participants talked about their desire for a focus on data and interventions that reflect the strengths and aspirations of whānau and communities. They also talked about wanting to see themselves reflected in teams that design, build, use and make decisions based on algorithms. They want opportunities to exert choice and control over how data is used. They want consent and buy-in that is actively sought rather than assumed or opt-out.

The report “Towards trust and trustworthiness in automated decision-making” can be found here

About the research

The Digital Council for Aotearoa commissioned scenario-based community workshops to gauge people’s level of trust in using automated decision-making in different New Zealand settings. They reached out to a wide range of people from diverse backgrounds including those from ethnic communities, women with migrant and refugee backgrounds, disabled people, Māori and Pacific youth, and young people with care experience.

For more information contact:

Please consider seeking the perspectives of those with lived experience of automated decision-making systems. Māori, Pasifika people, ethnic communities, disabled people and youth have particular rights and interests in this space. The people below are available for comment. 

Associate Professor Maui Hudson

Te Kotahi Research Institute

0272061183

maui@waikato.ac.nz 

Dr Jonathan Godfrey

National President, Blind Citizens NZ

029 538 9814

a.j.godfrey@massey.ac.nz

Mitchell Pham

Chairperson, Digital Council for Aotearoa New Zealand

Phone 021 623 334

Email: mitchell@digitalcouncil.nz

 

BACKGROUND INFORMATION

The Digital Council’s seven recommendations to Government

  1. Fund community groups to lead their own data and automated decision-making (ADM) projects (eg for community planning or monitoring local environments)
  2. Fund and support a public sector team to test and implement ADM best practice
  3. Establish a public sector ADM hub
  4. Work collaboratively to develop and implement private sector automated decision-making rules or best practice
  5. Build automated decision-making systems from te ao Māori perspectives
  6. Build a diverse digital workforce
  7. Increase the digital skills and knowledge of public sector leaders.

Who we talked to 

Workshops were attended by 187 people from the following groups: blind and vision impaired people, ethnic community leaders, ethnic community youth, women with migrant and refugee backgrounds, Māori and Pacific youth, Pacific youth leaders, Whānau Ora navigators, young people with care experience, general public.

The Council developed six scenarios based on real-world situations where algorithms are used: media (recommending music and movies); recruitment (talent selection algorithm); youth support (Not in Education, Employment or Training: NEET Algorithm); immigration (risk assessment algorithm for residency applications); health (surgery waiting list scoring system); criminal justice ((Risk of ReConviction x Risk of Imprisonment Algorithm — ROC*ROI).

About the Digital Council for Aotearoa New Zealand (www.digitalcouncil.govt.nz

The Digital Council for Aotearoa New Zealand was established in 2019. Its role is to provide independent advice to the Minister for Digital Economy and Communications about how to make a thriving, equitable digital future a reality for Aotearoa.  It is made up of seven diverse members, who bring a range of skills, expertise and connections to the table: Mitchell Pham, Roger Dennis, Marianne Elliott, Kendall Flutey, Rachel Kelly, Nikora Ngaropo, Colin Gavaghan.