The German case study will zoom into a specific micro scene of AI in public administration. The research will focus on studying the practices of implementing a specific AI system in the context of a specific local government office. Studying practices implies a detailed observation of human activities and routines in the interaction with the AI system but also whether and how the AI system impacts on human actors, that is studying the relationships within situations of AI based local governance taking into account all involved factors. The objective of the research is studying in microscopic detail the change of practices through AI-based social assessment.
In Hamburg, a major city in Northern Germany the agency for labor, social affairs, family and integration uses a software called JUT-IT. In 2009 the decision was made to replace the old software systems called PROJUGA, PROSA, and DIWOGE by a new, general purpose system. The old systems were developed in the 1990s and became meanwhile outdated. It even became precarious to find qualified staff for the maintenance of the system. The decision was made to buy JUS-IT, a system based on Cúram—a modular off-the-shelf product that can be tailored for specific needs. The system had been bought worldwide by a number of US states, Australia, New Zealand, or the UK. It is used to administer cases and automate payments and is equipped with interfaces that connect it to police reports and health insurance funds. However, it has been criticized for functioning inaccurately in Canada (CBC News 2014) and caused major problems for the healthcare insurance in Minnesota (Nord 2014). When the decision was made to buy JUS-IT, the project was managed by Hewlett Packard together with the company Cúram as software supplier. However, during the implementation, in 2011 Cúram was bought by IBM and the whole process of implementation was severely delayed and is still not finalized. At the same time the costs increased dramatically and reports to the senate attested a very low quality of the project management. For this reason, it was highly contested in public and political debate. In consequence the senate of Hamburg decided in 2012 to reduce the scope of the software. Originally it was planned that the software should be used by the whole agency whereas in 2012 it was used only by the childcare department and has not been implemented in other departments of the agency.
In the department the system had a very bad reputation. The administration staff complained about the malfunctioning of the software and an increasing workload for documentation due to a user interface that was not specifically tied to the needs of childcare. Case managers suspected that the software was merely an instrument of control. The system had been put in use in the childcare department specifically because in Hamburg a considerable number of children had died that were supervised by the childcare department. The implementation of the system was connected with the hope that it would help to better integrate the departments of the agency and thereby reduce the risk for disadvantaged children. However, shortly after the system was put in use a young girl died because of being severely beaten by her parents. While the mother was accused for murder also the childcare department was accused for being jointly responsible: Shortly before the incident the child had been ordered by the childcare department from a foster family back to her physical family. This was after a change of responsibility in the administration. Whereas in the electronic system JUS-IT it was recorded that the child had been beaten and abused by her parents, in the physical dossier it was not. Thus, it could be argued that the staff did not follow rules properly. However, the mismanagement of the case also had systemic causes: first, the duplication of files made the case management error prone. Initially, the intention for the implementation of JUS-IT was to switch to just one electronic file. However, due to the user-unfriendly design of the user interface and practical problems in the handling of the system this could not be realized, and it was more likely that case managers relied on the physical documents than on the electronic system.
The case has even covered the attention of nation-wide media such as “Die Welt” or “Zeit Online”. Mostly the case has been reported by the local press such as the “Hamburger Abendblatt”. In 2018, an expert commission on child protection found that JUS-IT lengthens administrative processes, leaving less time for much needed family visits. For instance, the expert commission cited a case manager asking whether “the PC has studied social work or us?” “Empathy is more important than control”, said the head of the expert commission in an interview. As a result, the commission recommends major revisions or a complete shut-down of the software.
For examining the functionality of software and detecting sources of malfunctioning or bias, a scheme has been developed by Danks and London (2017) and extended by Silva and Kenney (2018). It provides a kind of algorithmic “value chain” of the different stages of the development and applications of algorithms, starting from the input, via algorithms and the output to the end users. Sources of biases can be identified at every stage along this value chain (see http://ai-fora.de/resources/literature-analysis/scientific-analysis/ for more details). The objective of the German case study is investigating the JUS-IT software along the lines of this value chain and presumably even extending the model. As the reports indicate a poor project management (e.g. a too late specification) it is likely that already processes prior to the actual software development need to be considered. For this purpose, the case study will utilize the common AI-FOR A research approach of building a team of technical and social science partners for the case studies.
The research endeavor will integrate social sciences and technical sciences, including the following elements:
- Desk research to get an overview of the historical development of the case since the decision to acquire a new software system in 2009 and the first implementation of JUS-IT in 2012. This research will be done by the social science partner and serve as an input for the technical partner.
- In order to investigate the implementation and potential biases along the algorithmic value chain, the technical partners will investigate the functionality and decision-making processes of the software by means such as black-box analysis (Diakopoulos 2014). Subsequently, methods of Explainable AI (XAI) will be used to evaluate the impact of specific input values (features) on the software’s decisions and predictions. These insights will be used as an input for social science research that investigates the impact of the software on administrative procedures and the guiding social values.
- Participatory multi-stakeholder workshops organized by the social science partner at Intermediaries will bring together representatives from the local government and parliament, the Agency for Labor, Social Affairs, Family and Integration in Hamburg, NGOs such as Algorithmwatch, and research and media that is dealing with the case to collect data on values, perspectives, opinions and attitudes of decision makers and agenda setters. These workshops will identify actors and possible modes of action in the cases. This provides the input for developing scenario simulations.
- Focus-group discussions will be undertaken with affected persons, i.e. the clients of the Agency for Labor, Social Affairs, Family and Integration in Hamburg specifically regarding childcare services. As the JUS-IT system is meanwhile implemented for several years, the focus groups will enable to reveal if and how the practice of family support has changed and how this affects the families that are clients of the agency.
- Interviews with domain experts will help to understand the context dependencies of actors within these areas. Interviews will be undertaken with local decision makers, external experts in family care such as members of the 2018 commission on child protection, as well as practitioners in the administration such as case managers and administration of the agency. This will support the research in understanding the social, economic, political and administrative pressures that drive the debate around the case. This research will be undertaken by the social science partner.
- Since the media discourse strongly influences public opinion and sets agendas for public discourse, it will be analyzed which topics, arguments, and sentiments shape the media discourse in order to capture the lines of arguments. A first overview of the media discourse seems to indicate that the topic is discussed under the issues of bad project management, overspending money, and complaints by the case managers working with the system, but not so much under topics specifically to AI. So far just one report of the data protection ombudsman has been found who complaint differences in deleting content in JUS-IT and the JUS-IT data warehouse. The different implementation refers also back to the quality of the project management. However, thereby issues of privacy of data are affected that can easily be overlooked by a case manager working with the system in daily routine. This is less covered in the media discourse which implies the question whether the public debate is misleading. This refers to the question whether or inasmuch there is a gap between technological development and public discourse and what core values resonate with the technology. This research will be undertaken by the social science partner.
- Multi-stakeholder workshops and interviews with case managers undertaken by the social science partner will support the technical science partner in identifying issues of relevance for the development of software in a co-creation lab that takes into account the values and interests of the participating stakeholders.