ABSTRACT
Deployed AI systems often do not work. They can be constructed haphazardly, deployed indiscriminately, and promoted deceptively. However, despite this reality, scholars, the press, and policymakers pay too little attention to functionality. This leads to technical and policy solutions focused on “ethical” or value-aligned deployments, often skipping over the prior question of whether a given system functions, or provides any benefits at all. To describe the harms of various types of functionality failures, we analyze a set of case studies to create a taxonomy of known AI functionality issues. We then point to policy and organizational responses that are often overlooked and become more readily available once functionality is drawn into focus. We argue that functionality is a meaningful AI policy challenge, operating as a necessary first step towards protecting affected communities from algorithmic harm.
- [n.d.]. Wave control - Google Nest Help. https://support.google.com/googlenest/answer/6294727?hl=en.Google Scholar
- 2013. Zhang v. Superior Ct., 304 P.3d 163 (2013).Google Scholar
- 2019. Stipulated Order for Civil Penalty, Monetary Judgment, and Injunctive Relief, No. 1:19-cv-2184, Docket 2-1 (D.D.C. July 24, 2019) (fining Facebook $5 billion for violating a prior consent decree).Google Scholar
- [4] 12 U.S.C. § 5511 [n.d.].Google Scholar
- ACLU. 2018. ACLU Comment on New Amazon Statement Responding to Face Recognition Technology Test. https://www.aclu.org/press-releases/aclu-comment-new-amazon-statement-responding-face-recognition-technology-test. Accessed: 2022-1-12.Google Scholar
- ACLU. 2021. ACLU Comment on NIST’s Proposal for Managing Bias in AI. https://www.aclu.org/letter/aclu-comment-nists-proposal-managing-bias-ai. Accessed: 2022-1-6.Google Scholar
- Raag Agrawal and Sudhakaran Prabakaran. 2020. Big data in digital healthcare: lessons learnt and recommendations for general practice. Heredity 124, 4 (April 2020), 525–534.Google ScholarCross Ref
- Nur Ahmed and Muntasir Wahed. 2020. The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research. CoRR abs/2010.15581(2020). arXiv:2010.15581https://arxiv.org/abs/2010.15581Google Scholar
- AI Act [n.d.]. Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts (COM(2021) 206 final).Google Scholar
- Nil-Jana Akpinar, Maria De-Arteaga, and Alexandra Chouldechova. 2021. The effect of differential victim crime reporting on predictive policing systems. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency(Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York, NY, USA, 838–849.Google ScholarDigital Library
- Zaheer Allam, Gourav Dey, and David S Jones. 2020. Artificial intelligence (AI) provided early detection of the coronavirus (COVID-19) in China and will influence future Urban health policy internationally. AI 1, 2(2020), 156–165.Google ScholarCross Ref
- Ann Anderson. 2015. Snake oil, hustlers and hambones: the American medicine show. McFarland.Google Scholar
- Robert D Atkinson. 2018. ” It Is Going to Kill Us!” and Other Myths About the Future of Artificial Intelligence. IUP Journal of Computer Sciences 12, 4 (2018), 7–56.Google Scholar
- Pranjal Awasthi and Jordana J George. 2020. A case for Data Democratization. (2020).Google Scholar
- Solon Barocas and Andrew D Selbst. 2016. Big data’s disparate impact. Calif. L. Rev. 104(2016), 671.Google Scholar
- Solon Barocas, Andrew D Selbst, and Manish Raghavan. 2020. The hidden assumptions behind counterfactual explanations and principal reasons. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 80–89.Google ScholarDigital Library
- R Barker Bausell. 2009. Snake oil science: The truth about complementary and alternative medicine. Oxford University Press.Google Scholar
- Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 610–623.Google ScholarDigital Library
- Emily M Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 5185–5198.Google ScholarCross Ref
- Stan Benjamens, Pranavsingh Dhunnoo, and Bertalan Meskó. 2020. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ digital medicine 3, 1 (2020), 1–8.Google Scholar
- Paul Berger. 2019. MTA’s Initial Foray Into Facial Recognition at High Speed Is a Bust. The Wall Street Journal(2019).Google Scholar
- Joseph Bernstein. 2021. Bad News. https://harpers.org/archive/2021/09/bad-news-selling-the-story-of-disinformation/. Harper’s Magazine (2021).Google Scholar
- Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. 2020. Explainable machine learning in deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 648–657.Google ScholarDigital Library
- Deborah Blum. 2018. The Poison Squad: One Chemist’s Single-minded Crusade for Food Safety at the Turn of the Twentieth Century. Penguin.Google Scholar
- National Transportation Safety Board. 2017. Collision Between a Car Operating With Automated Vehicle Control Systems and a Tractor-Semitrailer Truck. https://ntsb.gov/investigations/Pages/HWY18FH010.aspxGoogle Scholar
- National Transportation Safety Board. 2017. Driver Errors, Overreliance on Automation, Lack of Safeguards, Led to Fatal Tesla Crash. https://www.ntsb.gov/news/press-releases/pages/pr20170912.aspxGoogle Scholar
- National Transportation Safety Board. 2018. Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian. https://ntsb.gov/investigations/Pages/HWY18FH010.aspxGoogle Scholar
- Meredith Broussard. 2018. Artificial unintelligence: How computers misunderstand the world. mit Press.Google Scholar
- Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, 2018. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228(2018).Google Scholar
- Joanna Bryson. [n.d.]. AI & Global Governance: No One Should Trust AI - United Nations University Centre for Policy Research. https://cpr.unu.edu/publications/articles/ai-global-governance-no-one-should-trust-ai.html. Accessed: 2022-1-6.Google Scholar
- Joy Buolamwini, Sorelle A Friedler, and Christo Wilson. [n.d.]. Gender shades: Intersectional accuracy disparities in commercial gender classification. http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf. Accessed: 2022-1-12.Google Scholar
- Ryan Calo. 2015. Robotics and the Lessons of Cyberlaw. Calif. L. Rev. 103(2015), 513.Google Scholar
- Carolyn L. Carter. 2009. Consumer Protection in the States. Technical Report. National Consumer Law Center.Google Scholar
- Robert Charette. 2018. Michigan’s MiDAS Unemployment System: Algorithm Alchemy Created Lead, Not Gold-IEEE Spectrum. IEEE Spectrum 18, 3 (2018), 6.Google ScholarDigital Library
- Charlie Pownall. 2021. AI, Algorithmic and Automation Incident and Controversy Repository (AIAAIC). https://www.aiaaic.org/.Google Scholar
- Bryan H Choi. 2019. Crashworthy code. Wash. L. Rev. 94(2019), 39.Google Scholar
- Danielle Keats Citron. 2007. Technological due process. Wash. UL Rev. 85(2007), 1249.Google Scholar
- Danielle Keats Citron. 2016. The Privacy Policymaking of State Attorneys General. Notre Dame L. Rev. 92(2016), 747.Google Scholar
- Consumer Product Safety Commission. [n.d.]. About Us. https://www.cpsc.gov/About-CPSC.Google Scholar
- Federal Trade Commission. 2014. In re Snapchat, Inc., File No. 132-3078, Docket No. C-4501 (consent decree).Google Scholar
- Federal Trade Commission. 2021. FTC Votes to Update Rulemaking Procedures, Sets Stage for Stronger Deterrence of Corporate Misconduct. https://www.ftc.gov/news-events/press-releases/2021/07/ftc-votes-update-rulemaking-procedures-sets-stage-stronger.Google Scholar
- Kate Crawford. 2016. Artificial intelligence’s white guy problem. The New York Times 25, 06 (2016).Google Scholar
- Russell C. Wald Christopher Wan Daniel E. Ho, Jennifer King. 2021. Building a National AI Research Resource: A Blueprint for the National Research Cloud. https://hai.stanford.edu/sites/default/files/2022-01/HAI_NRCR_v17.pdf.Google Scholar
- Andrea De Mauro, Marco Greco, Michele Grimaldi, and Paavo Ritala. 2018. Human resources for Big Data professions: A systematic classification of job roles and required skill sets. Inf. Process. Manag. 54, 5 (Sept. 2018), 807–817.Google ScholarCross Ref
- Ángel Díaz and Laura Hecht. 2021. Double Standards in Social Media Content Moderation. https://www. brennancenter.org/sites/default/files/2021-08/Double_Standards_Content_Moderation.pdf. New York: Brennan Center for Justice(2021).Google Scholar
- Digwatch. 2021. The COVID-19 crisis: A digital policy overview. https://dig.watch/trends/covid-19-crisis-digital-policy-overview/.Google Scholar
- Roel Dobbe, Thomas Krendl Gilbert, and Yonatan Mintz. 2019. Hard Choices in Artificial Intelligence: Addressing Normative Uncertainty through Sociotechnical Commitments. (Nov. 2019). arxiv:1911.09005 [cs.AI]Google Scholar
- Will Douglas Heaven. 2020. AI is wrestling with a replication crisis. MIT Technology Review (Nov. 2020).Google Scholar
- Nature Editorial. 2021. Greece used AI to curb COVID: what other nations can learn. Nature 597, 7877 (2021), 447–448.Google Scholar
- Lilian Edwards and Michael Veale. 2017. Slave to the algorithm: Why a right to an explanation is probably not the remedy you are looking for. Duke L. & Tech. Rev. 16(2017), 18.Google Scholar
- Paul Egan. 2019. State of Michigan’s mistake led to man filing bankruptcy. https://www.freep.com/story/news/local/michigan/2019/12/22/government-artificial-intelligence-midas-computer-fraud-fiasco/4407901002/.Google Scholar
- Alex C Engler. 2021. Independent auditors are struggling to hold AI companies accountable. FastCompany.Google Scholar
- Nora Freeman Engstrom. 2013. 3-D printing and product liability: identifying the obstacles. U. Pa. L. Rev. Online 162 (2013), 35.Google Scholar
- Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2017. Runaway Feedback Loops in Predictive Policing. (June 2017). arxiv:1706.09847 [cs.CY]Google Scholar
- Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics 8 (2020), 34–48.Google ScholarCross Ref
- Virginia Eubanks. 2018. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin’s Press, New York.Google ScholarDigital Library
- Todd Feathers. [n.d.]. Las Vegas Cops Used ‘Unsuitable’ Facial Recognition Photos To Make Arrests. Vice ([n. d.]). https://www.vice.com/en/article/pkyxwv/las-vegas-cops-used-unsuitable-facial-recognition-photos-to-make-arrestsGoogle Scholar
- [58] Federal Trade Commission Act, 15 U.S.C. § 45 [n.d.].Google Scholar
- Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and Removing Disparate Impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Sydney, NSW, Australia) (KDD ’15). Association for Computing Machinery, New York, NY, USA, 259–268. https://doi.org/10.1145/2783258.2783311Google ScholarDigital Library
- Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 259–268.Google ScholarDigital Library
- A G Ferguson. 2016. Policing predictive policing. Wash. UL Rev. (2016).Google Scholar
- Chaz Firestone. 2020. Performance vs. competence in human–machine comparisons. Proceedings of the National Academy of Sciences 117, 43(2020), 26562–26571.Google ScholarCross Ref
- Benjamin Fish, Jeremy Kun, and Ádám D Lelkes. 2016. A confidence-based approach for balancing fairness and accuracy. In Proceedings of the 2016 SIAM International Conference on Data Mining. SIAM, 144–152.Google ScholarCross Ref
- U.S. Food and Drug Administration. 2021. Good Machine Learning Practice for Medical Device Development: Guiding Principles. https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles.Google Scholar
- Coalition for Critical Technology. [n.d.]. Abolish the #TechToPrisonPipeline. https://medium.com/@CoalitionForCriticalTechnology/abolish-the-techtoprisonpipeline-9b5b14366b16.Google Scholar
- Karoline Freeman, Julia Geppert, Chris Stinton, Daniel Todkill, Samantha Johnson, Aileen Clarke, and Sian Taylor-Phillips. 2021. Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy. bmj 374(2021).Google Scholar
- Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2021. The (Im)Possibility of Fairness: Different Value Systems Require Different Mechanisms for Fair Decision Making. Commun. ACM 64, 4 (mar 2021), 136–143. https://doi.org/10.1145/3433949Google ScholarDigital Library
- Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P Hamilton, and Derek Roth. 2019. A comparative study of fairness-enhancing interventions in machine learning. In Proceedings of the conference on fairness, accountability, and transparency. 329–338.Google ScholarDigital Library
- Sidney Fussell. [n.d.]. An Algorithm That ‘Predicts’ Criminality Based on a Face Sparks a Furor. Wired ([n. d.]). https://www.wired.com/story/algorithm-predicts-criminality-based-face-sparks-furor/Google Scholar
- Colin K Garvey. 2017. On the Democratization of AI. In Datapower Conference Proceedings. 5–3.Google Scholar
- Clare Garvie. 2019. Garbage in, Garbage out. Face recognition on flawed data. Georgetown Law Center on Privacy & Technology (2019).Google Scholar
- Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. 2018. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 3–18.Google ScholarCross Ref
- Mark A Geistfeld. 2017. A roadmap for autonomous vehicles: State tort liability, automobile insurance, and federal safety regulation. Calif. L. Rev. 105(2017), 1611.Google Scholar
- Milena A Gianfrancesco, Suzanne Tamang, Jinoos Yazdany, and Gabriela Schmajuk. 2018. Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data. JAMA Intern. Med. 178, 11 (Nov. 2018), 1544–1547.Google ScholarCross Ref
- Elizabeth Gibney. 2018. The scant science behind Cambridge Analytica’s controversial marketing techniques. Nature (2018).Google Scholar
- Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572(2014).Google Scholar
- Ben Green. 2019. The smart enough city: putting technology in its place to reclaim our urban future. MIT Press.Google Scholar
- Nitzan Guetta, Asaf Shabtai, Inderjeet Singh, Satoru Momiyama, and Yuval Elovici. 2021. Dodging Attack Using Carefully Crafted Natural Makeup. CoRR abs/2109.06467(2021). arXiv:2109.06467https://arxiv.org/abs/2109.06467Google Scholar
- Benjamin Haibe-Kains, George Alexandru Adam, Ahmed Hosny, Farnoosh Khodakarami, Thakkar Shraddha, Rebecca Kusko, Susanna-Assunta Sansone, Weida Tong, Russ D. Wolfinger, Christopher E. Mason, Wendell Jones, Joaquin Dopazo, Cesare Furlanello, Levi Waldron, Bo Wang, Chris McIntosh, Anna Goldenberg, Anshul Kundaje, Casey S. Greene, Tamara Broderick, Michael M. Hoffman, Jeffrey T. Leek, Keegan Korthauer, Wolfgang Huber, Alvis Brazma, Joelle Pineau, Robert Tibshirani, Trevor Hastie, John P. A. Ioannidis, John Quackenbush, Hugo J. W. L. Aerts, and Massive Analysis Quality Control (MAQC) Society Board of Directors. 2020. Transparency and reproducibility in artificial intelligence. Nature 586, 7829 (2020), E14–E16. https://doi.org/10.1038/s41586-020-2766-yGoogle ScholarCross Ref
- Isobel Asher Hamilton. 2020. Facebook’s nudity-spotting AI mistook a photo of some onions for ’sexually suggestive’ content. https://www.businessinsider.com/facebook-mistakes-onions-for-sexualised-content-2020-10.Google Scholar
- M Harris. 2019. NTSB investigation into deadly Uber self-driving car crash reveals lax attitude toward safety. IEEE Spectrum (2019).Google Scholar
- Woodrow Hartzog. 2018. Privacy’s blueprint. Harvard University Press.Google Scholar
- [83] Sudhir Hasbe and Ryan Lippert.[n.d.]. ([n. d.]).Google Scholar
- John C Havens and Ali Hessami. 2019. From Principles and Standards to Certification. Computer 52, 4 (2019), 69–72.Google ScholarCross Ref
- Will Douglas Heaven. 2021. Hundreds of AI tools have been built to catch covid. None of them helped.Google Scholar
- Nicolaus Henke, Jordan Levine, and Paul McInerney. 2018. You Don’t Have to Be a Data Scientist to Fill This Must-Have Analytics Role. Harvard Business Review (Feb. 2018).Google Scholar
- Thomas A Heppenheimer and Ta Heppenheimer. 1995. Turbulent skies: the history of commercial aviation. Wiley New York.Google Scholar
- Alex Hern. 2018. Cambridge Analytica: how did it turn clicks into votes. The Guardian 6(2018).Google Scholar
- Matthew Herper. 2017. MD Anderson Benches IBM Watson In Setback For Artificial Intelligence In Medicine. Forbes Magazine (Feb. 2017).Google Scholar
- Kashmir Hill. 2020. Wrongfully accused by an algorithm. The New York Times 24(2020).Google Scholar
- Sharona Hoffman and Andy Podgurski. 2013. Big bad data: law, public health, and biomedical databases. J. Law Med. Ethics 41 Suppl 1 (March 2013), 56–60.Google Scholar
- Sharona Hoffman and Andy Podgurski. 2013. The use and misuse of biomedical data: is bigger really better?Am. J. Law Med. 39, 4 (2013), 497–538.Google ScholarCross Ref
- Chris Jay Hoofnagle. 2016. Federal Trade Commission: Privacy Law and Policy. Cambridge University Press.Google Scholar
- F Patrick Hubbard. 2014. Sophisticated robots: balancing liability, regulation, and innovation. Fla. L. Rev. 66(2014), 1803.Google Scholar
- Tim Hwang. 2020. Subprime attention crisis: advertising and the time bomb at the heart of the Internet. FSG originals.Google Scholar
- IEEE. 2006. IEEE Standard Dictionary of Measures of the Software Aspects of Dependability. IEEE Std 982. 1-2005 (Revision of IEEE Std 982. 1-1988) (May 2006), 1–41.Google Scholar
- Bilal Mateen Michael Wooldridge Inken von Borzyskowski, Anjali Mazumder. 2021. Data science and AI in the age of COVID-19. https://www.turing.ac.uk/sites/default/files/2021-06/data-science-and-ai-in-the-age-of-covid_full-report_2.pdf.Google Scholar
- Abigail Z Jacobs. 2021. Measurement as governance in and for responsible AI. (Sept. 2021). arxiv:2109.05658 [cs.CY]Google Scholar
- Abigail Z Jacobs and Hanna Wallach. 2021. Measurement and Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event Canada). ACM, New York, NY, USA.Google ScholarDigital Library
- David Jeans. 2020. ScaleFactor Raised $100 Million In A Year Then Blamed Covid-19 For Its Demise. Employees Say It Had Much Bigger Problems. Forbes Magazine (July 2020).Google Scholar
- Anna Jobin, Marcello Ienca, and Effy Vayena. 2019. The global landscape of AI ethics guidelines. Nature Machine Intelligence 1, 9 (Sept. 2019), 389–399.Google ScholarCross Ref
- Frederike Kaltheuner, Abeba Birhane, Inioluwa Deborah Raji, Razvan Amironesei, Emily Denton, Alex Hanna, Hilary Nicole, Andrew Smart, Serena Dokuaa Oduro, James Vincent, Alexander Reben, Gemma Milne, Crofton Black, Adam Harvey, Andrew Strait, Tulsi Parida, Aparna Ashok, Fieke Jansen, Corinne Cath, and Aidan Peppin. 2021. Fake AI. Meatspace Press.Google Scholar
- Margot E Kaminski. 2019. The Right to Explanation, Explained. Berkeley Technology Law Journal 34 (2019), 189.Google Scholar
- Margot E Kaminski and Jennifer M Urban. 2021. The right to contest AI. Columbia Law Review 121, 7 (2021), 1957–2048.Google Scholar
- Sayash Kapoor and Arvind Narayanan. 2021. (Ir)Reproducible Machine Learning: A Case Study. https://reproducible.cs.princeton.edu/., 6 pages. https://reproducible.cs.princeton.edu/Google Scholar
- Sean Kippin and Paul Cairney. 2021. The COVID-19 exams fiasco across the UK: four nations and two windows of opportunity. British Politics (2021), 1–23.Google Scholar
- Lauren Kirchner and Matthew Goldstein. 2020. Access Denied: Faulty Automated Background Checks Freeze Out Renters. The Markup (2020).Google Scholar
- Lauren Kirchner and Matthew Goldstein. 2020. How Automated Background Checks Freeze Out Renters. The New York Times 28 (May 2020).Google Scholar
- Kumba Kpakima. 2021. Tiktok’s algorithm reportedly bans creators using terms ’Black’ and ’BLM’. https://i-d.vice.com/en_uk/article/m7epya/tiktoks-algorithm-reportedly-bans-creators-using-terms-black-and-blm. The Verge (2021).Google Scholar
- P. M. Krafft, Meg Young, Michael Katell, Karen Huang, and Ghislain Bugingo. 2020. Defining AI in Policy versus Practice. Association for Computing Machinery, New York, NY, USA, 72–78. https://doi.org/10.1145/3375627.3375835Google ScholarDigital Library
- Mark Krass, Peter Henderson, Michelle M Mello, David M Studdert, and Daniel E Ho. 2021. How US law will evaluate artificial intelligence for covid-19. bmj 372(2021).Google Scholar
- NHS AI Lab. 2021. National Medical Imaging Platform (NMIP). https://www.nhsx.nhs.uk/ai-lab/ai-lab-programmes/ai-in-imaging/national-medical-imaging-platform-nmip/.Google Scholar
- Tom Lamont. 2021. The student and the algorithm: how the exam results fiasco threatened one pupil’s future.Google Scholar
- Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. (Aug. 2018). arxiv:1808.07048 [cs.CL]Google Scholar
- Colin Lecher. [n.d.]. What Happens When an Algorithm Cuts Your Health Care. The Verge ([n. d.]). https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsyGoogle Scholar
- Colin Lecher. 2018. What happens when an algorithm cuts your health care. The Verge (2018).Google Scholar
- David Lehr and Paul Ohm. [n.d.]. Playing with the data: What legal scholars should learn about machine learning. https://lawreview.law.ucdavis.edu/issues/51/2/Symposium/51-2_Lehr_Ohm.pdf. Accessed: 2021-8-10.Google Scholar
- Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. 2021. Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Pre-Proceedings). https://openreview.net/forum?id=mPducS1MsEKGoogle Scholar
- Xiaoxuan Liu, Samantha Cruz Rivera, David Moher, Melanie J Calvert, and Alastair K Denniston. 2020. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. bmj 370(2020).Google Scholar
- Kristian Lum and William Isaac. 2016. To predict and serve?Signif. (Oxf.) 13, 5 (Oct. 2016), 14–19.Google Scholar
- William McGeveran. 2018. The Duty of Data Security. Minn. L. Rev. 103(2018), 1135.Google Scholar
- Bruce Mellado, Jianhong Wu, Jude Dzevela Kong, Nicola Luigi Bragazzi, Ali Asgary, Mary Kawonga, Nalomotse Choma, Kentaro Hayasi, Benjamin Lieberman, Thuso Mathaha, 2021. Leveraging Artificial Intelligence and Big Data to optimize COVID-19 clinical public health and vaccination roll-out strategies in Africa. Available at SSRN 3787748(2021).Google Scholar
- Brian Menegus. 2019. Defense of amazon’s face recognition tool undermined by its only known police client.Google Scholar
- Shira Mitchell, Eric Potash, Solon Barocas, Alexander D’Amour, and Kristian Lum. 2021. Algorithmic Fairness: Choices, Assumptions, and Definitions. Annu. Rev. Stat. Appl. 8, 1 (March 2021), 141–163.Google ScholarCross Ref
- Milad Moradi and Matthias Samwald. 2021. Evaluating the robustness of neural language models to input perturbations. arXiv preprint arXiv:2108.12237(2021).Google Scholar
- Michael Muller, Melanie Feinberg, Timothy George, Steven J Jackson, Bonnie E John, Mary Beth Kery, and Samir Passi. 2019. Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–8.Google ScholarDigital Library
- Deirdre K Mulligan and Kenneth A Bamberger. 2019. Procurement as policy: Administrative process for machine learning. Berkeley Tech. LJ 34(2019), 773.Google Scholar
- Ralph Nader. 1965. Unsafe at any speed. The designed-in dangers of the American automobile. (1965).Google Scholar
- Myura Nagendran, Yang Chen, Christopher A Lovejoy, Anthony C Gordon, Matthieu Komorowski, Hugh Harvey, Eric J Topol, John P A Ioannidis, Gary S Collins, and Mahiben Maruthappu. 2020. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ 368(2020). https://doi.org/10.1136/bmj.m689 arXiv:https://www.bmj.com/content/368/bmj.m689.full.pdfGoogle ScholarCross Ref
- Arvind Narayanan. 2019. How to recognize AI snake oil. Arthur Miller Lecture on Science and Ethics(2019).Google Scholar
- Pandu Nayak. 2019. Understanding searches better than ever before. The Keyword 295(2019).Google Scholar
- Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. 2020. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning. 151–159.Google ScholarDigital Library
- Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447–453.Google Scholar
- [134] OED Online 2021. https://www.oed.com/view/Entry/54950742.Google Scholar
- National Audit Office. 2020. Investigation into the response to cheating in English language tests - national audit office (NAO) press release. https://www.nao.org.uk/press-release/investigation-into-the-response-to-cheating-in-english-language-tests/Google Scholar
- Catherine Olsson. 2019. Unsolved research problems vs. real-world threat models. https://medium.com/@catherio/unsolved-research-problems-vs-real-world-threat-models-e270e256bc9e. https://medium.com/@catherio/unsolved-research-problems-vs-real-world-threat-models-e270e256bc9eGoogle Scholar
- Steven Overly. 2020. White House seeks Silicon Valley help battling coronavirus.Google Scholar
- David G Owen. 2001. Manufacturing Defects. SCL Rev. 53(2001), 851.Google Scholar
- Jesse O’Neill. 2021. Facebook cracks down on discussing ‘hoes’ in gardening group. https://nypost.com/2021/07/20/facebook-cracks-down-on-discussing-hoes-in-gardening-group/.Google Scholar
- Mark A Paige and Audrey Amrein-Beardsley. 2020. “Houston, We Have a Lawsuit”: A Cautionary Tale for the Implementation of Value-Added Models for High-Stakes Employment Decisions. Educational Researcher 49, 5 (2020), 350–359.Google ScholarCross Ref
- Samir Passi and Solon Barocas. 2019. Problem Formulation and Fairness. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). Association for Computing Machinery, New York, NY, USA, 39–48.Google ScholarDigital Library
- Samir Passi and Steven J Jackson. 2018. Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects. Proc. ACM Hum.-Comput. Interact. 2, CSCW (Nov. 2018), 1–28.Google ScholarDigital Library
- Samir Passi and Phoebe Sengers. 2020. Making data science systems work. Big Data & Society 7, 2 (July 2020), 2053951720939605.Google ScholarCross Ref
- Jay Peters. [n.d.]. Researchers fooled Chinese facial recognition terminals with just a mask. The Verge ([n. d.]). https://www.theverge.com/2019/12/13/21020575/china-facial-recognition-terminals-fooled-3d-mask-kneron-research-fallibilityGoogle Scholar
- Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research: a report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research 22 (2021).Google Scholar
- Carina Prunkl and Jess Whittlestone. 2020. Beyond near-and long-term: Towards a clearer account of research priorities in AI ethics and society. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 138–143.Google ScholarDigital Library
- Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. 2019. Combating adversarial misspellings with robust word recognition. arXiv preprint arXiv:1905.11268(2019).Google Scholar
- Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy. 2020. Mitigating bias in algorithmic hiring: Evaluating claims and practices. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 469–481.Google ScholarDigital Library
- Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021. AI and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366(2021).Google Scholar
- Inioluwa Deborah Raji and Joy Buolamwini. 2019. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 429–435.Google ScholarDigital Library
- Inioluwa Deborah Raji, Sasha Costanza-Chock, and Joy Buolamwini. 2022. Change From the Outside: Towards Credible Third-Party Audits of AI Systems. Missing Links in AI Policy(2022).Google Scholar
- Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 33–44.Google ScholarDigital Library
- Inioluwa Deborah Raji and Jingying Yang. 2019. About ml: Annotation and benchmarking on understanding and transparency of machine learning lifecycles. arXiv preprint arXiv:1912.06166(2019).Google Scholar
- Alexander Ratner, Dan Alistarh, Gustavo Alonso, David G Andersen, Peter Bailis, Sarah Bird, Nicholas Carlini, Bryan Catanzaro, Jennifer Chayes, Eric Chung, 2019. MLSys: The new frontier of machine learning systems. arXiv preprint arXiv:1904.03257(2019).Google Scholar
- [155] Restatement (Third) of Torts: Products Liability § 3 [n.d.].Google Scholar
- Rashida Richardson. 2021. Best Practices for Government Procurement of Data-Driven Technologies. Available at SSRN 3855637(2021).Google Scholar
- Rashida Richardson. 2021. Defining and Demystifying Automated Decision Systems. Maryland Law Review, Forthcoming(2021).Google Scholar
- Rashida Richardson, Jason Schultz, and Kate Crawford. 2019. Dirty Data, Bad Predictions: How Civil Rights Violations Impact Police Data, Predictive Policing Systems, and Justice. (Feb. 2019).Google Scholar
- Rashida Richardson, Jason M Schultz, and Vincent M Southerland. 2019. Litigating Algorithms: 2019 US Report. AI Now Institute, September(2019).Google Scholar
- Samantha Cruz Rivera, Xiaoxuan Liu, An-Wen Chan, Alastair K Denniston, and Melanie J Calvert. 2020. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. bmj 370(2020).Google Scholar
- Michael Roberts, Derek Driggs, Matthew Thorpe, Julian Gilbey, Michael Yeung, Stephan Ursprung, Angelica I Aviles-Rivero, Christian Etmann, Cathal McCague, Lucian Beer, 2021. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nature Machine Intelligence 3, 3 (2021), 199–217.Google ScholarCross Ref
- Ronald E Robertson, Jon Green, Damian Ruck, Katya Ognyanova, Christo Wilson, and David Lazer. 2021. Engagement Outweighs Exposure to Partisan and Unreliable News within Google Search. arXiv preprint arXiv:2201.00074(2021).Google Scholar
- Harold E Roland and Brian Moriarty. 1991. System safety engineering and management. John Wiley & Sons.Google Scholar
- Casey Ross, Ike Swetlitz, Rachel Cohrs, Ian Dillingham, STAT Staff, Nicholas Florko, and Maddie Bender. 2018. IBM’s Watson supercomputer recommended ’unsafe and incorrect’ cancer treatments, internal documents show. https://www.statnews.com/2018/07/25/ibm-watson-recommended-unsafe-incorrect-treatments/?utm_source=STAT+Newsletters&utm_campaign=beb06f048d-MR_COPY_08&utm_medium=email&utm_term=0_8cab1d7961-beb06f048d-150085821. Accessed: 2022-1-13.Google Scholar
- David S Rubenstein. 2021. Acquiring ethical AI. Florida Law Review 73(2021).Google Scholar
- David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. Advances in neural information processing systems 28 (2015), 2503–2511.Google Scholar
- Andrew D Selbst and Julia Powles. 2017. Meaningful information and the right to explanation. International Data Privacy Law 7, 4 (2017), 233–242.Google ScholarCross Ref
- J Shane. 2019. Janelle Shane: The danger of AI is weirder than you think TED Talk, 10: 20. Katsottu 8.8, 2020.Google Scholar
- Shreya Shankar and Aditya Parameswaran. 2021. Towards Observability for Machine Learning Pipelines. arXiv preprint arXiv:2108.13557(2021).Google Scholar
- Nate Silver. 2012. The signal and the noise: why so many predictions fail–but some don’t. Penguin.Google Scholar
- George Simon, Courtney D DiNardo, Koichi Takahashi, Tina Cascone, Cynthia Powers, Rick Stevens, Joshua Allen, Mara B Antonoff, Daniel Gomez, Pat Keane, Fernando Suarez Saiz, Quynh Nguyen, Emily Roarty, Sherry Pierce, Jianjun Zhang, Emily Hardeman Barnhill, Kate Lakhani, Kenna Shaw, Brett Smith, Stephen Swisher, Rob High, P Andrew Futreal, John Heymach, and Lynda Chin. 2019. Applying Artificial Intelligence to Address the Knowledge Gaps in Cancer Care. Oncologist 24, 6 (June 2019), 772–782.Google ScholarCross Ref
- Mona Sloane, Rumman Chowdhury, John C Havens, Tomo Lazovich, and Luis Rincon Alba. 2021. AI and Procurement-A Primer. (2021).Google Scholar
- Mona Sloane, Emanuel Moss, and Rumman Chowdhury. 2022. A Silicon Valley love triangle: Hiring algorithms, pseudo-science, and the quest for auditability. Patterns 3, 2 (2022), 100425.Google ScholarCross Ref
- David Smith and Kenneth Simpson. 2004. Functional safety. Routledge.Google Scholar
- Jacob Snow. 2018. Amazon’s Face Recognition Falsely Matched 28 Members of Congress With Mugshots. https://www.aclu.org/blog/privacy-technology/surveillance-technologies/amazons-face-recognition-falsely-matched-28. Accessed: 2022-1-12.Google Scholar
- Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. Release Strategies and the Social Impacts of Language Models. arxiv:1908.09203 [cs.CL]Google Scholar
- Jay Stanley. [n.d.]. Pitfalls of Artificial Intelligence Decisionmaking Highlighted In Idaho ACLU Case. ACLU Blogs ([n. d.]). https://www.aclu.org/blog/privacy-technology/pitfalls-artificial-intelligence-decisionmaking-highlighted-idaho-aclu-caseGoogle Scholar
- Brian Stanton and Theodore Jensen. 2021. Trust and Artificial Intelligence. (March 2021).Google Scholar
- Luke Stark and Jevan Hutson. 2022. Physiognomic Artificial Intelligence. forthcoming in Fordham Intellectual Property, Media & Entertainment Law Journal XXXII (2022). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3927300Google Scholar
- Eliza Strickland. [n.d.]. IBM Watson Heal Thyself: How IBM Watson Overpromised And Underdeliverd On AI Health Care. https://spectrum.ieee.org/how-ibm-watson-overpromised-and-underdelivered-on-ai-health-care. Accessed: 2022-1-13.Google Scholar
- Andreas Sudmann. 2020. The Democratization of Artificial Intelligence. In The Democratization of Artificial Intelligence. transcript-Verlag, 9–32.Google Scholar
- Maia Szalavitz. 2021. The Pain Was Unbearable. So Why Did Doctors Turn Her Away?https://www.wired.com/story/opioid-drug-addiction-algorithm-chronic-pain/.Google Scholar
- Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. 2020. Measuring Robustness to Natural Distribution Shifts in Image Classification. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 18583–18599. https://proceedings.neurips.cc/paper/2020/file/d8330f857a17c53d217014ee776bfd50-Paper.pdfGoogle Scholar
- Chris Tennant and Jack Stilgoe. 2021. The attachments of ‘autonomous’ vehicles. Social Studies of Science 51, 6 (2021), 846–870.Google ScholarCross Ref
- Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. (Aug. 2018). arxiv:1808.10432 [cs.CL]Google Scholar
- Microsoft Translator. 2018. Neural Machine Translation reaches historic milestone: human parity for Chinese to English translations. https://www.microsoft.com/en-us/translator/blog/2018/03/14/human-parity-for-chinese-to-english-translations/. Accessed: 2022-1-12.Google Scholar
- [187] Uniform Commercial Code § 2-314 [n.d.].Google Scholar
- [188] Uniform Commercial Code § 2-315 [n.d.].Google Scholar
- Sam Varghese. 2021. How a Google search could end up endangering a life. https://itwire.com/home-it/how-a-google-search-could-end-up-endangering-a-life.html.Google Scholar
- Michael Veale and Frederik Zuiderveen Borgesius. 2021. Demystifying the Draft EU Artificial Intelligence Act—Analysing the good, the bad, and the unclear elements of the proposed approach. Computer Law Review International 22, 4 (2021), 97–112.Google ScholarCross Ref
- Lee Vinsel. [n.d.]. You’re Doing It Wrong: Notes on Criticism and Technology Hype. ([n. d.]). https://sts-news.medium.com/youre-doing-it-wrong-notes-on-criticism-and-technology-hype-18b08b4307e5Google Scholar
- Lee Vinsel. 2019. Moving Violations: Automobiles, Experts, and Regulations in the United States. JHU Press.Google ScholarCross Ref
- Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. 2017. Why a right to explanation of automated decision-making does not exist in the general data protection regulation. International Data Privacy Law 7, 2 (2017), 76–99.Google ScholarCross Ref
- Zhiyuan Wan, Xin Xia, David Lo, and Gail C. Murphy. 2021. How does Machine Learning Change Software Development Practices?IEEE Transactions on Software Engineering 47, 9 (2021), 1857–1871. https://doi.org/10.1109/TSE.2019.2937083Google ScholarCross Ref
- Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, 2021. Ethical and social risks of harm from Language Models. arXiv preprint arXiv:2112.04359(2021).Google Scholar
- Emily Weinstein. 2020. China’s Use of AI in its COVID-19 Response.Google Scholar
- Eric Weiss. 2019. ‘Inadequate Safety Culture’ Contributed to Uber Automated Test Vehicle Crash - NTSB Calls for Federal Review Process for Automated Vehicle Testing on Public Roads. https://www.ntsb.gov/news/press-releases/Pages/NR20191119c.aspxGoogle Scholar
- Michael Wick, swetasudha panda, and Jean-Baptiste Tristan. 2019. Unlocking Fairness: a Trade-off Revisited. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2019/file/373e4c5d8edfa8b74fd4b6791d0cf6dc-Paper.pdfGoogle Scholar
- Christo Wilson, Avijit Ghosh, Shan Jiang, Alan Mislove, Lewis Baker, Janelle Szary, Kelly Trindel, and Frida Polli. 2021. Building and auditing fair algorithms: A case study in candidate screening. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 666–677.Google ScholarDigital Library
- Nick Wingfield. 2014. Nest Labs Stops Selling Its Smoke Detector. The New York Times (Apr 2014). https://www.nytimes.com/2014/04/04/technology/nest-labs-citing-flaw-halts-smoke-detector-sales.htmlGoogle Scholar
- [201] Winter v. G.P. Putnam’s Sons, 938 F.2d 1033 (9th Cir. 1991) 1991.Google Scholar
- Natalia Wojcik. [n.d.]. IBM’s Watson ‘is a joke,’ says Social Capital CEO Palihapitiya. https://www.cnbc.com/2017/05/08/ibms-watson-is-a-joke-says-social-capital-ceo-palihapitiya.html. Accessed: 2022-1-13.Google Scholar
- Andrew Wong, Erkin Otles, John P. Donnelly, Andrew Krumm, Jeffrey McCullough, Olivia DeTroyer-Cooley, Justin Pestrue, Marie Phillips, Judy Konye, Carleen Penoza, Muhammad Ghous, and Karandeep Singh. 2021. External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Internal Medicine 181, 8 (08 2021), 1065–1070. https://doi.org/10.1001/jamainternmed.2021.2626 arXiv:https://jamanetwork.com/journals/jamainternalmedicine/articlepdf/ 2781307/jamainternal_wong_2021_oi_210027_1627674961.11707.pdfGoogle ScholarCross Ref
- Matt Wood. [n.d.]. Thoughts On Machine Learning Accuracy. https://aws.amazon.com/blogs/aws/thoughts-on-machine-learning-accuracy/.Google Scholar
- Eric Wu, Kevin Wu, Roxana Daneshjou, David Ouyang, Daniel E Ho, and James Zou. 2021. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nature Medicine 27, 4 (2021), 582–584.Google ScholarCross Ref
- Laure Wynants, Ben Van Calster, Gary S Collins, Richard D Riley, Georg Heinze, Ewoud Schuit, Marc MJ Bonten, Darren L Dahly, Johanna A Damen, Thomas PA Debray, 2020. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. bmj 369(2020).Google Scholar
- Karen Yeung. 2020. Recommendation of the council on artificial intelligence (oecd). International Legal Materials 59, 1 (2020), 27–34.Google ScholarCross Ref
Index Terms
- The Fallacy of AI Functionality
Recommendations
Measuring the Superfluous Functionality in Software Components
CBSE '15: Proceedings of the 18th International ACM SIGSOFT Symposium on Component-Based Software EngineeringOne of the drawbacks of a pragmatic, white box approach to the reuse of software is that reusable components often have more built-in functionality than is needed for a particular (re)usage scenario. This functionality either has to be invasively ...
The fallacy of composition
Although the fallacy of composition is little studied by theorists and trivially illustrated in textbooks, some view it as ubiquitous and highly significant. Furthermore, although definitions regard the concept as unproblematic, it contains three ...
Extracting Code Fragment that Implements Functionality
APSEC '99: Proceedings of the Sixth Asia Pacific Software Engineering ConferenceA program interacts with its environment through accepting and delivering information from and to its environment respectively. As such, the functionality in a program is achieved through its input/output statements. Based on this hypothesis, this paper ...
Comments