[NOTE: in the original email of this piece, I inexplicably wrote “Google’s Dario Amodei” instead of “Google’s Demis Hassabis.” As the reader who pointed this out to me suggested, I’m going to write this off as a subtle joke about AI hallucination. Apparently I went rogue.]
Thank you for the positive response to my recent pieces about child safety and AI doomerism. Given your demonstrated interest in these topics, I am going to spend the next few weeks talking about my views on software safety and ethics. While I will cite sources where I can, I find this topic is more opinion-driven than provable, so you can expect a higher ratio of ranting to homework than my usual stuff.
Software can be dangerous. Let’s stop treating it like a toy.
Fears around software safety are almost as old as the invention of software. Current worries like kids allegedly being corrupted by social media, AI agents going rogue, Flock and Ring cameras spying on you, Tesla autopilot causing accidents, and so forth are just the latest examples.
The AI companies are somewhat novel in the field for actively asking for government regulation, instead of fighting it. That’s better than nothing, but as I mentioned last week, I don’t love Demis Hassabis’s call to model AI regulation on the Financial Industry Regulatory Authority (FINRA). FINRA’s mission is “to protect investors and safeguard the integrity of our vibrant capital markets to ensure that everyone can invest with confidence.” It’s comical to say the solution to AI’s alleged existential threat to humanity is to emulate an authority meant to protect a vibrant marketplace for the potentially-murderous product in question.
As Mariana Rosenblat and I argued in our first Live Science op-ed, software is a lot like construction or medicine in that the end user can’t directly evaluate the quality of the product. One way to approach safety in an industry like this is by trying to control output by inspecting the end product. Mariana and I urged more rigorous inspecition of age-verification companies that have access to our most sensitive identity documents. But that’s just one type of software and one use case. We have a long way to go to define generally-applicable software safety standards. [I applaud Internet Safety Labs, a small group working hard to build this.]
Another path to improving safety is by controlling input, namely who is allowed to practice and what standards they need to follow. This is the part of FINRA which I like: mandatory testing, certification, and ethical standards for industry professionals.
Licensing is how we ensure that consumers can trust the skills, safety, and ethics of professionals who have expertise that we can’t evaluate directly. This starts with doctors, lawyers, engineers, and architects, but goes way deeper. Depending on how you count sub-specialties and federal mandates, my home state of New Jersey requires licensing for somewhere between 50 and 150 distinct professions.
Software engineering is a profession. Let’s license it like one.
Software developers are at least as pervasive and important to our lives and well-being as acupuncturists, landscape architects, and polysomnographers (sleep study technologists). And yet anyone can get a job building and deploying software. There’s no qualification exam, no continuing professional development requirements, and no code of conduct or ethical standards. Is it any surprise that we’re constantly struggling with what software is doing to us?
Consider just one example: the 2023 revelation that Tesla engineers built software that lied about their cars’ battery range.
Tesla years ago began exaggerating its vehicles’ potential driving distance – by rigging their range-estimating software. The company decided about a decade ago, for marketing purposes, to write algorithms for its range meter that would show drivers “rosy” projections for the distance it could travel on a full battery, according to a person familiar with an early design of the software for its in-dash readouts.
Then, when the battery fell below 50% of its maximum charge, the algorithm would show drivers more realistic projections for their remaining driving range, this person said. To prevent drivers from getting stranded as their predicted range started declining more quickly, Teslas were designed with a “safety buffer,” allowing about 15 miles (24 km) of additional range even after the dash readout showed an empty battery, the source said.
The directive to present the optimistic range estimates came from Tesla Chief Executive Elon Musk, this person said.
Given the absence of professional standards and oversight, it was logical for the software engineers at Tesla to build software that lied. If the lie was discovered, they wouldn’t face criminal charges because they weren’t the ones who made the decision to lie, and they could be pretty confident that it wouldn’t affect their career prospects. But if they refused, they’d probably have to look for a new job. [Or maybe become a whistleblower, and suffer the wrath of an army of terminally-online Tesla fanboys.]
If those software engineers had to sign an ethics pledge to enter the field, and were overseen by a licensing board that could convene a review and revoke their certification, writing dishonest software would be a huge personal risk. Like doctors or lawyers who commit malpractice, they could be disbarred and lose their livelihood. The licensing board would also be a powerful ally to help them stand up to dishonest employers like Musk.
I use the Tesla example because it’s a clear-cut case of dishonesty, but ethical standards would help with harder problems, too. The model rules of conduct for architecture state: “In practicing architecture, an architect’s primary duty is to protect the public’s health, safety, and welfare.” If Meta’s engineers made a similar pledge, would Mark Zuckerberg have been able to force his ads integrity engineers to tolerate pervasive fraud and scams, or his Instagram growth team to ignore signs of harm to children and teens?
Beyond ethical concerns, qualification and continuing education requirements could help mitigate egregious security and privacy failures like idscan.net leaking over 150 million drivers licenses, or any other of the constant stream of major data breaches tracked on Wikipedia. We deserve to know that anyone handling our data is only employing software engineers with a set of minimum required skills in security design and practices, and who update their knowledge regularly as the threat environment changes. Doctors in New Jersey need to complete 100 hours of continuing education every two years.
Licensing would be complicated. Let’s try it anyway.
I am, of course, glossing over a lot of specifics and debate here. There are variations and exceptions in what types of professionals need to be licensed depending on which field and where in the world you are. I saw one estimate that only 20% of practicing mechanical, electrical, etc engineers are actually licensed in the US. There are also a lot of different roles in any field that should be treated differently. Construction workers and hospital orderlies don’t need to be licensed like architects and doctors. In software, we would need to decide how to handle entry-level coders versus senior architects; how to handle adjacent roles like data science and product management; and so forth.
The existence of these edge cases and exceptions doesn’t invalidate my point. In these other fields, we have spent decades (or longer) in active discussion with regulators, professional societies, and the public over the appropriate qualifications and standards required to protect our safety without undue burden. We need to have that same conversation about software engineering now that it’s as important and influential as other professions.
[Note: there are efforts to bring professional standards to software engineering, but the absence of licensing requirements has meant it’s been strictly voluntary and thus almost totally marginalized.]
Engineers, in the fulfillment of their professional duties, shall:
Hold paramount the safety, health, and welfare of the public.
Perform services only in areas of their competence.
Issue public statements only in an objective and truthful manner.
Act for each employer or client as faithful agents or trustees.
Avoid deceptive acts.
Conduct themselves honorably, responsibly, ethically, and lawfully so as to enhance the honor, reputation, and usefulness of the profession.
Ideas? Feedback? Criticism? I want to hear it, because I am sure that I am going to get a lot of things wrong along the way. I will share what I learn with the community as we go. Reach out any time at [email protected].
