On the Existential Threat of AI

Q: how on earth should I begin to process the news cycle of the past week or so?

A: I want to start by saying that the following is my informed opinion as a leader who specializes in AI integration across Capacity and our client work. It does not necessarily reflect the opinion of every member of Capacity’s staff, or our constituency more broadly, but I want to share where my head is—because I think this conversation is only going to get louder.

What Happened

For anyone who hasn’t been following all of this, a quick overview: Jacob Coxon, an AI researcher who worked on developing models at both OpenAI and Anthropic, publicly quit Anthropic last week specifically because of his concerns about where AI is headed. 

The important distinction, and part of why I think this is worth talking about, is that they’re not saying the Claude or ChatGPT we’re using today is harmful to humanity. The concern is about what comes next, particularly AI becoming capable enough to help build better AI. If AI starts doing more of the research and development required to create its successors, development could accelerate significantly and give humans less time to understand, test and put guardrails around each generation.

Then, Anthropic CEO Dario Amodei publicly called for AI companies to slow the pace of frontier AI development so safety can catch up. Sam Altman agreed that the industry needs to pace frontier development, Elon Musk responded that “Dario is right,” and other AI leaders have expressed support for greater coordination and safety measures.

Take that in: the people competing to build some of the most capable AI systems are now publicly saying we may need to slow down how quickly we build them.

The Two Conversations

In the conversations I’ve been having and hearing since this all happened, I keep coming back to the fact that there are really two conversations happening at the same time.

There’s where we are now. ChatGPT, Claude, Gemini, AI search, agents, workflows, etc. These tools have risks and we need to use them thoughtfully, but I also think they’re incredibly useful and there’s a lot we should be learning about how to use them well.

And then there’s where this is going. Much more capable systems, greater autonomy, systems learning and improving on their own, and eventually capabilities that are very different from what we’re interacting with today. I think there are very real risks there, and I’m concerned that companies and countries are all racing each other without enough conversation about where the lines should be or where we may actually want to slow down. Personally, I want to see global regulation around this. And I would be completely fine if there were no major capability advancements from where we are today for a long time.

The hard part is that I think these two conversations are going to increasingly get mixed together. “There are potentially enormous risks from where advanced AI is headed” very quickly becomes “AI is bad.” And I don’t think those are the same thing.

What Responsible Use Means To Me

I think we have to be able to hold both ideas at once. We should take the risks seriously, and we also have an obligation to use the technology that exists today responsibly. It means having guardrails around how we use these tools. But AI is already changing marketing, search, how people work and how audiences find information, and ignoring that reality is not responsible, either.

I’m wrestling with all of this. I spend a lot of my time helping people use this technology, while also believing the warnings about where it could eventually go are credible. I don’t have a neat way to reconcile those things, and I’m not going to pretend that I do.

I will say: social media is another place where I think we already hold some of this tension. We can recognize the very real problems it has created and still acknowledge that arts organizations need to understand and use it to reach their audiences. I don’t mean that as an excuse or a reason to dismiss the risks of AI. The scale of the potential risk is very different. I mean it as an example of how recognizing that a technology can cause harm doesn’t automatically mean the responsible choice is to disengage from it entirely.

Talk soon,
Jen

Have a question for a future edition? Submit it here!