“Just because you do not take an interest in politics doesn’t mean politics won’t take an interest in you. ― Pericles
On May 21, the California Senate passed Bill 1047. This bill, which sets out to regulate AI at the model level, is now slated for a California Assembly vote in August. If passed, one signature from Governor Gavin Newsom could cement it into California law.
So here is what you need to know: Senate Bill 1047 is designed to apply to models trained above certain compute and cost thresholds. The bill also holds developers legally liable for the downstream use or modification of their models. Before training begins, developers would need to certify their models will not enable or provide “hazardous capabilities,” and implement a litany of safeguards to protect against such usage.
Overseeing the enforcement of the new laws would be a “frontier model division,” a newly formed oversight and regulatory agency funded by the fees and fines on developers. This agency would establish safety standards and advise on AI laws, and misrepresenting a model’s capabilities to this agency could land a developer in jail for perjury.
In this Q&A with a16z General Partner Anjney Midha (an edited version of a recent conversation on the a16z Podcast), he breaks down the compute threshold being targeted by SB 1047, historical precedents that we can look to for comparison, the implications of this bill on the startup ecosystem, and — most importantly — what you can do about it.
a16z: Why is SB 1047 such a big deal to AI startups right now?
Anjney Midha: It’s hard to understate just how blindsided startups, founders, and the investor community feel about this bill. When it comes to policy-making, especially in technology at the frontier, our legislators should be sitting down and soliciting the opinions of their constituents — which in this case, includes startup founders.
If this passes in California, it will set a precedent for other states and have rippling consequences inside and outside of the USA — essentially a huge butterfly effect in regards to the state of innovation.
a16z: SB 1047 will establish mandatory reporting requirements for models that are trained on 10 ^ 26 integer floating point operations per second, or FLOPs, of compute power. What exactly are FLOPs and why are they significant in this context?
Anjney: FLOPs in this context refers to the number of floating point operations used to train an AI model. Floating point operations are a type of mathematical operation that computers perform on real numbers, as opposed to integers. The amount of FLOPs used is a rough measure of the computing resources and complexity that went into training a model. For example, if models are like cars, training FLOPs might be the amount of steel or electricity used to make a car. It doesn’t really tell you much about what the car can and cannot do directly, but it’s one way to measure the difference between the steel required to make a sedan versus a truck. And this 10 ^ 26 FLOP threshold is significant because that’s how the bill is trying to define what a covered model is. It’s an attempt to define the scale at which AI models become potentially dangerous or in need of additional oversight.
But tying regulations to some fixed FLOP count or equivalent today is completely flawed, because as algorithmic efficiency improves, computing costs decline, and models that take far fewer resources than 10 ^ 26 FLOPs will match the capabilities of a 10 ^ 26 FLOP model of today within a fairly short time frame. So this threshold would quickly expand to cover many more models than just the largest, most cutting-edge ones being developed by tech giants. This would really hurt startups and burden small developers.
a16z: It’s a relevant touchpoint to remind people that a smartphone today has more FLOPs than a supercomputer did about 20 years ago. The Apple Macintosh G4 from the 1990s had enough computing power that, according to this bill’s parameters, would have been regulated as a national security threat.
Anjney: That’s a great historical example. We’ve been here before, and we know that when policymakers and regulators try to capture the state of a current technology that’s improving really fast, they also become obsolete incredibly fast. And that’s exactly what’s happening here.
a16z: Amendments to SB 1047 were published, one of which limits the scope of the bill to only models trained at [the 10 ^ 26] compute capacity and also costs more than $100 million to train. If we attach a dollar amount to this bill, doesn’t it make the compute threshold obsolete?
Anjney: Yes, this $100 million amendment to train might seem like a reasonable compromise at first, but when you really look at it, it has the same fundamental flaws as the original FLOP threshold.
The core issue is that both approaches are trying to regulate the model layer itself, rather than focusing on the malicious applications or misuses of the models. Generative AI is still super early and we don’t even have clear definitions for what should be included when calculating these training costs.
Do you include the data set acquisition or the researcher salaries? Should we include the cost of previous training runs or just the final ones? Should human feedback for model alignment expenses count? If you fine tune someone else’s model, should the costs of the base model be included? These are all open questions without clear answers, and forcing startups, founders, and academics to provide legislative definitions for these various cost components at this stage would place a massive burden on these smaller teams, many of whom just don’t have the resources to navigate these super complex regulatory requirements.
The outcome is clear, right? Most startups will simply have to relocate to more AI-friendly states or countries, while open source AI research in the U.S. will be completely crushed due to the legal risks involved. In essence, the bill is creating this disastrous regressive tax on AI innovation. Large tech companies that have armies of lawyers and lobbyists will be able to shape the definitions to their advantage, while smaller companies, open source researchers, and academics will be completely left out in the cold. It’s just blatant regulatory capture, and it’s one of the most anti-competitive proposals I’ve seen in a long time.
What we should be focusing on instead is regulating specific high-risk applications and malicious end users. That’s the key to ensuring that AI benefits everyone, not just a few.
a16z: You’ve mentioned that the purported goal of 1047 and some other AI-related bills is to prevent against “catastrophic harms” or existential risks from artificial intelligence. What do you think are the actual biggest threats posed by LLMs?
Anjney: What we have is a complete over rotation of the legislative community around entirely non-existent concerns in the name of AI safety, when what we should really be focused on is AI security. These AI models are no different than databases or other tools that have given humans more efficient ways to express themselves. They are neutral pieces of technology. And yes, they may be allowing bad actors to increase the speed and scale of their attacks, but the fundamental attack vectors — spear phishing, deep fakes, misinformation — are the same.
The notion that these models are going to autonomously go rogue to produce weapons of mass destruction or become Skynet from “The Terminator” is highly unlikely. The real threat here is our government not focusing on the malicious users of these models and instead putting the burden of doing that on startups, founders, and engineers.
What we really need is more investment in defensive artificial intelligence solutions, right? What we need is to arm our country, our defense departments, our enforcement agencies, with the tools they need to keep up with the speed and scale at which these attacks are being perpetuated — not slowing down the fundamental innovation that can actually unlock those defensive applications.
And look, the reality is America and her allies are up against a pretty stiff battle from adversarial countries around the world who aren’t stopping their speed of innovation, and so it’s almost an asymmetric warfare against ourselves that’s being proposed by SB 1047.
a16z: The other thing that SB 1047 would do is impose civil and in some cases criminal liability on model developers. If they build a model that’s covered by this bill, they would need to prove that what they’re building couldn’t be used for any of these types of attacks.
Anjney: The idea of imposing civil and criminal liability on model developers when downstream users do something bad is so misguided and such a dangerous precedent.
First off, the bill requires developers to prove that their models can’t possibly be used for any of the defined hazardous capabilities. But these definitions are way too vague, ambiguous, and subject to interpretation. How can a developer prove a negative, especially when the goalposts keep moving? It’s an impossible standard to meet.
Second, the bill holds developers responsible for any misuse of their models, even if that misuse comes from someone else who’s fine tuned or modified the model. It’s ridiculous. It’s like holding car manufacturers liable for every accident caused by a driver who’s modified their car. The practical effect of these liability provisions will be to drive AI development underground or offshore. No rational startup founder or academic researcher is going to risk jail time or financial ruin just to advance the state of the art in AI. They’ll simply move their operations to a jurisdiction with a more sensible regulatory environment, and the U. S. will lose out. Period.
a16z: Speaking of developers, we haven’t talked about open source projects and open source development, which have been a huge driver of innovation over the past couple of decades. We’re talking about very, very bootstrapped skeletal budgets on some of these things, but hugely important.
Anjney: I don’t think the current wave of modern generative scaling laws based AI would even exist without open source.
Looking back, transformers — the atomic unit of how these models learn — was an open source and widely collaborated-on development. In fact, it was produced at one lab, Google, who then allowed another lab, OpenAI — after open publishing and collaboration — to actually continue that work.
There’s no chance we’d be here without open source. The downstream contributions of open source continue to be massive today when a company like Mistral or Facebook open source models and releases their weights. That allows other startups to then pick up on their investments and build on top of them. Open source is the heart of software innovation and this bill slows it down.
a16z: If you were in charge of regulating AI, How would you approach it? Or how would you advise lawmakers who feel compelled to address what seem like concerns over AI?
Anjney: There should be zero liability at the model layer. What you want to do is target the misuses and malicious users of AI models — not the underlying models and not the infrastructure. And that’s the core battle here. I think that’s the fundamental flaw of this bill — it’s trying to regulate the model and infrastructure, instead of focusing on the misuses and malicious users of these models.
We should focus on concrete AI security and strengthening our enforcement and our defenses against AI security attacks, which are increasing at speed and scale. But fundamentally, these safety concerns that are largely science fiction and theoretical, are a complete distraction at the moment. And lastly, we have no choice but to absolutely accelerate open source innovation. We should be investing in open source collaboration between America and our allies to keep our national competitiveness from falling behind our adversarial countries.
And so the three big policy principles I would look for from regulators would be to regulate and focus and target misuses, not models, to prioritize AI security over safety, and to accelerate open source. But the current legislation is absolutely prioritizing the wrong things and is rooted in a bunch of arbitrary technical definitions that will be outmoded, obsolete, and overreaching fairly soon.
a16z: What can everyday people do about this? If I’m a founder, if I’m an engineer, if I’m just concerned, what can I do to voice my opinion about SB 1047?
Anjney: I think there are three steps here. The first would be to just read the bill. It’s not very long, which is good, but most people just haven’t had a chance to actually read it.
Second, especially for people in California, the most effective way to have this bill be opposed is for each listener to call their assembly rep and tell them why they should vote no on this bill in August. This is less than 90 days away. So we really don’t have much time for all of the assembly members to hear just how little support this bill has from the startup community, tech founders, academics.
And step three is to go online, make your voice heard on places like Twitter, where it turns out, a lot of both state level and national level legislators do listen to people’s opinions. I think if this bill passes in California. It sure as hell is going to create a ripple effect throughout other states, and then this will be a national battle.
To hear more about SB 1047, listen to the entire podcast below.
