A Playbook for Starting a Company in AI Safety
For-profit, philanthropic capital, and the door most founders don't know is open
A blogpost by Liran Markin
I spent a month on one question: if you're a strong technical founder who actually cares about AI safety, how do you start something? Not whether you should. How. What's the mechanism, where's the money, what do you build first.
Blog posts didn't get me far. What worked was a month of one-on-ones at EAG and around the LISA hub in London - grantmakers, researchers orbiting the frontier labs, founders who'd already raised - until the picture snapped into focus.
Everyone I talked to before that month framed the starting decision the same way I did: it's an identity choice. For-profit or philanthropic. Pick a lane, commit, move on.
And yet.
The founders who actually win don't pick. There are two funding doors in AI safety, philanthropic and venture, and they're nowhere near as symmetric as people pretend. Philanthropic capital for credible safety work is real, and the bar looks nothing like a seed round. A for-profit market barely exists: few buyers, and a lot of "products" that are honestly just research wearing a price tag. Startup people usually don't know the first door exists; research people raely reckon with how thin the second market is. You want to be the founder who understands both.
The playbook, compressed
1
Find a technical problem that matters for safety - one you can be genuinely good at.Founder-problem fit, not founder-market fit.
2
Write a theory of change, then raise philanthropic money on it.
You need neither traction nor a product for this.
3
Let the commercial layer come later.
Deliver the milestone and let the assets pile up; the product comes when the market does, unless you have the conviction to raise on the bet now.
All three steps are really one move: stop steering by today's market. Read where the technology is going, get funded philanthropically on that read, and make sure you're standing in the right spot when the market finally shows up.
Step 1
Find the most impactful problem you can be good at
The first move looks like any company's first move. The difference is that two of your startup instincts will betray you here: starting from a market, and starting from whatever hot category your resume points at. My thinking on this step got reshaped by two people who came at it from different angles and landed in the same place - Hugo Walrand of Catalyze Impact, who has run something like fifty founders through his program, and Geoffrey Irving, who ran alignment research at the UK AI Safety Institute and has since left to start Resolution.
Hugo runs Catalyze Impact, which has around fifty founders in its orbit and takes no equity and no legal hold over any of them. He's watched enough of them to be blunt: the market signal is a false compass right now. Safety spend today is a rounding error next to capabilities spend, so traction tells you almost nothing about whether the work matters. What he tells founders instead is to sit with one question - "what is the most impactful thing I can do with my knowledge?" - until a real answer shows up. He tried to shortcut that search himself and couldn't. In his experience only one or two interventions carry most of the impact, and finding them is painful. The part that surprised me is that the search itself is fundable: two recent grants of $500K, pre-seeds of $200K to $1M, seeds of $2M to $10M from funders who behave like VCs and take nothing. He'll fund a founder to go sit in SF or London for a while, so that being in the room is never a money problem.
Irving's lens is capability forecasting: aim two years out, and pick problems that shift the equilibrium rather than whatever the room is racing toward. Agentic security is the race, and its core technology cuts both ways - the same agent that patches can attack. What interests him is work that's defensive all the way down, and his case for why now is specific. The busy work formal verification demands has gotten cheap, because AI does it. He expects Clang and GCC could be formally proven correct within the year, and that someone determined could verify the Linux kernel free of memory bugs on a similar timeline. Then the opening: the formal-methods people chasing this don't understand real security and don't know where to point the methods. His other examples run the same shape. Unbreakable rollback, so that after a hack drains an account you can claw yourself back through time the way credit-card fraud already lets you. Hardware enclaves, compartmentalization, treaty-verification hardware that would physically stop the US and China from breaking an AI agreement, which only works if it's built years early.
Put them together and the test stops being "which corner of safety matches my background." It becomes: out of everything that shifts the equilibrium two years out, and everything you can be unusually good at, what are the one or two things only you would build? Security-native founders often land on defensive infrastructure, and often that's the honest answer. But they earn it through the exploration, not by grabbing "AI security" off the shelf.
Step 2
Write the theory of change
"Theory of change" is a term of art in the philanthropic world, and it took me a while to see that it is not just a pitch with different fonts. What it names is the explicit cause-and-effect chain running from the work you plan to do to the risk that goes down at the end, built backwards from the ultimate change you want - some category of catastrophic risk shrinks - to what has to be true for that to happen. For a safety org the spine reads something like:
We do X, X gives the field capability Y it lacks, labs and AISIs adopt Y, and risk Z goes down.
The chain itself is the cheap part. A grantmaker reads the assumptions hanging off each link. Who adopts this? Why would it work? How sure are you, and did you write that down honestly? Several people repeated the same rule to me: the chain has to end in actual risk reduction. Papers published is not an ending. Neither is tools shipped. The chain is what the funder is buying, the way a VC buys a deck and an ARR curve.
Coming from startups, this next part took me a while to believe. A lot of this money is explicitly hunting for work that cannot monetize - evals, interpretability tooling, formal verification, monitoring research. Hard to sell, important to build, and that combination is the whole reason the philanthropic side exists. A year of pure R&D with zero customers would scream red flag to any VC I know. Here it's just the normal shape of a grant.
I don't want to oversell this. Grantmakers screen hard, plenty of applications die in review, and a founder who can't speak the impact language fluently or whose assumptions fall apart under scrutiny won't get far. The narrower claim survives: if you're a strong technical person doing legible safety work, the bar sits somewhere below a venture seed, and the check can be just as large.
Step 3
The commercial layer comes later
The sequence, defined by what money you raise and what you do with it rather than by your legal structure:
Raise philanthropic money against a concrete milestone from your theory of change.
Deliver it. What's left over compounds: deep understanding of a hard problem, a team that's now world-class at it, relationships with labs, AISIs, funders.
Build the commercial product on those assets, when the market is there.
Not a bait-and-switch, and not a trick to play on funders - go in intending to build a great organization on the mission. Redwood, FAR, METR - and now Irving's just-launched Resolution - are well-funded non-profits that are the destination, not a runway to something else. Some of the best people in the field never take step three at all, and committing to the mission doesn't cost you the option: Apollo built its name on non-profit research and is now monetizing parts of it.
Now the commercial side, honestly. Irving ran alignment-research funding at UK AISI with two pools, one for grant work and one for commercial and equity bets, and could not spend the commercial one. Not for lack of money, for lack of things to fund. Revenue-generating safety work is genuinely hard to build, and the labs, the only ones who really need it today, build it in-house.
But as capability spreads - more deployers, more autonomy, more regulation - the set of organizations that must buy safety rather than build it gets much wider. The market gets created by where the technology goes.
Which leaves two timings. Raise commercial capital now, against your thesis about the equilibrium two years out, and spend the runway building for a market that isn't here - "The money is now. The commercial is in the future," as Irving put it. Or take the grant, deliver the milestone, and start the company once there are buyers. The first needs an investor with your conviction. The second needs patience from you. And occasionally the market isn't in the future at all - some un-sexy tool solving a problem a lab has today, with a budget. If that's what you found, skip the philanthropic detour and go sell it.
Some categories already have a credible commercial story. Marius Hobbhahn, Apollo's CEO, names agent observability and control, evals infrastructure, interpretability tooling, red-teaming, runtime guardrails, secure AI for regulated verticals. VCs who'd never fund "alignment research" fund "runtime AI security" without blinking.
The pattern I keep coming back to is an iceberg. Sell the thin layer above the waterline - a guardrail, a monitoring product - and let it fund the research below, which is the actual safety contribution and the actual moat. Philanthropy pays for what's underwater. Venture pays for the tip.
The Counterargument
"Grant-funded R&D with no customers is how founders fool themselves into not having a business. Traction is truth."
In a normal market I'd sign that with both hands. But here the buyers who'll need this work mostly don't exist yet - the market gets created by where the technology goes, not where it is. The philanthropic door lets you be early to a market that hasn't formed and still be alive when it does. You're not dodging the discipline of customers; you're refusing to let a market that's two years away talk you out of building what it will need.
With one condition, and it's the one that keeps this honest. "The market will exist later" only holds if you can name why it doesn't exist now. Three answers survive. It's too early, in which case deeptech investors who underwrite a decade-long thesis are also a live door. It's a public good nobody can own - formal verification is the clean case - and philanthropy is the only correct funder, permanently. Or the users are real and the budget isn't: labs, AISIs, evaluators, served the way METR and Apollo serve them until the money arrives. If none of those describe you, the counterargument wins and you should go find a customer.
See the sequence
The strange fact about this field: the first check comes on conviction rather than traction, and the same technical conviction that funds a mission today can fund a business tomorrow. Most founders only ever see one door. See both.
Who actually writes the checks - the philanthropic funders, the hybrids that grant and invest, the VCs circling safety - is mapped here.
Drawn from conversations at EAG and around LISA, London, late May 2026 - in particular with Geoffrey Irving (Resolution, Previously Chief Scientist @ UK AISI) and Hugo Walrand (Catalyze Impact) - and a month of mapping the AI-safety funding landscape.
