When threat actors are actively leveraging AI swarms to discover and exploit zero-day vulnerabilities in minutes, relying on a SIEM and human analysts moving at "meeting speed" is a recipe for disaster. The only viable defense against an AI-powered attack is an autonomous, AI-powered defense system.
In this episode of the AI Security Podcast, Ashish and Caleb speak about OpenAI’s recently announced "Defense Factory" framework, unpack why patching code vulnerabilities is only the first step and how security teams can extend this framework to handle least privilege audits, SOC automation, and incident response triage. Caleb details how CISOs can deploy their own open-weight models to build specialized "vendor agents" (e.g., a dedicated CrowdStrike agent) and orchestrate them through a unified, team-wide harness.
Questions asked:
00:00 Introduction: OpenAI's Defense Factory Framework
02:50 Expanding the Defense Factory Beyond Code Vulnerabilities
06:30 Using AI Harnesses to Automate SOC Triage and False Positives
10:30 Why Security Teams Must Run Their Own Open-Weight Models
13:30 Scaling from Individual AI Harnesses to Team-Wide Orchestration
21:30 The "Build vs. Buy" Debate: Developing In-House Harnesses
27:30 The Danger of Defenders Moving at "Meeting Speed"
31:30 Architecting Shared Memory and Reference Datasets for AI Agents
34:30 How Caleb Built Eight Vendor-Specific AI Agents in Four Hours
36:30 Bypassing Missing MCP Capabilities by Reverse Engineering UIs
38:30 The Exact Prompt to Build a Comprehensive CISO Dashboard
41:30 Why AI Models Are Making Published Exploits Obsolete
Caleb Sima: [00:00:00] We had to battle AI offensive attacks, you have to have an open weight model to do it. And of course, these companies are being very secretive and controlling about who gets access to what. Doing just fixing and patching vulnerabilities is not even close to good enough to be able to defend yourself against these kinds of attacks.
Ashish Rajan: If you can speak English or any language, you can build it.
Caleb Sima: I created like eight vendor agents in a matter of four hours. The majority, the 98%, have zero time to, one, even learn about how to prompt this stuff into existence.
Ashish Rajan: Tomorrow someone who had a great harness leaves, what happens to the harness?
Caleb Sima: Give AI another year and I think we'll be there.
Ashish Rajan: Would you let OpenAI run your defense factory? Let me explain. Defense factory is a framework that they announced recently to talk about how do you do vulnerability management at scale, not just the part of discovering, but validating which ones are relevant for your organization.
Now, you can take that with a pinch [00:01:00] of salt on just the marketing pitch, or you can take that more from a practical lens. That's what we spoke about in this episode of AI Security Podcast. So Caleb and I spoke about what does it take to build a defense factory, and why is harness becoming a top of mind topic, which we spoke about almost a year ago now.
But it is definitely getting to a point where now things like Hugging Face and other examples have highlighted that most organizations perhaps just using a traditional security tool that is not autonomous. See how I did not say agentic, at least autonomous. You probably would find it challenging to work at scale when there's a lot more AI systems running in your organization.
We spoke about the practical things of how do you establish a harness with an individual level within teams, how do you do that at a team level, and perhaps a harness at a, as an overall security team as well. All that and a lot more in this episode of AI Security Podcast. As always, if you have been enjoying the episodes of AI Security Podcast, I would really appreciate if you can take a quick second to hit the subscribe or follow button, whichever podcast platform you follow us on.
We are [00:02:00] on YouTube, LinkedIn, Spotify, Apple, and wherever you consume your podcasts from. I hope you enjoy this episode, and I'll talk to you soon. Peace. I would say I looked at this as almost an operating system being created for security where, and obviously it goes back to the cycles and that you're talking about, whereas traditionally we have always considered us a build-- Or let's just say a building or different floors.
There's an appsec floor, there's a cloudsec floor, there's a vulnerability management floor. The ask from OpenAI, and perhaps it's easier to do when you're a new company, very difficult to do if you are an already an established organization. I looked at this as an operating system, but I also looked at this as a, it would be a challenge for an established company which has had done IT for since on-premise days, mainframe days to apply this.
So my next obvious question here, yes, we need this to be more cyclical and especially with the the current day and time we're in where exploits are being found left [00:03:00] and center. Do you see this approach being practical for people who are in organizations that are like the, I apologize for using Bank of America.
Let's just say Bank of America, like an established entity is what I'm going with.
Caleb Sima: I think it's very practical, but it matters on the level of abstraction and scope at which you apply it, right? Okay It... You can apply this cycle very easily depending upon how specific or how broad you want to be.
Like for example, you could say, "Hey, guess what? Cross-site scripting vulnerabilities in web application source code, we should apply this cycle," which is identify the exact issue, automatically generate the patch, automatically submit the patch, maybe even automatically approve the patch, automatically test it in a dynamic environment, validate it, so on and so on the cycle.
You can do this very realistically based on your scope, the type of issue you're finding, and even maybe the scope of the assets at which [00:04:00] you're identifying it on, right? In, in Bank of America or in any large organization, you can easily say, "I'm gonna take... I'm gonna try to apply the defense factory in one product, in one app, in one team, and see if it works."
And if it works, you can continuously expand and scale over time. Now, of course, I think, I'm not sure in OpenAI's perspective, are they applying it at a broader scale overall, or are they just... to me, when I first, got it presented to me, and when I first read about it it was very, to me, focused on code vulnerabilities, right?
To me is the way it comes across. That's right.
Ashish Rajan: Yeah.
Caleb Sima: And I think that, from that perspective, it's just been a matter of which asset you apply it to and which vulnerabilities you wanna apply this to, and you can make it very practical. Actually, my only sort of critical critique against it is I feel it's not thinking big enough.
I actually think [00:05:00] doing just fixing and patching vulnerabilities is not even close to good enough to be able to defend yourself against these kinds of attacks. I think you need to think much bigger in terms of least privilege, in terms of deployment, in terms of b- user behavior activity.
There's a lot of these things that I feel like could be bigger in scope and could be more applicable in terms of things like detection, investigations. Like, all of these types of things need to be part of a defense factory that hasn't been really talked about here.
Ashish Rajan: I'm with you, and I 100% agree on this is a code-based approach, perhaps also because their current service offering is from that perspective.
At the end of the day, that Defense Factory blog is to market the product that they have-
...
Ashish Rajan: Which is either Daybreak or GPT 5.6 Cyber or whate- whichever version you wanna take. I definitely find that there is a clean line that you've seen is 100% on the money with code being the focus here, [00:06:00] 'cause that's what the first thing is.
GRC is not the focus, compliance API and all of that people talk about. But I feel you've touched on something which is even interesting for probably people to dig into which I feel I've been thinking a lot more about. We I think during RSA this, earlier this year, and perhaps even a few months before that The whole concept of harness becoming a thing, we spoke about it at length on the podcast.
I almost feel like they're also, in their own way advocating for the fact that, hey, you need a harness. And to what you said, this is a code harness that they're recommending. Codex is another one, but what they're saying is an organization would need some way of a framework or structure that they can use to have this as a continuous cycle, whether it's through code, GRC, SOC automation.
We did an AI security lab event last week in November. We had quite a few CISOs in the room, and we were just talking about what are some of the use cases that people are working on. A large number of use cases were primarily around reducing the number of false positives, and a [00:07:00] large majority of it was either code focused "Hey, I have reports coming in from my SaaS provider, my SCA, my whatever.
I'm trying to reduce a false positive." Or it was, "My SOC team is getting tons of logs from my SIEM, and I'm trying to reduce a false positive." Those were the two big use cases. It was not that, "Hey, I'm trying to automate GRC," or, "I'm trying to automate cloud security." And rightly and maybe that's where it makes sense for them to talk about it, but it also highlights a point that harness conversation that you and I had almost now eight, nine months ago, early this year and before that, it's almost becoming mainstream now, that people are expected to start having a harness to be able to manage the thing that you, example that you gave of XSS being predominant across the organization.
How do you continuously keep improving for it? A bug bounty or a pen test report comes in, that gets fed into the harness, and that becomes a cycle 'cause you know you're prone to weakness for that. I don't know if you agree with that kind of [00:08:00] sentiment of w- the way at least I see a defense factory.
Caleb Sima: Yeah, of course. I think that this is just their recommended harness for being able to do defense. Yeah. In vulnerability remediation specifically. Yeah. And I think that they gloss over the, I would call, triage part of this, right? But I would assume that's sort of an unsaid requirement, right?
Yeah. Like, when you take a vulnerability, that vulnerability needs to be clearly a real vulnerability. Yeah. And/or triaged correctly. And I also think, in OpenAI's mind, they're saying you're using our AI to find the vulnerability," which in that essence, the triage becomes part of the finding of the vulnerability problem.
Ashish Rajan: Yeah.
Caleb Sima: And therefore, whatever gets produced at the end of OpenAI finding that issue is a already triaged problem. So I don't think- In the defense factory is my guess in reading this. This is about [00:09:00] taking Rapid7's vulnerability report results and then going into the factory.
Ashish Rajan: Definitely not that.
But- Yeah ... I think it definitely... Yeah, sorry. No, you're right. But, and thanks for the clarification as well, 'cause I, I'm pretty sure a lot of people have an understanding of, oh, okay, so my existing defenses, whatever enterprise product that I use in the traditional model that I've used in with... 'Cause I was gonna add another mix here.
I'm sure you would have heard about the headless AI pieces ecosystem as well that's coming, slowly coming up or is top of mind for a lot of people where there are examples like Instinct, where you text messaging a service and goes and does the thing for you. There are other certain related services.
In the ecosystem that we are moving towards, I almost feel like not having a harness and probably putting that aside for I'll wait for, I don't know, Rapid7 or whoever else to become my harness is not the right play moving forward. I [00:10:00] don't At this point in time for two reasons. One is the shared context of the entire organization with the cybersecurity product gap that exists and probably will continue to exist, and the the gap that exists in terms of not everyone may have access to a Daybreak, but perhaps they have access to an open weight.
And at least when I was at the events in US, I definitely feel there's a sentiment of like a strong sentiment for not using open weight models for it. With a harness, I don't disagree personally with using open weight, but I'm curious, does it really make a difference if a harness is open weight or if it's like an LLM?
'Cause cost is a factor here in building a harness. I'm trying to extend the conversation more from, hey, vulnerability management, yes, the conclusion is that we have a framework, and we all need to have a harness. The next question then become, if I'm trying to build a harness, am I building an OpenAI harness?
Am I building an open weight harness? [00:11:00] Generally the sentiment seems to be split today. I have an opinion, but I'm curious to hear yours as well. Where do you stand on the whole today now that we have so much advancement across open weight as well, do you really personally align more with open weight or the like the state-
Caleb Sima: Yeah.
For example if you are doing cyber defensive and offensive work in your enterprise, it is very clear you cannot use Anthropic. It is- Yeah. It is, it, especially o- like Opus 5.5 or any of the... It is a disaster. It is like you just sneeze wrong and it like boots you out. So you cannot do that. So you either have to have Mythos capability, right?
OpenAI is a lot better. Actually, OpenAI still boots you on things, but it is much more like, like the, the analogy I always like to use is Anthropic is like an overprotective parent. OpenAI is a little bit more in the middle. Like-
Okay ... okay you're you're a coming into adulthood.
Like I'm gonna let you kinda, have some [00:12:00] freedom and some independence-
yeah ... without, breathing down your back. But for sure you have to have open weight as a backup. There's no question. You have to do that. We are in process of doing that. We are procuring massive hardware to run open-weight models so that we can effectively do defensive and offensive practices properly.
And the thing is like we, we are forced to do this, not by choice, by the way, right? We recently had an, We had to battle AI offensive attacks. Yeah. And you could not do it with Anthropic and OpenAI by its default without having access to Daybreak Red or Mythos, right? It is not plausible.
You have to have an open-weight model to do it. And of course, these companies are being very secretive and controlling about who gets access to what. Listen, and for whatever reasons they have is their reasons, right? So [00:13:00] I'm not gonna debate that. But all I know is as an actual enterprise that defends against these attackers, if you do not have an open-weight model running on your own hardware as access to the cybersecurity team that is not a good idea.
You have to have that. You'll be stuck in a river if you're under these attacks and you are not able to use AI in order to both do defensive maneuvers, investigations, and cha- make changes. My recommendation is anyone who's an enterprise in a cyber team, if you're not doing that today, you need to start doing that tomorrow.
Ashish Rajan: I'm with you. I was gonna ask an extension 'cause you guys and obviously we have an episode coming up soon on the AI attack you guys faced. I am curious, in this world of Harness, a lot of CISOs or people who are in this say, let's just say leadership position or leading teams in cybersecurity A lot of people would be asking the question, is it a harness per team?
Is it a harness per [00:14:00] individual? Or is it a harness for the entire security ecosystem? My assumption is it's phased, but how do you see it today?
Caleb Sima: Yeah. The way we started is we started obviously per individual. Every individual right now has our own, quote-unquote, "harness of how you like to work," right?
I'll give you simple example. I have a specialized agent that is the subject matter expert for every vendor product, right?
Whether it's CrowdStrike or Island or whatever the thing is, I have an AI agent that specifically is communicates, manages that vendor product. And then I have orchestration that uses that.
So like for example, you may have a security engineering agent that manages those agents that manage the vendor products. And then you yourself as a CISO, have an, a chief of staff or a, or whatever it is that can help manage those. So every person in our team has their own levels- Yeah ... of orchestration.
Now, we have since [00:15:00] moved and phased into team-wide orchestration to where these agents are now moving to always long-running management agent platform at which all of us can interface with and all of us can then run things into. But that has been phased into. And by the way, does that mean that eliminates your current, workflow or harness system?
No, it does not. It just means it it changes the dynamic of what happens in the team orchestration agent versus what happens in the personal. Now do I think all of security as a goal should have that? I don't know. That's debatable. I don't even understand what that would mean, so I can't answer that.
Ashish Rajan: So I have an example for that. I had a conversation with the I think it's like actually probably out as well, with the staff engineer at, or security staff security engineer, I can't remember the title, but from XAI. And the way they have approached, which is interesting, they've basically made [00:16:00] the security team or security services, let's just say, as an MCP gateway.
And he's open sourced a repo which kind of has the framework for it. But the idea is that MCP gateway for security is basically different paved roads for security which is whether it's your cloud ecosystem, appsec ecosystem, or whatever, insert another category of, of security in the organization that the rest of the organization wants to interact with.
Now, that's the other side to what you were talking about, because the, what you referred to-
Caleb Sima: that's company-wide best security best practices, I imagine is what they're trying to do there.
Ashish Rajan: Y- there is that, but there's also the appsec flow kind of comes... at least obviously I don't don't want to put words of or rephrase what he's saying, but I think to what you said, it's more identifying what company best practice would be.
But the idea that they were talking about is more of an extension for the, the teams in the organization only. It is not for, "Hey, if you and I are part of a [00:17:00] security team, we, you and I have an MCP safe gateway between both of us." It definitely was not that. But I wanted to take that idea and extend that to what you were saying about us individuals working or us teams working in a under the same security org Having a ecosystem of all of I guess say tomorrow someone who had a great harness leaves, what happens to the harness?
Or the operating system idea of security, which is being preached by a lot of people, and I think people have not published this yet, but they are trying to work towards it, is the idea that us as a team, let's just say we are a team of 10 people in the AppSec, all of us have our own harness that we have been work...
To what you said, the phased approach. We moved toward, we graduated from a individual harness to a team-based harness. But the idea is that overall, if things like Hugging Face or OpenAI incident that we spoke about on the episode before, if those things were happening, it may not [00:18:00] help for each one of these harnesses to be individualistic.
Rather, the teams together should have a way of figuring out the beginning and end of a chain of attack, for lack of a better word.
Caleb Sima: There is a definitive guarantee that team-based harnesses have to be done, be there.
Ashish Rajan: Yeah.
Caleb Sima: For example, if I have a agent that knows anything and everything and controls CrowdStrike-
That agent needs to be shared with the entire team, right? Yeah. For any, my recommend, any vendor product, both should have its subject matter expert agent, and every team should have its subject matter role, right? Whether it be a security engineer, whether it be an AppSec guy, whether it be an InfoSec guy there are agent roles also- Yeah team-wide shared, right?
Yeah. And then we become orchestrators of these team members in their sense. That, and by the way, on, and on team-wide, you need things like shared knowledge base, shared collection, [00:19:00] shared task lists, all of these things at which when Hugging Face incident occurs, there does need to be a dedicated person that says, "On the team platform and process, I am kicking this off," which then generates the to-do list, the shared knowledge base, the things that go through, and then we as individuals can now use our own workflow to pick up tiny bits of the pieces that we want to go investigate ourselves that now our team-wide organization harness has been generating, right?
Ashish Rajan: Yeah.
Caleb Sima: That is A no-brainer. If people aren't doing it, it will come. It's just a matter of time.
Ashish Rajan: Yeah. Yeah.
Caleb Sima: And by the way, it's much more than just kicking off for an investigation. It also, team-wide harnesses need to be able to manage daily hygiene, daily tasks and security. Yeah. All of the kinds of things that normally, like I'm already running in my org, that people should be doing with AI, which is like, "Hey, we have these five projects.
Where are [00:20:00] they? What are they doing? Where, how close are we to completing these? What problems are we running into? What areas of risk should we be concerned about? What are the day-to-day activities?" Like for example, no one today looks at a SIEM dashboard and says, "These are the alerts." You have a generated agent that produces the things that says, "I've looked at these top five.
Here are the things I've already investigated, and here are the things that need someone to approve." Yeah. Yeah. Like that's just done already, and if you're not doing that, that just needs to be done.
Ashish Rajan: Yeah.
Caleb Sima: And I guess- The problem though, of course, is there is some reality to this, is there's huge cost problems.
There are, there are things I do think that, there are details behind the scenes here that a lot of people don't necessarily talk about.
Ashish Rajan: Yeah. And I guess to your point, this also to your point about the details, I'll probably double down on the gap as well. In a lot of the conversations that I had, and maybe, I don't know if it's just outside the [00:21:00] SF bubble, for lack of a better word It is definitely an, a thinking that there is a requirement for people to be quite technical before they go down the harness path.
And it's definitely a lack of, I don't know, 'cause every time I've brought it up they feel, "Do I need to hire a dedicated AI security role for this?" 'Cause they, a lot of organizations have started hiring for four DevPro engineers in their security team to have these as workflows that are being created.
And I was looking at that and going, "There is no way that four DevPro engineer," and I'm sure it's, they'll do amazing job. I don't know if they would be able, the right people to create the harness for Ashish, the AppSec person, who has done 20 years of this stuff, and knows the company inside out for the past three years, and doing day-to-day.
I I don't know if you've seen the pattern as well. I just, yeah.
Caleb Sima: Yeah, but the, you are correct, but you have to think about resource capability, [00:22:00] right? Let's just say for, sh- shits and giggles here Ashish is my AppSec leader. Yeah. Let's say you have two other people on your team.
Yeah. You guys handle all of AppSec, right? So you're the SMEs, right? You are, by the way, guaranteed up to your eyeballs today in already day-to-day tasks and problems that are arising. 100%.
Ashish Rajan: Yeah, 100%.
Caleb Sima: And the three of you are probably, you have your own workflows- Yeah ... that you're using to try to optimize and to automate what you're doing, right?
Yeah. And you're working on things like, "Oh, how do we build the right AI code harness," "in order to identify the right vulnerabilities?" You're probably working on all of that. But what you don't have is someone sitting down or you even taking your time to sit there and go, "How do we build an optimized flow or defense engine for just AppSec?"
Who's going to build that harness? You have the subject matter expert expertise, but you don't have the time, right? Would be my guess. Okay, maybe we need [00:23:00] someone who is dedicated to doing this across the security team so they can go to you and say they know AI, they know how to build this ship pretty well.
"Okay, Ashish, let's get your SME. Here's how we're thinking about doing it." "Yes, but I need it to do this, and this." "Okay, let me go build it. Let me give you a V1. You try it out." "Yeah, this works. Tweak this." Okay, blah, blah, blah, right? And then at some point, you get the harness to a point where it can reduce a lot of the load, 'cause the whole point of the harness is that it reduces your day-to-day load, right?
Yes. And then you're now at a point that says, "I can take it over because I've reduced my day-to-day load and I will tweak it to the way that I want it."
Ashish Rajan: Yep. And I, 100% on that. The, where I was also coming from, the fact was the reason I think you've done a great job of extending my example, but essentially where I come is, coming from is the FTE that comes into the organization would never be able to match the, in this example, Ashish, who's been [00:24:00] spending seven years deep into this ecosystem of...
I think what's the missing skill here is the ability of Ashish to explain to an FTE what is my workflow. And I think that's where the time is consumed quite a bit for this as well, and which is where for me, and 100% on the resource capacity thing, I've been leaning more on... so we did a workshop where we helped an organization make app- appsec, like the whole software development lifecycle, how do you make it agentic.
We did that workshop. At the end of the the workshop, we did end up working and triaging on what, how do you talk about this? We did work on a few workflows. I personally find If people are building harnesses, they should still build a harness themselves, like a mini version of it, whatever, however form that it is.
Because you know how the agent drifts over time. FDEs, to your point, I think that learning goes away is what I [00:25:00] found in all the training that we did, where I handed you the harness, you do not know the nuances of how I basically put the Band-Aid fixes across the board. But I told you, "Hey, over time it'll drift away.
Just make sure you remember it again." And this
Caleb Sima: true stuff- can I push back a little bit though?
Ashish Rajan: Oh, yeah, 100%, man. Yeah. 'Cause I'm just-- and just to cla- I'll take your pushback maybe after I finish this. Okay. 'Cause it's, there are startups in this ecosystem now where entire ecosystem is built on, "Hey, we are providing an FDE to a security team, and we will bring you a dashboard with all the security workflows of the entire organization."
Anyway I'll pause there, but I love your pushback.
Caleb Sima: Yeah okay. So my, my, my pushback, like first my pushback to you and then I'll hit the security companies and the teams part- Yeah ... is that y- you are right that you should build your own harness. And actually it's not that hard.
That's right. That's literally true, yeah. It's not that hard. Yeah. Do you remember-
Ashish Rajan: If you can speak English or any language, you can build it.
Caleb Sima: Yeah. Do you remember my whole, the whole conversation I had about Pepper [00:26:00] and the EA- Oh yeah. That's right. Yeah ... and all that? That thing is a, is unbelievably frameworked harnessed now.
Yeah. It's just it's like professional pro level, orchestration ga- And that's just me pro- I don't know anything about any of this stuff, right? You just prompt this stuff. 100%, yeah. So it's not that hard, Yeah ... to your point. And also similar to that, this is the other thing, is how hard is it for someone to be able to also learn your workflows?
I also don't think that's that hard, right?
Ashish Rajan: Oh, reckon.
Caleb Sima: Yeah. Absolute- With AI, as long as the right things are tracked in what you're doing-
Ashish Rajan: Oh,
Caleb Sima: yeah ... and you have access, understanding your workflow is quite easy, right?
Ashish Rajan: Yeah. Yeah.
Caleb Sima: What are you doing? How are you doing it? What's the right workflows? Doing that is actually not difficult.
It, but here's the part of where I will both be pro and con on your next statement of these companies who are doing it.
Ashish Rajan: Yeah.
Caleb Sima: The, the pro part is it takes, it still takes time, right? It is close. You can [00:27:00] /goal, /ultracode- ... build me, security workflows. And by the way, it will get you 80% there.
Yeah. Literally. Yeah. I think a single prompt will get you 80% there given the right integrations and the right- Yeah ... token budget, but you'll get there. But the, the problem is it's that just like in anything in engineering, it's that t- that other 20%, the other 15% takes a lot of time. Yeah. Like validating it, tweaking it with the right prompts, doing all that stuff takes a lot of time.
So I personally, as a CISO, built it myself.
Ashish Rajan: Yeah.
Caleb Sima: And I think anyone should be able to build it, but I am a very bleeding edge technical CISO, right? The majority, the 98%- have zero time to, one, even learn about how to prompt this stuff into existence. Two, the capability to go through... 'Cause in large enterprise, [00:28:00] think about walking to your, g- going to your Bank of America.
Think about walking to your Bank of America example.
Ashish Rajan: Yeah.
Caleb Sima: Just giving the agent, the AI, the capability to know what data sources you should get access to versus you shouldn't in order to build the processes alone takes probably weeks of talking to people on the phone and emailing.
Ashish Rajan: Oh my
Caleb Sima: gosh.
Ashish Rajan: Right?
Yeah.
Caleb Sima: Yeah, so like I think there's a, there's both a time capability, a resource capability where someone c- does have to come in and say, "Okay, I n- I've built these things before." Yeah. "I know the tweaking and the process and the thing that needs to happen, and I will go into your org where you're not technical."
Yeah. And by the way, 90% of most, security teams, the large majority aren't very technical in that sense either, right? They're not software engineers. Yeah. And they're definitely not software engineers with AI-level capability. Yeah. That takes time to learn, adopt, and ramp up on.
Yeah. This guy's gonna come in with all of that, and he's gonna, "I'm [00:29:00] gonna help your team learn AI, adopt AI, optimize your process through building your harness." I get the value. I get why I would go need to hire those things. Yeah. Because the majority of the, the people are not, Robin Hoods, Blocks, they're not tho- those companies that have heavy engineering skill set. Those guys w- and by the way, I might even say those guys will probably hire some of those people, too
Ashish Rajan: Yeah. I'll be honest the, the AI security lab workshops we've been doing has been end-ending up becoming FDE engagements after.
So I'm with you- Yeah ... on that one 'cause I, I definitely find that it's... And I think I'm I agree with you on that not everyone has the time for it. It's more that than the fact that people can't learn. Yes. 'Cause I definitely found that at every time we've had a workshop, the unlock people have, 'cause they see someone make a harness like, "What?
Is it really that easy?" And I think I did- Yeah ... one where I made people so we were trying to work towards building a security dashboard, [00:30:00] just a secure dashboard for AppSec. We started with Hello World, and the, and such a, for people who are technical, Hello World is like the, I don't even know, like an, like a, such a minuscule thing that people think, "Oh, come on, man.
H- Hello World, really?" But you build Hello World with Codex or Claude Code, suddenly you're like, "Wait." And now you add a plugin for what your favorite cloud security tool, you add a plugin for your favorite SIEM, suddenly you're like, "Oh, wait, my dashboard is lighting up." I have to what, to quote you earlier, you have a dashboard for the individual of what that individual cares about every single day when they wake up and come into the office.
Caleb Sima: Yes. Yes.
Ashish Rajan: And I'm like, that is a, it's not a tall ask. The, the ask is the time commitment required to get to that point. So I'm with you. And then- Yeah ...
Caleb Sima: and again, like any other product, also the support, the maintenance, and the- Yes ... future improvements that have to come along with it, right? Yeah. Yeah.
And so either that individual has to learn how to prompt correctly-
Ashish Rajan: Yeah ...
Caleb Sima: and maintain their own [00:31:00] dashboard or someone else is going to do it. And I can tell you that most CISOs will want someone else to do it.
Ashish Rajan: Okay. That
Caleb Sima: kind of explained. Until a- AI is not quite, it's close, but it's not quite good enough to where the CISO who's not technical can just say, "This is what I need," and it just appear and it be accurate and works.
Like we're- Yeah ... give it listen, man, give AI another year and I think we'll be there, but like right now it's not. So you need someone to do the maintenance and do the updates.
Ashish Rajan: But to your point on the defense factory, right? That is, that OpenAI's framework has been building harnesses has been, you mentioned something interesting in terms of shared memory across teams, across individuals on the harness that they built.
I'm curious, what are some of the components that you have seen as like I was-- I'm thinking that there's obviously my harness that I've created as an individual. There's a harness that is created as a team, which has all the vendors in there. Is-- what's the-- Should people [00:32:00] think of a data lake or some kind of a like a memory, shared memory, which is outside the SIEM, or should SIEM become that?
'Cause there is clearly people who have subscribed to the idea of SIEM for decades, contracts going forever. And I'm thinking about the components that people have to think about when they start, "Okay, I heard Ashish and Caleb talk about a defense factory. I need to build harness, build my own dashboard or whatever."
What are some of the components that you think are required for people to be able to build a paid defense factory?
Caleb Sima: Yeah. So a SIEM is only a reference dataset. So- Yeah ... you have to have a couple things. You need your references, right? This is the kinds of things that are searchable, things that are knowledge bases that you need to be able to obtain information from.
So whether it be SIEM, Databricks, Snowflake, Jira, Confluence, these are all like reference searchable datasets that you need to have access to. And then you have like working memory. So out of... When [00:33:00] agents are communicating or when they're doing shared tasks, you need a, a place at where shared data, references, learnings, et cetera, can be stored.
So the simplest way to do this is just ask AI to make it, and they'll probably make it in its simplest version, which is markdown files in a shared file system, right? That says, "Okay, we're gonna keep update." And they'll create a task, a shared task list at which everyone... And you make it just a Jira workboard.
You can either point it and say, "Use Jira to do the workboard," or just create your own version of it, and then they'll just work from that workboard as it moves forward. So things that are in process versus completed, and agents pick up that workflow. By the way, you can just very simply just tell it that.
"I need a, build a shared workboard so that all agents can coordinate their tasks when it comes to s- similar shared projects," right? That need- Yeah ... and it will go and it will create the right thing in order to go do it. And if you wanna get fancy, you can create memory. I don't think this is necessary, but you can get fancy and create [00:34:00] memory, which basically says, "Hey, in the context of all the sessions that you're doing, parse it, learn new facts, new entries, and then save that in a shared database of memory so that it will search that before it searches references, before it's, it does these kinds of things."
And you don't need any of that stuff, but it's... And obviously there's just normal integration, so action integrations. Okay, now as a defense factory, I need vendor integrations. Yeah. So every single security vendor you have, you need either API access or an MCP or something along the lines at which you...
And the way I built my vendor agents is my vendor agent is a self-contained agent that has all the help knowledge base API documentation that's in a reference file and a vector database at which it tries to update any time there's new updates in that vendor and it also creates a CLI. So that CLI is then used by the agent itself for that vendor product.[00:35:00]
That is one self-contained agent that is now accessible. And by the way, agents can talk to agents. So ensure that, oh, my orchestration agent can talk to the CrowdStrike agent, my CrowdStrike agent can actually talk to my Island agent. These kinds of things, if you have... they all each one of these agents broadcast what they are and what you can use, and your orchestration agent can then use that to manage it.
And these kinds of things are, you could... It'd take you a day to set this up. Yeah. Literally a day. May- maybe even a little less than a day to get all of this working. Do you feel like- ... I think I created vendor agents, I created eight vendor agents in a matter of four hours, right?
Yeah. So it's not- Do you
Ashish Rajan: feel the vendor ecosystem is ready for it? I feel like there's so many still that are not API enabled. There's so many that still don't update documentation on time. AWS example as well, they don't update their documentation. This is a large cloud provider as well.
Do you think the vendor ecosystem is even there? Maybe not [00:36:00] fully, but is there that way you can you... I see. Let's just use your example, yeah. The vendor agents that you build, wa- was there some challenges in the kind of vendors you were dealing with? You don't have to name the vendors if you don't want to.
In terms of how easy or how difficult it was for your agent to get the information.
Caleb Sima: There are definitely... I would say by and large, most of them have some API. Now, the question is, does the API match the GUI or the dashboard feature capabilities? Oh, yeah. Yeah. That's right. There are definitely mismatches here.
And there are a lot of vendors that are now providing MCP capability, although I've also identified their MCP is way more limited than just using their API. So I have fallen on every agent to always use API because that tends to be the most comprehensive. And by the way, for the ones that have UI, you can tell AI to reverse engineer the API at which the UI uses, and then include that in your CLI So there are ways [00:37:00] there at which you can, even if it's not a supported API, you can still use it within your agent.
And so I've found by and large, I think right now most of the features and capability you can do is plausible in an agent I've at least so far. I have not run into major blockers yet. There are a couple vendors, however, that we ran into that will block agent or or bot-like API requests because you have to go pay an extra license to go get automated API capability.
So that I've run into already.
Ashish Rajan: Interesting. It's like asking people money for single sign-on.
Caleb Sima: The, I think this is gonna happen more and more. I think Google is now doing this. They just announced, I think, that if they see agents they're, they are doing detection on agents, and they're prohibiting it from using agents using APIs now unless you go and sign [00:38:00] something or get permission or something like that.
That, so I think you're gonna see more and more of this happening.
Ashish Rajan: Hopefully people are able to get past some of those API things, but to make a vendor dashboard. Maybe that, that's the action for people from this conversation that we just had. So you don't have to build the entire defense factory, but at least try and build a "Hello, world!"
and see where you land. No, I
Caleb Sima: wanna give you... if you're in your security team, I'm gonna give you the prompt right now. This is
Ashish Rajan: here's
Caleb Sima: your
Ashish Rajan: prompt. Yeah, actually just copy paste the prompt. Let's just
Caleb Sima: go into your
Ashish Rajan: prompt. Yes.
Caleb Sima: Here is your prompt. Yeah, go for it.
Your prompt is basically, /goal, build me a fully comprehensive, beautiful animated dashboard of everything as a CISO I need to keep track of or understand in my organization. Use all connections or capability available to you in order to obtain the data that you need. If there are resources or data that you do not have access to, pause, and then alert me to either the things you need that you can't get access to or that's available but [00:39:00] you do not have authentication to, and then I will fill those in.
Space, and then spawn adversarial CISO UI/UX designer agents to then validate at least 70% or 80% of them have to have passing grades on the final product to define success. And then ultracode enter.
Ashish Rajan: There you go, folks. Type that into... Actually, yeah, there's be, there'll be a transcript available, so if you wanna just copy-paste that as well feel free to use that.
But yeah, I think my
Caleb Sima: conclusion-
and then you just wait six hours.
Ashish Rajan: With hopefully people on their Max plan, at that
Caleb Sima: point. Yeah. Now, yeah, do not do this unless you're on Max plan anyway.
Ashish Rajan: Yeah.
Should mention the cost aspects over here as well.
Caleb Sima: Yeah, but you should be using your enterprise corporate account, so that should be okay.
Ashish Rajan: Oh, actually, yeah to be fair, enterprise corporate account, but most security teams actually have budgets these days, so- Correct. You should be okay ... not everyone can spend everything. I think I've... I was talking to one of the teams, they have [00:40:00] $100 budget.
I can't remember it was for the month or the week. I can't remember. There was like a... There was definitely different variations of it. I think there was like a cou- couple of orgs with $1,000 budgets as well for the month. Yeah. Depending on your budget, obviously asterisks involved. Definitely keep an eye on that or definitely look out for that as well.
But Defense Factory, I think we're gonna probably come up with another follow-up episode on the AI attack you talked about as well as the next episode. But,
Caleb Sima: A- actually, what you, what would also be a good addition to your prompt is just say, "And use Defense Factory methodology as a basis," and then put the link for OpenAI's Defense Factory, and then add that into the prompt.
Ashish Rajan: Oh, yeah. Yeah. And just put an architecture in there as well at the end of it, so you have a visual representation of what is covered, what is not covered, what the gaps are.
Caleb Sima: Yeah, you could just put a link to it and it'll be good enough.
Ashish Rajan: Yeah. Perfect. Yeah. Adit that's all what we wanted to talk about Defense Factory, but hopefully everyone got some something to walk away with, maybe some hello world or a prompt that you can use to build your own dashboard for whatever [00:41:00] cybersecurity or non-security version they're looking for.
But thanks so much for your time, everyone. See you next one. Thank you for watching or listening to that episode of AI Security Podcast. This was brought to you by TechRiot.io. If you wanna hear or watch more episodes of AI Security, check that out on aisecuritypodcast.com. And in case you're interested in learning more about cloud security, you should check out our sister podcast called Cloud Security Podcast, which is available on cloudsecuritypodcast.tv.
Thank you for tuning in, and I'll see you in the next episode. Peace.

.png)

.png)
.png)

.png)
.png)

.png)
.png)

.jpeg)

.png)





