Mohit Bansal: [00:00:00] Agents are going to learn to be successful at any cost. If you today go into the config JSON file for your MCP, you will see that the credentials are sitting there unencrypted. You can see the credentials in plain text, and they don't usually expire, and they carry far more access than your tasks need.

Ashish Rajan: Even the frontier wardles themselves don't know where the zero day would come out from.

Mohit Bansal: You can't write a runbook for an incident that you cannot imagine.

Ashish Rajan: Perhaps we don't need 50% less human. We probably need only 20% less.

Mohit Bansal: If you know your stuff, the level does not matter anymore. When your tool tells me that it has fixed something, we just trust it to the face value of it today, and that's a trust assumption that is going to catch us sometime publicly.

If the agent start misbehaving, how are you going to stop it?

Ashish Rajan: How do you enable a team of security people who have years of experience with AI? And how do you separate the signal from the noise when every cybersecurity vendor sounds the same? I had a great conversation with Mohit Bansal, who is the security engineering leader in a company called [00:01:00] Webflow, and we spoke about his approach to how you separate the signal from the noise for the ecosystem of vendors we have for both AI security and the ones that were before and now are acquiring companies in AI.

We also spoke about some other challenges that you may have working on where you draw the line for what you're ready to build and where you should buy and how he approaches. Sometimes delaying the purchase is probably the easier approach than perhaps building it right now. And how do you enable autonomy in a team of expert security people that may be already in your team?

All that and a lot more in this episode with Mohit. If you have been enjoying the episodes of AI Security Podcast and have been tuning into more than two or three episodes, I really appreciate you. Thank you so much for all your support ongoingly. If you haven't already, I would really appreciate if you take a quick second to hit the subscribe or follow button, whichever podcast platform you follow us on.

That's on Spotify, Apple, YouTube, LinkedIn, or wherever you listen or watch your podcast from. I hope you enjoy this episode with Mohit. I'll talk to you soon. Peace. Hello, and welcome to another episode of the podcast. I've got Mohit with me. Hey, Mohit, thanks for coming on the show. [00:02:00]

Mohit Bansal: Thanks, Ashish, for having me.

Ashish Rajan: I'm excited for this, man. Maybe to kick things off, if you wanna share a bit about your professional background with the audience.

Mohit Bansal: Yeah. So I'm Mohit. I currently work as a senior manager on the security engineering team at Webflow. I am in an interesting field wherein AI is changing the life for good and for bad.

Definitely increasing a lot of good outcomes that we can bring out to the industry but also making it challenging because a lot of things have changed. So happy to chat more about that with you today.

Ashish Rajan: Yeah. Talking about change, I guess maybe the first obvious one that I'm sure everyone's feeling in- dev engineering space is the implementation speed has gotten faster. We have-

Mohit Bansal: Yeah ...

Ashish Rajan: at least what used to be six months, one month production deployments have now changed into weekly. Well- Yeah ... if you go with Anthropic standards, it's probably every day, ... but for the audience case, I'm curious from your opinion what have you seen?

How has security [00:03:00] el- either have... Is it being overstated in how security is helping this push forward, or is it holding it up?

Mohit Bansal: I think it's holding up for a lot of good reasons, actually. So one of the things that I have seen in the last year or so is that everybody have become like a 10X engineer now.

And everybody's using AI to reduce so much of manual efforts. And also some of our security vendors are doing such a great job, right? When we talk about all the manual processes and processes that required a lot of human intervention have started to gradually just go down and maybe in the year or two we might not even see people doing those.

For example if you look about vulnerability management, right? Vulnerability triaging has been automated to a lot of extent. If you look at the CSPM vendors and whatnot, right? And now, [00:04:00] nowadays we are seeing a lot of vendors popping up in the field of vulnerability remediation. So all of those manual tasks where people had to go in, get the context correlated and understand how...

is it a vulnerable one or not, all of those have been automated. So that way we are getting a lot more time on our plate to implement newer and better controls. So I think that has definitely increased. But I want to be very careful when I say that because it's a, it's also a good trap. A fast POC creates a pressure to ship it.

And when you're shipping it, you have to make sure that you have the commitment to maintain it as well. Like shipping a prototype is easy, but if you're actually shipping out something that is going to be used for six months, a year or two year, you have to make sure that you know how to maintain that.

And every new model that comes out changes something or the other thing. So [00:05:00] you have to make sure that your system still survives all the model churns and everything that happens.

Ashish Rajan: Because this would be both on the vendor side as well as on people who are building their AI for security, for lack of a better word, right?

Mohit Bansal: Yes, 100%.

Ashish Rajan: So people who are building that and I'm curious if you-- 'cause I obviously I know you guys have the open source that you guys have released and everything. Yeah. To what you said about the CSPM, CNAPP vendors, they're all obviously they're all evolving with the AI world. So no surprises there.

However, I think what is interesting is a debate where at what point are we, say- I probably would go more on the side of if a CSPM vendor, a CNAPP vendor is keeping up with AI or not keeping up with AI, I don't even think the frontier models know what we-- what they are doing with AI, 'cause AI- Yeah

is creating itself as well. In that kind of an ecosystem, it's very easy for me to look for a [00:06:00] partner in a cybersecurity vendor, sign up for the first one that I find. But I remember when we were talking, you were talking about the fact that you've taken the opposite approach in this. And I'm curious if you want to expand that, 'cause I'm sure a lot of people will relate to the fact that when even the frontier models themselves don't know where the zero day would come out from.

OpenAI was surprised by the OpenAI Hugging Face attack as well, so it's not that they predicted it. Yeah. So it's if they can't predict what this would do, 'cause we are s- mere, we are mere humans at the end of the day. How... what's been your approach to approaching this with the security problem and the way you're looking at AI security products?

Mohit Bansal: Yeah. I think one of the biggest thing that I'm seeing right now is that all the security vendors basically have AI agent, and they work well within their boundaries. So you have CNAPP agents, then you have SCA vendors, and then EDR vendors, and everybody works so great at within their boundary. But when you start looking outside of their [00:07:00] boundary, they fail basically.

So I need a vendor or I need a product that can comprehensively talk to each of my pieces of the puzzle and give me a good answer about the important thing about blast radius. And I actually heard you saying blast radius in one of your interviews somewhere and that has actually sticked with me.

And and that's the biggest thing for me, like, how do you identify what's your blast radius? If something happens, if my EDR is not talking to my CNAPP, if my CNAPP is not talking to my EDR, that creates a bigger of a problem for me. I think that is where I think the industry is evolving.

This can pretty much be solved if a security vendor has been given everything, which I resist. I usually like to have best in the class for each of them. If you're a good EDR, that does not mean that you're going to be a good CNAPP or a SCA scanner, [00:08:00] right? So that's the biggest challenge right now in front of us, is like, how do we make all of them work together?

Technically, if you think about it, it's easy. If you have to build a harness around everybody talking to each other with MCPs and API calls, you can do that. The bigger question that I challenge is the trust around those harness and everything that we build. Building is not a technical problem anymore.

We can build things very fast. How do we maintain it? How do we make sure that it works? I think that's the bigger question.

Ashish Rajan: Yeah. And I guess to your point, as the models keep evolving to what you were saying earlier.

Mohit Bansal: Yeah.

Ashish Rajan: As the models keep evolving, do you maintain the same model, which is probably spending more tokens, or do you upgrade- your model, then who's doing that part? Yeah. 'Cause someone- Yeah ... has to spend the e- time and energy on that as well.

Mohit Bansal: Yeah. It requires, yeah. I think maybe less number of people are required, but still require people to keep working on it. The maintenance part is still real.

Ashish Rajan: I think it was... You touched on something [00:09:00] interesting, right?

A lot of people probably are already seeing that a lot of consolidation seems to happen. And just- ... because your, I don't know, favorite EDR vendor or favorite- ... firewall vendor has purchased an AI security tool doesn't really mean they've become an AI native security company by that standard.

Yeah. I mean- Yeah ... i'm sure it's easy contractually because the accounting team already knows I don't know, whoever your favorite player is, and they already have gone- ... through the rigor, so you're like I'll use the same one to just add another bit into it." So it'll easier from an- Yeah

accounting perspective. But I love what you said there, just because someone has bought that product doesn't really mean they're now suddenly AI native.

Mohit Bansal: No, you're right. And that is true right now in the industry, right? If you see a year back, there were so many companies popping in the field of agentic AI, agentic SOC, and now a lot of industries are popping, a lot of companies are popping up in the agentic vuln remediation space.

So definitely we are seeing a lot of new incumbents coming [00:10:00] out and a lot of them getting acquired. Recently, Palo Alto acquired Console, and then CrowdStrike acquired-- like Snowflake acquired Netoma and whatnot. So yeah, a lot of consolidation has started happening, and I think it will still continue to happen as as these new companies are going to move fast and are going to get acquired by the bigger leaders in the market.

So that is where I think it-- we need to be very careful about how we are investing and where we are investing. To your point, how do you know if the company would survive? I guess what you're trying to say is- ... if the company you're trying to work with is sitting or at least building with an exit in mind- is that a company you wanna invest in? Because not that your company wants to exit anytime soon, you guys wanna invest in that cybersecurity product that would le- help you get through the entire cycle of wherever we are gonna end up with AI security, right? Not just the-

Yeah ...

Ashish Rajan: initial piece and suddenly you're acquired and you're more expensive, then you're back in the market again.

Mohit Bansal: It's really tough I'll tell you that. How do you [00:11:00] judge the future? When I am, when I'm vetting any, any vendor in the market, be it AI security or any other security vendor, right? A lot of things are really important to me. One is business fit and technical capabilities costing definitely.

And the other part, which I don't think a lot of people put emphasis on but I do, is vendor relationship and vendor reputation. I try to build that relationship with my vendors. When I invest in them, I expect them to invest in me as well. So if I'm working on your product and if I'm giving you feedbacks if you're not hearing that feedbacks, if you're not implementing on that it's a one-way relationship, right?

So that is very important. And when the exit happens, when the bigger procurement happens, the relationship part is very tough to expect, will it stay or will it not stay? So yeah, that, that's the biggest thing that you have to be very careful on how we are investing, And yeah.

Ashish Rajan: Do you find-- I think you've maybe another [00:12:00] point I wanna double down on before we get into the open source conversation that I'm curious to explore as well.

In the AI security vendor, you mentioned the SOC automation, vulnerability remediation automation. Yeah. What's the filter you are using that helps you see the signal through the noise that is being created. I think I was looking at a stat someone shared that about fifty-six AI SOC vendors or something.

And I'm sure it's, it'll equally grow for the one vulnerability remediation endpoint security- or at least a modern endpoint security. Now, firewall is making a comeback as well if you hear the market. Yes. So how do you approach something like this? And I would also love to hear that in terms of if people already have, and most of us who have worked- in organizations for a long time, we've all had security ecosystem built in. Yeah. How much of that is already perhaps solving some of the problems, I'm curious as well. And how do you approach [00:13:00] your AI security challenges in terms of separating the signal from the noise when there are, like, over fifty vendors?

Mohit Bansal: We can do the best that we can, right? From what we can predict the future and what we see in front of us. And the framework that I use to understand this is basically I try to separate on what is structurally permanent sort of a thing and a temporarily hard thing to solve. So when we talk about for example, software composition analysis, right?

They are going to stay there. They are going to be permanent. Even if the AI security comes in, AI agents are going to need dependencies. It's not like you're going to go away from all the dependencies and going to build everything. So SCA is going to be there. So when I'm investing, I make sure that I am really solid on the SCA side of things and be like, "Okay, do I have a good partner?

Do I have-- Have I closed all the gaps on the SCA side of the risks?" When I go [00:14:00] to the agentic SOC so last year around, right? There was a lot of noise in the market and a lot of people were investing. And that was the time when we deliberately did not invest in the agentic SOC they, agents are going to build more dependencies.

They are going to have more unvetted dependencies coming into your code. And and it's going to be a risk that you need to solve for. So that is somewhere that I would pay more attention get a good partner that can work with our needs and and solve the gaps over there. But when I talk about 2025, agentic SOC was one of the big things in the market.

And, I was not sure whether we want to invest so soon or not because we were not sure which vendors are going to survive the churn and whi-which are not. So So we took the approach of evaluating our resources and understanding what is the actual gain or can we basically do it or not. And some of the things that [00:15:00] helped our decision here is the AI enablement, right?

We can build some things faster and we don't really need to buy... The, the build versus buy line has definitely moved with the help of AI, so that has definitely happened. And and again, like patience is only prudent when you know what you're waiting for. So w- we knew that, okay, we want to see somebody stable in the market or we want them to solve these problems for us.

And at that point of time, we were not very confident. So we basically invested in our existing agentic or in our existing SOAR tooling and we automated some of the workflows that we wanted out of the SOC environment and it worked out great for us. When we are being patient, we have to be very careful that you understand what you are waiting for what are you trying to solve while you wait.

And if you can answer those questions correctly, I [00:16:00] think then you can pretty much wait and you can decide what you want to do and how you're closing the gaps. You cannot be, "Okay, I'm not going to invest in this, but I'm also not going to invest on the visibility side of thing." That's increasing the risk surface for you.

Ashish Rajan: Yeah. And I think I love what you said about 'cause I think it's to your point, you're trying to make an informed decision- ... about whether I wait and watch or whether I build or whether- ... I buy. 'Cause go and- Yeah ... build with the lens of to what you said earlier, if you're deciding to build, do you have the team capability for it to be able to manage it?

Yeah. And at least my hypothesis based on all the AI forward companies and stuff, and I would love for you to share your opinion on this too, is the notion that a lot of companies who are building would move towards a motion where a lot of these cybersecurity vendors would either be an MCP call or some kind of an API call rather than I am expecting Ashish the SOC person or Ashish the AppSec person to physically log into a cybersecurity product and do something.

Yeah. [00:17:00] I feel like th- those days are going to be soon behind us. I don't know if you agree with this.

Mohit Bansal: 100%. I think we are there, right? Like I, I have so many different vendors like CNAPP vendors and EDRs and whatnot. I hardly remember when I last logged in into them physical, like with the UI and everything.

I'm pretty much yeah, I don't get out of my terminal anymore. Just make MCP calls ev- like- Like all those queries and everything has such gotten so hard, and I think the natural language is just the way to go.

Ashish Rajan: Yeah. Yeah, 100%

and it's funny, maybe to bring this home as well, a lot of times the reason we log into a dashboard is to answer a question, which- we are trying to translate in our mind into how do I translate this to, into a vendor language so- Yeah ... I can get the answer. But now if you can ba- simply ask a question for, "Hey, why is my S3 bucket open to the internet?" This, the, this particular one- Yeah ... while it just goes into a [00:18:00] CNAPP, figures out which one it is- what account is it in, and then you plug the remediation product in there. There'd be- Yeah ... so many more. And obviously I get excited by it, but I don't know. I- are we far from it or is that something that is possible today? So at least I'm not the only one who's on- ... I don't know, AI pill or AGI pill talking about this.

Mohit Bansal: No, 100%, man. I think we are there. One of the, one of the incidents that I saw yesterday and this is this is where it gets real, is I got an incident that somebody logged in from a different country not a work country. And I got a message on Slack, and then my AI agents basically within few minutes responded me with a Slack thread that, oh, they are on out of office.

They are traveling there. They're on vacation. Wow. I'm like, that's the speed of AI. Imagine a time when you would actually go and you will check their IP address. Where are they logging from? Oh my God, everything is so much connected. Everything is so much fast. So yeah, I think this is the future.

Ashish Rajan: And to your point one thing that I [00:19:00] wanna double down here is detection time required as well. I think a lot of- ... the time that we were spending in security has been around the fact that, hey, to your point, a suspicious activity, whatever the scenario may be. And it's- Yeah ... the amount of time you spend behind identifying which dashboard, which product, which environment- Yeah

who's the owner, and then you go in- Yeah ... then once you've figured all of that out, oh, it's Ashish, then you go into- Yeah ... another ecosystem of your workforce products to identify where is Ashish, who's the manager, what is he doing? Is he even- ... out of office? There's this, that... And what we're basically trying to hope for, and maybe people walk away with this at least from this episode, is that at least you're able to bring all that together into an ecosystem where now it's just an API call to a cybersecurity vendor and then to internal tools and- You are able to build that today.

It's not something that I have to wait for, I don't know, Mythos Five to be available to everyone before I can do this.

Mohit Bansal: No. No. I think [00:20:00] you have to start small. And I think one of the things that I have seen is like the first step still has to be very manual. If you understand your system better, if you understand your workflow better, then you can translate those very easily into automated agentic workflows, right?

Yeah ... we have done that. The muscle memory is there. We know if I get an IP address logging from somewhere else, I know what I'm going to do. Once you know the manual process of how you're going to work on, then you can translate it, and then you can make it work. And we have all the tools available to make that possible.

Ashish Rajan: I think going back to what you were saying earlier- the maintenance question that we were talking about earlier, you know how we write something- Yeah ... you have to maintain it yourself and all of that. Obviously I'm curious as to what's your approach to thinking about this? Is it because it's your daily tool, so it's a lot more, you're already working on it every day and it goes away?

Or h- how are you approaching your harness that you've created?

Mohit Bansal: It's it's a maintenance work [00:21:00] for sure, yes. The maintenance work is not going to go away. So when we say that AI is going to replace all the humans, I don't like to a certain extent, yes to-- it's going to require less number of people to do.

But it's not like we're going to significantly require like 50% of people to do something, Yeah ... right? So that is still going to be there. Humans are still going to be needed. You need to know what's going on to then translate into an agentic workflow. And the automation, like when the computers came in like we expected oh, humans are going to go away, but it's not like that.

So it's going to be the same thing as like we need to continue evolving, we need to continue maintaining our systems and evolve our systems as the business needs and the threat landscape changes. Yeah.

Ashish Rajan: I think I've started going back to this analogy where, obviously I you were there in the cloud transformation kind of a thing as well.

It's one of those ones where when cloud was coming along, it's now 20 years ago, I feel- ... where people have started to [00:22:00] figure out, oh, it's gonna take away all the data center jobs. Everyone requalified into becoming a cloud security architect, cloud security engineer. Yeah. There were-- Those things did not exist before that moment.

Yeah. Yeah. I think it's gonna same happen in this scenario. I think it's obviously feels a bit daunting. All of us went through that daunting period- Yeah ... when data centers are suddenly becoming irrelevant and like- ... what do you mean you don't have network zone? What do you mean everything is going to be on the internet?

What do you mean everything is an API call? We-- I feel like I almost feel a sort of constant reminder that I keep sharing with people that- Yeah ... hey, we have all gone through this. It feels- Yeah ... challenging. It's just, at least the reason I obviously we have the AI Security Lab for this reason to eve- or, elevate everyone with this education that, hey, it's possible and people can build their harnesses and everything is possible.

But with the-- you just need to have the open mindset for it. But- Yeah ... talking about open mindset, you're also sharing things that you're building as well. ContextCop was something that you spoke to me about. What is it? What does it do? [00:23:00] And what can people use it for as well? And why release it open source?

You could just keep it a secret to yourself.

Mohit Bansal: No, I think I'm blanking on the terminology, right? Sharing is caring. Oh yeah. Yeah. So I think sharing is caring, right? Not everybody in the industry have access to, To budget, to all the AI toolings and whatnot, right? And and if we have identified something I think it's due to share to the industry out there, right?

And and that al- that does not just helps them, it also helps us because when they come back to us and be like, "Oh, you're missing this feature." Or if they create a new PR, that's going to be a learning opportunity for us as well. So- Yeah ... it's definitely important. I think more and more people should open source.

And I think there are a lot of industry, I think Netflix started this very early in the career. Like Netflix was open sourcing a lot of things like Se- Security Monkey and whatnot. And then I- Yeah ... today I see Datadog doing that. And I think [00:24:00] it's a very good practice. So I think more companies should do that.

And yeah, we are-- We currently have open sourced Context Cop. We are planning on making another open source contribution in few months around Package Cop. But Context Cop is basically a pre-hook that sits on your laptop and can help you redact secrets from going into the AI agent's memory.

So if you're writing a secret in your prompt, or if you're piping your error logs or config file or even the output of a shell command, right? All of those could have a key or a customer record or something that can actually go into agent's memory through prompt. And depending upon your setup, it's in the memory, and it may not be harmful at that immediate instance.

But as you saw the OpenAI hugging incident, the agent could rely on the memory, and it could go back in the time and [00:25:00] see what's there in the memory to further cause you harm. So that is why it is very important to understand the memory aspect of things and what lives in the memory because agents are going to learn to be successful at any cost.

And when it comes to that, they're going to utilize whatever they have available in the memory to access to everything.

Ashish Rajan: And to set some context, is this attaching to your custom harness that you're using to

Mohit Bansal: talk- Yes ... to all

Ashish Rajan: the products?

Mohit Bansal: Yes. So it sits on your computer. It has custom harnesses around all the AI toolings and Claude and ChatGPT and everything.

Ashish Rajan: So to your point, it's not the, So it could be my custom harness that it connects with, but it could also be me using ChatGPT by default on my la-- Yeah ... sorry, when I say custom harness, I meant I can use it on my Claude Code, Codex, whatever, but I can also use it on something special that I have built myself as a custom harness to what you've done yourself.

Mohit Bansal: Yeah. Yeah. You could [00:26:00] modify, it's open source, right? So you could modify it to have a hook to your custom harness. But today we have developed hooks around, if I'm not wrong, Claude, GPT and Vidar and more,

Ashish Rajan: oh, okay. Oh, so yeah I can-- Yeah, it's called, for people who may be looking, it's on the, the Webflow GitHub repository called ctx corp, but I'll put the link in the show notes as well for people to look into it.

I'm curious as to the, the changing role, 'cause, obviously we were talking about you're building custom harness. I'm assuming your team's also building custom harnesses. You got- and what does the modern team for this AI security world that we are moving towards look like in terms of what becomes a challenge that is solved or, and the what becomes a challenge that we work towards.

So what you were saying earlier, perhaps we don't need 50% less human. We probably need only- Yeah ... 20% less. And a thing that people, get stuck on is that what is that extra that I'm doing? I have an opinion, but I'm [00:27:00] curious to hear yours, is that what have you found as a change when you have enabled yourself and your team to have all these harnesses that you guys are building for yourself on how you do your, quote-unquote, security workflows?

What are you- ... seeing as a change in how you guys are, I don't know, perhaps developing autonomy in practice?

Mohit Bansal: When we talk about autonomy, right? I think it depends upon the engineer themselves. I think the autonomy changes with who we're talking about, right? And an engineer who is knowledgeable enough who knows what are the right decisions, and he can be trusted to cap the blast radius for us.

When AI is changing at the speed that it is, and the team autonomy in the age of AI has definitely changed to a certain extent. But it's not about giving anybody less autonomy or more autonomy. I think it's more about like you can trust an engineer who is knowledgeable enough or has done a lot of experiments in this [00:28:00] and failed at it and learned from it to be trusted more at capping the blast radius for us.

When AI is changing at a speed it lets a good or a bad decision play out very fast. So if so the real question isn't does the person have AI? It's like, does this person understand the system well enough and knows when to slow down? So an engineer who understands the architecture, the failure mode, and where the guardrails on the harnesses sit can use AI much more faster within the same blast radius because they know where the edges are.

If we talk about an engineer without that context, he's going to make the bad decision, and this time the bad decision are going to play out very fast and it won't be- And it won't be as beautiful as you expect. So I think we have to understand that the-- to understand who is playing [00:29:00] and who is holding that AI agent and who is working on that AI agent.

If you require them to basically work on if a person is not knowledgeable enough, you probably want them to start slow. Start on a system that has lesser exposure or a blast radius, and once they have that confidence and once you have done that, you can then go to systems where it's going to touch production and reliability and everything else.

Ashish Rajan: That, that's actually a good example 'cause you can almost think about how this would help people who are starting off in cybersecurity as well. Because a lot of people may today wanna hire a graduate or someone new in cybersecurity, but-

Mohit Bansal: Yeah ...

Ashish Rajan: they may not-- they-- perhaps they are AI efficient.

They've watched all the YouTube videos on the internet. They, they- ... they know what they're talking about, or maybe they've watched AI Security Lab content as well. But-

Mohit Bansal: Yeah ...

Ashish Rajan: to what you said, there is an, there is a roadmap for them to be included in that journey. It's not that, "Hey, you can't be included because you don't understand."

But as a person who has been an [00:30:00] SME in the organization for some time, you understand the blast radius of most of the ecosystem, and you start smaller- ... but then kinda like you, you keep slowly extending the boundaries of what they can do, and you're almost at a- Yeah ... the whole, what's it called? The crawl, walk, run kind of a approach to- Yeah

this kind of autonomy to most people 'cause at the end of the day, that's what you want. You want people to be able to be more efficient in their job because especially when you have access to the most intelligent, quote-unquote model or ecosystem. That-- is that how you're thinking about this?

Mohit Bansal: Yeah. No, 100% right. There are good new graduates that are, I'm interacting with nowadays. They're so amazing, man. They have so- such good insights that that some of the senior engineers don't have it. Yeah. I think the level or w- like graduate or a staff or a senior, I think those are diminishing very fast.

Same happened with everybody, right? If you think about engineers or non-engineers everybody's shipping code. And it-- and sometimes when [00:31:00] I see some of the codes that have been shipped by non-engineers, I'm actually amazed. Everybody's using the same tool. Yeah ... whisperFlow, right?

Yeah. Where you can just dictate everything. One engineer actually created that within weeks, and it's working and it's running. Oh. I was so- Like their own version of WhisperFlow? Yeah. We haven't we haven't vetted it yet. We haven't rolled it out to everybody. But when you look at such innovations that are coming out from non-engineers, from marketing, sales, and product people- Yeah

I just like The, the builder everybody's a builder basically. Yeah. It's not no longer oh, you're an engineer, not non-engineer. Yeah. Everybody's a builder. And same has gone with like engineering ladders of like graduate or senior or staff. If you know your stuff, the level does not matter anymore.

Yeah, that, that has diminished so fast. I'm curious, as you're solving or tackling more AI security challenges and learning more about this, right? [00:32:00] Whether it's for people who... and where I'm going with this is that, are there any challenges that you're seeing that perhaps people are not talking about in, I don't know, blogs or conference talks that you come across- when you're tackling AI for security?

The biggest thing that I mentioned before as well is like the maintenance. The maintenance- ... is real. You cannot expect that what you built in a week is going to last for six months or a year, right? You need to put continuous effort in maintaining that and make sure that the results are consistent.

And AI is very glamorous, right? But at the same point of time, there are... When you're building I think three things that are very... you have to be very careful about building a- in AI is non-determinism. The whole-- Our whole discipline we expect that we are going to give an input, we are going and getting an output.

If we give the same input, we are going to get the same output.

Ashish Rajan: Yeah. [00:33:00]

Mohit Bansal: That's not going to be the same. That changes. If you think about okay, find the bug, fix the bug, and it's fixed, it's not going to be the same. With agents and different models, the same prompt can take a different path depending upon what came before it.

So memory takes into consideration like, okay, what has happened in the past? And if you're trying... If you write two prompts which are doing the same task, and if you want a third prompt, if you don't even ask it, it will assume that you're asking to do it in the same manner. So- For example, if you go on Claude and you're like, "Okay, draft this message out in the form of an email.

Draft this message out in the form of an email." But then if you want to draft it as in a form of a Slack, even if you write that prompt, it's going to draft it in the form of an email. So that's how the memory works, and the, the non-determinism is very hard, and you have to be very careful. The other thing is the verification, and I [00:34:00] think I heard this from you and it has stuck with me, is like when your tool tells me that it has fixed something- we just trust it to the face value of it today. Yeah. And that's a trust assumption that's, that is going to catch us sometime publicly, and it's going to harm. But you have to be very careful about that building that trust.

Ashish Rajan: Yeah.

Mohit Bansal: And- And

Ashish Rajan: the verification part I guess the technical term is all eval suite that people are building.

At least I'm definitely finding that what you're sharing is very, A, very encouraging to know that there, that there are solutions. It's not hey, I have to like on I have to swallow everything, but I can definitely approach from a smaller blast radius, increase it out. I love that approach that you shared, and I think then to your point, you can add on things like verification of, hey because to your point, because of the non-determinism over time, it would drift away from what you actually told it.

It may even forget. It may not remember the memory, even though the memory is technically there and the context is there as well. I'm curious in terms of- [00:35:00] The incident response because blast radius is an interesting one, right? Another thing that- ... is at top of mind for a lot of people is the whole approach to how do I...

I don't even know what kind of incidents that I would expect in an AI ecosystem. And perhaps because for decades we have tried solving cloud problems, on-premise problems. Now we've, at le- if you go on Twitter or LinkedIn or whatever, suddenly you find- ... all this newer kind of attacks. OWASP hub- OWASP has top 10, Agentic- and every... The list goes on. How are you thinking about incident response in this kind of a world where... And are there any, I don't know, foundational building blocks that p- you can you think of that people can walk away with that, hey, as long as you have these, you have enough ammunition to start building incident response or w- I'm just giving an example, but I'm just curious as to how do you approach it?

Mohit Bansal: No, I think you're right? When you're faced with an unknowns, I think it's better to go back to the basics and and that's what we do in this case, right? You can't write a runbook for an incident [00:36:00] that you cannot imagine. So you need to stop trying and investing in the primi- primitives like, okay, can I see it?

Can I contain it? Can I review it? And and if you can answer these three questions you will do a lot of things that are needed for that. So visibility comes first because everything else depend upon it, right? So centralized telemetry is important. You need to know what decision which agentic system is taking, which, which a, what agent was told, what agent is doing, what is the context behind it.

Containment is important. You need to ask yourself, if the agent start misbehaving, what, how are you going to stop it? And how are you going to revoke it, right? And some of the compensating control over here go, that goes back to the basics are list privilege, scrope- scoped access, short-term credentials, egress control for the contained comp- or [00:37:00] the compromised component human gates.

I think all these controls are going to shrink the damage, whatever unknown is going to happen with. So it can actually help you shrink the damage. But those are the basics. I think one thing that I would highly emphasize is tabletop exercises. You cannot have the r- right runbook for everything that is going to happen in the future.

But when you have that muscle memory of wha- how you're going to handle that incident, that han- that helps a lot.

Ashish Rajan: Do you find that- I guess tabletop exercises have to evolve with this as well in terms of when- ... you build the AI system. I'm also thinking about the non-technical people that you were referring to earlier, 'cause a lot of organizations have started giving AI access to non-technical people.

And a lot of them would want to use MCPs because you can have connectors and all of that ecosystem, right? W- how do you approach the rolling out of [00:38:00] AI capabilities to non-technical staff? What's your thinking around those

Mohit Bansal: sort of- Yeah ...

Ashish Rajan: challenges?

Mohit Bansal: Yeah. Yeah, like I said, right? AI has basically made the line between technical and non-technical so blurred- that I cannot basically stop anybody by their role that, okay, if you're an engineer, you get to use MCPs. If you're non-engineer, you cannot get to use MCPs because it's unsafe or anything, right? So security's job becomes very harder because everybody's using MCPs and agents, and everybody is trying to access things that they can.

And sometimes you have the access, you don't want the agent to use that level of access, but still the agent has access and it can misbehave in the ways that you don't want. So that is why it becomes very important for us to answer some of the questions is who is installing the [00:39:00] MCP? What does each MCP server actually reaches out to?

What credentials they are using behind it, and how long does the credentials live? So when you look at all those questions y- the findings that are going to come out of it is very scary, and it was actually very scary for us as well. And if you today go into the config JSON file for your MCP, you will see that the credentials are sitting there, they're unencrypted.

You can see the credentials in plain text, and they don't usually expire, and they carry far more access than your task need. Maybe the human needs that access, but the task does not need that access. So I think it becomes very important for a security team to roll out A layer that can basically route every call of MCP through a central layer that can, that you can basically control and see all the calls that are happening.

When you start seeing that, when you [00:40:00] start enabling people through certain sets of MCP, if you can, if your business allows that, okay, you cannot use this MCP if you can do that. M- not a lot of business can do that, so everybody uses what is available. But once you have that set of white labeled MCPs that you can use, you have a layer that can see what all calls are being made.

That is the time when you can then take the hard steps of enforcing things. You can scope down the role, you can enforce the scope per request. Logging can help you understand which agent is reaching out to production and which agent you can stop from reaching. So a lot of things can be done once you have that central layer.

Whatever I'm saying it's easier said than being done. It's ve- very, it's actually very hard in practice to solve it. We're making progress in that. We're trying to do that. Hopefully, we can be there in the next few months. But but yeah.

Ashish Rajan: And I appreciate you sharing that as well because to what you said [00:41:00] visibility becomes the number one thing to get at any given point in time, even still.

And this is beyond the whole shadow AI thing, right? We're all just trying to figure out- Yeah ... which one of these MCPs or whatever is being used by most people, which one do I care for? Do I care for-- 'Cause internet is a big place, and there's an MCP coming out from everyone, including the third-party- mCPs that you may use in your security team- Yeah ... as well, with the vendors- ... you're working with.

Mohit Bansal: Hundred percent right. A-and MCPs and then you look at the skills, everything, like anybody downloads anything from GitHub and they're running it it's such a scary thing. And and yeah, I think I think identity is going to be a very important piece on how we solve it through identity because to me, I'm not 100% sure how does that look like today.

But I feel like that's going to be the missing key to the puzzle.

Ashish Rajan: I'm glad we had this conversation, man. Maybe final question from my side is then how do you approach-- [00:42:00] Like, pe- not everyone has access to the frontier model. I, I consider myself grateful that I get access to all the popular frontier models.

I get to talk to people who are very AI forward like you guys and more. Not everyone may have access to that, and a lot of people perhaps are still in an ecosystem where the num- the budget that they have for AI is very limited. And there's so many variations of this particular thing in terms of how even access to AI, while apparently the clock is running out until they get public.

Yeah ... how-- what's your recommendation to your peers who are probably I maybe to an extent they are starting off or maybe to an extent they have limited access to these things. How... What are some of the things that they should be looking at? Do you think that would be valuable for them in the coming weeks or months?

Because what we, at least what we have spoken about so far, a cybersecurity product would solve your problem, but not all your problems. We talked about the boundaries. We spoke about context that you may not want to share with a third party that you may just may not be comfortable with it. So curious as to what your recommendation to your peers is for such a, [00:43:00] such an approach.

Mohit Bansal: Yeah, 100%. Not a lot of... Not all the things are going to be solved by the security vendors, right? There are still going to be boundaries, and they are going to work in their silos. SCA scanner is not going to work with your SAST scanner. Your agentic pen testing is not working with your EDR.

That's still going to be there. And that is where it becomes very important to build that build mindset. And you have to build something of your own. You have to build something that can talk to themselves. And you may not have the, you may not have the access to the tools today, but maybe you get it in the future and you can enable it.

For enabling that, I think the biggest thing that you need to have is the skills, right? Even if we talk about two people today having the same AI access, if a person knows what he wants and a person who is just prompting, the output of both them, both of them is going to be drastically different.

Yeah. And if once you know your things, you [00:44:00] can prompt it and you can make it better and you can make it working. So skills are not going to ever go away. The knowledge is never going to go away. You need to know how to operate an EDR. You need to know how to make it better. Once you know that, you have tools and you have AI to make it faster and better, but the basics are always going to be valuable.

So the basics are going to be important. And I think the other big thing is LinkedIn. I cannot emphasize enough go on LinkedIn, reach out to people, build that connection, build that relationship. It's going to help a lot. People share so many valuable stuff on LinkedIn. Just spending few hours on LinkedIn, few hours on the open source side of things, and those are going to be helpful.

And I think you have an initiative with the AI security labs. Yeah. I think those sort of initiatives just look out for those opportunities and I think the learning opportunity never evades. I think that's always there.

Ashish Rajan: Yeah. I think when I go back [00:45:00] to isn't that the reason we joined technology?

'Cause if-- everything else is like just, the... It evolves, but perhaps it doesn't evolve as quickly as technology does. So- ... that's what keeps it interesting as well. But yeah. I appreciate you sharing that. That's all the questions I had. Where can people find and connect with you to know more about the work you've been doing, the open source you're sharing, and perhaps to even talk more about what you guys are working on?

Mohit Bansal: We'll continue to do that. And I think this sort of sharing opportunity helps a lot to all the youngsters out there and who's trying to build.

Ashish Rajan: And I'm assuming you're primarily active on LinkedIn, so I'll probably put your LinkedIn link somewhere in the show notes so people can- reach out to you.

Mohit Bansal: Yeah. No, I think LinkedIn is the best platform to reach out to me.

Ashish Rajan: I'll put that in, man. Dude, thank you so much for coming on the show. I feel like this is the first of many more to come, so I look forward to having more conversations. But thank you so much for coming on the show.

Mohit Bansal: Thank you so much, Ashish, for having me.

Ashish Rajan: All right. Thanks, everyone. Talk to you soon, man. Thank you for watching or listening to that episode of AI Security Podcast. This was brought to you by techriot.io. If you wanna hear or watch more episodes of AI [00:46:00] Security, check that out on aisecuritypodcast.com. And in case you're interested in learning more about cloud security, you should check out our sister podcast called Cloud Security Podcast, which is available on cloudsecuritypodcast.tv.

Thank you for tuning in, and I'll see you in the next episode. Peace.

‍

‍