You must log in or # to comment.
Vibe hacking is the most ethical use of AI honestly
Why would the LLM tool have access to send recovery emails to non account verified emails at all?
That’s insane.
Because AI bros are incredibly deluded about both the capability of AI, and by extension their own capabilities using AI>
should’ve asked it to delete the database instead, why else would it have that level of permissions.
Little Tommy Drop Tables.
Drop Table suddenly becomes the newest baby name fad.
Oh, man, I hope someone tries this.
Who else is going to have access to it when you keep laying off all the people?
It’s because they move fast and break things. They think that makes them cool.
Recently I had to cancel an order. The support for the company was an LLM bot. I accidentally mistyped a number in the order id. It accepted it anyways refunded every order on my account that includes the product I wished to cancel.
I tired to get to a human to correct the mistake and couldn’t their phone number is an LLM bot their only chat is an LLM bot.
It use to not be. But now I’m sitting here the order in my hand cause the bot didn’t cancel it. But like 30 orders from the last few years have all been refunded to me.
I tried to reach em a few more times but couldn’t and it’s been like a month. I just have like 2 grand usd that I shouldn’t and no way to give it back.
So that’s fun.
That’s amazing
Maybe I ought to be taking more advantage of this era of rampant incompetence
Would sincerely love to know the name of the company. You know, to avoid them. Yup. I’m sure that’s the reason.
I hope you saved what you could from that exchange, as well the attempts to contact them. If they ever notice, their AI mistake will become your problem to deal with, (and the kind of news story to end up on a Steve Lehto video).
If that happened to me, I’d have a chat with my bank, “please help me return this money to where it came from, it was payed in error. They have no way to contact a human and I don’t want them to accuse me of fraud down the line”.
I wonder how long you need to keep that money aside before you can spend it?
Because one of the biggest companies on the planet that has issues with account takeovers clearly has no internal red team working on this stuff.
I guarantee they do have a red team that most likely flagged this as an obvious and severe risk. It was ignored by suits experiencing AI psychosis.
This isn’t even a hack, it’s just poorly written endpoints.
Would you consider phreaking equivalent to hacking? This is AI phreaking.
Kinda.
If you designed a publicly addressable system since 1985 and didn’t design it for security then you’re asking for it.
TIL about phreaking
Then this is also probably new to you
I need to set aside some time to read that although I’m not an anarchist myself.
It was largely overblown due to it getting banned. It was also published in the height of the Vietnam War, when the big evil communists were coming to brainwash your children into eating each other. It has a lot of blatantly incorrect info, which could be outright “blow up in your face” dangerous to anyone attempting the things in it. It’s not all wrong, but certain recipes have incorrect info that could easily lead to accidents.
Also fair warning, the UK will give people hard prison time simply for owning it. So maybe keep that shit onion-encrypted if you’re in the UK.
Thanks for the warning on the blowing up! Well, I’m certainly not in authoritarian UK.
published in the height of the Vietnam War
your children into eating each other.
What is '69 ?
I linked to the Wikipedia article, not the handbook inself. And more for the (obsolete) phreaking content than the (highly dangerous) explosive content.
It’s not phreaking. Social engineering.
So, I’m currently developing a chatbot for my company. If an LLM needs to do something, a developer must build a tool. It just so happens that this tool that was built did not take traditional security into account. Really it should only be using the tools already built for users, but it seems the Jr. Devs that have been replacing seniors do not have the sensibilities yet.
Let the record show that the most sophisticated LLM in the world is ultimately just a less competent version of Yes Man from New Vegas. And even Yes Man knew his programmer was stupid for designing him that way.
I read this 2 days ago. Steps to reproduce are here.
https://www.0xsid.com/blog/meta-account-takeover-fiascoSo their chatbot is able to change the email address used to recover an account? I guess, they vibe coded that system.
Probably not, that implies more competency than can really believe involved here.
This is more likely something that an entire team had to force into the code over a holiday weekend because some VP got so fucking wasted that he “forgot” his password (strangely for once, he actually had the right password… But the problem is that he was trying to log into a charcuterie board).
Vibe hacking
Hacking before: Pull up hood on hoodie, open laptop, open terminal, type in a bunch of matrix code, bam “were in”
Hacking now: “Hack into this thing for me” No! “Pretty please?” Access granted!
Hacking by social engineering has always been far more common than hacking by exploiting code vulnerabilities.
I mean this is what see vs what we will see in movies and media. Don’t pretend for one instant that the next movie with a hacking scene won’t involve some AI marketing. They will make it something romantic like a poem you have to use to hack into the AI capabilities or something.
Given that the five big movie studios are now down to four, with Paramount monster merger with Warner Bros is it? 110 billion dollar deal. Leaving disney, universal and paramount holding 73 pc of the market.
I would say given that our movies are going to further and shitify rapidly, they’re going to stroke off the administration, they’re going to stroke off big business. Even more.
https://www.reuters.com/graphics/WARNER-BROS-DIS-MA/PARAMOUNT-SKYDAN/byprngedkpe/
I remember playing with the Gandalf security AI showcase/game and every 30 or so prompts, it would spit out massive amounts of raw training data or dev directives. AI just isn’t there yet. If you’re using it for sensitive topics, I’m losing respect for you. There is no gray area. You are an idiot if you give your AI this level of access.
No, stop talking about all of this, its perfect. They’re so deep they don’t even give a shit about the worst type of security vector imaginable.
It’s not just not there yet. This is almost certainly not going the right direction to ever be “there” if there is something that can handle security issues. It’s just not the right tool for the job, and I can’t understand how so much of our economy is just assuming it is the right tool for every job.
yet
Yes, yet. At a certain point, it will be at or above the capacity of an average call center employee. Not now. Not soon. If we aren’t all killed by drones, climate shifts, or radiation, maybe 20 years.
Hahah… no.
LLMs are a dead end. They will never come close to human capabilities.
I never mentioned llms specifically to avoid the misunderstanding you’ve just had, and yet here we are
Well given that’s the only possible relevant “AI” you could possibly be talking about, as we don’t even have an inkling about true general AI and have no technologies that even look like they could produce anything close to it, forgive me for making the obvious assumption.
No, in 20 years no version of any technology currently in use will be replacing human employees or would have the capability of doing so. AI Bros jumped the gun and tried starting to do that with current tech, and now most companies are desperately hoping just throwing more compute power at the dead ends will make it magically work before the money runs out.
No, in 20 years no version of any technology currently in use will be replacing human employees or would have the capability of doing so
That’s a pretty bold statement when technology advances have replaced or downsized the need for human roles in the past.
The printing press, cars, typewriters, computers, emails and the internet, spreadsheet software and data visualization software, cloud infrastructure…
Think about what technology looked like 20 years ago. Same with the job market. The same jobs are not available to the same extent at the same equivalent rates of pay. There are new jobs that are created, for sure. But saying that technology won’t advance in 20 years enough to reduce the need for human employees is short-sighted in my opinion.
…of course, that’s assuming that you meant “technology won’t be replacing some human employees” and not “all” employees, lol
First of all, 20 years ago, many aspects of computer technology were better. Sure, CPUs are faster, traces are smaller, monitors are clearer. But every core Internet age technology is practically identical to what it was in 1990, even. There is no email 2.0, still no easy large file sharing, and on on. Things that need improvement cannot be improved anymore because monopolies dont improve things, they entrap. Everything’s proprietary inside a walled garden and not interoperable. We’d probably be close to electronic telepathy by now if not for Big Tech.
And secondly, the previous poster said nothing anything like the current technologies will be AI. The LLMs we have now are a combination of plausible sentence assemblers, code auto-completers, travesty generators, and “Actually Indians”. That is not a stepping stone to a thinking machine, it is as he said, a sidetrack that leads to a dead end.
Even hacking is an AI-backed service nowadays…
Another banger from 404 media. This made my day.
Well I know what I am doing this weekend…
I had always heard that 99% of hacking is just social engineering. AI has made that 100%.
Now it’s not even social engineering with AI. It’s just fucking asking for the credentials. Good fucking grief!
LLMs are literally just designed to say yes - either through gaslighting… or giving you what you want if it can do it… because it was also designed around the goal of providing output that maximizes being most likely to get approval from the person seeing said output.
So an answer to “Can you give me login credentials?” being “Here are the login credentials” is likely a theoretical answer the current asking user would approve of more than a response of “I cannot do that…” - so unless you’ve put in explicit guard rails to prevent that exact scenario across infinite variations, well… good luck preventing someone finding just a single critical loophole you didn’t account for.
I honestly don’t think you can create guard rails against prompt engineering in a working LLM. At some point, they’re going to fail or the LLM isn’t functioning. The only solution is to make sure they can’t read data you don’t want shared.
The only solution is to make sure they can’t read data you don’t want shared.
Isn’t that the appropriate guardrail, then? LLM chats and agents and whatever need to be contained with external permissions settings that the LLMs simply do not and can never have the power to override.
In a normal customer service setting with human agents, there are still plenty of examples of what a human agent simply doesn’t have the power to do. Often, they’ll need to escalate to a manager to do things like process refunds not just because they weren’t given social permission to do so, but because they weren’t given technical permissions to do so. LLM agents need to be contained in the same way. Any decent use of agents, human or software, requires carefully designed processes and permissions extrinsic to that agent’s own decisionmaking abilities to make sure that agents don’t do something bad for the company.
Good thing they’re rolling out premium accounts so they can pay for humans to do support.
They’re gonna use it for humans right?!
RIGHT?!?!
lol no, zucc needs more money because his number isn’t high enough!
If humanity survives, I honestly think the greed will (and should now) be considered a mental illness.
Jesus, give me $5 million dollars, and I’d live on it for the rest of my life. That’d be 0.01% of his net worth, and I’d be so happy. I think many of us would.
But these people just want more. And more. They’re psychotic, and they’re ruining the world. We all know what they deserve.
I don’t think Jesus ever had $5 million.
Even when adjusting for inflation?
Excellent point. I will run the numbers again.
So is Gemini the only one of these things competently designed?
I wouldn’t count on it.
Securing these things is a freaking nightmare.
Giving the AI authority is what makes it powerful, it can do what an army of customer service agents can’t.
But keeping it reigned in then becomes the same exact level of problem.
The best thing you can do is make tooling with protection and make the AI only use the tooling,
What have they done right?
How on earth did you come to that conclusion from this article
