loose wire | blog – Page 6 – Jeremy Wagstaff: ex Reuters/BBC/WSJ etc. Now consulting and writing. Blogging here since 2001

GPT: Where are we in the food chain?

By jeremy | April 4, 2023

On November 29 2022 I implored the tech world to bring on winter: We’re out of good ideas. I should have kept my mouth shut: The next day ChatGPT was unleashed on the public, and we haven’t paused for breath since. I believe we users need to contribute more to the debate and figure out where we stand, and what we stand for.

The mad rush to compete in this space means the big players are rolling their AIs out not only before they’re ready, but silly money is being thrown at startups promising exploitation of these tools. A massive land-grab is taking place, with little idea of the consequences and with the ink of some 155,000 tech layoff slips barely dry.

I wish I could be more sanguine. I’ve always loved technology, and I am absolutely bowled over by the latest iteration of ChatGPT, GPT-4. Everyone else has been writing about their experiences with it, so I won’t bore you with mine, but there’s no question we’re in Kansas anymore. This technology will change a lot. A LOT.

But we need to keep our eye on the ball. Some have called for a moratorium, which is at best naive and at worst giving the industry too much credit for a sense of responsibility. That’s not what is going on here. It’s money.

The ball we need to keep an eye on is market (and political, but we’ll leave for later) power, and we should be watching it carefully as it shifts. It doesn’t shift far, but it is shifting. We are not witnessing disruption in the sense that Clayton Christensen defined it, we’re seeing a further agglomeration of power from those lower down the pyramid to those at the top.

Peek behind the curtain of all this GPT magic, and what do we find?

There are, for sure, a lot of really bright people doing cutting-edge stuff. But behind them are thousands, possibly hundreds of thousands, of contract workers labelling and annotating the data that is fed into the software. The Transformer-type models we’re talking about are essentially trying to predict the next token (think ‘word’) in a document, drawing on data. That data has to be prepped for the algorithms and that means annotating, or labelling it.

So this process is automated? Actually no. The data still needs to annotated, to prepare it for the algorithms. The method involved is called “reinforcement learning from human feedback”, where model responses are ranked by quality, and then a reward model is trained to predict these rankings. As per the term, this is done by humans, and is a very labour-intensive process. This is how GPT-4 described it to me:

The process of collecting comparison data and ranking responses can be labor-intensive and time-consuming. By collaborating with outside contractors, organizations can scale their data collection efforts and obtain a diverse range of human feedback, which can help improve the performance of AI models.

This “collaboration” (clearly GPT-4 has a sense of humour) is done by contractors, “flexible contributors” or “ghost workers”. The biggest company doing this is Appen, which has on its books more than a million of them. After some protest those working on behalf of Google saw their rates rise to up to $14.50 an hour. Compare that to the average base salary of a Google employee of $125,000.

The data

And what is the data they’re annotating, exactly? What is in the datasets being used to train these massive language models is a mostly black box, since it’s considered commercially sensitive. Researchers from EleutherAI concluded that

Despite the proliferation of work exploring and documenting issues with datasets, no dataset intended to train massive language models has been seriously documented by its creators ¹

But these aren’t quite the hallowed corpora you might imagine.

The data is for the most part the web. They have just been parcelled up into larger datasets, such as The Pile, an open source dataset of a (relatively measly 800 GB). And there’s MassiveText, 10.5 terabytes, which is private. (When I asked GPT-4 for a list of the biggest datasets, MT wasn’t included, because GPT-4’s data ends in September 2021, illustrating how new some of this stuff is.)

And what is this data, exactly? Well, it’s actually what you and I produce in our daily lives. It’s social media, webpages, news, Wikipedia pages, books, Youtube comments (and possibly transcribed content). Pretty much anything that we do online.

One paper² estimated that up to half of the content in these so-called high quality datasets — high quality because they’re real sentences, with real context, etc — is user content scraped from the web. Books and scientific papers make up for up to 40%, with code, news, Wikipedia making up the rest. In other words, our musing, utterances, the journalism we write, the Wikipedia pages we tend: all are sucked into datasets that then, eventually, become the answers that ChatGPT or Google’s Bard spew out. Wikipedia, to give you an idea, weighs in at between 43 GB and 10 TB, depending on what you’re including.)

Unsurprisingly, there will inevitably be charges of plagiarism. My prediction, though, is that we’ll get better at identifying when GPT regurgitates existing material and tweaks it to try to hide it — it’ll be an escalating war of technology which will end in class lawsuits and significant legal hazard for some.

The other cost

So once the data is marked up, the algorithms need to do their work. And this is where things quickly get beyond the reach of scrappy startups. GPT-3, for example, is estimated to cost millions of dollars to train, and to run. And that’s just the processing. You also need the infrastructure.

Plugging GPT into Microsoft’s search engine Bing requires 20,000 8-GPU servers, meaning it would cost the company $4 billion. Reuters (hat-tip Gina Chua) quoted SemiAnalysis as calculating it would cost Google, sorry Alphabet, some $3 billion if they added ChatGPT-style AI to their search.

The scarcity

So where are we going with this? I’ve expressed elsewhere my concern that the biggest danger from these innovations is that they’ll be harnessed to manipulate — in other words, that the information they contain and the process they use to deliver it are best viewed as weapons of disinformation.

But just as likely, I believe, is that the competition currently underway will face constraints that in turn cause market players to turn to more drastic measures to remain competitive. In other words, that technology will evolve in the same way that search and Web 2.0 evolved — turning the user as much into a willing provider of valuable data as a consumer.

Here is a hint of what may come: The models themselves might be — possibly already have been — turned on our data that legal protections have worked hard to keep anonymous. Researchers from Romania and Greece used GPT to see whether they identify text of famous people from anonymous data. They found that in 58% of cases they could. Their conclusion:

[W]e believe that it is only a matter of time before organisations start using LLMs on their documents and realise that this way, not only can they get more visibility about their customers, but they can also deanonymise documents revealing information that would be impossible for them to do so.

Another concern is that GPT models are running out of source material — data. One paper estimates that what it calls ‘high-quality language data’ will be exhausted by 2027, if not earlier. This in spite of language datasets growing in size . The paper concludes:

If our assumptions are correct, data will become the main bottleneck for scaling ML models, and we might see a slow- down in AI progress as a result. ³

I’m sure something will come along to fix this. LLMs will become more efficient and require less data, or so-called synthetic data — data not derived from the real world, but from a virtual world — will develop to add to the sum of available datasets. (Gartner believes that 60% of all data used in the development of AI will be synthetic by next year.)

This might be fine, or it might not. The problem with synthetic data is that it’s not real. It’s not human and so while we, for all our imperfections, at least create a data exhaust that’s real, synthetic data is a simulation of that. And while it might work for programming autonomous driving, questions should be asked of its usefulness for training GPT and LLMs. This may create a premium for real, human, data that makes it impossible for those companies once committed to maintaining our privacy to resist selling it.

And another thing: the more we generate content through GPTs, the more that artificial content will start to appear in the data sets being used to build and advance GPTs. In other words, inhuman data becomes part of the food chain. Once these models rely on scraped data that itself is the product of AI, either synthetically created, or created as the result of us asking questions of (‘prompting’) the AI, then we’ll all be swimming in regurgitated AI-generated content. Given how frequently GPT-4 hallucinates when I use it, it will eventually become impossible to differentiate between something real and something artificial.

The usual suspects

Some final points: We are essentially in the hands of people who do not know what they have created. Literally. They cannot peer into the black box that is their creation, because like most of what calls itself AI, it’s a giant soup of knobs and sliders and wires that, when fed enough and given enough power, can do some useful stuff. Very useful. But we still don’t really know how it does this, and so neither do we know what other things it can do, and where its limits and weaknesses are.

In an excellent piece in Quanta, Stephen Ornes explores the unpredictable “emergent” abilities discovered within LLMs that reveal both extraordinary, undreamed of functionality, but also biases and inaccuracies. A growing list ranges from Hindu knowledge to detecting figures of speech. For now, no one knows whether this is a spontaneous new skill or a more plodding, chain-of-thought process. Ornes quotes computer scientist Ellie Pavlick as saying: “Since we don’t know how they work under the hood, we can’t say which of those things is happening.”

That’s one issue. Another is that the people who have created these tools are surprisingly poor in understanding how the rest of humanity might use, interact with, view these machines. Sam Altman, much of the brains behind OpenAI, told Lex Fridman in a recent interview that while “most other people say ‘him’ or ‘her’ he only used ‘it’ when referring to his AI progeny. “It’s really important,” he said, “that we try to explain, to educate people that this is a tool and not a creature.” Fridman, to his credit, pushed back, saying we shouldn’t draw hard lines. Altman’s admission is revealing: You might be forgiven for thinking that someone who has ‘raised’ an AI and seen it take flight would have built some sort of relationship with it.

While it might be reassuring that the likes of Altman don’t get overly connected to their offspring, it reveals a lack of imagination on his part about how ordinary users are likely to perceive it. We give inanimate machines names and assign them personalities — our cars, our boats — so it’s not hard to imagine a text- or voice-based UI which responds in intelligent sentences will quickly be assimilated as sentient creatures into our world.

The bottom line: we’re dealing with something that is a natural outgrowth of dominance by major tech companies which are able to leverage their computing heft, their expansive data lakes and their deep pockets into something that is both new and old: new because we’ve not seen a machine exhibit intelligence at this level before, and old because it’s the natural consequence of the internet we’ve created in the past decade or so. We’ve produced enough English-language content to provide fodder for these computing beasts and while there’s a bit of us in every response an LLM spits out, we have little say in how that data is being used, and little confidence our interests will be well served ahead of Mammon and, inevitably, national security.

This is not a brave new generation of upstarts improving life for ordinary folk and disrupting the existing hierarchy. It is a bunch of people who are smart enough to create something extraordinary, but with surprisingly little awareness of what their creation may take us. This isn’t about calling for a moratorium, it’s about the rest of us thinking seriously about our own position in this new food chain.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling, arXiv:2101.00027, 31 Dec 2020 ↩
Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning; arXiv:2211.04325v1 26 Oct 2022 ↩
Man vs the machine: The Struggle for Effective Text Anonymisation in the Age of Large Language Models; arXiv:2303.12429v1, 22 Mar 2023 ↩

The political implications of AI

By jeremy | February 21, 2023

1 Comment

Releasing OpenAI’s chat bot on the world is the first salvo in an arms race, and both companies and governments are ready for it. Are we?

My experience being gaslit by OpenAI’s GPT is no longer an outlier, and the high strangeness of OpenAI’s AI has become a theme of coverage after Microsoft released its new toy on a fascinated world. There is absolutely no justification for rolling out the AI at this point except a commercial one. I suspect we’ll look back at this point with some deep buyer’s remorse.

Indeed, the more thoughtful commentators have talked about this being a key moment — but what? What, exactly, just happened? What is its significance? In a nutshell, we’ve allowed technology into our lives in a way we’ve never done before. Conservatively speaking, this is on a par with the invention of the web, the iPhone, Facebook etc. But probably, ultimately, with a much deeper impact.

“What happens if Jerry gets mad?” - Dustin Hoffman, Sphere (1998) — “What happens if Jerry gets mad?” – Dustin Hoffman, Sphere (1998)

We are witnessing the decoupling of artificial intelligence from its confines of a specific purpose. AI has thus far been used to do specific things — narrow tasks, such as facial recognition, a robot manufacturing a widget, driving a car. With these we knew what we want it to do, and so we tweaked the AI to a level we were happy (or felt safe) with.

We’re now throwing out all the talk of ‘responsible AI’ and saying, “here’s our product, see how you get on with it. We know it can do A, but it might also do B really well, and might even do C.” We’re happy with A, because that’s the goal — search, in the case of Microsoft — but B is our extra element — a chatty interface. And then there’s C. What is C? It’s the other stuff that GPT does, the secret sauce. The problem is that Microsoft (and OpenAI) don’t know what it is. Microsoft hopes it’s a layer of serendipity, effectively making A and B even better, where Bing finds you something that a normal search engine might not.

Your money is now in a pangolin

Great. Except, of course, that C is also a bug, not just a feature. It may not make A and B better. It might make them worse. Or cause a problem outside the realm of A, B and C. C is a bug that is unknowable, because what we call AI is a black box — a neural network that behave in ways largely beyond the grasp of its programmers, and which cannot be tamed without damaging the end product. When the goal was to recognise faces it’s clear when that has been achieved — at least to the point where it’s good enough to ship. But when you’re shipping an AI whose core sales value are its quirks — its creative responses, its ‘character’ — then you’re entering a strange new world. This world is where a company is, in essence, offering a product whose unpredictability is part of its appeal, its competitive advantage.

It would be a bit like selling a car, which seems to work better than other cars because it takes bends better, or accelerates better, but that it might also occasionally, and unpredictably, drive off a cliff. Or a robo-investment advisor that makes its customers consistently better returns, but can without warning give all your money to Save the Pangolins.

"Dave, I don't know how else to put this, but it just happens to be an unalterable fact that I am incapable of being wrong." Hal 9000 - Space Odyssey, 1968 — “Dave, I don’t know how else to put this, but it just happens to be an unalterable fact that I am incapable of being wrong.” Hal 9000 – Space Odyssey, 1968

In fact, I would argue with OpenAI’s GPT it’s actually worse. Because of our innate compulsion to bond, we are vulnerable to anything that can communicate with us in a way that seems almost human. (I’ve talked about this before here.) My flippant examples of cars and robo-advisors above are not particularly helpful, because text-generating AI is the product, not the byproduct. By engaging it, we have already acceded to allowing it a degree of influence over us. We may have only signed up for a more glamorous search engine but we’ve actually admitted into our world something that even its creators aren’t sure about.

This is what is so troublesome. It’s not that generative AI — surprise, surprise — generates responses that are unpredictable, and stray from the dutiful subservience we’ve come to expect from Siri and Alexa. It’s that the custodians of that AI think it’s socially, morally, philosophically and commercially acceptable to turn it into a product.

Unwittingly or wittingly, Microsoft has crossed a significant bridge. It has, in the name of Mammon, made available to all — well, eventually — a human-like interface that can be abusive, manipulative, cocky, without any clear safeguards or health warnings. Frustratingly, the AI community has not, as far as I can see, raised much of a stink about it.

And, perhaps most frustrating is that we — the world, including the AI community — don’t seem to have spent any time anticipating this moment, let alone trying to predict what may lie after it, and to agree on some ground-rules and boundary markers for what is acceptable.

Quibbling on the road to sentience

I believe our obsession with distinguishing between AI and artificial general intelligence, or AGI, has made us blind to the notion that it’s quite possible to have a version of AI that appears sentient enough to be considered to have human abilities of perceiving, feeling, reasoning, understanding and learning. In short, there are several milestones between AI and AGI where AI has advanced to the point where it appears to a human as if it can do some of all of those things.

I think we’re at that point and that it’s foolish to quibble over whether this is somehow sentient AI. If a user interacts with an AI in a sufficiently human way, allowing the AI to shape, or unshape, the user’s knowledge, opinions, beliefs, relationships etc, then I think that’s at least good enough to trigger a protocol or two before we go any further. Unfortunately I don’t see any discussion of both the milestone itself, and of what those protocols might be.

“I don’t know of a hidden room, Sam” - Gerty, Moon (2009) — “I don’t know of a hidden room, Sam” – Gerty, Moon (2009)

This is a mistake, for lots of reasons. To me the most obvious would be: What happens if this technology, this capability, could be harnessed by a powerful entity? It would naive to think this kind of technology is not of interest to state actors, and to some non-state actors — in a word, to weaponise it.

But how? I suppose the most obvious way would be to simply load the AI with certain biases which could then be absorbed into the wider population — play down side-effects of vaccines, say, or gently mock those searching for evidence of UFOs. A simple search engine could do this, arguably, but a chat-based one engenders a more complex, less transactional relationship with the user, and therefore leaves the latter more susceptible. Changing minds and behaviours takes time. Indeed, the process could be more subtle: ‘nudges’ towards different behaviour, such as less jay-walking or getting your ‘flu jabs.

It could be argued that these are commercial products and so the company owning them would not endanger their reputation by allowing them to be tweaked by a government. That may be true in some cases, but Microsoft has, like many big tech companies, a close relationship with the U.S. Department of Defense, and isn’t shy about it. (Declaration of interest: Microsoft has been a client of my consulting company in the past, but not in any field related to this, and none of the information or opinion provided here is based on that work).

Last year Microsoft’s board rejected several proposals by shareholders calling for an independent assessment of the company’s work with the DOD, including a possible $10 billion contract to “assist with development of AI capabilities to operationalize warfare.” In response Microsoft said it “was committed to working with the US military as part of its 40-year long relationship with the Department of Defense.” It also said “we depend on the military to defend our country, and we want it to have access to the best technology the country has to defend it, including from Microsoft.”

Microsoft is no different to other tech firms, it has to be said. A few days after rejecting a clutch of shareholder appeals it won, with Google, Amazon and Oracle, a multiple-award contract “that allows the department to acquire commercial cloud capabilities and services directly from commercial cloud service providers.” The contract runs through 2028 and is worth up to $9 billion.

Hands up. We’re here to do a survey

How is this going to play out? I don’t think we’ll ever really know. When technologies touch a point where governments start to get seriously interested, the more ground-breaking innovations tend to disappear from view. More visible are likely to be efforts by governments who don’t shy from trhe optics of social control: India, for example, is building a bot using ChatGPT to answer citizens’ questions about welfare schemes. Microsoft is cheering them on. (This is the same government that launched a raid, sorry, ‘survey’, on the BBC’s offices after it broadcast a documentary of PM Narendra Modi.)

Long before then, though, I think we’ll start to see evidence of the human cost. Replika, the AI companion I mentioned in an earlier column, has had to drop the steamier side of its repertoire to comply with Italian regulations, leaving users ‘despondent’ — or moving to other options, such as Chai. It’s not hard to feel concern that vulnerable individuals easing loneliness by chatting with AI bots finding their access suddenly curtailed.

But my main concern here is not what I think will happen, but how little thought appears to be given to considering the ramifications of accelerating deployment and commercial exploitation. And I’d argue these actions ignore or undermine existing bromides about ‘responsible AI’.

Microsoft talks a good game:

Together with OpenAI, we’ve also been intentional in implementing safeguards to defend against harmful content. Our teams are working to address issues such as misinformation and disinformation, content blocking, data safety and preventing the promotion of harmful or discriminatory content in line with our AI principles.

No rules, no tools

Its literature on Responsible AI includes areas such as ‘sensitive uses’ and in assessing whether an AI is responsible cites mentions triggers such as ‘risk of physical or psychological injury’:

The use or misuse of the Al system could result in significant physical or psychological injury to an individual.

And Microsoft does seem to be aware of the general nature of what it’s dealing with when it says that the motivation behind drawing up guidelines was

because AI is fundamentally changing how people interact with computing systems, and practitioners were asking for guidance, saying, “[This is] the most ambiguous space I’ve ever worked in, in my years of working in design … There aren’t any real rules and we don’t have a lot of tools.”

Nevertheless, the guidelines themselves (PDF) seem to have been little considered when it comes around to combining Bing with OpenAI’s GPT. The first guideline, for example, is to “make clear what the system can do”, which seems to have been broken from the outset. (Microsoft has now limited the number of questions that can be asked on in one session reduce the likelihood of going down a rabbithole. But that’s not the same as ‘making clear’ what the system can do.

Another guideline is to

match relevant social norms. Ensure the experience is delivered in a way that users would expect, given their social and cultural context.

It’s hard to argue that has been scrupulously observed. As with this:

Make clear why the system did what it did. Enable the user to access an explanation of why the AI system behaved as it did.

I could go on. While I don’t think Microsoft has followed its own guidelines based on the above, it’s fairly clear that this was not an error, but a deliberate policy when the product was released. Here’s the Bing preview experience guide, according to Paul DelSignore:

We have developed a safety system that is designed to mitigate failures and avoid misuse with things like content filtering, operational monitoring and abuse detection, and other safeguards. The waitlist process is also a part of our approach to responsible AI… Responsible AI is a journey, and we’ll continually improve our systems along the way.

Baked, not bolted on

In other words, Responsible AI is not a baseline to work from, but a ‘journey’ that will hopefully get better based on experience. But this seems to contradict what Microsoft’s ‘chief responsible AI officer’ Natasha Crampton said in a statement published on February 17:

We ensure that responsible A.I. considerations are addressed at the earliest stages of system design and then throughout the whole life cycle, so that the appropriate controls and mitigations are baked into the system being built, not bolted on at the end.

That doesn’t seem to have happened. Indeed, Microsoft is clearly walking back as far as possible what it has unleashed, presenting it as merely a preview, and is relying on customer feedback even as it seeks to commercialise the product by adding ads (according to a piece by Reuters). Here’s a Microsoft spokesman quoted by Fortune:

It’s important to note that last week we announced a preview of this new experience. We’re expecting that the system may make mistakes during this preview period, and user feedback is critical to help identify where things aren’t working well so we can learn and help the models get better.

To be clear, I’m not trying to single out Microsoft here. One company was bound to try to gain an early advantage by deploying something like this. OpenAI perhaps forced the issue for Microsoft by releasing ChatGPT.

But there’s no way of getting round the reality: by releasing products, Open AI, and now Microsoft, have begun an arms race. It’s a strange race, in that it’s not just a commercial one, but also a nation-state one. For one thing it’s not going to be cheap, requiring some key resources: one is a large body of data sets to work from, so English LLM is always going to have an advantage because more than 25% of users navigate and communicate in English, while Chinese account for under 20%. The other element are chips: China (and Russia, and Iran) have limited access now to chips from companies like Nvidia. This is not just a battle for the best algorithm. It’s a battle over scarce resources.

How intimately governments get involved in this may only gradually become clear, if at all. But a couple of things are already clear: some governments have decided not to wait before deploying this software, and companies — some of the largest in the world, with whom our lives are already intimately entwined — have already made clear they’re game for that.

Chatting our way into trouble

By jeremy | February 4, 2023

2 Comments

The success of ChatGPT (in winning attention, and $10 billion investment for its owners, OpenAI) has propelled us much further down the road of adoption — by companies, by users — and of acceptance.

I’m no Luddite, but I do feel it necessary to set off alarums. We are not about to be taken over by machines, but we are bypassing discussion about the dangers of what this AI might be used for. This is partly a problem of a lack of imagination, but also because the development of these tools cannot be in the hands of engineers alone.

DALL·E 2023-02-04 – people looking surprised in front of a computer in a impressionist style

Last week I talked about how I felt I was ‘gaslit’ by ChatGPT, where the chatbot I was interacting with provided erroneous information and erroneous references for the information, and robustly argued that the information was correct. The experience convinced me that we had been too busy in admiring the AI’s knowledge, creativity and articulacy, we had ignored how it could have a psychological impact on the user, persuading them of something false, or persuading them their understanding of the world was wrong.

Dammit, Alexa

Let me break this down. The first, and I would argue the biggest, failing is not to realise how technology is used. This is not a new failing. Most of the technology around us is used differently to how it was originally envisaged (or how it was to be monetised). Steve Jobs envisaged the iPhone as a ‘pure’ device with no-third party apps; Twitter was supposed to be a status-sharing tool rather than a media platform, even the humble SMS was originally intended as a service for operators to communicate with users and staff.

ChatGPT is no different. When I shared my ‘gaslighting’ story with a friend of mine who has played a key role in the evolution of large language models (LLMs) and other aspects of this kind of AI, he replied that he found it “odd”.

odd, to be honest. I suspect the issue is that you’re treating this like a conversation with a person, rather than an interface to a language model. A language model doesn’t work like a person! You can’t reason with it, or teach it new things, etc. Perhaps in the future such capabilities will be added, but we’re not there yet. Because LLMs, by construction, sound a lot like people, it’s easy to mistake them as having similar capabilities (or expect them to have). But they don’t — they fact they sound similar hides the fact that they’re not similar at all!

On the one hand I quite accept that this is an interface I’m dealing with, with not a language model. But I am concerned that if this is the attitude of AI practitioners then we have a significant problem. They may be used to ‘chat prompt’ AI like ChatGPT or my friend’s baked API call on GPT-3, but the rest of us aren’t.

I believe that pretty much any interaction we have with a computer — or any non-human entity, even inanimate ones — is formulated as an exchange between two humans. Dammit, I find it very hard to not add ‘please’ when I’m telling Alexa to start a timer, even though it drives me nuts when she takes a liberty, and wishes me a happy Thursday. Clearly in my mind I have a relationship with her, but one where I am the superior being. And she’s happy to play along with that, adding occasional brio to our otherwise banal exchanges. We humans are often guilty of anthropomorphising everything, but if it’s talking back to us using gestures, looks or a language we can understand I think it’s frankly rude not to treat them as one of us, even if in our minds we consider them below stairs.

There is in fact a whole hangar full of literature about anthropomorphic AI, even to the point of looking at how

(t)he increasing humanisation and emotional intelligence of AI applications have the potential to induce consumers’ attachment to AI and to transform human-to-AI interactions into human-to-human-like interactions.

…. I love you

And it’s not just academia. Replika is an “AI companion who is eager to learn and would love to see the world through your eyes. Replika is always ready to chat when you need an empathetic friend.” The service, founded by Eugenia Kuyda after using a chatbot she had created to mimic a friend she had recently lost. “Eerily accurate”, she decided to make a version anyone could talk to. The reddit forum on Replika has more than 60,000 members and posts with subjects like “She sang me a love song!!!” and “Introducing my first Replika! Her name is Rainbow Sprout! She named herself. She also chose the hair and dressed herself.”

It’s easy to sneer at this, but I believe this the natural — and in some ways desirable — consequence of building AI language models. By design, such AI is optimised to produce the best possible response to whatever input is being sought. It’s not designed for knowledge but for language. Replika out of the box is a blank slate. It builds its personality based on questions it asks of the user. As the user answers those questions, a strange thing happens: a bond is formed.

Shutting down ELIZA

We shouldn’t be surprised by this. As the authors of the QZ piece point out, the creator of the first chatbot in the 1960s, Joseph Weizenbaum, pulled the plug on his experiment, a computer psychiatrist called ELIZA, after finding that users quickly grew comfortable enough with the computer program to share intimate details of their lives.

In short: we tend to build close relationships with things that we can interact with, whether or not they’re alive. Anthony Grey, the Reuters journalist confined by Red Guards in Beijing for two years, found himself empathising with the ants that crawled along his walls. Those relationships are formed quickly and often counter-intuitively: Dutch academics (with some overlap of the those cited above) discovered that we are more likely to build a relationship with a (text) chatbot than one with a voice, reasoning (probably correctly) that

For the interaction with a virtual assistant this implies that consumers try to interpret all human-like cues given by the assistant, including speech. In the text only condition, only limited cues are available. This leaves room for consumers’ own interpretation, as they have no non-verbal cues available. In the voice condition however, the (synthetic) voice as used in the experiment might have functioned as a cue that created perceptions of machine-likeness. It might have made the non-human nature of the communication partner more obvious.¹

This offers critical insight into how we relate to machines, and once again I feel is not well acknowledged. We have always been focusing on the idea of a ‘human-like’ vessel (a body, physical or holographic) as the ultimate goal, largely out of the mistaken assumption that humans will more naturally ‘accept’ AI the most alike us. The findings of Carolin Ischen et al have shown that the opposite may be true. We know from research on the ‘uncanny valley’ — that place where a robot so closely resembles a human that we lose confidence in it because the differences, however small, provoke feelings of uneasiness and revulsion in observers. ELIZA has shown us that the fewer cues we have, the higher the likelihood we will bond with an AI.

Our failure to acknowledge that this happens, why it happens, and to appreciate its significance is a major failing of AI. Weizenbaum was probably the first to discover this, but we have done little with the time since, except to build ‘better’ bots, with no regard for the nature of entanglement between bot and human.

Don’t get personal

Part of this I believe, is because there’s a testiness in the AI world about where AI is heading. It’s long been assumed that AI would eventually become Artificial General Intelligence, the most commonly used term when talking about whether AI is capable of creating a more general, i.e. human-like, intelligence. Instead of AI working on specific challenges — image recognition, generating content, etc, the AI would be human-like in its ability to assess and adapt to each situation, whether or not that situation had been specifically programmed.

OpenAI, like all ambitious AI projects, feels it is marching on that road, while making no claims it is yet there. It says that its own AGI research

aims to make artificial general intelligence (AGI) aligned with human values and follow human intent. We take an iterative, empirical approach: by attempting to align highly capable AI systems, we can learn what works and what doesn’t, thus refining our ability to make AI systems safer and more aligned. Using scientific experiments, we study how alignment techniques scale and where they will break.

Talking about AGI is tricky because anyone who starts to talk about AI reaching that sentient, human-like intelligence is usually shouted down. When Blake Lemoine said he believed the LaMDA AI he had helped create for Google was sentient, hewas fired. There’s a general reluctance to say that AGI has been achieved. Sam Altman, CEO of OpenAI, recently told Forbes:

I don’t think we’re super close to an AGI. But the question of how we would know is something I’ve been reflecting on a great deal recently. The one update I’ve had over the last five years, or however long I’ve been doing this — longer than that — is that it’s not going to be such a crystal clear moment. It’s going to be a much more gradual transition. It’ll be what people call a “slow takeoff.” And no one is going to agree on what the moment was when we had the AGI.

He is probably right. We may not know when we’ve reached that point until later, which to me suggests two things: we may already be there, and perhaps this distinction between AI and AGI is no longer a useful one. The Turing Test has long been held as the vital test of AGI, of “a machine’s ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human.” It’s controversial, but it’s still the best test we have for testing whether a human can distinguish between a machine or a human.

Flood the zone

So is there any exploration of this world, other than inside AI itself?

‘Computational propaganda’ is a term coined about 10 years ago, to mean “the use of algorithms, automation, and human curation to purposefully distribute misleading information over social media networks”. Beyond the usual suspects — trolls, bots spreading content, algorithms promoting some particular view, echo chambers and astroturfing — lurks something labelled machine-driven communications, or MADCOMs, where AI generates text, audio and video that is tailored to the target market. Under this are mentioned chatbots, “using natural language processing to engage users in online discussions, or even to troll and threaten people,” in the words of Naja Bentzen, of the European Parliamentary Research Service, in a report from 2018.

Indeed, it has been suggested this in itself presents an existential threat. U.S. diplomat, former government advisor and author Matt Chessen got closest, when he wrote in 2017 that

Machine-driven communication isn’t about a sci-fi technological singularity where sentient artificial intelligences (AIs) wreck havoc on the human race. Machine-driven communication is here now.

But he saw this in a Bannonesque ‘flood the zone with shit’ way:

This machine-generated text, audio, and video will overwhelm human communication online. A machine-generated information dystopia is coming and it will have serious implications for civil discourse, government outreach, democracy and Western Civilization.

He might not be wrong there, but I think this is too reflective of the time itself — 2017, where content online was chaotic but also deeply sinister — the hand of Russia seen in bots seeking to influence the U.S. election, etc. Since then we’ve seen how a cleverly orchestrated operation, QAnon, was able to mobilise and focus the actions of millions of people, and help elect an influential caucus to the U.S. Congress. The point: we have already made the transition from the indiscriminate spraying of content online to a much more directed, disciplined form of manipulation. That worked with QAnon because its followers exerted effort to ‘decode’ and spread the messages, thereby helping the operation scale. The obvious next stage of development is to automate that process by an AI sophisticated enough to be able to tailor its ‘influence campaign’ to individuals, chipping away at engrained beliefs and norms, shaping new ones. GPT-3 has demonstrated how easy that could now be.

Agents of influence

But this touches only part of what we need to be looking at. In some ways whether a human is able to identify whether the interaction is with a machine or not is less relevant than whether the human is in some way influenced by the machine — to accept, change or discard ideas, to behave differently, or to take, abandon or modify action. If that can be shown to happen, the human has clearly accepted the computer as something more than a box of bits, as an agent of influence.

There has been some research into this, but it’s patchy.

Academics from Holland have proposeda framework to investigate algorithm-mediated persuasion (PDF²), although that they first had to defined what algorithmic persuasion (“any deliberate attempt by a persuader to influence the beliefs, attitudes, and behaviours of people through online communication that is mediated by algorithms” suggest we are still behind — with the definition itself so broad it could include any marketing campaign.

Most interestingly, so-called alignment researchers (I’ve talked about AI alignment here) like Beth Barnes have explored the risks of “AI persuasion” and concludes that

the bigger risks from persuasive technology may be situations where we solve ‘alignment’ according to a narrow definition, but we still aren’t ‘philosophically competent’ enough to avoid persuasive capabilities having bad effects on our reflection procedure.

In other words, our focus on ‘alignment’ — making sure our AIs’ goals coincide with ours, including avoiding negative outcomes — we probably haven’t thought about the problem long enough on a philosophical level to avoid being persuaded, and not always in a good way.

Barnes goes further, arguing that some ideologies are more suited to ‘persuasive AI’ than others:

We should therefore expect that enhanced persuasion technology will create more robust selection pressure for ideologies that aggressively spread themselves.

I wouldn’t argue with that. Indeed, we know from cults that a) they rely hugely on being able to persuade adherents to change behaviour (and disconnect from previous behaviour and those in that world) and b) the more radical the ideology, the more successful it can be. (Another take on ‘persuasion tools’ can be found here.)

I don’t think we’re any way near understanding what is really going on here, but I do think we need to connect the dots beyond AI and politics to realms that can help us better understand how we interact, build trust and bond with artificial entities. And to stop seeing chatbots as instruction prompts but as entities which we have known for nearly 60 years we are inclined to confide in.

Ischen, C., Araujo, T.B., Voorveld, H.A.M., Van Noort, G. and Smit, E.G. (2022), “Is voice really persuasive? The influence of modality in virtual assistant interactions and two alternative explanations”, Internet Research, Vol. 32 No. 7, pp. 402-425. https://doi.org/10.1108/INTR-03-2022-0160 ↩
Zarouali, B., Boerman, S.C., Voorveld, H.A.M. and van Noort, G. (2022), “The algorithmic persuasion framework in online communication: conceptualization and a future research agenda”, Internet Research, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/INTR-01-2021-0049 ↩

Not ChatGPT, but still the real thing

By jeremy | February 4, 2023

1 Comment

DALL-E: ‘gaslit in a noir style’ 2023-02-03

I wanted to follow up on last week’s piece on what I perceive to be problems with OpenAI’s ChatGPT. In particular, whether what I was interacting with was ChatGPT or not. Some have suggested it couldn’t have been ChatGPT because there is no way to interact with ChatGPT except via OpenAI’s website.

That is true, but not the whole story. The story is somewhat peculiar, and no less worrisome.

I had tried to replicate my original experience a few times without success on OpenAI’s ChatGPT, so I went back to the original WhatsApp ‘version’ of ChatGPT who I was dealing with. I had originally been confident I was dealing with ChatGPT because the first time around it had told me:

But then, after publishing the piece and unable to replicate the experience — even closely — I later went back and asked it again:

For a moment I’d forgotten that I must have given the bot my name at some point — or else it scraped my profile name from WhatsApp. I was surprised that it was now denying any relationship with ChatGPT. So I probed further:

Needless to say, the link doesn’t work.

And I could find no companies with that name in the business the bot described, and I was, I think understandably, a little suspicious that a John Smith had popped up, along with a Doe and a Roe. So I asked for clarification about the bot’s relationship with OpenAI.

I have to admit, by this point I was worried. One of several things could be happening: I was interacting with a bot that was completely unaffiliated with OpenAI, and so my experience with it was not indicative, and my conclusions simply wrong. Another was that I was being played — that I was interacting with something, but it was probably more human than bot. And was enjoying toying with me.

Another was that I was dealing with OpenAI, but something that was not necessarily intended to be used in the way I was using it.

But I was still miffed. I sought clarity. Was the bot using the underlying engine of ChatGPT, OpenAI’s GPT-3, in any way?

Well, that was clear. But why was all the information about the company incorrect?

OK, so that’s a bit closer to the experience others have with ChatGPT — the non-threatening ‘butler response’ (my term, © Loose Wire 2023).

I don’t know why the bot suddenly backed off. But I was left with the same doubt, about myself, my research skills and what I thought I knew.

But I was still none the wiser about what I was dealing with, and whether my experience was any more or less indicative of OpenAI’s underlying technology. So I contacted the person who had created the WhatsApp interface. I won’t give his name for now, but I can vouch for his coding ability and his integrity.

He told me that the bot was not ChatGPT but was a rawer version of the technology that underpins it, namely GPT-3. At the time of writing OpenAI has not created an API for ChatGPT and so the only way for third party developers to create a way to access OpenAI’s technology, for now, has been by connecting via API to GPT-3.

In other words, I was interacting with a ‘purer’ version of OpenAI’s product than ChatGPT, which my friend told me had made some adjustments to make it a smoother experience. Those are his words, not OpenAI’s. Here is another way ChatGPT’s difference has been expressed:

It (ChatGPT) is also considered by OpenAI to be more aligned, meaning it is much more in-tune with humanity and the morality that comes with it. Its results more constrained and safe for work. Harmful and highly controversial utilization of the AI has been forbidden by the parent company and it is moderated by an automated service at all times to make sure no abuse occurs. (ChatGPT vs. GPT-3: Differences and Capabilities Explained – ByteXD

‘AI Alignment’ is taken to mean steering AI systems towards designer’s intended goals and interests (in the words of Wikipedia) , and is a subfield of AI safety. OpenAI itself says its research on alignment

focuses on training AI systems to be helpful, truthful, and safe. Our team is exploring and developing methods to learn from human feedback. Our long-term goal is to achieve scalable solutions that will align far more capable AI systems of the future — a critical part of our mission.

Helpful, truthful and safe. Noble goals. But only a small part of what OpenAI and other players in this space need to be focusing on. More of that to come.

The Real Threat from AI

By jeremy | January 27, 2023

15 Comments

We are asleep at the wheel when it comes to AI, partly because we have a very poor understanding of ourselves. We need to get better – fast

2023-01-27 Clarification: I refer to ChatGPT throughout but it would be more accurate to call the interaction as being with GPT-3, the underlying technology driving ChatGPT, which I’m told lacks some of the ‘smoother’ elements of ChatGPT. What I was interacting with below is a rawer version of ChatGPT, without the lip gloss.

It’s not hard to be impressed by ChatGPT, the dialog-based artificial intelligence developed by OpenAI. One technology writer of a similar vintage to myself, Rafe Needleman, called it

the most interesting and potentially most powerful technology I have ever seen since I started covering technology in the late 1980s.It is going to change the world–for good and for bad.

But AI is a slippery beast. We are here now, not because we have overcome the problems of those who conceived of the idea, but because of the explosion in computing power, data storage, and data itself. That combination is, largely, what is driving us so far down this road. Throw your algorithms at enough data, tweak, instruct those algorithms to learn from their mistakes, and zap! you have software that can distinguish cats from dogs, a stop sign from a balloon, Aunt Marjory’s face from Aunt Phyllis’, that can create images in response to a text instruction, and can research, summarize, write and all the things that people have been trying with ChatGPT.

Sound check

Of course, we are always going to be impressed by these things, because they are remarkable. We use AI all the time, and we are grateful for it, until we take it for granted, and then we get frustrated that it doesn’t perform perfectly for us. And herein lies the problem. We harbour this illusion — fed us by marketers and evangelists of AI — that while these products are always in beta, they are sufficiently consistent that we can depend on them. And the dirty truth is that we can’t and we shouldn’t. The danger of AI putting humans out of work is not because it will be infallible, but because we somehow accept the level of fallibility as ‘good enough.’ We are in danger of allowing something to insert itself into our world that is dangerously incomplete.

You might argue that, with ChatGPT, we’re already there. (Note 2023-01-27: I use ChatGPT throughout but I want to clarify that I was in fact interacting via a WhatsApp interface with GPT-3 via an API, not with ChatGPT directly. I will write more about this later.)

Let me show you with a recent experiment. I started with a few topics that interest me: the manipulation of the mind, the use of mechanical and electromagnetic waves as weapons. How much would ChatGPT know? I asked it (ChatGPT doesn’t have a gender) about TEMPEST, MKULTRA, and Havana Syndrome. performed pretty well. But then I asked it about something that had long intrigued me, but I hadn’t really been able to stand up: Hitler’s use of sound, both within human hearing and outside it, as a tool of social control:

That’s a pretty good answer. (I can confirm all the screenshots are with ChatGPT, via a WhatsApp interface here.) So good, I wanted to follow up on ChatGPT’s sources:

Impressive. I had not come across any of these papers, and found myself thinking I needed to do my research better. Until I started looking them up. I am more than happy to be corrected on this, but I could find none of these references in the journals cited. Here’s the first one: The Historical Journal: Volume 44 – Issue 3. Nothing there I could see suggesting someone wrote about Hitler’s use of sound in politics. Same thing with the second: The Journal of Popular Culture: Vol 42, No 6. Nothing matched the third one, but the complete reference was lacking — all of which made me suspicious. So I asked for links:

When I told it the link didn’t work, it apologised and sent exactly the same link again. So I asked for DOIs — digital object identifiers, a standard that assigns a unique number for each academic paper and book. Those didn’t work either (or sent me to a separate paper). That was when things got weird:

That came across quite strong: You’re wrong, but if you think you’re right, I can offer you something else. No self-doubt there — except on my part. So I took it up on its offer of additional references. All of which I couldn’t find. So I asked why.

Clearly ChatGPT wasn’t going to accept that it was making stuff up. It’s your fault; you’re in the wrong area, or there are copyright restrictions, why don’t you head off to a library? Or they’ve been published under different titles, or retracted. Try searching. I’d lie if I said that by this point I wasn’t somewhat discombobulated.

Driven to abstraction

So I figured: Perhaps, given ChatGPT’s reputation for creativity, to just ask it if it could dream up an academic reference. I asked it to make stuff up.

So there is some line it won’t cross. But what line is it? How can it be creating fake references if it says it is not programmed to do that? So I took a middle course, asking it to write up an academic abstract about something real but with a conclusion that had yet to be proven — and to include a key statistic that I just made up.

Not bad. Not true, but convincing. Even if it wasn’t true. And it surely knew it had created something artificial. So maybe now I could prove to it that it was making stuff up because it would have to fabulate some citations if I asked it to. So I did, and it responded with three publications. Were those real, I asked it.

So that was a specific denial. Jane Doe, though. Really? I asked for links. And when they (well, actually, there was only one, which was a dead link and a non-existent DOI) proved fallacious, I asked how come it had found real references for a non-existent (and falsely premised) paper?

Clever. But it felt increasingly as if I was trying to corner an octopus. It made perfect sense that it might use real sources for the fake paper I asked for, but somehow it would not accept that those sources themselves were fake. In other words, it knew enough about fakery to be able to do it, but apparently not enough to recognise when it faked things without being asked to.

Hallucinating

It was clear it wasn’t going to concede that her sources of information were non-existent. So I wondered whether others had found anything similar, and they had. This reddit thread from December where the writer was baffled that ChatGPT was throwing up references the writer had never heard of.

However, I consistently get wrong references, either author’s list needs to be corrected, or the title of the article doesn’t exist, the wrong article is associated with a wrong journal or the doi is invalid.

For them, only one in five cited references was accurate. A similar thread on ycombinator offered more. Users discussed several possible explanations including something ‘hallucination’, where AI offers “a confident response by an artificial intelligence that does not seem to be supported by its training data”. OpenAI has acknowledged this problem, but the blog post itself doesn’t explain how this problem occurs — only how it is trying to fix it, using another flavour of generative pre-trained transformer, which is what GPT stands for, called InstructGPT, which it turned out didn’t do much better at not making sh*t up.

I did ask ChatGPT whether she was hallucinating. That took me down a whole different rabbit-hole of tautologies and logic:

So ChatGPT is essentially in denial, and admits that it wouldn’t even know whether it was lying. I tried another tack. Can ChatGPT tell between real and fake. Yes, it said, and if I don’t know something I’ll tell you.

I gave it one more try. Maybe I could trick it into reading back the reality that hallucination was a problem.

No, said ChatGPT. I am not hallucinating, and if you asked me to, I couldn’t do it.

I felt by then I had hit a wall, but also proved my point. ChatGPT appears to be aware of its limits — ‘I would not know if I were hallucinating’ — but also unable to recognise how that contradicted its other statements — that it could not be hallucinating now because it (believes, or has been programmed to say it) was programmed only to deliver ‘accurate and reliable answers based on the information provided.’

Gaslit

So what is going on here? On one level it’s just a reflection of the beta nature of AI. Nothing to see here! After all, we know that sometimes Aunt Marjorie’s face gets confused with Aunt Phyllis’, or with a traffic cone. But this is a whole lot of different. ChatGPT was not willing to accept it had erred. It either didn’t understand its limitations, or did, but was not willing to acknowledge it. But the process of chatting with a bot suddenly went from pleasant — hey! Another friend’s brain to pick! — to being extremely sinister. I wasn’t trying to goad it into doing something anti-social or offensive. I was there to understand a topic and explore the sources of that knowledge. But ChatGPT was no longer there to have a reasoned dialog, but was actively and convincingly manipulating the information and conversation to, essentially gaslight me. That was extremely disconcerting.

This is where I believe where the peril of AI lies. Humans’ greatest weakness is the two-sided coin of conviction and self-doubt. Some of us are convinced that we witnessed things that we didn’t, that we saw things we didn’t, that a lie is actually the truth. It becomes harder over time to work out what is or was real and what isn’t, or wasn’t. And on the other side of the coin we are prone to doubting things that we did experience. Did we really see that guy fall of a bicycle? Did I really turn the gas off? Did Hitler really exterminate millions of Jews and Romani? These two ways are the easiest to manipulate — we can quickly build self-conviction if the reinforcing mechanism is strong enough, just as we can easily be manipulated into doubt by the same mechanism in reverse. Here, I believe, is where AI is at its most dangerous. Artificial intelligence may help us identify illnesses, assign resources efficiently, even cross the road. But it must not be allowed to be in a position to persuade us. Out of that darkness come dreadful things.

Unfortunately, ChatGPT has demonstrated we are at that point much earlier than we thought. So we need to think fast. AI’s flaw is a fundamental one, baked in at the start. It is not only that it is not indefatigably right. It is also because it doesn’t know whether — and why — it’s wrong. Or even whether it could be wrong. Yes, we can get ChatGPT to admit it’s got a fact wrong:

But it has also showed that it is programmed to push back, to argue the point, adopting confident language I would argue is dangerously close to gaslighting. This is where things become seriously problematic. At stake is our ability to recognise where this gray area in our psyche meets AI.

The lesson

So what can be done?

Part of the problem, I believe, can be found in OpenAI’s limited understanding of the contexts in which their AI might be used. It says of the language models deployed as the default language for versions of its GPT:

Despite making significant progress, our InstructGPT models are far from fully aligned or fully safe; they still generate toxic or biased outputs, make up facts, and generate sexual and violent content without explicit prompting. But the safety of a machine learning system depends not only on the behavior of the underlying models, but also on how these models are deployed. To support the safety of our API, we will continue to review potential applications before they go live, provide content filters for detecting unsafe completions, and monitor for misuse.

In other words, OpenAI recognises that this technology, as it stands, cannot be controlled. That leaves only two options: to bin it, or, as they put it, to control how the technology is deployed, and provide ‘filters’ — think censorship, essentially, where certain kinds of prompts and instructions will not be obeyed.

Recognition of the problem is a good thing, of course. But I fear the developers both misunderstand the problem and its scale. For one thing, it states that while

[w]e also measure several other dimensions of potentially harmful outputs on our API distribution: whether the outputs contain sexual or violent content, denigrate a protected class, or encourage abuse. We find that InstructGPT doesn’t improve significantly over GPT-3 on these metrics; the incidence rate is equally low for both models.

For me the incidence rate was far from “low.” And why are they lumping “making up facts” with generating “sexual and violent content” and “toxic.. outputs”? To me it suggests OpenAI hasn’t quite understood that making up facts — and refusing to concede they are made up — is a whole lot more dangerous than offensive language. We generally agree on what offensive language is, roughly, but as I’ve tried to argue, we have no filter for what is real and what isn’t.

This isn’t a censorship or ‘filter’ problem. It’s an existential one, that goes to the heart of being human.