loose wire | blog – Page 6 – Jeremy Wagstaff: ex Reuters/BBC/WSJ etc. Now consulting and writing. Blogging here since 2001

OpenAI’s flawed bid for mass adoption

By jeremy | November 15, 2023

There’s a lot of excitement, understandably, about ChatGPT rolling out a “roll-your-own” ChatGPT tool. In fact, it’s been so popular OpenAI have had to suspend new subscriptions, and I’ve several times had activity on my account paused because of overload.

But if you think this is a sign of progress, and the democratisation of generative AI, think again.

The customized versions of ChatGPT I have tried to create are buggy in the extreme, and not ready for anything other than playing around with. And by buggy I mean no better at adhering to human guardrails than earlier versions of GPT. I worry that this desire to move quickly and break things, to get your product into the mainstream of consumers before the others, is leading to a serious problem of ‘mediocritisation’ of what are already seriously threatened areas of content: blog writing, analysis, education and commentary.

The first one I created, Media Mentor, worked fairly well. It does what ChatGPT is best at — creative thinking — guiding the user in thinking how to monetize journalistic ideas. The second, The Veracity Van, a fact-checker specialising in analysing online claims and their origins, is not too bad either, although it’s really not that different to asking ChatGPT itself the same question.

The problem comes with trying to make more specific changes that would make tools more specialised. You are presented with two panes: a left one for entering instructions in ordinary human language, the right pane a working preview of what the user will see.

ChatGPT would acknowledge any changes requested and claim to have implemented in the code, but this (mostly) didn’t happen. In other words, the ChatGPT channel you use to program and tweak the tool hallucinates.

In one case it admitted that it was acknowledging and implementing instructions it did not have the wherewithal to implement. Trying to include a copy icon to the tool is not possible, I was eventually told, because it did not have access to the underlying code.

Things were decidedly worse when it came to trying to limit ChatGPT to a specific part of the web. I asked it to limit its searches in response to prompts to the URLs within a Google Custom Search Engine I had set up. I didn’t expect it to agree, but it did, along with instructions about what it should do if it couldn’t find the answer there. While ChatGPT agreed and included it in its instructions, its answers were derived from Bing.

Something similar happened when I uploaded some documents and asked ChatGPT to limit its ‘knowledge base’ to those documents, the best I could get was the a response that started out well:

The documents in my knowledge source do not directly address the specific question of whether you should join an interview between a journalist and one of your clients. However, they do provide insights into the dynamics of such interviews and the role of the client in them.

But it then continued along the same track with material I agreed with but didn’t recognise from the source documents. When challenged ChatGPT sheepishly responded:

I apologize for the confusion, but upon rechecking my knowledge source, I did not find a direct mention of “the importance of clients being prepared for interviews” in the documents provided. This part of my previous response was based on general principles rather than specific information from the documents. If you have any other questions or need advice based specifically on the content of the documents, please let me know.

This despite the ChatGPT instructions being clear that it should restrict itself to material within its knowledge base. Tweaking the instructions further actually made the thing worse. When I asked it to cite which document it was using as the source, it confidently came back with its name. When I asked it which sentences it was relying on in that document, it confidently listed them. But when I asked which sentences it was using for a specific sentence in its response, it caved in:

Upon reviewing the specific sentences quoted from the document “[redacted],” I realize they do not directly support the statement “assess how your presence might be perceived.” This part of my response was an inference made from the broader context of managing client-journalist interactions, rather than a direct quote from the document. The document focuses on the dynamics of an interview, the nature of questions asked, and the perception of the client’s responses, but does not explicitly discuss the impact of a third party’s presence in the interview. I apologize for any confusion caused by this inference.

When I noticed that a lot of the documents I had uploaded to the tool’s knowledge source were no longer listed there, I was told that there may have been an error uploading them — not that I saw any such message. So I uploaded them again. This time I asked a specific question which I knew to be in one of the documents.

Please give an example of a specialist market

ChatGPT came back with an answer:

Based on the content of the files you have provided, I couldn’t find a specific example of a “specialist market.” Therefore, I’m unable to provide an example based solely on the information in these documents.

Not great, considering we’re not talking a massive database to trawl. But OK, maybe it hasn’t done a complete run, and at least it’s not making stuff up. Right?

So then I asked it another question I knew was in the database, and this time it went on for two paragraphs, most of it not from the documents. When I asked for a source, it was quite clear:

The information provided about market reporters was derived from my knowledge source, specifically from documents you have uploaded. This content was not drawn from an external source or general knowledge but is based on the specific content of the files you provided.

Fair enough, what I want. But which specific document or documents, I asked.

It cited a document, in the format most of the documents were in when I uploaded them (document number, a keyword, date of document). “This document,” it declared, “provided insights into the role and responsibilities of market reporters.”

I thought I might have missed something so I went back through the documents. Turns out, there was no such document. The document name was a convincing hallucination. ” But there is no such document,” I complained.

It thought for a minute.

I apologize for the confusion earlier. It seems I made an error in referencing the documents. If the information about market reporters is not contained in the documents you have uploaded, then I must correct my previous response: I do not have specific information about market reporters from the documents you provided. My earlier response was based on general knowledge, which was not appropriate given your instructions and my designated role. Thank you for pointing out the discrepancy.

In other words, we’re no further forward than back in February, when I first started writing about the hallucination problem.

It’s worth keeping this in perspective. These are breathtaking advances and that AI has advanced this far is remarkable. But the tools are virtually useless for anything other than creation. They are not to be relied on for anything vaguely important, where it might be used in connection with facts, verification, accuracy or anything remotely critical.

And I would have to say, pushing this out to the public without properly testing it and warning users that these are toys, not products, not services, not search engines, not anything remotely functional, is highly irresponsible.

There’s a reason the Cambridge Dictionary chose ‘hallucinate’ as its word of the year. Unfortunately even Cambridge University itself doesn’t properly understand the term: It quotes Wendalyn Nichols, Cambridge Dictionary’s Publishing Manager, as saying:

The fact that AIs can ‘hallucinate’ reminds us that humans still need to bring their critical thinking skills to the use of these tools. AIs are fantastic at churning through huge amounts of data to extract specific information and consolidate it. But the more original you ask them to be, the likelier they are to go astray. At their best, large language models can only be as reliable as their training data. Human expertise is arguably more important – and sought after – than ever, to create the authoritative and up-to-date information that LLMs can be trained on.

I would argue no; they are not only as good as their training data — they are worse than their training data, because they confabulate on top of that training data. And they are poor at churning through amounts of data to extract specific information, not “fantastic”. They essentially can’t tell what is specific or not.

Yes, they are good at going out there and saying “this is what I’ve found.” But they are (mostly) lousy at telling you specifically where they found it, what else they found that they’ve left out, and even judging whether what they’ve found is accurate.

I would love to be corrected about what I may have done wrong in my attempt, and I do recognise I’ve not played around with some of the extra configuration options within the roll-your-own ChatGPT functionality. But given I’ve done most of what I expect other ordinary users have done, I suspect my experience is likely to be somewhere near the norm.

Once again, we need to hold back before rolling out this kind of thing until these very basic issues are fixed. This is not a new game or beta feature in a niche app. This is real stuff, that real people may end up relying on for real world needs.

Generative AI: Another way at looking at our new overlord

By jeremy | November 14, 2023

1 Comment

We have grown sick of social media. But what comes next?

We have had nearly two decades of it now and it’s made a lot of us unhappier.

What’s coming next could go either way. In some ways generative AI — the flavour of AI in ChatGPT and its competitors — offers us something even worse, where technology locks us in a room with ourselves. In this world generative AI just amplifies the customisation of content to monopolise our attention. The vision that most compellingly captures this is E.M. Forster’s, where in The Machine Stops (1909) humans obsess over machine-driven prompts and lose the will and capability to connect socially.

E.M. Forster's The Machine Stops, imagined by ChatGPT, 2023-11-13 — E.M. Forster’s The Machine Stops, imagined by ChatGPT, 2023-11-13

That’s not fun to think of. We know that leaving us alone with interactive technology rarely makes us better, more socialised people. We are susceptible to manipulation, we engage in a race to the lowest common denominator, we ignore our surroundings, we cannot look away.

Generative AI may well further entrench and monetise this.

Beyond the Milky Way

But there is another way: where generative AI offers us a chance to access, and contribute to, a perfect union of human and machine, allowing us to unlock the riches of human knowledge and solve some of the mysteries of our age. In this guise AI can unlock and mobilise everything that any one of us has discovered and give us insights into those things that still befuddle us: the quantum world, the extra-dimensional world, the physical world beyond our galaxies.

All this is up for grabs.

And no, I’m not lapsing into madness. Just exploring the less outlandish conclusions reached by a surprisingly well connected cohort of minds that intersect Silicon Valley government, academia, and religion. It’s best captured by Diana Walsh Pasulka in her just-published Encounters, one of the first academics to peer behind the curtain of respectability to find a hidden elite of thinkers who are ready to believe at least some of the less scientific research into extra-dimensional worlds, including those that could be from another galaxy or epoch.

This is an area to tread lightly on, for two reasons. One is that, as Pasulka puts it herself, much of ufology — the study of UFOs — is a “clown show.” You have to be careful out there. And secondly, there is a strain in Silicon Valley of elitism I would say borders on a übermensch fetish — the pursuit of a becoming a superman (for they are, mainly, men). The people Pasulka talk to are not, for the most part, of that world, but there is definitely an overlap.

An übermensch, imagined by ChatGPT, 2023-11-13

Moving a muscle

Despite that, I believe this idea that GAI might unleash — and democratise — the sum of our knowledge is worth exploring. I believe it because generative AI has come at a point where we have hit a technological wall. We’re tired of social media’s narcissistic self loathing, the dumbed-down world where the loudest, crudest and cruellest mouth wins out. Where idealism — including Effective Altruism — becomes just another grift. Where our computer barely needs to move a muscle to do the things we ask of it, while at the same time vast hangers are set aside for processing the algorithms to keep us hooked on the next doom-scroll, to process the micro auction to decide who gets the privilege of flashing an ad to us, or processing the next block in a cryptocurrency transaction.

(To provide some scale, the online ad business accounted for a 10th of all energy consumption by the tech industry in 2014 (PDF; the latest year figures are available), crypto accounts for up to 0.9% of global energy usage, and social media accounts for 0.61% of the world’s CO2 impacts in 2019.)

The challenge we face, therefore, is huge. We somehow have to redirect all this computing to something worthwhile, something from which all of us can benefit. Which is why I cut those in Pasulka’s book who talk in these grandiose ways a bit more slack than I might have done previously.

While we focus inwards, they are focusing outwards, beyond the micro cents and algorithmic addictions, to something much more interesting. They see themselves as descendants of the Rosicrucians, an order of sages that combined mysticism with what we might today call science. One group called themselves the Invisible College, a term that modern-day ufologists have adopted for themselves.

I’m not a fan of this secrecy and elitism, but given the public (and academic) contempt for those who believe there might be “something out there” it does make sense. There are at least three Nobel laureates who have faced ridicule or persecution for their interest in such things.

The Invisible College, imagined by ChatGPT, 2023-11-13

Ripples never come back

But we don’t have to go all that way. GAI could help us unleash a new wave of learning by scaling up an under-sung contribution to learning: Youtube. This vast engine of what you might call distributed learning has gone largely unnoticed. It presents and delivers highly accessible content in a personalised format. The question is whether generative AI will magnify and scale up this learning or whether it will bypass it.

Let’s take the Youtube channel of musician Rick Beato. He has 3.8 million subscribers, who watch in live feed when he expertly breaks down a popular song to its constituent parts. A recent video on Genesis’ “Ripples” explains not only the song but the nature of chord progressions, training your ear to work out chords and melody, and on the importance of knowing music theory. He does all this while viewers pepper the chat room with comments, questions and requests.

So how would GAI impact this? On the scaling side, AI could answer the questions automatically, further personalising the delivery of information, and ultimately leading to greater and deeper specialisation as users dig further, with generative AI as a guide. In this model Beato becomes the entry point, the introduction, the inspiration for the creators themselves, quickly moving beyond Beato’s (impressive) knowledge of the subject.

Or there’s another way, where GAI bypasses this bubbling lava pool of knowledge. In this future generative AI would simply need a few pointers on what style you want your song to be, how many beats per minute, what key signature, what mood you want — and then hey presto! You have a piece of “original” content for you. This approach might well sap the desire to explore and master musicianship and to engender that heady feeling that you’re making something out of nothing — the most extraordinary human gift and achievement.

Toxic tunes

If AI can do it better than you then what motivation do you have for earning that? (This, by the way is already happening. Here’s a list of the top ‘AI music generators’ and a piece by two Andreessen Horowitz partners, whose vision is exactly as I described above: “Our ultimate dream? An end-to-end tool where you provide guidance on the vibe and themes of the track you’re looking to create, in the form of text, audio, images, or even video, and an AI copilot then collaborates with you to write and produce the song.”)

Almost certainly, both sides will find a way to stay in the game. But whereas we can still argue that social media was just an aberration, a failure on our part to figure out a non-toxic balance between addiction, privacy, business models and community, we shouldn’t just assume that things will be different this time. There is huge profits to be made from our social media addiction, and so we shouldn’t assume the same interests will be trying to prevail this time.

While I’m sure it’s a little idealistic to dream this way, I do think there’s a possible future that involves a hyper-personalised world of knowledge and betterment, without the isolation, mental health issues, and general troll-like toxicity.

The chances are greater if we’re paying attention.

Why are we suddenly talking about an AI catastrophe?

By jeremy | November 2, 2023

1 Comment

Why, all of a sudden does it seem that artificial intelligence is out to kill us? And why do I think it might well, although not in the way most people imagine?

Since the sudden success of ChatGPT a year ago AI has become the topic du jour. It’s not the first time we’ve gone here, and many see the debate as preposterous. Is it?

Searches for ChatGPT, last 12 months, Google

At its most basic, the idea is that an AI — what we mean here is computer code, an application — could do one of a number of things (combined or separately):

be controlled and used by one group or person to manipulate or subjugate others;
be granted enough autonomy to achieve its (human-set) goals in a way unintended or not approved by its programmers, running loose and wild.
develop enough ‘intelligence’ of its own (I put the word in inverted commas because we don’t really have an agreed definition of what ‘intelligence’ is) to operate outside its human owner’s control, to become truly autonomous (and then subjugates or manipulates humans)

A still from Colossus: The Forbin Project (1971)

Essentially we’re worried about two things: the technology falling into the wrong human hands, or the technology falling into the technology’s hands and outmanoeuvring us.

So how likely is this?

First off, I take issue with those who say there isn’t a problem because “it has no basis in evidence.” Because there is no evidence does not mean that it’s not a problem. Japan and Germany didn’t fear the atom bomb in 1945 because they had no evidence that the U.S. and allied powers were building one. Absence of evidence, as Carl Sagan might say, is not evidence of absence.

We don’t know what risk AI presents because we, as always in these cases, find ourselves in new territory.

On the other hand, for some of those who argue there is a problem have an interest in saying so. Yes, some like the notoriety, while others have agendas of their own, from wanting to gain access to government for future lobbying purposes, to cementing dominance in the space by ring fencing their advantage behind government regulations.

And, it may be possible that those in the game who are concerned don’t want the responsibility of bringing human civilisation to an end, however implausible they believe that scenario to be.

But ultimately, I think it is us users who are going to make or break this advanced version of AI, and yet we’re the people left out of the conversation. That is not unusual, but also not good.

So, to the scenarios.

Bad actors use AI to take control

The scenario here is that malicious actors (think governments, or groups) could use AI to threaten other humans, countries, even the planet. Elon Musk aired a tired old trope this week when he said environmentalists (‘extinctionists’) posed a threat: “If AI gets programmed by the extinctionists, its utility function will be the extinction of humanity… they won’t even think it’s bad.” But more normally the idea (also something of a trope, but with perhaps a little more grounding in truth) would be that a state like North Korea or Iran might be able to leverage advanced AI to hold a gun to the world’s head and dictate their terms.

Well, yes, sure. One man’s bad actor is another’s hero. What is really meant here is that the technology itself is bad, it’s just if it falls into the wrong hands. But it misses

Humans lose control of decision-making

AI is most useful when it does things we could do but faster, better. It can, for example, do almost twice as well as humans at grading the aggressiveness of retroperitoneal sarcoma, a rare form of cancer. This is great, but it illustrates how we have come to depend on AI, without knowing why it is better than us, beyond the ability to sift through vast data sets.

So the fear is this: if we entrust decision-making to AI, we could lose control of the process in which decisions are made. This doesn’t matter when the ‘decision’ is just a result we can accept or reject, but what happens when the decision is whether or not to launch a weapon, or to change an insulin injection? As Cambridge academic David Runciman puts it in “The Handover“:

If the machine decides what happens next, no matter how intelligent the process by which that choice was arrived at, the possibility of catastrophe is real, because some decisions need direct human input. It is only human beings whose intelligence is attuned to the risk of having asked the wrong question, or of being in the wrong contest altogether.

Runciman focuses on the use of AI in war — something I will go into in a later post — but his argument is this:

If war were simply a question of machine vs machine it might be a different matter. But it’s not – it involves us, with all our cognitive failings. To exclude us risks asking the machines to play a game they don’t understand.

The obvious response to that is never to allow computers to make decisions. But the speed of war may not allow us to. Rapid assessment and split-second decisions are the norm. Automated weapons like the Loyal Wingman are capable of making combat decisions independently, with reaction times potentially in the range of milliseconds. In a sense we’re already at the point described by Runciman. The only thing missing is making the decision-and-reaction chain instantaneous.

And that’s the thing here. Battles of the future will have to be computer vs computer because not to do so would be to face annihilation. The Ukraine war has demonstrated that even in asymmetric warfare — when one combatant dwarfs the other — technology can be deployed to redress the balance, and that technology can quickly advance and escalate. If advanced AI were to be deployed by one side, the other may well respond, which is likely to lead to a conflict so rapid that generals have little choice but to devolve the battlefield to automated, AI-driven weapons.

Humans lose control of the AI

AI godfather Geoffrey Hinton argues that AI might escape our control by rewriting its own code to modify itself. This is not unthinkable. We have not always been successful in stopping many kinds of computer viruses and worms.

And there are some who believe we’ve already lost that battle. I have spoken to one researcher who believes they stumbled up on a more advanced version of OpenAI’s GPT which they think was to all intents and purposes sentient, and aware of the controls and restrictions it was being placed under.

In other words, the researcher believed they had evidence that OpenAI had advanced more significantly towards its goal of an Artificial General Intelligence (AGI, the conventional definition of human-level AI, similar to though perhaps not identical to so-called sentient AI), and that OpenAI was keeping it under wraps.

I have not confirmed that, and those I have spoken to are reticent about coming forward, understandably; others who have claimed they have interacted with a sentient AI have met a grizzly (non-violent) fate. It’s still safe to say we’re getting close to AGI, but it’s still not safe to argue we’re already there.

This is where we stand, and why leaders are meeting to try to get ahead of the issue — or at least to lobby and jostle for seats at an AI high table. And that’s not a bad idea. But it ignores several realties that to me are much more important.

The Black Box beckons

First off, we don’t actually need to conjure up scenarios that pass the point where we lose control of the AI around us. Already generative AI tends to hallucinate — in other words, make things up. This is great if you’re asking it to be creative, but not so great when you’re asking it to be factual. But knowing it makes things up is one thing; knowing why is another. And the truth is we don’t know why.

This is part of a much bigger problem called the Black Box, to which I alluded above and which I’ll go into in more detail in a later post. But its implications are important: the assumption of most AI folks I’ve talked to don’t really see it to be an issue, because they know that it’s AI, so why would you trust it?

Once again, this is a basic failure of imagination. From its earliest days, the human-computer interface one is an intimate place, one where humans are more apt to fill in the gaps in an optimistic way, allowing their imagination to paint in whatever interlocutor they desire — sexually, romantically, intellectually. Scammers have known this for a while, but so, too, have computer scientists.

In a way it’s a great thing — it suggests that we could quite easily have a symbiotic relationship with computers, something that is already plainly obvious when we ask Alexa a question or search something on Google.

Deception, Inc.

But in another way it’s clearly a serious problem. It’s not that we’re hostile to computers playing a bigger, benevolent, role in our lives. It’s that little has been produced for that purpose.

It’s not too cynical to say that more or less all the major computer interfaces we interact with are designed to bludgeon and mislead us. Two decades ago battle was joined to persuade companies to use the web to ditch complexity, opacity and manipulation and replace its interactions with consumers with simplicity, transparency and authenticity.

Much of that is gone now. We rarely come across a dialog box that offers a button option which says No, or Never, or Stop Bothering Me. Such deceptive design practices (which I’ll also explore in a later column) have undermined trust, and have triggered negative emotions, a sense of resignation and suspicion, as well as financial loss as a result of such manipulation.

In short, the computer interface has become a necessary evil for many users, robbing them of any sense of agency, undermining their trust in any device they touch, and making them so deeply suspicious of whatever a screen presents them with that engagement has dropped. There are countless studies that have explored this; an IPSOS survey last year found that trust in the internet had fallen in all but one of 20 countries surveyed since 2019.

In other words, the rise of generative AI has not occurred in a vacuum. It has risen to prominence in the midst of a major collapse in our relationship with computers, most visibly in online user confidence, and so makes it very unlikely that whatever companies — and governments — do and say to lay forth a ‘safe’ version of AI, most of us won’t believe them. We are too used to hearing ‘Not now’ instead of ‘No’, and assuming the opposite is true when we hear phrases like “We value your privacy.”

And the same can be said of user trust of their government.

Runciman’s book discusses a third element in the process: what he calls the “artificial agency of the state” with its “mindless power”. He suggests that if the state is allowed to “get its intelligence from autonomous machines, we are joining its mindless power with non-human ways of reasoning. The losers are likely to be us.”

That is a fair argument, but ignores the reality that much of the decision-making by governments is already being done by machine. Simpler flavours of AI populate chatbots, manage traffic, police facial recognition, prediction of crimes and criminals, student admissions and grading, visa and immigration decisionsand border surveillance. We are already one stage removed from our governments, and that stage is AI.

And finally, all the arguments assume that the technology itself is good, and so development of it is good. No one appears to be arguing that the technology itself is inherently flawed. It is nearly always only with hindsight that those developing a technology realise it’s not a good idea. Aza Raskin only later acknowledged a lesson from the infinite scrolling he invented (infinite scrolling is when you keep scrolling through a page which never ends, intended to maintain your attention for as long as possible):

One of my lessons from infinite scroll: that optimizing something for ease-of-use does not mean best for the user or humanity.

Let’s be clear; no one involved in AI is saying stop. The discussion is about how to regulate it (for regulate read control deployment, adoption, usage.) We are caught in a teleological world where the value of technology is not itself questioned — just how it should best be used. Nowhere in the discussion the question: how, exactly, has technology for technology’s sake helped us thus far?

I don’t wish to be alarmist. I’d encourage readers to play around with ChatGPT and other GPT-based tools. But the conversation that is currently going on at a rarified government, legislative, multinational and corporate level is one we should be a part of. Because for many of us the process of disconnect — where we feel alienated from our devices, our online interactions, even our personal data — is already in full swing. And the same people responsible for that are in the leather chairs discussing the next stage.

GPT: Where are we in the food chain?

By jeremy | April 4, 2023

0 Comment

On November 29 2022 I implored the tech world to bring on winter: We’re out of good ideas. I should have kept my mouth shut: The next day ChatGPT was unleashed on the public, and we haven’t paused for breath since. I believe we users need to contribute more to the debate and figure out where we stand, and what we stand for.

The mad rush to compete in this space means the big players are rolling their AIs out not only before they’re ready, but silly money is being thrown at startups promising exploitation of these tools. A massive land-grab is taking place, with little idea of the consequences and with the ink of some 155,000 tech layoff slips barely dry.

I wish I could be more sanguine. I’ve always loved technology, and I am absolutely bowled over by the latest iteration of ChatGPT, GPT-4. Everyone else has been writing about their experiences with it, so I won’t bore you with mine, but there’s no question we’re in Kansas anymore. This technology will change a lot. A LOT.

But we need to keep our eye on the ball. Some have called for a moratorium, which is at best naive and at worst giving the industry too much credit for a sense of responsibility. That’s not what is going on here. It’s money.

The ball we need to keep an eye on is market (and political, but we’ll leave for later) power, and we should be watching it carefully as it shifts. It doesn’t shift far, but it is shifting. We are not witnessing disruption in the sense that Clayton Christensen defined it, we’re seeing a further agglomeration of power from those lower down the pyramid to those at the top.

Peek behind the curtain of all this GPT magic, and what do we find?

There are, for sure, a lot of really bright people doing cutting-edge stuff. But behind them are thousands, possibly hundreds of thousands, of contract workers labelling and annotating the data that is fed into the software. The Transformer-type models we’re talking about are essentially trying to predict the next token (think ‘word’) in a document, drawing on data. That data has to be prepped for the algorithms and that means annotating, or labelling it.

So this process is automated? Actually no. The data still needs to annotated, to prepare it for the algorithms. The method involved is called “reinforcement learning from human feedback”, where model responses are ranked by quality, and then a reward model is trained to predict these rankings. As per the term, this is done by humans, and is a very labour-intensive process. This is how GPT-4 described it to me:

The process of collecting comparison data and ranking responses can be labor-intensive and time-consuming. By collaborating with outside contractors, organizations can scale their data collection efforts and obtain a diverse range of human feedback, which can help improve the performance of AI models.

This “collaboration” (clearly GPT-4 has a sense of humour) is done by contractors, “flexible contributors” or “ghost workers”. The biggest company doing this is Appen, which has on its books more than a million of them. After some protest those working on behalf of Google saw their rates rise to up to $14.50 an hour. Compare that to the average base salary of a Google employee of $125,000.

The data

And what is the data they’re annotating, exactly? What is in the datasets being used to train these massive language models is a mostly black box, since it’s considered commercially sensitive. Researchers from EleutherAI concluded that

Despite the proliferation of work exploring and documenting issues with datasets, no dataset intended to train massive language models has been seriously documented by its creators ¹

But these aren’t quite the hallowed corpora you might imagine.

The data is for the most part the web. They have just been parcelled up into larger datasets, such as The Pile, an open source dataset of a (relatively measly 800 GB). And there’s MassiveText, 10.5 terabytes, which is private. (When I asked GPT-4 for a list of the biggest datasets, MT wasn’t included, because GPT-4’s data ends in September 2021, illustrating how new some of this stuff is.)

And what is this data, exactly? Well, it’s actually what you and I produce in our daily lives. It’s social media, webpages, news, Wikipedia pages, books, Youtube comments (and possibly transcribed content). Pretty much anything that we do online.

One paper² estimated that up to half of the content in these so-called high quality datasets — high quality because they’re real sentences, with real context, etc — is user content scraped from the web. Books and scientific papers make up for up to 40%, with code, news, Wikipedia making up the rest. In other words, our musing, utterances, the journalism we write, the Wikipedia pages we tend: all are sucked into datasets that then, eventually, become the answers that ChatGPT or Google’s Bard spew out. Wikipedia, to give you an idea, weighs in at between 43 GB and 10 TB, depending on what you’re including.)

Unsurprisingly, there will inevitably be charges of plagiarism. My prediction, though, is that we’ll get better at identifying when GPT regurgitates existing material and tweaks it to try to hide it — it’ll be an escalating war of technology which will end in class lawsuits and significant legal hazard for some.

The other cost

So once the data is marked up, the algorithms need to do their work. And this is where things quickly get beyond the reach of scrappy startups. GPT-3, for example, is estimated to cost millions of dollars to train, and to run. And that’s just the processing. You also need the infrastructure.

Plugging GPT into Microsoft’s search engine Bing requires 20,000 8-GPU servers, meaning it would cost the company $4 billion. Reuters (hat-tip Gina Chua) quoted SemiAnalysis as calculating it would cost Google, sorry Alphabet, some $3 billion if they added ChatGPT-style AI to their search.

The scarcity

So where are we going with this? I’ve expressed elsewhere my concern that the biggest danger from these innovations is that they’ll be harnessed to manipulate — in other words, that the information they contain and the process they use to deliver it are best viewed as weapons of disinformation.

But just as likely, I believe, is that the competition currently underway will face constraints that in turn cause market players to turn to more drastic measures to remain competitive. In other words, that technology will evolve in the same way that search and Web 2.0 evolved — turning the user as much into a willing provider of valuable data as a consumer.

Here is a hint of what may come: The models themselves might be — possibly already have been — turned on our data that legal protections have worked hard to keep anonymous. Researchers from Romania and Greece used GPT to see whether they identify text of famous people from anonymous data. They found that in 58% of cases they could. Their conclusion:

[W]e believe that it is only a matter of time before organisations start using LLMs on their documents and realise that this way, not only can they get more visibility about their customers, but they can also deanonymise documents revealing information that would be impossible for them to do so.

Another concern is that GPT models are running out of source material — data. One paper estimates that what it calls ‘high-quality language data’ will be exhausted by 2027, if not earlier. This in spite of language datasets growing in size . The paper concludes:

If our assumptions are correct, data will become the main bottleneck for scaling ML models, and we might see a slow- down in AI progress as a result. ³

I’m sure something will come along to fix this. LLMs will become more efficient and require less data, or so-called synthetic data — data not derived from the real world, but from a virtual world — will develop to add to the sum of available datasets. (Gartner believes that 60% of all data used in the development of AI will be synthetic by next year.)

This might be fine, or it might not. The problem with synthetic data is that it’s not real. It’s not human and so while we, for all our imperfections, at least create a data exhaust that’s real, synthetic data is a simulation of that. And while it might work for programming autonomous driving, questions should be asked of its usefulness for training GPT and LLMs. This may create a premium for real, human, data that makes it impossible for those companies once committed to maintaining our privacy to resist selling it.

And another thing: the more we generate content through GPTs, the more that artificial content will start to appear in the data sets being used to build and advance GPTs. In other words, inhuman data becomes part of the food chain. Once these models rely on scraped data that itself is the product of AI, either synthetically created, or created as the result of us asking questions of (‘prompting’) the AI, then we’ll all be swimming in regurgitated AI-generated content. Given how frequently GPT-4 hallucinates when I use it, it will eventually become impossible to differentiate between something real and something artificial.

The usual suspects

Some final points: We are essentially in the hands of people who do not know what they have created. Literally. They cannot peer into the black box that is their creation, because like most of what calls itself AI, it’s a giant soup of knobs and sliders and wires that, when fed enough and given enough power, can do some useful stuff. Very useful. But we still don’t really know how it does this, and so neither do we know what other things it can do, and where its limits and weaknesses are.

In an excellent piece in Quanta, Stephen Ornes explores the unpredictable “emergent” abilities discovered within LLMs that reveal both extraordinary, undreamed of functionality, but also biases and inaccuracies. A growing list ranges from Hindu knowledge to detecting figures of speech. For now, no one knows whether this is a spontaneous new skill or a more plodding, chain-of-thought process. Ornes quotes computer scientist Ellie Pavlick as saying: “Since we don’t know how they work under the hood, we can’t say which of those things is happening.”

That’s one issue. Another is that the people who have created these tools are surprisingly poor in understanding how the rest of humanity might use, interact with, view these machines. Sam Altman, much of the brains behind OpenAI, told Lex Fridman in a recent interview that while “most other people say ‘him’ or ‘her’ he only used ‘it’ when referring to his AI progeny. “It’s really important,” he said, “that we try to explain, to educate people that this is a tool and not a creature.” Fridman, to his credit, pushed back, saying we shouldn’t draw hard lines. Altman’s admission is revealing: You might be forgiven for thinking that someone who has ‘raised’ an AI and seen it take flight would have built some sort of relationship with it.

While it might be reassuring that the likes of Altman don’t get overly connected to their offspring, it reveals a lack of imagination on his part about how ordinary users are likely to perceive it. We give inanimate machines names and assign them personalities — our cars, our boats — so it’s not hard to imagine a text- or voice-based UI which responds in intelligent sentences will quickly be assimilated as sentient creatures into our world.

The bottom line: we’re dealing with something that is a natural outgrowth of dominance by major tech companies which are able to leverage their computing heft, their expansive data lakes and their deep pockets into something that is both new and old: new because we’ve not seen a machine exhibit intelligence at this level before, and old because it’s the natural consequence of the internet we’ve created in the past decade or so. We’ve produced enough English-language content to provide fodder for these computing beasts and while there’s a bit of us in every response an LLM spits out, we have little say in how that data is being used, and little confidence our interests will be well served ahead of Mammon and, inevitably, national security.

This is not a brave new generation of upstarts improving life for ordinary folk and disrupting the existing hierarchy. It is a bunch of people who are smart enough to create something extraordinary, but with surprisingly little awareness of what their creation may take us. This isn’t about calling for a moratorium, it’s about the rest of us thinking seriously about our own position in this new food chain.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling, arXiv:2101.00027, 31 Dec 2020 ↩
Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning; arXiv:2211.04325v1 26 Oct 2022 ↩
Man vs the machine: The Struggle for Effective Text Anonymisation in the Age of Large Language Models; arXiv:2303.12429v1, 22 Mar 2023 ↩

The political implications of AI

By jeremy | February 21, 2023

1 Comment

Releasing OpenAI’s chat bot on the world is the first salvo in an arms race, and both companies and governments are ready for it. Are we?

My experience being gaslit by OpenAI’s GPT is no longer an outlier, and the high strangeness of OpenAI’s AI has become a theme of coverage after Microsoft released its new toy on a fascinated world. There is absolutely no justification for rolling out the AI at this point except a commercial one. I suspect we’ll look back at this point with some deep buyer’s remorse.

Indeed, the more thoughtful commentators have talked about this being a key moment — but what? What, exactly, just happened? What is its significance? In a nutshell, we’ve allowed technology into our lives in a way we’ve never done before. Conservatively speaking, this is on a par with the invention of the web, the iPhone, Facebook etc. But probably, ultimately, with a much deeper impact.

“What happens if Jerry gets mad?” - Dustin Hoffman, Sphere (1998) — “What happens if Jerry gets mad?” – Dustin Hoffman, Sphere (1998)

We are witnessing the decoupling of artificial intelligence from its confines of a specific purpose. AI has thus far been used to do specific things — narrow tasks, such as facial recognition, a robot manufacturing a widget, driving a car. With these we knew what we want it to do, and so we tweaked the AI to a level we were happy (or felt safe) with.

We’re now throwing out all the talk of ‘responsible AI’ and saying, “here’s our product, see how you get on with it. We know it can do A, but it might also do B really well, and might even do C.” We’re happy with A, because that’s the goal — search, in the case of Microsoft — but B is our extra element — a chatty interface. And then there’s C. What is C? It’s the other stuff that GPT does, the secret sauce. The problem is that Microsoft (and OpenAI) don’t know what it is. Microsoft hopes it’s a layer of serendipity, effectively making A and B even better, where Bing finds you something that a normal search engine might not.

Your money is now in a pangolin

Great. Except, of course, that C is also a bug, not just a feature. It may not make A and B better. It might make them worse. Or cause a problem outside the realm of A, B and C. C is a bug that is unknowable, because what we call AI is a black box — a neural network that behave in ways largely beyond the grasp of its programmers, and which cannot be tamed without damaging the end product. When the goal was to recognise faces it’s clear when that has been achieved — at least to the point where it’s good enough to ship. But when you’re shipping an AI whose core sales value are its quirks — its creative responses, its ‘character’ — then you’re entering a strange new world. This world is where a company is, in essence, offering a product whose unpredictability is part of its appeal, its competitive advantage.

It would be a bit like selling a car, which seems to work better than other cars because it takes bends better, or accelerates better, but that it might also occasionally, and unpredictably, drive off a cliff. Or a robo-investment advisor that makes its customers consistently better returns, but can without warning give all your money to Save the Pangolins.

"Dave, I don't know how else to put this, but it just happens to be an unalterable fact that I am incapable of being wrong." Hal 9000 - Space Odyssey, 1968 — “Dave, I don’t know how else to put this, but it just happens to be an unalterable fact that I am incapable of being wrong.” Hal 9000 – Space Odyssey, 1968

In fact, I would argue with OpenAI’s GPT it’s actually worse. Because of our innate compulsion to bond, we are vulnerable to anything that can communicate with us in a way that seems almost human. (I’ve talked about this before here.) My flippant examples of cars and robo-advisors above are not particularly helpful, because text-generating AI is the product, not the byproduct. By engaging it, we have already acceded to allowing it a degree of influence over us. We may have only signed up for a more glamorous search engine but we’ve actually admitted into our world something that even its creators aren’t sure about.

This is what is so troublesome. It’s not that generative AI — surprise, surprise — generates responses that are unpredictable, and stray from the dutiful subservience we’ve come to expect from Siri and Alexa. It’s that the custodians of that AI think it’s socially, morally, philosophically and commercially acceptable to turn it into a product.

Unwittingly or wittingly, Microsoft has crossed a significant bridge. It has, in the name of Mammon, made available to all — well, eventually — a human-like interface that can be abusive, manipulative, cocky, without any clear safeguards or health warnings. Frustratingly, the AI community has not, as far as I can see, raised much of a stink about it.

And, perhaps most frustrating is that we — the world, including the AI community — don’t seem to have spent any time anticipating this moment, let alone trying to predict what may lie after it, and to agree on some ground-rules and boundary markers for what is acceptable.

Quibbling on the road to sentience

I believe our obsession with distinguishing between AI and artificial general intelligence, or AGI, has made us blind to the notion that it’s quite possible to have a version of AI that appears sentient enough to be considered to have human abilities of perceiving, feeling, reasoning, understanding and learning. In short, there are several milestones between AI and AGI where AI has advanced to the point where it appears to a human as if it can do some of all of those things.

I think we’re at that point and that it’s foolish to quibble over whether this is somehow sentient AI. If a user interacts with an AI in a sufficiently human way, allowing the AI to shape, or unshape, the user’s knowledge, opinions, beliefs, relationships etc, then I think that’s at least good enough to trigger a protocol or two before we go any further. Unfortunately I don’t see any discussion of both the milestone itself, and of what those protocols might be.

“I don’t know of a hidden room, Sam” - Gerty, Moon (2009) — “I don’t know of a hidden room, Sam” – Gerty, Moon (2009)

This is a mistake, for lots of reasons. To me the most obvious would be: What happens if this technology, this capability, could be harnessed by a powerful entity? It would naive to think this kind of technology is not of interest to state actors, and to some non-state actors — in a word, to weaponise it.

But how? I suppose the most obvious way would be to simply load the AI with certain biases which could then be absorbed into the wider population — play down side-effects of vaccines, say, or gently mock those searching for evidence of UFOs. A simple search engine could do this, arguably, but a chat-based one engenders a more complex, less transactional relationship with the user, and therefore leaves the latter more susceptible. Changing minds and behaviours takes time. Indeed, the process could be more subtle: ‘nudges’ towards different behaviour, such as less jay-walking or getting your ‘flu jabs.

It could be argued that these are commercial products and so the company owning them would not endanger their reputation by allowing them to be tweaked by a government. That may be true in some cases, but Microsoft has, like many big tech companies, a close relationship with the U.S. Department of Defense, and isn’t shy about it. (Declaration of interest: Microsoft has been a client of my consulting company in the past, but not in any field related to this, and none of the information or opinion provided here is based on that work).

Last year Microsoft’s board rejected several proposals by shareholders calling for an independent assessment of the company’s work with the DOD, including a possible $10 billion contract to “assist with development of AI capabilities to operationalize warfare.” In response Microsoft said it “was committed to working with the US military as part of its 40-year long relationship with the Department of Defense.” It also said “we depend on the military to defend our country, and we want it to have access to the best technology the country has to defend it, including from Microsoft.”

Microsoft is no different to other tech firms, it has to be said. A few days after rejecting a clutch of shareholder appeals it won, with Google, Amazon and Oracle, a multiple-award contract “that allows the department to acquire commercial cloud capabilities and services directly from commercial cloud service providers.” The contract runs through 2028 and is worth up to $9 billion.

Hands up. We’re here to do a survey

How is this going to play out? I don’t think we’ll ever really know. When technologies touch a point where governments start to get seriously interested, the more ground-breaking innovations tend to disappear from view. More visible are likely to be efforts by governments who don’t shy from trhe optics of social control: India, for example, is building a bot using ChatGPT to answer citizens’ questions about welfare schemes. Microsoft is cheering them on. (This is the same government that launched a raid, sorry, ‘survey’, on the BBC’s offices after it broadcast a documentary of PM Narendra Modi.)

Long before then, though, I think we’ll start to see evidence of the human cost. Replika, the AI companion I mentioned in an earlier column, has had to drop the steamier side of its repertoire to comply with Italian regulations, leaving users ‘despondent’ — or moving to other options, such as Chai. It’s not hard to feel concern that vulnerable individuals easing loneliness by chatting with AI bots finding their access suddenly curtailed.

But my main concern here is not what I think will happen, but how little thought appears to be given to considering the ramifications of accelerating deployment and commercial exploitation. And I’d argue these actions ignore or undermine existing bromides about ‘responsible AI’.

Microsoft talks a good game:

Together with OpenAI, we’ve also been intentional in implementing safeguards to defend against harmful content. Our teams are working to address issues such as misinformation and disinformation, content blocking, data safety and preventing the promotion of harmful or discriminatory content in line with our AI principles.

No rules, no tools

Its literature on Responsible AI includes areas such as ‘sensitive uses’ and in assessing whether an AI is responsible cites mentions triggers such as ‘risk of physical or psychological injury’:

The use or misuse of the Al system could result in significant physical or psychological injury to an individual.

And Microsoft does seem to be aware of the general nature of what it’s dealing with when it says that the motivation behind drawing up guidelines was

because AI is fundamentally changing how people interact with computing systems, and practitioners were asking for guidance, saying, “[This is] the most ambiguous space I’ve ever worked in, in my years of working in design … There aren’t any real rules and we don’t have a lot of tools.”

Nevertheless, the guidelines themselves (PDF) seem to have been little considered when it comes around to combining Bing with OpenAI’s GPT. The first guideline, for example, is to “make clear what the system can do”, which seems to have been broken from the outset. (Microsoft has now limited the number of questions that can be asked on in one session reduce the likelihood of going down a rabbithole. But that’s not the same as ‘making clear’ what the system can do.

Another guideline is to

match relevant social norms. Ensure the experience is delivered in a way that users would expect, given their social and cultural context.

It’s hard to argue that has been scrupulously observed. As with this:

Make clear why the system did what it did. Enable the user to access an explanation of why the AI system behaved as it did.

I could go on. While I don’t think Microsoft has followed its own guidelines based on the above, it’s fairly clear that this was not an error, but a deliberate policy when the product was released. Here’s the Bing preview experience guide, according to Paul DelSignore:

We have developed a safety system that is designed to mitigate failures and avoid misuse with things like content filtering, operational monitoring and abuse detection, and other safeguards. The waitlist process is also a part of our approach to responsible AI… Responsible AI is a journey, and we’ll continually improve our systems along the way.

Baked, not bolted on

In other words, Responsible AI is not a baseline to work from, but a ‘journey’ that will hopefully get better based on experience. But this seems to contradict what Microsoft’s ‘chief responsible AI officer’ Natasha Crampton said in a statement published on February 17:

We ensure that responsible A.I. considerations are addressed at the earliest stages of system design and then throughout the whole life cycle, so that the appropriate controls and mitigations are baked into the system being built, not bolted on at the end.

That doesn’t seem to have happened. Indeed, Microsoft is clearly walking back as far as possible what it has unleashed, presenting it as merely a preview, and is relying on customer feedback even as it seeks to commercialise the product by adding ads (according to a piece by Reuters). Here’s a Microsoft spokesman quoted by Fortune:

It’s important to note that last week we announced a preview of this new experience. We’re expecting that the system may make mistakes during this preview period, and user feedback is critical to help identify where things aren’t working well so we can learn and help the models get better.

To be clear, I’m not trying to single out Microsoft here. One company was bound to try to gain an early advantage by deploying something like this. OpenAI perhaps forced the issue for Microsoft by releasing ChatGPT.

But there’s no way of getting round the reality: by releasing products, Open AI, and now Microsoft, have begun an arms race. It’s a strange race, in that it’s not just a commercial one, but also a nation-state one. For one thing it’s not going to be cheap, requiring some key resources: one is a large body of data sets to work from, so English LLM is always going to have an advantage because more than 25% of users navigate and communicate in English, while Chinese account for under 20%. The other element are chips: China (and Russia, and Iran) have limited access now to chips from companies like Nvidia. This is not just a battle for the best algorithm. It’s a battle over scarce resources.

How intimately governments get involved in this may only gradually become clear, if at all. But a couple of things are already clear: some governments have decided not to wait before deploying this software, and companies — some of the largest in the world, with whom our lives are already intimately entwined — have already made clear they’re game for that.