There’s a lot of excitement, understandably, about ChatGPT rolling out a “roll-your-own” ChatGPT tool. In fact, it’s been so popular OpenAI have had to suspend new subscriptions, and I’ve several times had activity on my account paused because of overload.
But if you think this is a sign of progress, and the democratisation of generative AI, think again.
The customized versions of ChatGPT I have tried to create are buggy in the extreme, and not ready for anything other than playing around with. And by buggy I mean no better at adhering to human guardrails than earlier versions of GPT. I worry that this desire to move quickly and break things, to get your product into the mainstream of consumers before the others, is leading to a serious problem of ‘mediocritisation’ of what are already seriously threatened areas of content: blog writing, analysis, education and commentary.
The first one I created, Media Mentor, worked fairly well. It does what ChatGPT is best at — creative thinking — guiding the user in thinking how to monetize journalistic ideas. The second, The Veracity Van, a fact-checker specialising in analysing online claims and their origins, is not too bad either, although it’s really not that different to asking ChatGPT itself the same question.
The problem comes with trying to make more specific changes that would make tools more specialised. You are presented with two panes: a left one for entering instructions in ordinary human language, the right pane a working preview of what the user will see.
ChatGPT would acknowledge any changes requested and claim to have implemented in the code, but this (mostly) didn’t happen. In other words, the ChatGPT channel you use to program and tweak the tool hallucinates.
In one case it admitted that it was acknowledging and implementing instructions it did not have the wherewithal to implement. Trying to include a copy icon to the tool is not possible, I was eventually told, because it did not have access to the underlying code.
Things were decidedly worse when it came to trying to limit ChatGPT to a specific part of the web. I asked it to limit its searches in response to prompts to the URLs within a Google Custom Search Engine I had set up. I didn’t expect it to agree, but it did, along with instructions about what it should do if it couldn’t find the answer there. While ChatGPT agreed and included it in its instructions, its answers were derived from Bing.
Something similar happened when I uploaded some documents and asked ChatGPT to limit its ‘knowledge base’ to those documents, the best I could get was the a response that started out well:
The documents in my knowledge source do not directly address the specific question of whether you should join an interview between a journalist and one of your clients. However, they do provide insights into the dynamics of such interviews and the role of the client in them.
But it then continued along the same track with material I agreed with but didn’t recognise from the source documents. When challenged ChatGPT sheepishly responded:
I apologize for the confusion, but upon rechecking my knowledge source, I did not find a direct mention of “the importance of clients being prepared for interviews” in the documents provided. This part of my previous response was based on general principles rather than specific information from the documents. If you have any other questions or need advice based specifically on the content of the documents, please let me know.
This despite the ChatGPT instructions being clear that it should restrict itself to material within its knowledge base. Tweaking the instructions further actually made the thing worse. When I asked it to cite which document it was using as the source, it confidently came back with its name. When I asked it which sentences it was relying on in that document, it confidently listed them. But when I asked which sentences it was using for a specific sentence in its response, it caved in:
Upon reviewing the specific sentences quoted from the document “[redacted],” I realize they do not directly support the statement “assess how your presence might be perceived.” This part of my response was an inference made from the broader context of managing client-journalist interactions, rather than a direct quote from the document. The document focuses on the dynamics of an interview, the nature of questions asked, and the perception of the client’s responses, but does not explicitly discuss the impact of a third party’s presence in the interview. I apologize for any confusion caused by this inference.
When I noticed that a lot of the documents I had uploaded to the tool’s knowledge source were no longer listed there, I was told that there may have been an error uploading them — not that I saw any such message. So I uploaded them again. This time I asked a specific question which I knew to be in one of the documents.
Please give an example of a specialist market
ChatGPT came back with an answer:
Based on the content of the files you have provided, I couldn’t find a specific example of a “specialist market.” Therefore, I’m unable to provide an example based solely on the information in these documents.
Not great, considering we’re not talking a massive database to trawl. But OK, maybe it hasn’t done a complete run, and at least it’s not making stuff up. Right?
So then I asked it another question I knew was in the database, and this time it went on for two paragraphs, most of it not from the documents. When I asked for a source, it was quite clear:
The information provided about market reporters was derived from my knowledge source, specifically from documents you have uploaded. This content was not drawn from an external source or general knowledge but is based on the specific content of the files you provided.
Fair enough, what I want. But which specific document or documents, I asked.
It cited a document, in the format most of the documents were in when I uploaded them (document number, a keyword, date of document). “This document,” it declared, “provided insights into the role and responsibilities of market reporters.”
I thought I might have missed something so I went back through the documents. Turns out, there was no such document. The document name was a convincing hallucination. ” But there is no such document,” I complained.
It thought for a minute.
I apologize for the confusion earlier. It seems I made an error in referencing the documents. If the information about market reporters is not contained in the documents you have uploaded, then I must correct my previous response: I do not have specific information about market reporters from the documents you provided. My earlier response was based on general knowledge, which was not appropriate given your instructions and my designated role. Thank you for pointing out the discrepancy.
In other words, we’re no further forward than back in February, when I first started writing about the hallucination problem.
It’s worth keeping this in perspective. These are breathtaking advances and that AI has advanced this far is remarkable. But the tools are virtually useless for anything other than creation. They are not to be relied on for anything vaguely important, where it might be used in connection with facts, verification, accuracy or anything remotely critical.
And I would have to say, pushing this out to the public without properly testing it and warning users that these are toys, not products, not services, not search engines, not anything remotely functional, is highly irresponsible.
There’s a reason the Cambridge Dictionary chose ‘hallucinate’ as its word of the year. Unfortunately even Cambridge University itself doesn’t properly understand the term: It quotes Wendalyn Nichols, Cambridge Dictionary’s Publishing Manager, as saying:
The fact that AIs can ‘hallucinate’ reminds us that humans still need to bring their critical thinking skills to the use of these tools. AIs are fantastic at churning through huge amounts of data to extract specific information and consolidate it. But the more original you ask them to be, the likelier they are to go astray. At their best, large language models can only be as reliable as their training data. Human expertise is arguably more important – and sought after – than ever, to create the authoritative and up-to-date information that LLMs can be trained on.
I would argue no; they are not only as good as their training data — they are worse than their training data, because they confabulate on top of that training data. And they are poor at churning through amounts of data to extract specific information, not “fantastic”. They essentially can’t tell what is specific or not.
Yes, they are good at going out there and saying “this is what I’ve found.” But they are (mostly) lousy at telling you specifically where they found it, what else they found that they’ve left out, and even judging whether what they’ve found is accurate.
I would love to be corrected about what I may have done wrong in my attempt, and I do recognise I’ve not played around with some of the extra configuration options within the roll-your-own ChatGPT functionality. But given I’ve done most of what I expect other ordinary users have done, I suspect my experience is likely to be somewhere near the norm.
Once again, we need to hold back before rolling out this kind of thing until these very basic issues are fixed. This is not a new game or beta feature in a niche app. This is real stuff, that real people may end up relying on for real world needs.