Warning: This post is a bit of a ramble, with new sections being added to the end, as further thoughts come to mind.
Contents
- Introduction to ChatGPT, by ChatGPTI asked ChatGPT this question: “Describe ChatGPT in 200 words, for a general, non-specialist reader”The response: “ChatGPT is a language model developed by OpenAI. It is based on a deep learning technique called transformer, which allows it to generate human-like text. It has been trained on a large dataset of text from the internet, and can be fine-tuned for specific tasks such as language translation, text summarization, and text generation. ChatGPT is capable of understanding and generating text in multiple languages, and can respond to a wide range of questions, prompts, and commands. It can also generate creative content such as stories and poetry. The model can be used for various applications like virtual assistants, chatbots, automated writing and more. However, it is important to note that while ChatGPT is highly advanced, it is not a human and may not always understand context or nuances in language and may produce errors or biases present in the training data”.Here is an introductory video providing a beginners guide to how to access and use ChatGPT: https://www.youtube.com/watch?v=JTxsNm9IdYUUsing Chat GPT to analyses bodies of textWhy: This is my current interest, where I think ChatGPT is already useful in its current formThe challenge: Evaluators often have to deal with large volumes of text data, includingManual analysis of this material can be very time consuming. In recent years a number of different software packages have been developed which are useful for different forms of content analysis. These are generally described as text analytics, text mining and Natural Language Processing (NLP) methods. I have experimented with some of these methods, including clustering tools like Topic Modelling, sentiment analysis methods, and noun and key word extraction tools.From my limited experience to date, ChatGPT seems likely to leave many of these tools behind. Primarily on criteria such as flexibility and usability. I am less certain on criteria such as transparency of process and replicability of results. I need to give these more of my attentionHow it works – from a user perspectiveHere below is the user interface, seen after you have logged on. You can see prompt I have written in the top of the white section. Then underneath is the ChatGPT response. I then have two options.There is a similarity between these two functions and March’s (1991) distinction between two complimentary approaches to learning: Exploration and Exploitation. With regeneration being more exploratory and refined prompts being more exploitative.But bear in mind that ChatGPT is using data that was available up to 2021. It does not (yet) have real time access to data on the internet. When it does, that will be another major step forward. Fasten your seat belts!.The design of promptsThis is the key to the whole process. Careful design of prompts will deliver more rewards. The more clearly specified your request, the more likely you will see results which are useful.I will now list some of the prompts, and kinds of prompts, I have experimented with. These have all been applied to paragraphs of text generated by a ParEvo exercise (which I cant quote here for privacy reasons).
- 2023 03 03 Postscript. Check out Pickaxe. Here you can design a simple (no code required) app that you and others can use to carry out specific AI based inquiries. Such as sentiment analysis or actor extraction. Try my Pickaxe AI Pile Sorting Exercise Vs 2Extracting causal theories – Steve Powell’s explorationsSteve Powell, a colleague of mine (who developed the Causal Map app) has been testing the following prompts, with the intention to identify if ChatGPT can usefully identify causal connections between events in a text:I have not quoted the original text that was presented for analysis, nor all the results. But here is the response to the last prompt:Steve also notes that… another important challenge I forgot to mention is to get it to abstract a little, so that it uses the same labels for similar concepts across paragraphs and cases. This is a task that I also felt needs further exploration through better designed prompts, to see if ChatGPT can do concept abstraction or not.He also reports that he will work a prompt to get it to format its output for more easy upload into Causal Map. That could be very useful for many people, including myself. I have been doing the same kind of prompt development with text descriptions of network relationships to be converted into a data format that can be imported into social network analysis software – see further above.PS: Read more of Steve’s thoughts and experiment with ChatGPT and causality on his blog here: Causal Map: Making ConnectionsOther applications that may be useful to evaluatorsThe one feature that seems to have received the least attention, within the generalist media, is how ChaptGP and its related products Open AI Codex, could have a major impact on software development . “Proficient in more than a dozen programming languages, Codex can now interpret simple commands in natural language and execute them on the user’s behalf” – executing them meaning write relevant functional computer code.In response a query I made, ChatpGP replied“As an AI language model, I can help users with coding suggestions and provide information on a wide range of programming languages. Some of the most common programming languages that I can assist with include:Python: a high-level, interpreted language known for its simplicity and ease of use, often used in data science and machine learning.Java: a popular object-oriented language used for developing large-scale applications.JavaScript: a client-side scripting language used for creating interactive web pages and web applications.C++: a powerful, high-performance language used for building operating systems, games, and other resource-intensive applications.Ruby: a dynamic, object-oriented language used for web development, especially with the Ruby on Rails framework.PHP: a server-side scripting language used for building dynamic websites and web applications.Swift: a modern programming language developed by Apple for iOS, macOS, watchOS, and tvOS development.C#: an object-oriented language developed by Microsoft, used for developing Windows desktop applications, games, and other applications.These are just a few examples of the many programming languages I can provide assistance with, and I can also provide guidance on other languages based on the user’s needs and preferences.What it does not also mention is that it can provide advice on relevant Excel formula, macros and VBA code. A capacity likely to be relevant to a wider group of evaluatorsOne point to note about this capacity, is that testing the answers is straightforward in most cases. It either works or does not, and if it does work it should be easy enough to identify if the results are correct or not.There are a few videos available online that explain what can be done by combining use of ChatGPT and Excel:Quotes of note..“As the novelty of that surprise wears off, it is becoming clear that ChatGPT is less a magical wish-granting machine than an interpretive sparring partner”Crypto was money without utility,” he argued, while tools such as ChatGPT are, “for now, utility without money.”“It’s going to be fascinating to see how people incorporate this second brain into their job,”“…you’re curious how GPT and other AI tools are going to change “the way people talk about talking, write about writing, and think about thinking.”“If the old line was “Learn to code,” what if the new line is “Learn to prompt”? Learn how to write the most clever and helpful prompts in such a way that gives you results that are actually useful.”“Your job won’t be replaced by AI but it may be replaced by someone who knows how to use AI better than you…”In summary…as of 22 February 2023Seeing ChatGPT as “…an interpretive sparring partner…” is a good approximate description. Another is that working with ChatGPT is (as others have already said) like working with an intern that has at least a Masters degree (or more) in every subject you need to be working with. The trouble is that this intern is not above bluffing and bullshitting when it cant find any thing better (i.e. more informed/detailed/accurate) to say. So you need to get past the understandable “Wow” reaction to its apparent intelligence and creativity, and lift your own game to the level where you are ready and able to critically review what ChapGPT has responded with. Then, through further dialogue with ChatGPT, get it to know when some of its answers are not acceptable and, through further feedback, to improve on its own performance thereafter.Which will of course mean you will then (again) need to get past any (additional) “Wow” reaction to its (additional) apparent intelligence and creativity, and lift your own game to (an additional) another level where you are ready and able to critically review what ChapGPT has responded with”…. 🙂 The ball comes back into your court very quickly. And it does not show evidence of tiring, no matter how long the dialogue continues.Lame criticisms – a miscellany with some responsesPS: There seem to be significant current limitations to ChatGPT’s ability to build up self-knowledge from user responses. Each time a new Chat is started no memory is retained of the contents of previous chats (which include users responses). Even within a current chat there appears to be a limit on how many prior prompts and associated responses (and the information they all contain), can be accessed by ChatGPT.