Gang Fang's Blog

Here is a collection of experiments I ran on ChatGPT. This is more explorative and the direction is to get the chatbot to produce results that reveal more of its nature.

1. Can ChatGPT produce texts that don’t display regularities of English language?

Regularities here refer to anything that gives senses to a text, like the consistency it has with the real-world, the organizing principles of a language, its grammar, the uses of correct words and etc. I am experimenting with this because the whole point of the machine is in capturing regularities and producing outputs that are based on those regularities. So requesting it to return texts that don’t display regularities, note that irregularities used in training of the model is still a kind of regularities, should produce insights into the inner working of the model.

What I found is this is indeed a challenge for the chatbot, it felt bound by regular English language and had difficulties in consistently generating texts that are nonsensical, opposite to humans, who are good at doing that. But this is also tricky in asking this to the chatbot: its failure might be due to my inability to clearly express what I wanted.

But here are some examples:

Book sky under, shoes over whispers loudly, moon singing clocks dance whispering. Rain bounces, sunlight bakes ocean wears, stars jumping library cookies, chair flies music taste colors, purple dances on green melodies, fish walk through dreaming clouds, trees listen to underground stars, flowers paint the wind with silent songs. Piano eats the rainbow, curtains follow the river, mountains chase the breeze, letters find their way in a sea of numbers, shadows play hide and seek with the dawn, dreams weave patterns in the fabric of reality, time dances around the echo of its own heartbeat, words melt into the canvas of silence, creating a symphony of mismatched thoughts.

For this one, the beginning does a better job being nonsensical. For some reason, it became more coherent, and hence wrong, as it went on. I asked it to generate this one that is longer because all the preceding examples I got were one sentence long. So I thought there could be some engineered bias towards shorter answers when it is less confident in its text. And yes, as it went on, it returned to its old pattern of generating texts that are more predictable and coherent.

One more thing is that the most basic pattern in English: noun -> verb is still preserved throughout. The chatbot apparently couldn’t break out of it.

Flibber flabber, zib zob zub, wibble wobble, drib drab dribble, klick klack kloke, blibber blubber blabble, snick snack snook.

I asked it to write some gibberish. As you can see, these gibberish words are sill following some patterns.

Zib zup zot, qix qux. Lip lim lop, nif nux. Bim bop, kix kux. Zim zom, qip qup. Wib wob, dix dux. Fim fop, gix gux. Hix hux, jip jup.

Then I asked it to write something that completely uses words that it invents (meaning I shouldn’t be able to find them on google) and these words have to be less than 4 characters. Again, there are clear patterns in the words it generated. These 3-letter words all have a vowel in the middle. But of course, there are only less than 20k 3-letter combinations so this isn’t entirely fair.

https://chat.openai.com/share/a2ec2c52-d45f-4bff-81f0-0de7d336082a

2. Can ChatGPT generate any useful diagrams?

This experiment came about when I realized all diagrams (i.e., symbolic representations of information using visualization techniques) from ChatGPT are literally, trash. Like this one:

show me a example Guice object graph

More to do this week.