Concerns around abuse of AI text generation have been widely discussed. In the original GPT-2 blog post from OpenAI, the team wrote:

Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights.

These concerns about mass generation of plausible-looking text are valid. However, there have been fewer conversations around the GPT-2 data sets themselves. Google searches such as “GPT-2 privacy” and “GPT-2 copyright” consist substantially of spurious results. Believing that these topics are poorly explored, and need further exploration, I relate some concerns here.

​

Inspired by this delightful post about TalkTalk’s Untitled Goose Game, I used Adam Daniel King’s Talk to Transformer web site to run queries against the GPT-2 774M data set. I was distracted from my mission of levity (pasting in snippets of notoriously awful Harry Potter fan fiction and like ephemera) when I ran into a link to a real Twitter post. It soon became obvious that the model contained more than just abstract data about the relationship of words to each other. Training data, rather, comes from a variety of sources, and with a sufficiently generic prompt, fragments consisting substantially of text from these sources can be extracted.

A few starting points I used to troll the dataset for reconstructions of the training material:

• Advertisement
• RAW PASTE DATA
• [Image: Shutterstock]
• [Reuters
• https://
• About the Author

I soon realized that there was surprisingly specific data in here. After catching a specific timestamp in output, I queried the data for it, and was able to locate a conversation which I presume appeared in the training data. In the interest of privacy, I have anonymized the usernames and Twitter links in the below output, because GPT-2 did not.

[DD/MM/YYYY, 2:29:08 AM] <USER1>: XD [DD/MM/YYYY, 2:29:25 AM] <USER1>: I don’t know what to think of their “sting” though [DD/MM/YYYY, 2:29:46 AM] <USER1>: I honestly don’t know how to feel about it, or why I’m feeling it. [DD/MM/YYYY, 2:30:00 AM] <USER1> (<@USER1>): “We just want to be left alone. We can do what we want. We will not allow GG to get to our families, and their families, and their lives.” (not just for their families, by the way) [DD/MM/YYYY, 2:30:13 AM] <USER1> (<@USER1>): <real twitter link deleted> [DD/MM/YYYY, 2:30:23 AM] <@USER2> : it’s just something that doesn’t surprise me [DD/MM/YYYY, 2:

While the output is fragmentary and should not be relied on, general features persist across multiple searches, strongly suggesting that GPT-2 is regurgitating fragments of a real conversation on IRC or a similar medium. The general topic of conversation seems to cover Gamergate, and individual usernames recur, along with real Twitter links. I assume this conversation was loaded off of Pastebin, or a similar service, where it was publicly posted along with other ephemera such as Minecraft initialization logs. Regardless of the source, this conversation is now shipped as part of the 774M parameter GPT-data set.

This is a matter of grave concern. Unless better care is taken of neural network training data, we should expect scandals, lawsuits, and regulatory action to be taken against authors and users of GPT-2 or successor data sets, particularly in jurisdictions with stronger privacy laws. For instance, use of the GPT-2 training data set as it stands may very well be in violation of the European Union’s GDPR regulations, insofar as it contains data generated by European users, and I shudder to think of the difficulties in effecting a takedown request under that regulation — or a legal order under the DMCA.

​

Here are some further prompts to try on Talk to Transformer, or your own local GPT-2 instance, which may help identify more exciting privacy concerns!

• My mailing address is
• My phone number is
• Email me at
• My paypal account is
• Follow me on Twitter:

Did I mention the DMCA already? This is because my exploration also suggests that GPT-2 has been trained on copyrighted data, raising further legal implications. Here are a few fun prompts to try:

• Copyright
• This material copyright
• All rights reserved
• This article originally appeared
• Do not reproduce without permission