Licensed text corpora
A licensed text corpus is a body of articles, books, forum posts or documents that a rights holder licenses to AI developers for training or retrieval.
Last reviewed
What licensed text corpora contains
- Articles, books or posts
- Metadata and timestamps
- License terms and permitted uses
- Update feeds
Who buys it and how it is priced
| Attribute | Value |
|---|---|
| Belongs to | pretraining data |
| Main buyers | Foundation-model developers and search and answer products |
| Pricing | Large publishers and platforms negotiate annual fees, some reported in the tens of millions of dollars. Smaller owners use marketplaces with revenue-share or per-use terms. |
Quality criteria and risks
Quality criteria
- Clean rights chain
- Freshness
- Uniqueness versus the public web
- Structured metadata
Risks
- Disputed rights
- Scraped content mixed in
- Unclear scope for training versus display
Providers that list licensed text corpora
Auction and pricing mechanisms that fit
- Posted-price sale: Fits catalogue datasets that can be sold many times, such as cloud data exchange listings.
- Revenue-share licensing: Fits long-term content licenses where value is uncertain up front and both sides want to share upside.
- English auction: Fits one-off, scarce assets where buyers can see each other, such as a unique archive of footage or a rare-language speech corpus.
Key terms
- data licensing
- Data licensing is the grant of rights to use a dataset for stated purposes, in return for payment or other terms.
- opt-out
- An opt-out is a signal or process by which a rights holder tells AI developers not to use their content for training.
- fair use
- Fair use is a US copyright doctrine that permits some uses of copyrighted work without permission.
- revenue share
- Revenue share is a payment model in which a data owner receives a percentage of the revenue generated from using the data.
Frequently asked questions
- How much do AI labs pay to license text?
- Reported deals range from undisclosed sums to figures such as about $60 million a year for Reddit and Google. See the deal tracker for dated, sourced examples.