For the non developers, the web industry is using .txt files at the root of a domain to provide guidance for different kind of bots wandering on the Internet and indexing/scrapping data. The bots can ignore the instructions, declare a different User-Agent or other kind of evasion methods.
- robots.txt - it instructs crawlers on which resources can be indexed
- ads.txt - used for serving ads and can be used to provide transparency on the approved platforms
- humans.txt - to provide credit to the members of the team that are behind a website
- security.txt - contact information in case of security incidents and for responsible disclosures
All the above are industry attempts to set guidelines and common sense.
Currently the AI industry is like a wild west. No rules, no regulation and everybody is training models on data that "I found it on the Internet". Well, if it is on Internet it doesn't mean that it is free or that it can be used in models that most of the times have a commercial scope.
We are experiencing a new era, a shift from legacy code to end to end machine learning. For this to work we need a new hardware architecture, a new paradigm and huge amounts of data. It looks like all the big players and foundation models developers are throwing everything at the wall and see what sticks while using any data that is "on the Internet". Just watch the interview with Mira Murati (CTO OpenAI) for WSJ:
The results are mindblowing but pretty sure the model used data that was not licensed for this. We already see actions in justice for the improper use of copyrighted materials for the training of LLMs. This time is way harder to prove that some material was used for training. Remember when Bing was using Google results to serve them to Bing users?
I am expecting to see the industry pushing a new .txt file, maybe ai.txt or ml.txt in an initial effort to bring a bit of order in this chaos.
