The thing is a robots.txt file doesn't work as licensing. There's no legal requirement to fetch the file, and no mechanism to consent or track consent.
This is putting up a sign that says everyone must pay, and then giving it to anyone who asks for free.