I have the following code in robots.txt to allow crawling from everyone for now
User-agent: * Disallow:
Before I changed this the layout of the file was this below. I've been looking for details about unique and I can't find it. Anyone see this before and what is "unique" doing exactly?
User-agent: * Disallow: /unique/