Trick or Treat on Specter Street: Widow's Web
I am very stucked in Trick or Treat on Specter Street: Widow's Web I can't do none of the questions, but in any case I start by 4th that is the first answerable one Your first task is to simulate the loyal Crawlers. Run legitimate-crawler and inspect the output in Lab-Files to observe their behavior. To simulate the rogue Crawlers, you must discover the hidden paths on the website. Read the blog posts – they contain clues. Disallow these in Website-Files/robots.txt and run malicious-crawler. Inspect the output in Lab-Files. What is the token? I have created the robots.txt file since I understand that malicious-crawler goes expressedly there. My robots.txt contains all url's I can imagin Disallow: /secret Disallow: /treat Disallow: /hidden Disallow: /crypt Disallow: /warden Disallow: /rituals Disallow: /witch-secrets Disallow: /admin Disallow: /vault Disallow: /uncover Disallow: /post1 Disallow: /post2 Disallow: /post3 Disallow: /post4 Disallow: /contact Disallow: /drafts/rituals But the result of malicious-crawler.txt doesn't give me either a token nor a hint I have curl-ed all pages looking for words as token and nothing. I have found some key words in http://127.0.0.1:3000/witch-secrets as intercepted-incantations, decoded them and nothing. I have searched in spider-sigthings.log what hapened at 3.00 am but nothing Can someone gime me a hint?
