robots.txt — allow the paths that earn money
Write a short file. Allow everything public, disallow admin, cart, internal search and checkout, and never disallow CSS or JavaScript that the page needs to render. Publish it at the domain root, not in a subfolder, and keep one copy per environment.