Hyrax - Configuration
When you install Hyrax for the first time it is pre-configured to serve test data sets that come with each of the installed data handlers. This will allow you to test the server and make sure it is functioning correctly. After that you can customize it for your data.
To correctly deploy a robots.txt file for Hyrax, you must correctly deploy it for Tomcat. This means that your robots.txt file must be accessible here:
Placing it lower in the URL path doesn't seem to work.
In order to get Tomcat to serve the file from that location you must place it in $CATALINA_HOME/webapps/ROOT
If you still find that you system is being cruelly burdened with robot traffic then you might want to try the BotBlocker handler for the OLFS.