Most precisely, if you specify a path that XMetaL cannot otherwise locate it checks the Rules folder.
So, if you have a SYSTEM id that just specifies the DTD filename with no path AND the DTD is not in the same folder as the XML document then the Rules folder is used.
I only mention this in case you happen to have the DTD and XML file in the same folder (this may be common while developing customizations for XMetaL).
Very detailed explanation of the lookup is located here:
http://www.slideshare.net/XMetaL/deploying-schemas-and-xmetal-customization-filesSee slide #8 for my flowchart. That document is likely worth reading from start to end if you have questions about how the product locates DTDs and the associated customization files.
Unfortunately, the Webex recording that goes with that slide deck was corrupted so we do not have it, however, I think the slides themselves answer any questions you might have.