A good place to start is with the parameters that XMetaL provides for adjusting PDF output. If what you need to do can be done via a parameter, it will probably be easier than editing XSL. See here:
http://na.justsystems.com/webhelp/en/xmetalauthor/enterprise/6.0/XMEE.html#Appendix%20D:%20Configuring%20PDF%20outputIf you want to modify anything in the Open Toolkit folder, be sure to read this first to make sure you're working in the right folder:
http://forums.xmetal.com/index.php/topic,939.0.htmlXMetaL does not have a UI for editing XSL files. There are various good third-party applications for editing XSL, or you could use a free text editor such as Notepad++.
Regarding PDF customization in general, there is a good whitepaper here:
www.scriptorium.com/whitepapers/ditaotpdf/DITA-PDF-tweaks.pdf . Note that the folder for customizing "XMetaL Enhanced PDF" output is
demo/xmfo, not
demo/fo.
Good luck!
Su-Laine