I'm not sure XSLT is a good idea for validation, or if it is I'm not sure it makes sense to try to build something from scratch that would use XSLT when there are alternatives.
Regardless, I think your best bet, if you are consistently receiving invalid XML from some source is to modify the tool creating that source so that it creates proper output. Doesn't it seem wrong to rely on a second tool to help fix up issues another tool is creating? If this happens once or twice for the odd file then sure, that makes sense, but if the issue is ongoing then I think it makes the most sense to address the issue at the source.
If these are DITA documents perhaps the easiest thing to do is to run them through the DITA OT. By default the DITA OT validates input documents (the ANT parameter named "validate" defaults to "true"). Just generate output then check for errors in the output log file. Of course this will not fix the errors, it will just identify them. You'd need to fix them separately, and ideally (if this is an ongoing problem) correct the tool that is introducing the errors.
Note that if you try to reference an invalid DITA document by adding a topicref to a DITA map in XMetaL Author Enterprise that will fail, so your maps will also need to have been generated by this other source, or your maps would need to have been created in XMetaL Author Enterprise when your topics were valid. If at some point after that the topic is modified to be invalid XMetaL Author Enterprise isn't going to catch that because the reference has already been made. Validating all the topics referenced in a map is considered redundant because the DITA OT already does that.
I'm attaching a screenshot of the kind of thing you can look for in the output log file. The referenced file noted in these errors begins as follows and contains an element named <bogus> which is not allowed and also improperly nested:
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<title>Welcome to XMetaL Author</title>