You are referencing 'fileName' as a string, but it should be a variable. So replace it to this:
var f = file.ReadFromFile(filename);
Also note variable names are case sensitive.
Then, check your file contents has xml first, use dlg.ShowMessage at various points, for example verify the path that is used, verify your file contains the xml. It can be difficult to debug in the script editor so doing this can help a lot.
I have not been following it closely because it really hasnt interested me but just my first thought is your making something extremely complicated that could be handled with simple actions and rules in SambaPOS. Instead your trying to script everything…
Sorry, I misread your script, o.toString() will not work as o is an XML object, not a string. The error message you posted is a generic one when you try to pass a non string to dlg.ShowMessage().
Try
var s = o.getElementByTagName('ds:DigestValue');
That may return an object, so I think you can try one of these:
It all points to the fact that may not be XML data there. Did you test the file name and that you have XML returned first? If you don’t do that, there is no point debugging further until you confirm your text from the file is actually loaded into f.