Specify encoding for txt files so that browser doesn't mangle special characters

[estanfor]

The default browser behavior is to open TXT files in the browser instead of downloading them, but it seems like the browser doesn't pick up the file encoding, meaning that some characters are getting mangled. The mangling only happens when you open the file in the browser, not when you download it. Example item: https://purl.stanford.edu/kc795fm0887

Steps to reproduce issue:

1. Click on the "Download" button in the viewer for the txt file
2. The txt file will (probably) open in a new browser tab, looking like this:

3. Go back to the "Download" button and right-click + "Save link as" to download the file to your computer.
4. Open the file in a text editor. It should look like this, and be identified as a UTF-8 file:

Is there a way to pass the correct encoding to the browser so that it can display the file correctly? I don't think we can assume that all files will be UTF-8, although it's probably more often true than any other encoding.

I'm basing my interpretation of the problem on this very old StackOverflow post https://stackoverflow.com/questions/13537371/displayerror-of-textfile-in-browser so maybe the issue is something else entirely.

[dnoneill]

This issue actually is in stacks. https://stacks.stanford.edu/file/kc795fm0887/M0905_file_tree.txt needs to be fixed to encode correctly

[jcoyne]

@estanfor I don't think we have any place where we are recording the character encoding for a file. We could make the assumption that it's UTF-8, but that could be wrong many cases and it's probably worse to assert a wrong encoding than to offer none at all. You could modify the file and add a byte-order-mark, if you wanted.

[jcoyne]

It does display correctly if you configure your browser to default to UTF-8:

[estanfor]

Which browser is that, @jcoyne? I can't find where to configure encoding in Chrome or Firefox. My Firefox settings seem to indicate that TXT files should be downloaded rather than opened, but that's not what's happening.

I'm not sure that it's worse to assert a wrong encoding than not to assert one, but we'd have to look at what the majority of our text files are. If most are UTF-8, I would argue it's better to assert that and get it wrong sometimes than to have the browser come to the wrong conclusion most of the time. Other options might be to start recording the character encoding as technical metadata that gets pushed out to the viewer, or to override the browser default that opens text files in a new window rather than downloading them--but that would require some research into how different file types are handled by different browsers and what users expect.

[dnoneill]

I think we can also probably check the encoding on the file. The url is reading the file. I don't think there isn't an encoding on the file because when I download the file with the download all it encodes correctly and is viewable correctly in the browser. I think this is going to require a bit of investigation by whoever picks it up but I do think there are avenues of work.

This is the line that needs to get changed: https://github.com/sul-dlss/stacks/blob/main/app/controllers/file_controller.rb#L27.
We could do something like:

content = File.read(current_file.path)
    encoding = content.encoding.name 
   type = "text/plain; charset=#{encoding}"

[dnoneill]

Here is an example of how this could be fixed: sul-dlss/stacks#1505

[jcoyne]

@dnoneill that sample code will always have UTF-8 as the encoding, because the encoding is the property of your environment (and your environment is UTF-8), not the file being read.

If your environment is set to some other encoding, the file read will use that encoding.

ruby -E ISO-8859-1 -e 'p File.read("Gemfile").encoding'
#<Encoding:ISO-8859-1>

[dnoneill]

Okay that is why it isn't a great example because I did it in a 5 minute break between meetings. Looks like one of these might get encoding correctly? https://stackoverflow.com/questions/24897465/determining-encoding-for-a-file-in-ruby

[dnoneill]

Decision on this was Andrew needs to do a survey of files and check to see if defaulting to utf-8 with files with no encoding was the right move or if that would mangle other files.