ESC currently handles binary data by coercing binary values to a (potentially-invalid) strings. This works okay internally, since Go does not validate the wellformedness of strings when coercing from binary data. However, the standard Go JSON encoder ensures that all strings are coerced to valid UTF-8 during marshaling, which replaces any invalid bytes with the replacement character. This corrupts the data.

For example, the following environment should write the bytes dcc5 43bc 1b4f f4fb 8263 d630 a199 2688 166b c084 5b71 1240 24c3 a944 d937 e2fc to a temporary file:

values:
  files:
    MY_FILE:
      fn::fromBase64: 3MVDvBtP9PuCY9YwoZkmiBZrwIRbcRJAJMOpRNk34vw=

Instead, it writes the bytes efbf bdef bfbd 43ef bfbd 1b4f efbf bdef bfbd efbf bd63 efbf bd30 efbf bdef bfbd 26ef bfbd 166b efbf bdef bfbd 5b71 1240 24c3 a944 efbf bd37 efbf bdef bfbd, because the decoded value of MY_FILE is marshaled as "��C�\u001bO���c�0��\u0026�\u0016k��[q\u0012@$éD�7��" due to the presence of invalid UTF-8 bytes in the decoded value.

The most reasonable proposal I can think of is:

1. Stop coercing binary values to strings in the evaluator
   • This may require allowing []byte values in esc.Value
2. Marshal all []byte values as base64 (this is the standard behavior of encoding/json)
3. Extend the behavior of files so that it allows the user to supply an object with the key data that accepts base64-encoded data which is then decoded prior to writing the file

Another alternative for (3) is to attempt to decode all values in files as base64 and then use the decoded bytes if decoding succeeds. However, the likelihood of values decoding as valid base64 may be high enough that this would result in breaks in existing environments.