50 fix boundary issue in multipartform data (#52)

bpbecker · web-flow · commit b8629e8dbb73 · 2024-04-03T14:15:49.000-04:00
* Fix #50 Address boundary issue * v5.5/5.6 documentation
diff --git a/docs/content-types.md b/docs/content-types.md
@@ -0,0 +1,67 @@
+If your HTTP request has a payload and you do not specify the content type, `HttpCommand` will attempt to determine whether to use a content type of `'application/json'` or `'application/x-www-form-urlencoded'`.
+
+This may be fine for interactively tinkering in the APL session. But when running `HttpCommand` under program control you should **explicitly specify the content type** for the payload by either setting `ContentType` or adding a `content-type` header. 
+
+The exception to this is when using `GetJSON` which is specifically intended to interact with JSON-based web services and will use a default content type of `application/json`. 
+
+### Special Treatment of Content Type `application/json` 
+If you specify a content type of `'application/json'`, `HttpCommand` will automatically convert a non-JSON `Params` setting to JSON. In the rare case where `Params` is an APL character vector that happens to represent valid JSON, you should convert it yourself using `1 ⎕JSON`.
+
+### Special Treatment of Content Type `multipart/form-data`
+Content type `multipart/form-data` is commonly used to transfer files to a server or to send multiple types of content in the request payload. If you specify a content type of `'multipart/form-data'`:
+
+* `Params` must be a namespace with named elements.
+* Each element in `Params` consists of the data for the element optionally followed by a content type for the element.
+* To send a file, prefix the file name with either:
+    * `@` to upload the file's content and its name
+    * `<` to upload just the file's content
+* If you do not specify a content type for the file, `HttpCommand` will use a content type of `'text/plain'` if the extension of the file is .txt, otherwise it will use a content type of `'application/octet-stream'`. 
+
+In the example below: 
+
+* Extra newlines have been removed for compactness.
+* The file `/tmp/foo.txt` contains `Hello World`.
+* We create 4 parts to be sent with the request:
+    * a simple string
+    * a named file - both the content and file name will be sent
+    * an unnamed file - only the content will be sent
+    * a JSON array (with content type 'application/json')
+```
+      h←HttpCommand.New 'post' 'someurl.com'
+      p←⎕NS ''                     ⍝ create a namespace
+      p.string←'/tmp/foo.txt'       ⍝ just a value
+      p.namedfile←'@/tmp/foo.txt'   ⍝ @ = include the file name
+      p.unnamedfile←'</tmp/foo.txt' ⍝ < = do not include the file name
+      p.json←'[1,2,3]' 'application/json' ⍝ value and content type
+      h.Params←p                    ⍝ assign the request Params
+      h.ContentType←'multipart/form-data' ⍝ 
+      h.Show
+POST / HTTP/1.1
+Host: someurl.com                                                                                              
+User-Agent: Dyalog-HttpCommand/5.6.0
+Accept: */*
+Accept-Encoding: gzip, deflate
+Content-Type: multipart/form-data; boundary=rgt7DIuxBqLFsveaLM0fBcR7gdvahhUfbtmuQ9UMEZvv9kDVrd
+Content-Length: 631
+
+--rgt7DIuxBqLFsveaLM0fBcR7gdvahhUfbtmuQ9UMEZvv9kDVrd
+Content-Disposition: form-data; name="json"
+Content-Type: application/json                                                                                              
+[1,2,3]
+
+--rgt7DIuxBqLFsveaLM0fBcR7gdvahhUfbtmuQ9UMEZvv9kDVrd
+Content-Disposition: form-data; name="namedfile"; filename="foo.txt"
+Content-Type: text/plain
+Hello World
+
+--rgt7DIuxBqLFsveaLM0fBcR7gdvahhUfbtmuQ9UMEZvv9kDVrd
+Content-Disposition: form-data; name="string"                                                 
+/tmp/foo.txt
+
+--rgt7DIuxBqLFsveaLM0fBcR7gdvahhUfbtmuQ9UMEZvv9kDVrd
+Content-Disposition: form-data; name="unnamedfile"
+Content-Type: text/plain
+Hello World
+
+--rgt7DIuxBqLFsveaLM0fBcR7gdvahhUfbtmuQ9UMEZvv9kDVrd--
+```
diff --git a/docs/css/main.css b/docs/css/main.css
@@ -53,4 +53,26 @@ pre + pre , hr + pre {
 }
 pre + pre > code, hr + pre > code {
     padding-top: 0.2em!important;
+}
+
+/* make admonition styling a la Dyalog */
+.md-typeset .admonition,
+.md-typeset details { 
+	border-color: #ED7F00 !important;
+	box-shadow: #ED7F00 !important;
+}
+
+.md-typeset .admonition-title {
+	background-color: #F3AD5b !important;
+} 
+
+.md-typeset .admonition-title:before,
+.md-typeset summary:before{
+	background-color: #F8F8F8 !important;
+}
+
+.md-typeset pre > code {
+	overflow: scroll !important;
+	scrollbar-color: black;
+	scrollbar-width: thin;
 }
diff --git a/docs/index.md b/docs/index.md
@@ -1,24 +1,36 @@
+`HttpCommand` is a utility is designed to make it easy for the APL user to send requests to and receive responses from HTTP servers like web servers and web services.
 
-__Note__: While `HttpCommand` itself is `⎕IO` and `⎕ML` insensitive, the examples in this documentation assume an environment of `(⎕IO ⎕ML)←1`.
+!!! note
+    While `HttpCommand` itself is `⎕IO` and `⎕ML` insensitive, the examples in this documentation assume an environment of `(⎕IO ⎕ML)←1`.
 
 ## Loading `HttpCommand`
-`HttpCommand` is a utility is designed to make it easy for the APL user to send requests to and receive responses from HTTP servers like web servers and web services. `HttpCommand` is included with Dyalog APL as a loadable utility. To bring it into your workspace, simply do:
+ `HttpCommand` is included with your Dyalog APL installation. To bring it into your workspace:
+
+### Dyalog APL Version 19.0 and later
 ```APL
-      ]Load HttpCommand
+      ]import {ns} HttpCommand
 ```
 or, under program control, do:
 ```APL
-      ⎕SE.SALT.Load 'HttpCommand' 
+      ⎕SE.Link.Import {ns} 'HttpCommand' 
 ```
-Beginning with Dyalog v18.2, you can also use the `]get` user command:
+where `{ns}` is an optional namespace name or reference.
+
+### Dyalog APL versions before 19.0
 ```APL
-      ]Get HttpCommand
+      ]load HttpCommand {-target=ns}
 ```
+or, under program control, do:
+```APL
+      ⎕SE.SALT.Load 'HttpCommand {-target=ns}' 
+```
+`{-target=ns}` optionally specifies the namespace in which to load `HttpCommand`. `ns` is the namespace name. 
+
 
-## Upgrading to `HttpCommand` Version 5
-This documentation describes version 5 of `HttpCommand`.  Dyalog versions 18.2 and 18.0 have earlier versions of `HttpCommand` pre-installed. To upgrade your in-workspace copy of `HttpCommand` to version 5, you may have to run [`HttpCommand.Upgrade`](./misc-methods.md#upgrade) once or twice depending on how recent your Dyalog installation is.
+## Upgrading to the Latest `HttpCommand`
+This documentation describes the latest version of `HttpCommand`, which may be more recent than the version of `HttpCommand` that came pre-installed with your Dyalog APL. Use [`HttpCommand.Upgrade`](./misc-methods.md#upgrade) upgrade your in-workspace copy of `HttpCommand`. Note that this only upgrades your in-workspace copy of `HttpCommand` and does not overwrite the version included in your Dyalog installation.
 
-If `HttpCommand.Version` reports a version earlier than 4.0.14, `HttpCommand.Upgrade` will upgrade to the latest version 4. Then running `HttpCommand.Upgrade` once more will upgrade to version 5.
+If `HttpCommand.Version` reports a version earlier than 4.0.14, `HttpCommand.Upgrade` will need to be run twice to upgrade to the latest version.
 
 `HttpCommand.Upgrade` does not work with Classic interpreters. See [`Upgrade`](./misc-methods.md#upgrade) for more information.
 
@@ -33,9 +45,12 @@ For example, `HttpCommand` version 3.4 may be found in Dyalog version 18.0.
       HttpCommand.Version  ⍝ verify that we upgraded to v4.0.14
  HttpCommand  4.0.14  2022-09-04 
       HttpCommand.Upgrade  ⍝ now upgrade to v5
-0  Upgraded to HttpCommand 5.0.7 2022-08-27 from HttpCommand 4.0.14 2022-09-04 
+0  Upgraded to HttpCommand 5.6.0 2024-03-17 from HttpCommand 4.0.14 2022-09-04 
 ```
 
+!!! note
+    When deploying `HttpCommand` as a part of your application, you should save a copy with your application to ensure you know precisely what version of `HttpCommand` is being used. You don't want to dynamically load or upgrade `HttpCommand` in your application.
+
 ## `HttpCommand` is Available as a Tatin Package 
 [Tatin](https://tatin.dev) is a package manager for APL-based packages. `HttpCommand` is available as a Tatin package. If you have the Tatin client installed, you can load `HttpCommand` using:
 ```APL
@@ -44,7 +59,7 @@ For example, `HttpCommand` version 3.4 may be found in Dyalog version 18.0.
 The Tatin client will be included in your Dyalog installation beginning with Dyalog version 19.0. For earlier versions of Dyalog, refer to the [Tatin website](https://tatin.dev) for instructions on installing the Tatin client.
 
 !!! warning "Load Once, Use Often"
-     It is **strongly recommended** that you save your own copy of `HttpCommand` in your application rather than dynamically loading or upgrading it at runtime. In particular, it is bad practice to repeated load `HttpCommand` as this may cause Conga to  reinitialize each time `HttpCommand` is loaded and may interfere with other components of your application that also use Conga. If `HttpCommand` is being used within your application (as opposed to ad hoc usage in your APL session) it is recommended that all Conga-using components refer to the same Conga namespace:
+     It is **strongly recommended** that you save your own copy of `HttpCommand` in your application rather than dynamically loading or upgrading it at runtime. In particular, it is bad practice to repeated load `HttpCommand` as this may cause Conga to  reinitialize each time `HttpCommand` is loaded and may interfere with other components of your application that also use Conga. If `HttpCommand` is being used within your application (as opposed to ad hoc usage in your APL session) it is recommended that all Conga-using components refer to the same Conga namespace. See [HttpCommand and Conga](./conga.md)
 
 ## Typical Usage Patterns
 
diff --git a/docs/release-notes.md b/docs/release-notes.md
@@ -1,3 +1,5 @@
+## Version 5.6
+* Added support for content type "multipart/form-data". See [Content Types](./userguide.md#content-types).
 ## Version 5.5
 * Added configuration setting [`Secret`](./operational-settings.md#secret) which will suppress the display of credentials in the authorization header. This is primarily so that one can demo using authenticated requests without having their credentials displayed in the session.
 * Added ability for HTTP header names and values to reference environment variables by enclosing the environment variable name in % (e.g. `%MyPassword%`). This provides additional security by reducing the need to store sensitive or variable information inline. This is particularly useful when setting [`Auth`](./request-settings.md#auth).  
diff --git a/docs/userguide.md b/docs/userguide.md
@@ -25,13 +25,13 @@ Typical use of `HttpCommand` might follow this pattern.
 ```
  resp ← HttpCommand.Get 'some-url'
 :If resp.IsOK ⍝ use the IsOK function from the result namespace
-    ⍝ code to handle bad request
-:Else
     ⍝ code to process the response 
+:Else
+    ⍝ code to handle bad request
 :EndIf
 ```
 
-If you expect to make several `HttpCommand` calls, you may want to create an instance and then update the necessary settings and execute the `Run` method for each call.  This is particularly useful when the requests are made to the same host as the connection to the host will be kept open, unless the host itself closes it.
+If you expect to make several `HttpCommand` calls, you may want to create an instance and then update the necessary settings and execute the `Run` method for each call.  This can be useful when the requests are made to the same host as the connection to the host will be kept open, unless the host itself closes it.
 
 ```
  hc←HttpCommand.New 'get'  ⍝ here we expect all requests to be HTTP GET
@@ -47,21 +47,12 @@ If you expect to make several `HttpCommand` calls, you may want to create an ins
   :EndIf
 :EndFor 
 ```
-## Content Types
-👉 If your HTTP request has a payload and you do not specify the content type, `HttpCommand` will attempt to determine whether to use a content type of `'application/json'` or `'application/x-www-form-urlencoded'`.
-
-This may be fine for interactively tinkering in the APL session. But when running `HttpCommand` under program control you should **explicitly specify the content type** for the payload by either setting `ContentType` or adding a `content-type` header. 
 
-The exception to this is when using `GetJSON` which is specifically intended to interact with JSON-based web services and will use a default content type of `application/json`. 
-### Special Treatment of `application/json` Content Type
-If you specify a content type of `'application/json'`, `HttpCommand` will automatically convert a non-JSON `Params` setting to JSON. In the rare case where `Params` is an APL character vector that is valid JSON, you should convert it yourself using `1 ⎕JSON`.
 ## `Timeout` and Long-running Requests
 The default `Timeout` setting (10 seconds) is adequate for most requests. There are a couple of patterns of long running requests.  `Timeout` can be set to a positive or negative number.
 
 - Setting `Timeout` to a positive number means that `HttpCommand` will time out after `Timeout` seconds with a return code (`rc`) of 100.  Any partial payload received will returned in `Data` element of the result. 
-- Setting `Timeout` to a negative number means that `HttpCommand` will not time out as long as data is being received.  This is useful in the case where a large payload may be received but you are uncertain of how long it will take to receive. If no data is received within a period of `Timeout` seconds, `HttpCommand` will time out with a return code (`rc`) of 100. Any partial payload received will be returned in the `Data` element of the result. 
-
-Using a negative `Timeout` value is useful in the case where a large payload is being received in chunks but has no benefit if the entire payload is sent in one chunk or if the host takes more than `|Timeout` seconds to begin sending its response. In that case, you'll need to set `|Timeout` to a larger value.
+- Setting `Timeout` to a negative number means that `HttpCommand` will not time out as long as data is being received.  This is useful in the case where a large payload may be received but you are uncertain of how long it will take to receive. If no data is received within a period of `|Timeout` seconds, `HttpCommand` will time out with a return code (`rc`) of 100. Any partial payload received will be returned in the `Data` element of the result.<br/><br/> Using a negative `Timeout` value is useful in the case where a large payload is being received in chunks but has no benefit if the entire payload is sent in one chunk or if the host takes more than `|Timeout` seconds to begin sending its response. In that case, you'll need to set `|Timeout` to a larger value.
 
 ## Compressing Response Payload ##
 `HttpCommand` can accept and process response payloads that are compressed using either the gzip or deflate compression schemes. To accomplish this, you need to set the `accept-encoding` header to `'gzip, deflate'`.
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -2,7 +2,7 @@ site_name: HttpCommand
 repo_url: https://github.com/dyalog/httpcommand
 repo_name: Dyalog/HttpCommand
 dev_addr: 127.0.0.1:22222
-copyright: Made with <a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener"><strong>Material for MkDocs</strong></a><br/>Copyright &copy; 2015-2023 <strong><a href="https://dyalog.com" target="_blank" rel="noopener">Dyalog, LTD</a></strong>
+copyright: Made with <a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener"><strong>Material for MkDocs</strong></a><br/>Copyright &copy; 2015-2024 <strong><a href="https://dyalog.com" target="_blank" rel="noopener">Dyalog, LTD</a></strong>
 nav:
     - Overview: 'index.md'
     - Usage:
@@ -27,6 +27,7 @@ nav:
         - 'Response Elements': 'result-response.md' # complete
       - 'Messages and Return Codes': 'msgs.md' # complete
     - Advanced Topics:
+      - 'Request Content Types' : 'content-types.md'
       - 'HttpCommand and Conga' : 'conga.md' # complete
       - 'Secure Communications' : 'secure.md' # complete
       - 'Using a Proxy Server': 'proxy.md' 

Original file line number	Diff line number	Diff line change
`@@ -1,3 +1,5 @@`
	`1`	`+## Version 5.6`
	`2`	`+* Added support for content type "multipart/form-data". See [Content Types](./userguide.md#content-types).`
`1`	`3`	`## Version 5.5`
`2`	`4`	* Added configuration setting [`Secret`](./operational-settings.md#secret) which will suppress the display of credentials in the authorization header. This is primarily so that one can demo using authenticated requests without having their credentials displayed in the session.
`3`	`5`	* Added ability for HTTP header names and values to reference environment variables by enclosing the environment variable name in % (e.g. `%MyPassword%`). This provides additional security by reducing the need to store sensitive or variable information inline. This is particularly useful when setting [`Auth`](./request-settings.md#auth).