Sending email via sendgrid fails

[smokedsalmonbagel]

I put my api key into the cocalc admin settings and tried the forgot password email. It seems to fail with this message in the logs.

debug: send_email(to:MY@EMAIL.com) -- sending email to MY@EMAIL.com -- error = {
  "message": "Unauthorized",
  "code": 401,
  "response": {
    "headers": {
      "server": "nginx",
      "date": "Fri, 22 May 2020 04:37:01 GMT",
      "content-type": "application/json",
      "content-length": "116",
      "connection": "close",
      "access-control-allow-origin": "https://sendgrid.api-docs.io",
      "access-control-allow-methods": "POST",
      "access-control-allow-headers": "Authorization, Content-Type, On-behalf-of, x-sg-elas-acl",
      "access-control-max-age": "600",
      "x-no-cors-reason": "https://sendgrid.com/docs/Classroom/Basics/API/cors.html"
    },
    "body": {
      "errors": [
        {
          "message": "The provided authorization grant is invalid, expired, or revoked",
          "field": null,
          "help": null
        }
      ]
    }
  }
}

Is there a way to find and display the API key cocalc is using (or at least check its length) to make sure it is actually set? I think there might be a an issue with some of these email related settings saving properly... I also noticed the container needs to be restarted for some settings to take effect.

I tried sending via curl from within the container - that worked just fine.

[williamstein]

Is there a way to find and display the API key cocalc is using (or at least check its length) to make sure it is actually set?

Query the server_settings table in the PostgreSQL database.

Here's what a properly configured table with basically all options set might look like, where I put xxx through private info:

smc=# select * from server_settings;
             name             |                                 value                                 
------------------------------+-----------------------------------------------------------------------
 commercial                   | yes
 stripe_secret_key            | sk_test_xxxx
 stripe_publishable_key       | pk_test_xxx
 iframe_comm_hosts            | dev.xxxx
 google_analytics             | UA-xxx-1
 organization_email           | help@xxx
 organization_name            | Sagemath, Inc.
 site_description             | Testing CoCalc
 site_name                    | TestCalc
 sendgrid_key                 | SG...xxxxXg
 password_reset_smtp_password | exxxxqr
 account_creation_token       | 
 kucalc                       | onprem
 dns                          | xxx
 project-tag                  | 2020-05-22-125332-nodejs-12
 email_enabled                | yes
 verify_emails                | yes
 email_backend                | sendgrid
 password_reset_override      | smtp
 password_reset_smtp_from     | pasxxxxom
 password_reset_smtp_login    | servxxxm
 password_reset_smtp_server   | smtp-relay.gmail.com
 help_email                   | help+test@xxxx
 ssh_gateway                  | yes

[williamstein]

You should be able to just type psql as root to get the database shell.

[williamstein]

You can look in /var/log/hub.log I think for more details about what went wrong too.

[smokedsalmonbagel]

Thanks - the setting look good.

             name             |                                 value
------------------------------+-----------------------------------------------------------------------
 site_description             | my name
 site_name                    | my name
 logo_square                  | https://****.png
 organization_url             | https://****.org/
 logo_rectangular             | https://****.png
 help_email                   | ****@****.****
 organization_email           | ****@****.****
 organization_name            | ****
 dns                          | cocalc.****.com
 iframe_comm_hosts            | ****@****.com
 verify_emails                | no
 email_smtp_login             | 5866****bb6d1
 email_smtp_password          | 5866****bb6d1
 email_smtp_port              | 587
 email_enabled                | yes
 email_smtp_server            | smtp.postmarkapp.com
 email_smtp_from              | info@****.org
 password_reset_smtp_from     | info@****.org
 password_reset_smtp_login    | 5866****b6d1
 password_reset_smtp_password | 5866****b6d1
 password_reset_smtp_port     | 587
 email_backend                | sendgrid
 password_reset_override      | default
 password_reset_smtp_server   | smtp.postmarkapp.com
 sendgrid_key                 | SG.****I

The output in my first post are from the hub log. Here is the full output after I send the test forgot password email:

2020-05-22T19:33:40.077Z - debug: send_email(to:myname@mydomain.com) -- sendgrid not configured, starting...
2020-05-22T19:33:40.077Z - debug: send_email(to:myname@mydomain.com) -- ... using site settings/sendgrid_key
2020-05-22T19:33:40.077Z - debug: send_email(to:myname@mydomain.com) -- started sendgrid client
2020-05-22T19:33:40.077Z - debug: send_email(to:myname@mydomain.com) -- sending email to myname@mydomain.com starting...
2020-05-22T19:33:40.078Z - debug: send_email(to:myname@mydomain.com) -- sending email to myname@mydomain.com -- data -- {"from":{"email":"CoCalc Help <info@somedomain.com>","name":"my org"},"subject":"my org Password Reset","content":[{"type":"text/html","value":"<div>Hello,</div>\n<div>&nbsp;</div>\n<div>\nSomebody just requested to change the password of your my org account.\nIf you requested this password change, please click this link:</div>\n<div>&nbsp;</div>\n<div style=\"text-align: center; font-size: 120%;\">\n  <b><a href=\"https://cocalc.mydomain.com/app?forgot=88fxx97c\">https://cocalc.mydomain.com/app?forgot=88fxxc97c</a></b>\n</div>\n<div>&nbsp;</div>\n<div>If you don't want to change your password, ignore this message.</div>\n<div>&nbsp;</div>\n<div>In case of problems, email\n<a href=\"mailto:info@somedomain.com\">info@somedomain.com</a> immediately!\n<div>&nbsp;</div>"}],"template_id":"037xx78","personalizations":[{"subject":"my org Password Reset","to":[{"email":"myname@mydomain.com"}]}],"substitutions":{"#title#":"my org Password Reset"},"categories":["password_reset"]}
2020-05-22T19:33:40.253Z - debug: send_email(to:myname@mydomain.com) -- sending email to myname@mydomain.com -- error = {"message":"Unauthorized","code":401,"response":{"headers":{"server":"nginx","date":"Fri, 22 May 2020 19:33:40 GMT","content-type":"application/json","content-length":"116","connection":"close","access-control-allow-origin":"https://sendgrid.api-docs.io","access-control-allow-methods":"POST","access-control-allow-headers":"Authorization, Content-Type, On-behalf-of, x-sg-elas-acl","access-control-max-age":"600","x-no-cors-reason":"https://sendgrid.com/docs/Classroom/Basics/API/cors.html"},"body":{"errors":[{"message":"The provided authorization grant is invalid, expired, or revoked","field":null,"help":null}]}}}
2020-05-22T19:33:40.254Z - debug: send_email(to:myname@mydomain.com) -- error sending email -- {"message":"Unauthorized","code":401,"response":{"headers":{"server":"nginx","date":"Fri, 22 May 2020 19:33:40 GMT","content-type":"application/json","content-length":"116","connection":"close","access-control-allow-origin":"https://sendgrid.api-docs.io","access-control-allow-methods":"POST","access-control-allow-headers":"Authorization, Content-Type, On-behalf-of, x-sg-elas-acl","access-control-max-age":"600","x-no-cors-reason":"https://sendgrid.com/docs/Classroom/Basics/API/cors.html"},"body":{"errors":[{"message":"The provided authorization grant is invalid, expired, or revoked","field":null,"help":null}]}}}

(I replaced some of my specific values from the log.)
Is this some sort of CORS issue? I am running cocalc behind a reverse proxy for SSL - but I don't think this matters for requests coming from cocalc. I know it is not my key or network as I can send via curl in the cocalc container. Also - why does the log say "sendgrid not configured"?

[smokedsalmonbagel]

I fixed the authorization issue by going through the "integration guide" in sendgrid.
For some reason keys generated for node.js in the integration guide in the sendgrid console work, but keys generated through the "API keys" page of the settings in the console don't work. My guess is sendgrid is checking some header sent by their SDK so requests for that key can only come from applications running that sdk. Hope that helps anyone having key issues with sendgrid keys.

Unfortunately sendgrid seems to now be "dropping" my emails. The reason is: "This email could not be sent." - On the sengrid console the "to:" field is blank. Is there anything else I need to do on the sendgrid side? Like create a template?

Here is m new log output:

2020-05-23T01:34:29.242Z - debug: send_email(to:myuser@something.com) -- <div>Hello,</div>
<div>&nbsp;</div>
<div>
Somebody just requested to change the password of your My org account.
If you requested this password change, please click this lin...
2020-05-23T01:34:29.242Z - debug: send_email(to:myuser@something.com) -- sending email to myuser@something.com starting...
2020-05-23T01:34:29.243Z - debug: send_email(to:myuser@something.com) -- sending email to myuser@something.com -- data -- {"from":{"email"     :"CoCalc Help <info@mydomain.com>","name":"name"},"subject":"My org Password Reset","conten     t":[{"type":"text/html","value":"<div>Hello,</div>\n<div>&nbsp;</div>\n<div>\nSomebody just requested to change the password of your org account.\nIf you requested this password change, please click this link:</div>\n<div>&nbsp;</div>\n<div styl     e=\"text-align: center; font-size: 120%;\">\n  <b><a href=\"https://cocalc.mydomain.com/app?forgot=c0236exx4a1b2cd6\">     https://cocalc.mydomain.com/app?forgot=c0236eb8-xxa1b2cd6</a></b>\n</div>\n<div>&nbsp;</div>\n<div>If you don't want to      change your password, ignore this message.</div>\n<div>&nbsp;</div>\n<div>In case of problems, email\n<a href=\"mailto:info@mydomain.com\">info@mydomain.com</a> immediately!\n<div>&nbsp;</div>"}],"template_id":"0375d02xx411e2a78","personalizations":     [{"subject":"My org Password Reset","to":[{"email":"myuser@something.com"}]}],"substitutions":{"#title#":"my org Password Reset"},"categories":["password_reset"]}
2020-05-23T01:34:29.516Z - debug: send_email(to:myuser@something.com) -- sending email to myuser@something.com -- success -- undefined

[haraldschilly]

Unfortunately sendgrid seems to now be "dropping" my emails. The reason is: "This email could not be sent." - On the sengrid console the "to:" field is blank. Is there anything else I need to do on the sendgrid side? Like create a template?

hi, there could be something missing in cocalc's code to send from sendgrid via templates. in any case, it is possible to transform an sendgrid API key to a password for using SMTP. at least, this page claims so: https://sendgrid.com/docs/API_Reference/SMTP_API/getting_started_smtp.html then, just switch use SMTP.

[smokedsalmonbagel]

Finally got this working with SMTP, but unfortunately I'm not sure what fixed it. This is what I did

• Used sendgrid's "integration guide" to create a key specifically for node.js (Don't just generate ti through the basic "generate key" option)
• Added a map parameter for nginx socket connections (I am behind a reverse proxy which does my SSL cert)
• Switched to SMTP for sendgrid. (I had tried this before with no luck so I am thinking one of the above may have fixed the issue)

Still no luck with non SMTP emails.

Relevant parts of my nginx conf:
(related to https://github.com/sagemathinc/cocalc-docker/issues/68)

 map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
 }
  location / {
      ...
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    location /api/websocket {
        proxy_set_header Host $host;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
   }

[haraldschilly]

ok, thank you for the update. My take is that this is a valid workaround. I'll move this ticket to the actual cocalc repository, because a fix this the problem here will not happen in the codebase for cocalc docker.