Unable to verify signature for testing purposes

[gotchahn]

Issue Summary

I'm trying to do some tests in rails to check if my sendgrid webhook endpoint is working. I'm in using the same values for PUBLIC KEY and SIGNATURE key shared on the fixtures page:

sendgrid-ruby/spec/fixtures/event_webhook.rb

Lines 5 to 20 in 865a2ac

	PUBLIC_KEY = 'MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE83T4O/n84iotIvIW4mdBgQ/7dAfSmpqIM8kF9mN1flpVKS3GRqe62gw+2fNNRaINXvVpiglSI8eNEc6wEA3F+g=='.freeze
	FAILING_PUBLIC_KEY = 'MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEqTxd43gyp8IOEto2LdIfjRQrIbsd4SXZkLW6jDutdhXSJCWHw8REntlo7aNDthvj+y7GjUuFDb/R1NGe1OPzpA=='.freeze
	SIGNATURE = 'MEUCIGHQVtGj+Y3LkG9fLcxf3qfI10QysgDWmMOVmxG0u6ZUAiEAyBiXDWzM+uOe5W0JuG+luQAbPIqHh89M15TluLtEZtM='.freeze
	FAILING_SIGNATURE = 'MEUCIQCtIHJeH93Y+qpYeWrySphQgpNGNr/U+UyUlBkU6n7RAwIgJTz2C+8a8xonZGi6BpSzoQsbVRamr2nlxFDWYNH3j/0='.freeze
	TIMESTAMP = '1600112502'.freeze
	PAYLOAD = "#{[
	{
	email: 'hello@world.com',
	event: 'dropped',
	reason: 'Bounced Address',
	sg_event_id: 'ZHJvcC0xMDk5NDkxOS1MUnpYbF9OSFN0T0doUTRrb2ZTbV9BLTA',
	sg_message_id: 'LRzXl_NHStOGhQ4kofSm_A.filterdrecv-p3mdw1-756b745b58-kmzbl-18-5F5FC76C-9.0',
	'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>',
	timestamp: 1_600_112_492
	}
	].to_json}\r\n".freeze # Be sure to include the trailing carriage return and newline!

But overall the verify method returns false to me.

Code Snippet

If I do this manually in the console, using the same values, the verify returns false:

irb(main):003:0> 
irb(main):004:0> ew = SendGrid::EventWebhook.new
=> #<SendGrid::EventWebhook:0x00007faa52aab758>
irb(main):005:0> 
irb(main):006:0> 
irb(main):007:0> public_key = 'MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE83T4O/n84iotIvIW4mdBgQ/7dAfSmpqIM8kF9mN1flpVKS3GRqe62gw+2fNNRaINXvVpiglSI8eNEc6wEA3F+g=='.freeze
=> "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE83T4O/n84iotIvIW4mdBgQ/7dAfSmpqIM8kF9mN1flpVKS3GRqe62gw+2fNNRaINXvVpiglSI8eNEc6wEA3F+g=="
irb(main):008:0> 
irb(main):009:0> signature = 'MEUCIGHQVtGj+Y3LkG9fLcxf3qfI10QysgDWmMOVmxG0u6ZUAiEAyBiXDWzM+uOe5W0JuG+luQAbPIqHh89M15TluLtEZtM='.freeze
=> "MEUCIGHQVtGj+Y3LkG9fLcxf3qfI10QysgDWmMOVmxG0u6ZUAiEAyBiXDWzM+uOe5W0JuG+luQAbPIqHh89M15TluLtEZtM="
irb(main):010:0> 
irb(main):011:0> timestamp = '1600112502'.freeze
=> "1600112502"
irb(main):012:0> 
irb(main):013:0> payload = "#{[
irb(main):014:1>       {
irb(main):015:2>         email: 'hello@world.com',
irb(main):016:2>         event: 'dropped',
irb(main):017:2>         reason: 'Bounced Address',
irb(main):018:2>         sg_event_id: 'ZHJvcC0xMDk5NDkxOS1MUnpYbF9OSFN0T0doUTRrb2ZTbV9BLTA',
irb(main):019:2>         sg_message_id: 'LRzXl_NHStOGhQ4kofSm_A.filterdrecv-p3mdw1-756b745b58-kmzbl-18-5F5FC76C-9.0',
irb(main):020:2>         'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>',
irb(main):021:2>         timestamp: 1_600_112_492
irb(main):022:2>       }
irb(main):023:1>     ].to_json}\r\n".freeze
=> "[{\"email\":\"hello@world.com\",\"event\":\"dropped\",\"reason\":\"Bounced Address\",\"sg_event_id\":\"ZHJvcC0xMDk5NDkxOS1MUnpYbF9OSFN0T0doUTRrb2ZTbV9BLTA\",\"sg_message_id\":\"LRzXl_NHStOGhQ4kofSm_A.filterdrecv-p3mdw1-756b745b58-kmzbl-18-5F5FC76C-9.0\",\"smtp-id\":\"\\u003cLRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net\\u003e\",\"timestamp\":1600112492}]\r\n"
irb(main):024:0> 
irb(main):025:0> ec_public_key = ew.convert_public_key_to_ecdsa(public_key)
=> #<OpenSSL::PKey::EC:0x00007faa52ab8160>
irb(main):026:0> 
irb(main):027:0> ew.verify_signature(ec_public_key, payload, signature, timestamp)
=> false
irb(main):028:0> 

Also this is my rspec that I was coding:

require "rails_helper"

ENV["SENDGRID_EVENT_HOOK_KEY"] = "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEc1xJU2qhLFMxcOLdKWQIA2OZdmUTlNRT5xFEipLDnGkO0uhW8aJiIQxJGglBRiKWxNqm3jjRbUpaAaDH9WHkng==".freeze

describe Webhooks::SendgridController, type: "request" do
  describe "POST 'create'" do

    let(:example_sendrid_payload) do
      "#{[
        {
          email: 'hello@world.com',
          event: 'dropped',
          reason: 'Bounced Address',
          sg_event_id: 'ZHJvcC0xMDk5NDkxOS1MUnpYbF9OSFN0T0doUTRrb2ZTbV9BLTA',
          sg_message_id: 'LRzXl_NHStOGhQ4kofSm_A.filterdrecv-p3mdw1-756b745b58-kmzbl-18-5F5FC76C-9.0',
          'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>',
          timestamp: 1_600_112_492
        }
      ].to_json}\r\n".freeze
    end

    let(:sendgrid_signed_header) do
      {
          "#{SendGrid::EventWebhookHeader::SIGNATURE}" => "MEYCIQCxfnpzX3CftRSaaA4wHWHOKEyHbCf5jbwL/z/QQpncqQIhANu8Ug7FTuQhgGzhSVQvDSIS64rp+fXz2AloRnV5qcTL".freeze,
          "#{SendGrid::EventWebhookHeader::TIMESTAMP}" => "1600112502".freeze
      }
    end

    it "responds with success when provided signature matches" do
      post webhooks_sendgrid_path, params: example_sendrid_payload, headers: sendgrid_signed_header

      expect(response).to have_http_status(200)
    end
end

Exception/Log

 1) Webhooks::SendgridController POST 'create' responds with success when provided signature matches
     Failure/Error: expect(response).to have_http_status(200)
       expected the response to have status code 200 but it was 401

Technical details:

• sendgrid-ruby version: 6.3.8
• ruby version: 2.6.5

[thinkingserious]

Hello @gotchahn,

Thank you for reporting this issue to us. I have been unable to reproduce. Could you please provide some more detail about your setup? Here is the code I used to test:

require 'sendgrid-ruby'
include SendGrid

ew = SendGrid::EventWebhook.new
public_key = 'MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE83T4O/n84iotIvIW4mdBgQ/7dAfSmpqIM8kF9mN1flpVKS3GRqe62gw+2fNNRaINXvVpiglSI8eNEc6wEA3F+g=='.freeze
signature = 'MEUCIGHQVtGj+Y3LkG9fLcxf3qfI10QysgDWmMOVmxG0u6ZUAiEAyBiXDWzM+uOe5W0JuG+luQAbPIqHh89M15TluLtEZtM='.freeze
timestamp = '1600112502'.freeze
payload = "#{[
      {
        email: 'hello@world.com',
        event: 'dropped',
        reason: 'Bounced Address',
        sg_event_id: 'ZHJvcC0xMDk5NDkxOS1MUnpYbF9OSFN0T0doUTRrb2ZTbV9BLTA',
        sg_message_id: 'LRzXl_NHStOGhQ4kofSm_A.filterdrecv-p3mdw1-756b745b58-kmzbl-18-5F5FC76C-9.0',
        'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>',
        timestamp: 1_600_112_492
      }
    ].to_json}\r\n".freeze
ec_public_key = ew.convert_public_key_to_ecdsa(public_key)
value = ew.verify_signature(ec_public_key, payload, signature, timestamp)
print value

This returned true.

With best regards,

Elmer

[gotchahn]

Hi @thinkingserious, I copy pasted your code in my rails console, and gave me false hmm, weird. I'm using Rails 6.0.3.2, Ruby 2.6.5 and Sendgrid-ruby 6.3.8

$ rails c
Running via Spring preloader in process 58835
Loading development environment (Rails 6.0.3.2)
irb(main):001:0> ew = SendGrid::EventWebhook.new
=> #<SendGrid::EventWebhook:0x00007faa4fe0c468>
irb(main):002:0> public_key = 'MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE83T4O/n84iotIvIW4mdBgQ/7dAfSmpqIM8kF9mN1flpVKS3GRqe62gw+2fNNRaINXvVpiglSI8eNEc6wEA3F+g=='.freeze
=> "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE83T4O/n84iotIvIW4mdBgQ/7dAfSmpqIM8kF9mN1flpVKS3GRqe62gw+2fNNRaINXvVpiglSI8eNEc6wEA3F+g=="
irb(main):003:0> signature = 'MEUCIGHQVtGj+Y3LkG9fLcxf3qfI10QysgDWmMOVmxG0u6ZUAiEAyBiXDWzM+uOe5W0JuG+luQAbPIqHh89M15TluLtEZtM='.freeze
=> "MEUCIGHQVtGj+Y3LkG9fLcxf3qfI10QysgDWmMOVmxG0u6ZUAiEAyBiXDWzM+uOe5W0JuG+luQAbPIqHh89M15TluLtEZtM="
irb(main):004:0> timestamp = '1600112502'.freeze
=> "1600112502"
irb(main):005:0> payload = "#{[
irb(main):006:1>       {
irb(main):007:2>         email: 'hello@world.com',
irb(main):008:2>         event: 'dropped',
irb(main):009:2>         reason: 'Bounced Address',
irb(main):010:2>         sg_event_id: 'ZHJvcC0xMDk5NDkxOS1MUnpYbF9OSFN0T0doUTRrb2ZTbV9BLTA',
irb(main):011:2>         sg_message_id: 'LRzXl_NHStOGhQ4kofSm_A.filterdrecv-p3mdw1-756b745b58-kmzbl-18-5F5FC76C-9.0',
irb(main):012:2>         'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>',
irb(main):013:2>         timestamp: 1_600_112_492
irb(main):014:2>       }
irb(main):015:1>     ].to_json}\r\n".freeze
=> "[{\"email\":\"hello@world.com\",\"event\":\"dropped\",\"reason\":\"Bounced Address\",\"sg_event_id\":\"ZHJvcC0xMDk5NDkxOS1MUnpYbF9OSFN0T0doUTRrb2ZTbV9BLTA\",\"sg_message_id\":\"LRzXl_NHStOGhQ4kofSm_A.filterdrecv-p3mdw1-756b745b58-kmzbl-18-5F5FC76C-9.0\",\"smtp-id\":\"\\u003cLRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net\\u003e\",\"timestamp\":1600112492}]\r\n"
irb(main):016:0> ec_public_key = ew.convert_public_key_to_ecdsa(public_key)
=> #<OpenSSL::PKey::EC:0x00007faa4ffe7f30>
irb(main):017:0> value = ew.verify_signature(ec_public_key, payload, signature, timestamp)
=> false
irb(main):018:0> print value
false=> nil
irb(main):019:0> 

[gotchahn]

BUT, noticed that if I do that code in a plain irb console, it works! but not on rails console or in my rails testing suite. Do you have any idea why it works only in a irb console??

$ irb
irb(main):001:0> require 'sendgrid-ruby'
=> true
irb(main):002:0> include SendGrid
=> Object
irb(main):003:0> 
irb(main):004:0> ew = SendGrid::EventWebhook.new
=> #<SendGrid::EventWebhook:0x00007fb08aa6cdd0>
irb(main):005:0> public_key = 'MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE83T4O/n84iotIvIW4mdBgQ/7dAfSmpqIM8kF9mN1flpVKS3GRqe62gw+2fNNRaINXvVpiglSI8eNEc6wEA3F+g=='.freeze
=> "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE83T4O/n84iotIvIW4mdBgQ/7dAfSmpqIM8kF9mN1flpVKS3GRqe62gw+2fNNRaINXvVpiglSI8eNEc6wEA3F+g=="
irb(main):006:0> signature = 'MEUCIGHQVtGj+Y3LkG9fLcxf3qfI10QysgDWmMOVmxG0u6ZUAiEAyBiXDWzM+uOe5W0JuG+luQAbPIqHh89M15TluLtEZtM='.freeze
=> "MEUCIGHQVtGj+Y3LkG9fLcxf3qfI10QysgDWmMOVmxG0u6ZUAiEAyBiXDWzM+uOe5W0JuG+luQAbPIqHh89M15TluLtEZtM="
irb(main):007:0> timestamp = '1600112502'.freeze
=> "1600112502"
irb(main):008:0> payload = "#{[
irb(main):009:1>       {
irb(main):010:2>         email: 'hello@world.com',
irb(main):011:2>         event: 'dropped',
irb(main):012:2>         reason: 'Bounced Address',
irb(main):013:2>         sg_event_id: 'ZHJvcC0xMDk5NDkxOS1MUnpYbF9OSFN0T0doUTRrb2ZTbV9BLTA',
irb(main):014:2>         sg_message_id: 'LRzXl_NHStOGhQ4kofSm_A.filterdrecv-p3mdw1-756b745b58-kmzbl-18-5F5FC76C-9.0',
irb(main):015:2>         'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>',
irb(main):016:2>         timestamp: 1_600_112_492
irb(main):017:2>       }
irb(main):018:1>     ].to_json}\r\n".freeze
=> "[{\"email\":\"hello@world.com\",\"event\":\"dropped\",\"reason\":\"Bounced Address\",\"sg_event_id\":\"ZHJvcC0xMDk5NDkxOS1MUnpYbF9OSFN0T0doUTRrb2ZTbV9BLTA\",\"sg_message_id\":\"LRzXl_NHStOGhQ4kofSm_A.filterdrecv-p3mdw1-756b745b58-kmzbl-18-5F5FC76C-9.0\",\"smtp-id\":\"<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>\",\"timestamp\":1600112492}]\r\n"
irb(main):019:0> ec_public_key = ew.convert_public_key_to_ecdsa(public_key)
=> #<OpenSSL::PKey::EC:0x00007fb08aadf4e8>
irb(main):020:0> value = ew.verify_signature(ec_public_key, payload, signature, timestamp)
=> true
irb(main):021:0> print value
true=> nil

[thinkingserious]

Hello @gotchahn,

Here are few things to check:

1. What is the return of RUBY_VERSION while in the irb console vs rails console.
2. Perhaps the rails console is using a different version of irb under the hood.
3. What happens when you run outside of the console?

Thanks!

With best regards,

Elmer

[gotchahn]

Hi @thinkingserious,

1. Both gave me 2.6.5 as the RUBY_VERSION.
2. The ONLY difference I see is that the rails console is using a Spring preloader Running via Spring preloader in process.... dunno if that has something to do.
3. Well, I tested it in my test specs, and the result is the same as the rails console.

I tested in production, and the endpoint works well with the validation of the REAL key and signature. Is just the made of ones for the tests that gives me false the verifier.

UPDATE:

I tested the rails console without the Spring preloader, and it returns false still.

[gotchahn]

Hi @thinkingserious,
I figured it out, it's the payload the cause of the issue, and mainly, the .to_json method with the 'smtp-id' section, look:

in irb console:

$ irb
irb(main):001:0> require 'sendgrid-ruby'
=> true
irb(main):002:0> include SendGrid
=> Object
irb(main):017:0> "'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>'".to_json
=> "\"'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>'\""

in rails console:

irb(main):001:0> "'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>'".to_json
=> "\"'smtp-id': '\\u003cLRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net\\u003e'\""
irb(main):002:0> 

The < and > are converted, so that's why the verifier doesn't returns true. Using JSON.generate worked:

irb(main):004:0> JSON.generate("'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>'")
=> "\"'smtp-id': '<LRzXl_NHStOGhQ4kofSm_A@ismtpd0039p1iad1.sendgrid.net>'\""

But I have another question, If I slightly change the payload, the verifier returns false, any idea of how I can use a custom payload with a different type of event for example, and get the right signature for the public key??

[TastyPi]

I realise this is an old issue, but I wanted to post a solution to the issue of how to generate signatures for custom payloads:

def sendgrid_key
  @sendgrid_key ||= OpenSSL::PKey::EC.generate("prime256v1")
end

def sendgrid_public_key
  Base64.strict_encode64(sendgrid_key.public_to_pem)
end

def sendgrid_signature(payload, timestamp)
  Base64.strict_encode64(sendgrid_key.dsa_sign_asn1(Digest::SHA256.digest("#{timestamp}#{payload}")))
end

def sendgrid_headers(body)
  timestamp = Time.now.to_i
  {
    "Content-Type" => "application/json;charset=utf-8",
    ::SendGrid::EventWebhookHeader::SIGNATURE => sendgrid_signature(body, timestamp),
    ::SendGrid::EventWebhookHeader::TIMESTAMP => timestamp
  }
end

Note that you have to generate a new key for your tests since SendGrid (correctly) does not provide the private key for generating signatures. We use an environment variable in our controller to get the public key, a gem that modifies ENV such as climate_control can be used to override it in tests, e.g.

ClimateControl.modify SENDGRID_PUBLIC_KEY: sendgrid_public_key do
  post "/sendgrid", params: payload, headers: sendgrid_headers(payload)
end