Connect Webhook to Elasticsearch

gavinbeattie at November 12th, 2020 16:00 — #1

I want to connect our OnePageCRM Webhooks to our Elasticsearch instance but as the service is hosted there is no option to change the security from standard curl requests. Could the option to control the headers of the request be added and passing of credentials e.g. apikey.

I'm not sure how useful the current secret key is within the JSON body, why not just have it as the normal curl security credentials in the header? The body structure in JSON is fine and would be ingested easily. Elasticsearch is a very popular platform and this would make a good feature to use for analysis.

vladimir at November 24th, 2020 13:26 — #2

Hi @GavinBeattie!

Thanks for your question. Could you please describe your use case more in details?
At the moment there are several ways you can pass the API key along with your request into OnePageCRM API, all via headers. You can find the way you do it here https://developer.onepagecrm.com/api

If you use OnePageCRM webhooks, then it is OnePageCRM --> YourSystem integration, i.e. in read-only mode. If you want to modify the data, then you have to use OnePageCRM API, you can find the docs here https://developer.onepagecrm.com/api

I hope it helped, if you have more questions please feel free to ask them

Thank you,
Vlad

gavinbeattie at November 27th, 2020 04:39 — #3

Hi,

To clarify, the api key I want to pass is the one required to access the destination, in our case Elasticsearch, so .../api is not currently what I want to use. The push option from OnePageCRM via webhooks appears close to fitting our needs.

Elasticsearch also has capability for webhooks in a push direction. The options they provide to add headers I think would be a really good feature to add to OnePageCRM and close to what I (and potentially others) would need OnePageCRM to have.

To help my explanation, Elasticsearch's webhook connector has the following fields. In brackets I've added the OnePageCRM equivalent:

Connector name (Optional name):
Method (Data format): {POST/PUT} ({Regular POST/JSON})
URL (Webhook URL):
Username (--):
Password (--):
Switch to add HTTP header (--):
Add header (--): [Key] [Value]

The last fields provide the flexible integration with existing web applications. In our case ES is hosted, so we cannot make changes to fit the inbound requests from the OnePageCRM webhook. I would assume most use cases would be similar to integrate with an existing application rather than configuring from scratch.

Thanks,

Gavin

vladimir at November 27th, 2020 06:46 — #4

Hi @GavinBeattie!

As far as I understood your explanation you need downstream integration from OnePageCRM to your system/elasticsearch. In this case you can subscribe to receiving webhooks in OnePageCRM apps page and you will receive updates on the changes happened in your account.
If you need to transform the data/format from OnePageCRM webhook, then you can use some middleware/3rd party tools (like Amazon Lambda for example) to transform it to what you can consume on your end. At the moment we cannot change the format of our output data because we have many customers consuming data in this format.

If you need more help then we should discuss exactly your case and it would be helpful if you provided a integration architecture diagram or constraints that you have to face when you design/implement your solution. So we knew what exactly is to be supported on your side.

Well I hope my suggestions helped you.
Thank you,
Vlad

gavinbeattie at August 3rd, 2021 08:31 — #5

Hi @vladimir,

I came across the Elasticsearch Webhooks connector used in the Alerts App (the reverse direction I know, but please bear with me). As shown in the details below this is a great example of the controls that I think are essential to make the Webhooks functionality in OnePageCRM useful.

Webhook connector (very similar to OnePageCRM):
--> Connector name
Connector settings
Method
POST/PUSH
URL

--> Authentication (common/standard for http requests)
Require authentication for this webhook [tick box]
Username
Password

--> Add HTTP header
Headers in use [tick box]
key-1
value-1
Add header [tick box]
key-2
value-2
Add header [tick box], etc.

The "Secret key" field is effectively a bespoke option and requires the downstream application to be customised (hence defeats the point of Webhooks). This is solved by the Elasticsearch connector design total flexibility for any header desired and common username and password, etc.

Your thoughts?

vladimir at August 3rd, 2021 09:27 — #6

Hi @GavinBeattie!

Well, the webhooks work the same way in OnePageCRM. For authentication purpose we don't use username/password pair, we use secret key value instead. Also we don't provide an ability to define custom headers.
So if you subscribe to webhooks in OnePageCRM and define URLs to your server handlers you will receive updates from OnePageCRM.

For the information about the format of incoming messages to your handlers please refer to https://developer.onepagecrm.com/webhooksmore/

Thank you,
Vlad