Webhook POST data parsing

pitchforkdave at January 23rd, 2019 10:23 — #1

Hi all,

I've searched for an answer on this but couldn't find anything definitive.

Using Automate.io, I have a Google Sheet successfully receiving data from a OnePage webhook when a deal is created. The following is a sample of the data being received...

{:id=>"5c486fc01787fa4517d31d13", :contact_id=>"5bdc64061787fa14ea8392b2", :owner_id=>"59391112c5a1f8d208e26024", :owner=>{:id=>"59391112c5a1f8d208e26024", :name=>"Dave Turner", :email=>"hello@epicbeers.co.uk"}, :name=>"DK", :text=>"1 x Wash Gold\n1 x Pilcrow\n1 x Red India", :date=>"2019-01-23", :contact_info=>{:contact_name=>"", :company=>"David Turner (BS24)"}, :author=>"Dave T.", :amount=>3.0, :months=>1, :total_amount=>3.0, :cost=>0.0, :total_cost=>0.0, :margin=>3.0, :commission=>0.0, :commission_percentage=>0.0, :commission_type=>"none", :commission_base=>"amount", :status=>"won", :has_related_notes=>false, :attachments=>[], :deal_fields=>[{:deal_field=>{:id=>"5aab7603aa7620144c5aa813", :name=>"Deliver WC", :type=>"date", :position=>0}, :value=>"2019-01-28"}], :has_deal_items=>false, :deal_items=>[], :created_at=>"2019-01-23T13:44:32Z", :modified_at=>"2019-01-23T13:44:34Z", :close_date=>"2019-01-23"}

I need to now parse this with a Google script and wasn't having much luck (I'm very new to JSON too by the way), so pasted the info in to jsonlint.com to check the data and it returns an error...

Error: Parse error on line 1:
{: id => "5c486fc0178
-^
Expecting 'STRING', '}', got ':'

It says it may be down to an extra comma at the end of a list but I don't think there is.

As a result, I can't seem to use the JSON.parse function in my script to get at the info.

Anyone know what I'm missing here please?

Hugely big thanks in advance

Dave

vladimir at January 25th, 2019 06:09 — #2

Hi Dave!

Thank you for your feedback and sorry for delayed response.
This data you received is a string representation of a ruby hash object with keys as ruby symbols. It looks like a bug on our side. But we can't fix it right away, because we have many customers using our webhooks API and this change can break their integrations.
First of all we have to find those customers which use it and we'll ask them if this change/fix can break their code. If there aren't many customers who use it, we can declare current implementation as deprecated and then fix this issue in a while. Otherwise we will have to implement a side API with fixed JSON object in order to leave current customers' integrations intact. But hopefully our customers will be okay with the fix in the current API.

Well, I don't know what our customers do with the JSON object you've mentioned, because you're the first who reported this bug. But I'd like to advise you in your case: given JSON will be parsed successfully if you find all entries of ":{some_string}=>" substrings and replace it with "{some_string}:" before giving the data to JSON parser. This can be done using regexps: you find all entries with /:(\w+)=>/ regexp and then replace found entry with "{matched word}" + ':', so at the end you'll get a valid JSON string which can be successfully parsed by JSON parser

Vladimir

pitchforkdave at January 28th, 2019 01:36 — #3

Thanks Vladimir, appreciate the reply and your suggestion. I've done something similar to your suggestion now so I can get the info I need, thanks ever so much and good luck with any fixes!

praneel at May 13th, 2024 13:21 — #4

Hello Vladimir,
I am encountering difficulties in parsing the data received from webhooks, particularly when the client includes the 'company' field while adding contact. The webhook response format I'm receiving is as follows:

{:id=>"663c9c452e8be4183bf22ddf", :name=>"jasghdah", :description=>"", :phone=>"", :url=>"", :company_fields=>[], :syncing_status=>false, :synced_status_id=>nil, :syncing_tags=>false, :synced_tags=>nil, :address=>{:address=>nil, :city=>nil, :state=>nil, :zip_code=>nil, :country_code=>nil}, :created_at=>"2024-05-09T09:49:57Z", :modified_at=>"2024-05-09T09:49:57Z", :contacts_count=>1, :won_deals_count=>0, :total_won_amount=>0.0, :total_pending_amount=>0.0, :pending_deals_count=>0, :contacts=>[#<Api::V3::ContactSerializer:0x00007f73c93982f0 @options={:scope=>#<WebhookWorker:0x00007f73c8d1e190 @jid="465b91bd2b7b62fd4b11a01a", @current_user=#<User _id: 65f7a528cbd21b252c7e38f7, created_at: 2024-03-18 02:21:34.774 UTC, updated_at: 2024-05-09 06:36:26.834 UTC, unread_tracked_emails_count: 0, google_directions_api_usage_count: 0, sequence_id: 0, firstname: "Pranil", lastname: "Khadka", email: "pranil@krispcall.dev", account_owner: true, account_role: "owner", reseller_id: nil, team_details: {}, phone: nil, google_plus_id: nil, photo_url: nil, api_keys: {}, signup_from_google: nil, password: nil, unknown_password: false, password_changed_at: 2024-05-07 15:27:33 UTC, api_key: "GD3d5C3EVIdEnpr4AE4ppc9BCMwCSe379RwPzj6Unow=", bcc_email: "65f7a52e7241d135568f6f68@users.onepagecrm.com", bcc_email_enabled: true, used_features: {"countries"=>["US"], "next_action_magic"=>true, "week_start"=>"monday"}, enabled_apps: ["webhooks"], timezone_id: "Asia/Kathmandu", active: true, currency: "USD", country_code: "NP", send_push_notifications: false, separator: ",", delimiter: ".", listing_data: {"hide_na_time_popup"=>false, "listing_size"=>25, "use_infinite_scroll"=>true, "hide_sales_figures"=>true, "hide_activity"=>true, "hide_new_features2"=>true, "hide_new_activities"=>true, "hide_new_account_view_popup"=>true, "hide_new_logo_tip"=>true, "hide_responsive_menu_popup"=>true, "hide_qa_with_date_popup"=>true, "hide_new_features"=>true}, account_rights: [], last_login: 2024-05-09 06:19:29.44 UTC, last_seen: 2024-05-09 09:45:53.624 UTC, date_format: "%d/%m/%Y", system_tags_me: {}, system_tags_all: {}, deal_stages: [], reminder_type: "new_task", email_reminder_hour: 6, mobile_reminder_hour: 8, sales_data: {"display_kanban_view"=>true, "current_pipeline"=>BSON::ObjectId('65f7a52e7241d135568f6f79'), "deals_list_options"=>{"deals_list_filter"=>"", "deals_list_sort_field"=>"expected_close", "deals_list_sort_order"=>"asc"}}, mailing_data: {}, custom_filters: [{"id"=>BSON::ObjectId('65f7a5307241d135568f701f'), "name"=>"Pending deals", "url"=>"/filter/65f7a5307241d135568f701f/pending-deals", "is_readonly"=>false, "admin_only"=>false, "mailchimp"=>false, "as_focus"=>false, "pseudo"=>false}, {"id"=>BSON::ObjectId('65f7a5307241d135568f7020'), "name"=>"Prospects with pending deals", "url"=>"/filter/65f7a5307241d135568f7020/prospects-with-pending-deals", "is_readonly"=>false, "admin_only"=>false, "mailchimp"=>false, "as_focus"=>false, "pseudo"=>false}], next_action_dates: [], custom_call_results: {"interested"=>"Interested", "not_interested"=>"Not interested"}, az_counts: {}, tag_counts: {}, stream_tag_counts: {}, system_tag_counts: {}, stream_system_tag_counts: {}, status_counts: {}, stream_status_counts: {}, lead_source_counts: {}, stream_lead_source_counts: {}, title_choices: [], title_enabled: nil, is_reseller: false, system_metadata: {"last_seen_ip"=>"202.166.220.144", "call_handler"=>{"id"=>"custom", "command"=>"https://qa-dialer.safefamilyapp.com?call=[number]"}, "generic-api"=>true}, deleted_at: nil, update_counts: {}, team_sequence_ids: {"0"=>BSON::ObjectId('65f7a528cbd21b252c7e38f7')}, product_updates_count: 0, unread_product_update_ids: [BSON::ObjectId('65fd7575cbd21b399b11cd11')], browser_utc_offset: 5, pinned_apps: ["email_sync", "call_handler"], enforce_logout_at: [], time_with_ampm: false, last_tell_a_friend_access: nil, mc_api3_batch_operation_id: nil, show_google_contacts_connect_box: true, custom_max_card_scans_limit: nil, worker_sid: nil, legacy_login_used_at: nil, product_updates_consent: false, empty_account_notified: true, survey_perfect_score_email_sent: false, aircall_default_country_code: nil, account_id: BSON::ObjectId('65f7a528cbd21b252c7e38f8'), group_id: BSON::ObjectId('65f7a52e7241d135568f6f67'), default_pipeline_id: BSON::ObjectId('65f7a52e7241d135568f6f79'), virtual_group_id: BSON::ObjectId('65f7a52e7241d135568f6f66')>, @response_service=#<Api::V3::ResponseService:0x00007f73c9398458 @request=nil, @view_context=#<WebhookWorker:0x00007f73c8d1e190 ...>>>}, @object=#<Contact _id: 663c9c452e8be4183bf22dde, created_at: 2024-05-09 09:49:57.867 UTC, updated_at: 2024-05-09 09:49:57.867 UTC, title: nil, firstname: "Addrerere", lastname: "asdghsa", photo_url: nil, job_title: "asdasyd", search_meta: ["addrerere asdghsa", "jasghdah", "hello@gmail.com"], phones_meta: ["9889876787656"], contact_search_meta: ["addrerere asdghsa", "asdghsa", "jasghdah", "hello@gmail.com", "gmail.com"], external_id: nil, address: "I live here 200", city: nil, state: nil, zip_code: nil, country_code: nil, country_name_text: nil, description: nil, urls: [{"type"=>"website", "url"=>"url webshite"}], phones: [{"type"=>"work", "number"=>"9889876787656"}], emails: [{"type"=>"work", "address"=>"hello@gmail.com"}], status: "lead", status_updated_at: 2024-05-09 09:49:57.867 UTC, starred_by: [], lead_source: nil, pending_deal: false, total_pendings: 0.0, is_private: false, next_actions: {}, as_weights_me: {}, as_weights_all: {}, action_names: {"all"=>[], "top"=>""}, action_dates: {"all"=>[]}, stream_users: [BSON::ObjectId('65f7a528cbd21b252c7e38f7')], is_waiting: false, waiting_for: [], waiting_dates: {}, closing_comments: {}, closing_dates: {}, sales_closed_for: [], unread_email_counts: {}, tags: ["abcccyo"], custom_fields: {"65f7a52e7241d135568f6f99"=>{"date"=>2024-05-10 00:00:00 UTC, "month"=>5, "idx"=>134}}, custom_fields_ids: [BSON::ObjectId('65f7a52e7241d135568f6f99')], next_anniversary_reminder: nil, last_reminder_sent: nil, last_viewed: 2024-05-09 09:49:57.991 UTC, last_activity_date: 2024-05-09 09:49:52.89 UTC, last_timeline_update: 2024-05-09 09:49:57.867 UTC, bulk_ids: [], sortable_fullname: "asdghsa addrerere", first_letter: "a", sortable_company_name: "jasghdah", first_company_letter: "j", company_phone: nil, company: "jasghdah", company_size: 1, contact_stream_sort_meta: "addrerereasdghsajasghdah", company_stream_sort_meta: "jasghdahaddrerereasdghsa", mailchimp_list_id: nil, mailchimp_email_id: nil, mailchimp_error: {}, xero_id: nil, sageone_id: nil, all_emails_md5: "b59a741183b7e1990156a46faa29b60c", emails_md5: [<BSON::Binary:0x158020 type=md5 data=0xb59a741183b7e199...>], names_md5: <BSON::Binary:0x158040 type=md5 data=0xb9018e4597167313...>, dummy_contact: false, mc_api3_batch_operation_id: nil, google_contact_ids: {}, enhanceable: false, enhance_hidden: false, initial_status: "lead", address_type: "work", user_id: BSON::ObjectId('65f7a528cbd21b252c7e38f8'), group_id: BSON::ObjectId('65f7a52e7241d135568f6f67'), import_id: nil, merge_import_id: nil, owner_id: BSON::ObjectId('65f7a528cbd21b252c7e38f7'), company_id: BSON::ObjectId('663c9c452e8be4183bf22ddf'), webforms_form_id: nil, author_id: BSON::ObjectId('65f7a528cbd21b252c7e38f7')>>], :pending_deals=>[]}

I'm seeking advice on whether there are any viable workaround solutions for creating contacts, considering this challenge. Should I proceed with parsing the data in its current format, or are there alternative approaches you would recommend?

vladimir at May 14th, 2024 07:14 — #5

Hello PraNeel!

Could you please tell me what is the web hook you've subscribed to? Or an endpoint you're querying if you're making an API request. What is the type of your web hook: post or json? I need the details from you in order to understand what you're doing and what data you're receiving from which webhook.

Thank you,
Vladimir

praneel at May 15th, 2024 00:16 — #6

Hello Vladimir,
It appears there's an issue with the webhook subscription regarding Contact creation events. The response format received after a new contact is created differs, making it difficult to parse effectively. The response type is a POST request.
My primary objective is to seamlessly retrieve the contact data whenever user creates new contact in OnePage CRM.
Thank you

vladimir at May 15th, 2024 04:28 — #7

Hi PraNeel!

Thank you, I'll investigate POST web hook type response for a contact creation use case. As a work around I can propose you to switch to JSON type of the webhook. In this case you should receive parseable json and your problem should be solved. Please tell me if it worked for you.

Thank you,
Vladimir

praneel at May 15th, 2024 11:43 — #8

Hello Vladimir,
Thank you for your response.
Currently, I am unable to utilize the JSON type of webhook as it has been created programmatically. Is there an alternative method or workaround to subscribe to a JSON type webhook programmatically?
Thank you

vladimir at May 15th, 2024 19:52 — #9

Hello PraNeel!

Yes, you can add a JSON web hook programmatically via our API:

POST https://app.onepagecrm.com/api/v3/webhooks
name={my_webhook_name}
secretkey={my_secret_key}
hook_url={my_hook_url}
type=json

If you want to update the existing web hook and if you know its ID in OnePageCRM you can update it as the following:

PUT https://app.onepagecrm.com/api/v3/webhooks/:your_webhook_id
name={my_webhook_name}
secretkey={my_secret_key}
hook_url={my_hook_url}
type=json

You can also delete a web hook by ID:

DELETE https://app.onepagecrm.com/api/v3/webhooks/:webhook_id

All API requests must be sent on behalf of account owner user or administrator user, i.e. when you're composing Authorization header with Basic {AuthKey}, then you should use account owner user or administrator user when you compose {AuthKey}.

But if you have access to OnePageCRM in web browser, it's way easier to configure the web hooks manually in https://app.onepagecrm.com/app/webhooks.
I hope I could help you, tell me if it worked for you or if you have any more questions on it.

Thanks,
Vladimir

praneel1 at May 16th, 2024 05:47 — #10

Hello Vladimir,
Thank you for your prompt response. Upon sending the request with the 'type' parameter set as JSON, I observed that it yielded a well-structured JSON response, facilitating easier parsing.

Regarding another query, I noticed that when a contact is deleted, the data format received is:

{
    "data": {
        "id": "6645cd0aa7c36c0aa4bc396c"
    },
    "type": "contact",
    "reason": "deleted",
    "secretkey": "*******************",
    "timestamp": 1715850875
}

Is there a possibility to enhance this data format to include additional pertinent details such as 'owner_id' or 'owner_email' for better context.

Thank you!

vladimir at May 17th, 2024 12:24 — #11

Hello PraNeel1!

Sorry for the delayed response. At the moment when we're sending the contact deletion web hook when the contact is already deleted, so that's why we're sending only the ID. If you are subscribed to the update web hooks you can fetch owner email info on your side and notify owner on your behalf. And also delete the contact on your side (if necessary) in order to sync with OnePageCRM data.

Thank you,
Vladimir