Auto indexing issue with WPAllImport and API to Posts plugins

[nosh02]

Describe the bug
I am experiencing auto indexing issues using WPAllImport and WPAPItoPosts plugins. Publishing posts natively in WP or performing a full reindex has been working completely fine. When checking the API logs in Algolia it looks like there is a deleteObject call first and then an updateObject call afterward on the same record when imports are performed from either of the plugins above. Is that typical? I am actually seeing them same when updating a post directly in WP so maybe nothing to worry about...Just thought it was odd that it was doing a delete and then an update.

Looking at the payload sent to updateObject I can clearly see my ACF fields that I am adding using the algolia_post_shared_attributes hook but the index never updates the object. I am using algolia_should_index_post hook to filter out particular posts based on ACF field values. I am not sure how sporadic it is yet but it seems to be happening often. My hunch is this is something between the API client and Algolia, maybe the speed in which tasks are being added the their queue? I have a ticket out to them to see why the tasks are not being processed. I am not sure if this plugin utilizes the wait method https://www.algolia.com/doc/api-reference/api-methods/wait-task/?

Any ideas on how I could troubleshoot this further would be helpful. I will see what Algolia has to say.

[tw2113]

Regarding All Import, I know we did a lot of work last summer to do our best to support "Fast Mode" https://github.com/WebDevStudios/wp-search-with-algolia/blob/main/includes/watchers/class-algolia-post-changes-watcher.php#L128-L130

However, I am wondering now if our change of c458917 in December potentially broke some things, assuming that WP All Import doesn't do its things on wp_after_insert_post However, I believe the hook should still run.

Yes it's typical that the plugin does a deleteObject call first, last time I dug into this topic. If I recall right, it's in part because it's easier to just push the entire new data instead of trying to calculate what differences there are between the originally indexed, and the new data coming in.

Regarding the wait method, we tried that as part of our original WP All Import release, but we got reports that things were timing out and generally much slower, so we had to revert that a bit, but we did add a filter to allow site owners to use wait() if they really want.

It can be seen at https://github.com/WebDevStudios/wp-search-with-algolia/blob/main/includes/indices/class-algolia-searchable-posts-index.php#L374-L387C11 and https://github.com/WebDevStudios/wp-search-with-algolia/blob/main/includes/indices/class-algolia-posts-index.php#L393-L405 and the filter should be the same for both.

[nosh02]

Thanks for your response, sorry for this turning into multiple subjects.

After doing more investigation, at least on the API to Posts plugin side, whats happening is the updateObject payload contains old data, so when "algolia_post_shared_attributes" is fired the post hasn't actually been updated yet so all the ACF fields im getting have old values. So I think everything is OK on the Algolia side and i confirmed looking at a handful of task ids that they are in the published state. I need to confirm if the same thing is happening on the WPAllImport end.

That being said it sounds like at least for the API to Posts plugin, it probably is doing its things AFTER whatever watcher fires on updating an existing post (save maybe?). It seems like when this plugin adds a new post, everything is fine. I will have to dig into the plugin to see what it is doing on update (im not even going to bother reaching out to their support...waste of time). https://wpgetapi.com/downloads/api-to-posts/. I can only speculate what its doing, maybe its saving the post and then updating custom fields afterwards? Maybe there is another watcher hook that should be used when updating existing posts? Maybe I need to use algolia_watch_post_meta_keys and add all my custom fields and not use the default watchers?

On the subject of using deleteObject and updateObject, I completely understand avoiding partially updating an object, would need a lot of supporting logic for that. However, couldnt you just send the entire object as you are now to updateObject and just not do deleteObject. Only reason I care is to reduce the number of Algolia Index operations.

[nosh02]

I dug into API to Posts a bit and it looks like they are looping through an array of fields and calling update_post_meta for each one. I haven't looked into the WPAllImport ACF addon yet but my guess is its doing the same thing. So I am assuming that means I cannot rely on "wp_after_insert_post" alone and will need to use "algolia_watch_post_meta_keys".

[tw2113]

Maybe I need to use algolia_watch_post_meta_keys and add all my custom fields and not use the default watchers?

Definitely worth a shot, I don't think we broke that functionality when we switched away from save_post hook for pushing content.

[tw2113]

Possibly related https://wordpress.org/support/topic/slow-re-indexing-due-to-individual-deleteobject/