brainstorming
brainstorming
You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).
apify-webhooks-eventsImplement Apify webhooks for Actor run notifications and event-driven pipelines. Use when setting up run completion alerts, building event-driven scraping pipelines, or configuring ad-hoc webhooks for individual runs. Trigger with "apify webhook", "apify events", "actor run notification", "apify run succeeded webhook", "apify ad-hoc webhook".
Independent GitHub snapshot · Automated classification · Original documentation governs
name: apify-webhooks-events description: 'Implement Apify webhooks for Actor run notifications and event-driven pipelines.
Use when setting up run completion alerts, building event-driven scraping pipelines,
or configuring ad-hoc webhooks for individual runs.
Trigger with "apify webhook", "apify events", "actor run notification",
"apify run succeeded webhook", "apify ad-hoc webhook".
' allowed-tools: Read, Write, Edit, Bash(curl:*) version: 1.5.0 license: MIT author: Jeremy Longshore jeremy@intentsolutions.io tags:
Configure webhooks to receive notifications when Actor runs complete, fail, or time out. Apify supports both persistent webhooks (for all runs of an Actor) and ad-hoc webhooks (for a single run). Event-driven architecture is the recommended pattern for production Apify integrations.
npm install apify-client in your project (and express if you build an HTTP handler)APIFY_TOKEN — read it from the environment
(process.env.APIFY_TOKEN) or pass Authorization: Bearer $APIFY_TOKEN on REST
calls, never hard-code itapify-sdk-patterns| Event | Fired When |
|---|---|
ACTOR.RUN.CREATED | A new Actor run starts |
ACTOR.RUN.SUCCEEDED | Run finishes with SUCCEEDED status |
ACTOR.RUN.FAILED | Run finishes with FAILED status |
ACTOR.RUN.ABORTED | Run is manually or programmatically aborted |
ACTOR.RUN.TIMED_OUT | Run exceeds its timeout |
ACTOR.RUN.RESURRECTED | A finished run is resurrected |
Persistent webhooks fire for every run of an Actor. Set condition.actorId, list
the eventTypes you care about, and shape the delivered body with
payloadTemplate:
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const webhook = await client.webhooks().create({
eventTypes: ['ACTOR.RUN.SUCCEEDED', 'ACTOR.RUN.FAILED', 'ACTOR.RUN.TIMED_OUT'],
condition: { actorId: 'YOUR_ACTOR_ID' },
requestUrl: 'https://your-app.com/api/webhooks/apify',
payloadTemplate: JSON.stringify({
eventType: '{{eventType}}',
actorRunId: '{{actorRunId}}',
defaultDatasetId: '{{resource.defaultDatasetId}}',
status: '{{resource.status}}',
statusMessage: '{{resource.statusMessage}}',
}),
isAdHoc: false,
});
console.log(`Webhook created: ${webhook.id}`);
The full payload template (all run fields) and the complete variable table are in payload templates & local testing.
Ad-hoc webhooks are created at run time and fire only for that specific run — pass
a webhooks array when starting the Actor:
const run = await client.actor('username/my-actor').start(
{ startUrls: [{ url: 'https://example.com' }] },
{
webhooks: [{
eventTypes: ['ACTOR.RUN.SUCCEEDED', 'ACTOR.RUN.FAILED'],
requestUrl: 'https://your-app.com/api/webhooks/apify',
}],
},
);
The equivalent REST/curl form is in payload templates & local testing.
Your endpoint must return a 2xx within 30 seconds, so acknowledge immediately and
process asynchronously. On SUCCEEDED, fetch the dataset via defaultDatasetId; on
FAILED/TIMED_OUT, pull the run log and alert:
app.post('/api/webhooks/apify', async (req, res) => {
res.status(200).json({ received: true }); // ack first
try {
await processWebhook(req.body); // switch on eventType, fetch dataset, alert
} catch (error) {
console.error('Webhook processing failed:', error);
}
});
The full processWebhook switch (dataset fetch, log tail, oncall alerting) is in
full implementation.
Apify may deliver a webhook more than once, so dedupe on
${actorRunId}:${eventType} before doing work. You can also chain Actors (Stage 1
SUCCEEDED → start Stage 2) and manage the webhook lifecycle (list, update,
delete, inspect dispatches). All three patterns — idempotent processing,
event-driven pipeline, and lifecycle management — are in
full implementation.
id (persistent webhooks are visible under
client.webhooks().list(); ad-hoc webhooks are scoped to one run)payloadTemplate JSON to your
requestUrlclient.webhook(id).dispatches().list() returns the delivery history — each entry
carries a status, createdAt, and the endpoint's responseStatus so you can
confirm delivery or diagnose retriescurl creation, and local testing with ngrok — payload templates & local testing| Issue | Cause | Solution |
|---|---|---|
| Webhook not delivered | URL unreachable | Verify HTTPS, check firewall |
| Duplicate processing | Webhook retry on non-2xx | Implement idempotency |
| Slow processing | Handler takes >30s | Respond 200 immediately, process async |
| Missing data in payload | Wrong template vars | Check template variable spelling |
| Webhook disabled | Too many failures | Re-enable in Console or via API |
For performance optimization, see apify-performance-tuning.