Automate Yandex SERP Pipelines: Webhooks, Postgres & CRM Sync

Yandex commands over 60% of search traffic across CIS and Eastern European digital ecosystems. For cross-border ecommerce brands, competitive intelligence analysts, and SEO professionals, capturing organic Yandex Search Engine Results Pages (SERPs) is essential. However, manually downloading CSV exports fails when decision-makers need real-time competitive shifts, rank-drop alerts, and market share tracking directly inside their operational stack.

By pairing the madeingermany/scrape-yandex Actor with automated webhooks and scheduled pipelines, you can transform ad-hoc SERP queries into continuous market intelligence feeds flowing directly into PostgreSQL, Airtable, Google Sheets, or CRM systems like HubSpot.

Serverless Extraction on Apify Cloud

Building custom scrapers for Yandex requires dealing with regional geofencing, dynamic anti-bot algorithms, and proxy rotation pools. The madeingermany/scrape-yandex Actor offloads all underlying browser orchestration, rotating residential IPs, and page rendering to Apify’s cloud infrastructure.

You do not need an enterprise budget or dedicated servers to run this pipeline. Apify provides every account with a complimentary $5 monthly starter credit allowance renewing every month—with no credit card required to begin. Because individual SERP queries consume only tiny fractions of a cent in cloud compute, this free allowance allows engineering teams and growth marketers to extract thousands of organic search records, rich snippets, and competitor links every month at zero cost.

Output Data Schema

The Actor produces clean, structured JSON items for each organic ranking, ad, or rich feature detected on the SERP. The table below outlines the primary fields delivered to your destination storage:

Programmatic Execution with Python

The following script executes madeingermany/scrape-yandex via the apify-client SDK, monitors the run to completion, and pushes the extracted records directly into a PostgreSQL database using psycopg2 with an upsert pattern.

import os
from apify_client import ApifyClient
import psycopg2
from psycopg2.extras import execute_values

# Initialize clients
APIFY_TOKEN = os.getenv("APIFY_TOKEN", "YOUR_APIFY_API_TOKEN")
DATABASE_URL = os.getenv("DATABASE_URL", "postgresql://user:pass@localhost:5432/market_db")

apify_client = ApifyClient(APIFY_TOKEN)

def run_yandex_pipeline():
    # 1. Define run input parameters
    run_input = {
        "queries": ["b2b logistics software", "warehouse automation api"],
        "maxPagesPerQuery": 2,
        "region": "213", # 213 = Moscow
        "language": "en"
    }

    print("Triggering madeingermany/scrape-yandex Actor...")
    run = apify_client.actor("madeingermany/scrape-yandex").call(run_input=run_input)

    dataset_id = run["defaultDatasetId"]
    print(f"Extraction finished. Fetching records from Dataset: {dataset_id}")

    dataset_items = apify_client.dataset(dataset_id).list_items().items
    if not dataset_items:
        print("No items found.")
        return

    # 2. Persist data into PostgreSQL
    conn = psycopg2.connect(DATABASE_URL)
    cursor = conn.cursor()

    create_table_query = """
    CREATE TABLE IF NOT EXISTS yandex_serp_records (
        id SERIAL PRIMARY KEY,
        query TEXT,
        position INT,
        title TEXT,
        url TEXT UNIQUE,
        snippet TEXT,
        scraped_at TIMESTAMP
    );
    """
    cursor.execute(create_table_query)

    records = [
        (
            item.get("query"),
            item.get("position"),
            item.get("title"),
            item.get("url"),
            item.get("snippet"),
            item.get("scrapedAt")
        )
        for item in dataset_items if item.get("url")
    ]

    upsert_query = """
    INSERT INTO yandex_serp_records (query, position, title, url, snippet, scraped_at)
    VALUES %s
    ON CONFLICT (url) DO UPDATE SET
        position = EXCLUDED.position,
        snippet = EXCLUDED.snippet,
        scraped_at = EXCLUDED.scraped_at;
    """

    execute_values(cursor, upsert_query, records)
    conn.commit()
    cursor.close()
    conn.close()

    print(f"Successfully synced {len(records)} SERP records to PostgreSQL.")

if __name__ == "__main__":
    run_yandex_pipeline()

Receiving Webhooks with Node.js & Express

Rather than polling the Actor status, you can configure Apify to trigger an HTTP POST webhook upon run completion (ACTOR.RUN.SUCCEEDED). Below is an Express-based microservice that receives the webhook payload, pulls dataset items, and routes competitive domain discoveries directly to Airtable.

import express from 'express';
import { ApifyClient } from 'apify-client';

const app = express();
app.use(express.json());

const apifyClient = new ApifyClient({
    token: process.env.APIFY_TOKEN || 'YOUR_APIFY_API_TOKEN',
});

// Endpoint triggered by Apify Webhook
app.post('/api/webhooks/yandex-run', async (req, res) => {
    try {
        const { eventType, eventData } = req.body;

        if (eventType !== 'ACTOR.RUN.SUCCEEDED') {
            return res.status(200).send('Ignoring non-success event.');
        }

        const defaultDatasetId = eventData.defaultDatasetId;
        console.log(`Processing completed run dataset: ${defaultDatasetId}`);

        // Pull items directly from the assigned cloud dataset
        const { items } = await apifyClient.dataset(defaultDatasetId).listItems();

        const transformedRecords = items.map((result) => ({
            fields: {
                Keyword: result.query,
                Rank: result.position,
                Title: result.title,
                URL: result.url,
                Snippet: result.snippet,
                ExtractedAt: result.scrapedAt || new Date().toISOString(),
            },
        }));

        // Send payload to external CRM or Airtable endpoint
        if (process.env.AIRTABLE_WEBHOOK_URL) {
            await fetch(process.env.AIRTABLE_WEBHOOK_URL, {
                method: 'POST',
                headers: { 'Content-Type': 'application/json' },
                body: JSON.stringify({ records: transformedRecords }),
            });
            console.log(`Dispatched ${transformedRecords.length} records to Airtable.`);
        }

        res.status(200).json({ status: 'success', processed: transformedRecords.length });
    } catch (error) {
        console.error('Webhook execution failed:', error);
        res.status(500).json({ error: error.message });
    }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`SERP Webhook listener running on port ${PORT}`));

Strategic Business Use Cases

1. Brand Infringement & SERP Hijacking Detection

Grey-market operators, unauthorized resellers, and counterfeit vendors frequently bid on high-intent brand queries or employ black-hat SEO to outrank official localized properties. Scheduling a daily pipeline allows security teams to catch unauthorized landing pages instantly and file automated takedowns.

2. Market Share & Visibility Tracking

Track changes in domain visibility for target high-volume enterprise software or consumer product keywords. By persisting rankings daily into PostgreSQL or Snowflake, data teams can compute rolling Share of Voice (SoV) indexes and identify when a competitor launches a new content cluster.

3. CRM Lead Enrichment & Inbound Qualification

When prospective enterprise accounts search for related services in regional markets, identifying which regional agencies, consultancies, or vendor directories dominate top results allows sales intelligence teams to enrich CRM accounts in HubSpot or Salesforce with direct market contextual data.

Production Best Practices

• Idempotency Keys: Use the combination of query + url + date string as a unique composite index in your database to prevent duplicate writes during retried runs.
• Geo-targeting Precision: Yandex heavily customizes SERPs based on the user’s LR (Location Region) code. Pass precise numeric region IDs (e.g., 213 for Moscow, 2 for Saint Petersburg) in the Actor inputs to maintain consistent ranking tracking over time.
• Payload Splitting: When scraping broad keyword lists exceeding 500 search queries, configure the Actor using multiple concurrent batch runs to reduce latency and isolate run-level error boundaries.