A Telegram bot that delivers Upwork jobs in real time
A Telegram bot that tracks Upwork RSS feeds and pushes new job postings to thousands of users within seconds — each with their own filters, at millions of messages a day.
- Stack
- Python · RabbitMQ · asyncio · Telegram Bot API · Docker
- Scale
- thousands of users · millions of messages/day
- Delivery
- within seconds, exactly-once
A job tracker is trivial until scale hits
A job tracker sounds trivial: poll a feed, forward new items. It stops being trivial the moment you have thousands of users, each with their own filters, and a single new posting that can match hundreds of them at once.
Now one job is not one message; it is hundreds, and they all need to go out within seconds — without duplicates, without tripping Telegram's rate limits, and without a slow RSS feed taking the whole pipeline down.
Overlapping RSS windows, processed exactly once
Upwork exposes new jobs through per-search RSS feeds, but treating them as a clean stream is naive: feeds return overlapping windows on every poll, items arrive out of order, and polling too aggressively risks throttling.
The ingestion layer polls each feed on a schedule, normalizes every entry into a canonical job record, and deduplicates against recently-seen IDs so a posting is processed exactly once. Everything downstream is decoupled from polling through RabbitMQ, so a failing or slow feed never blocks delivery.
One posting, hundreds of matches, one deep queue
Thousands of users each define their own searches and keyword filters, and one posting can match hundreds simultaneously. Instead of matching inline, new jobs are published to RabbitMQ and consumed by a pool of workers that evaluate each posting against user filter sets and fan out one delivery task per matched user.
The queue absorbs bursts: a wave of postings at peak does not overwhelm the system, it just deepens the queue, and workers drain it at a controlled rate. This is what pushes throughput into millions of messages a day.
High-load delivery under Telegram's rate limits
Telegram enforces per-chat and global rate ceilings; crossing them gets the bot throttled. The delivery layer respects those limits with rate-controlled workers, backpressure, and automatic retry-with-backoff on 429s, so notifications are neither lost nor sent out of order.
Failed sends are requeued rather than dropped, and transient errors are handled gracefully. The result is a pipeline that stays reliable at thousands of users and millions of messages a day, delivering matches within seconds of a job appearing on Upwork.
See what this would look like in your operations.
30 minutes · we map your 3 best automation opportunities · no obligation