We Built Our Own Lead Generator. Here’s What Actually Matters (Hint: It’s Not the Scraping)

🚀 Lessons from turning “0 leads across the board” into thousands of real, callable businesses in a single afternoon.

Scroll TikTok for five minutes and you’ll see it: “I made $10K with this one lead-gen hack.” Scrape this, automate that, plug in some AI and watch the leads roll in.

We actually built one. A real, working lead generator for our own outreach — search, filter, de-duplicate, enrich, review, and hand off to sales. Along the way we learned that the flashy part (grabbing data off the internet) is the easy part. The parts that decide whether you get ten leads or ten thousand are boring, unglamorous engineering: choosing the right data source, using your API properly, pagination, timing, and measuring what’s really happening.

We’re not going to hand over our whole playbook (more on that at the end 🤫), but here’s what we learned that you can use today.

1. 🔎 Your data source matters more than your filters

Our first version searched social media for small business pages. For a while it worked. Then one day, nearly every search came back with zero leads.

We spent hours tuning filters and squeezed it up to about one usable lead per search. Then we stepped back and asked a better question: where do real, operating local businesses actually list themselves? Switching to local map-style business results took us to roughly 13–20 new leads per search — and they came with phone numbers, addresses, categories, and websites already attached.

Takeaway: If you’re fighting your data source, stop polishing the filter. Find a better source. Search engines change how they handle tricks like site: operators without warning, and a technique that worked last year can quietly die.

2. 📊 Measure before you fix

“0 results” told us nothing. Was the API broken? Were we out of credits? Were the results all duplicates?

The fix that mattered most was tiny: we made every search report why each result was kept or thrown away — how many came back, how many were already in our database, how many were directories, how many failed a rule. Within one run, the real culprit was obvious: the search provider was wrapping results in redirect links, and our filter didn’t recognize them.

Takeaway: A few honest counters in your logs will save you more time than any clever algorithm. When something says “0,” make it tell you 0 out of what, and why.

3. 📄 Pagination: stop paying for page one over and over

Here’s a mistake almost every DIY scraper makes: it searches the same keywords again and again, and every time it gets the same first page of results. You pay for the search, find nothing new, and assume the market is tapped out.

What works:

Track the page for every query. Each keyword + location combo remembers which page it’s on, so a re-run asks for page 2, not page 1 again.
Go breadth-first. Run page 1 for every combo before anyone’s page 2. Page 1 is almost always the most valuable page per call, so cover your whole map before digging deeper.
Only advance when the provider says there’s more. Most APIs tell you if a next page exists. Respect it, and cap the depth (we stop at a few pages).
Know when to quit. Deeper pages drift farther from the town center and overlap more with neighbors. When a page stops paying for itself, stop.

4. ⏱ API utilization and timing: respect the limits (they’re there either way)

Paid search APIs usually have two limits: a monthly quota and an hourly throughput cap. Ignore either one and your tool falls over at the worst moment.

Trust the provider’s numbers, not your own counter. Our tool once said we’d used 942 searches; the provider’s dashboard said 431. Our counter was counting requests the provider didn’t bill (like cached repeats). Read usage from the source of truth.
Pace yourself and resume automatically. When you hit the hourly cap, don’t crash — wait, then pick up exactly where you left off.
Failures must not eat your work. A combo should only be marked “done” after a successful search. A timeout shouldn’t silently burn through your queue with fake zeros.
Retry with a backoff. Networks hiccup. Pause, retry, and give up gracefully after a sensible number of tries.
Use caching to your advantage. Many providers serve identical recent searches from cache for free — perfect for debugging without spending credits.
Clean your inputs. One business listing had a 500+ character website URL stuffed with tracking codes, and it broke a database insert. Strip the junk before you store it.

5. 📧 Free enrichment beats paid enrichment

Lots of lead tools charge extra to “enrich” a lead with an email or details. When your source is good, you often don’t need to. Once a lead has a website, a polite visit to the homepage and contact page finds a surprising number of email addresses — no API credits required.

We also capture a screenshot of every business website (📸 over 4,000 on day one). For a web design company, that’s gold: in a few seconds you can tell whether a site is modern, dated, broken, or missing entirely. Flip through them gallery-style, tag the ones worth a conversation, and move on.

6. ⚖️ Compliance is a feature, not an afterthought

The TikTok version of lead gen rarely mentions this part. It should be front and center:

Honor Do Not Call lists — automatically, every time, not “when someone remembers.”
Put your outreach rules in the code. If your team can only contact certain regions or channels, the tool should enforce it so nobody has to think about it.
Deleted means deleted. If someone asks to be removed, or a lead is junk, it should never sneak back in on the next search.
Different channels, different rules. A mailed postcard and a phone call aren’t governed the same way — build your filters to match.

7. 👥 Build for a team, not a spreadsheet

Spreadsheets are where lead lists go to die: five copies, three versions, nobody knows who called whom. We kept everything in one live tool:

Assign leads, add notes, and track call status in one place
🔥 Mark hot leads (with the date) and filter to your call list for tomorrow
Tag leads for different postcard campaigns and export clean mailing lists
Separate admin and staff views, so the team sees what they need and nothing they shouldn’t
A one-click hand-off that sends a qualified lead straight into the sales process

🤫 What we’re not telling you

Our target industries, our territories, our search keywords, our scripts, and exactly how the pieces fit together — that’s the secret sauce, and it stays in the kitchen. 😉

But the lessons above are universal. Whether you’re building your own tool or buying one, ask: Is the data source solid? Does it paginate properly? Does it respect API limits and timing? Can I see why it keeps or drops a result? Does it enforce my compliance rules? If the answer to any of those is “no,” no amount of AI magic will fix it.

Want a tool like this for your business?

We’ve been building custom websites and web tools since 1996. If you need a lead pipeline, a custom CRM, or just a website that doesn’t look like it’s from 2009, let’s talk.