5 Lessons learned building a web application crawler
Building a web application crawler came with plenty of challenges—here’s what we learned. Recently, we built a web application crawler from scratch—which had some scratching their heads, asking why we’d undertake such a thing. Here’s our answer to that, plus some interesting technical challenges we ran into and how we tackled them.