Sybix Web Crawler

A general purpose web crawler with customisable configuration and processing options for discovering content on websites.

After working on a number of projects that require data to be scraped from across the web, I have developed a custom crawler written in NodeJS and using MySQL as a database. Sybix is a flexible solution and can be applied to many crawling situations - it can crawl and discover websites continually without the need for any human input. It’s a responsible crawler that respects rate-limits and robots.txt files and is currently being used in the back-end for a variety of projects including Time to Nom, Oracle and Car Lookout.

Date: 2019
Technology: NodeJS, MySQL