Man in front of computer screens

IWF Crawler

Our intelligent web crawler uses pioneering technology to scan web pages on the internet searching out images and videos showing the sexual abuse of children so our analysts can work to have them removed. In 2020 alone it crawled almost 42 million webpages, and over half a billion images. 

Our Hotline assessed and took action on over 150,000 webpages containing images or videos of child sexual abuse – each page could contain hundreds or thousands of images and videos. The majority of those reports were found by proactive searching by our analysts.

One of the essential tech tools helping our analysts find these webpages is the IWF Crawler. Utilising our child sexual abuse imagery Hash Database, the IWF Crawler finds images matching the digital fingerprints (or hashes) of images our analysts have already assessed and categorised as criminal. By comparing each image our crawler finds to the hashes of known child sexual abuse material, it means we can find duplicate child sexual abuse images hidden across the internet and eliminate them.

We use our crawler as one operational tactic in a suite of tools designed to find, remove, and disrupt the availability of child sexual abuse material online. It’s deployed in a considered and targeted manner in order to be most effective. In addition, it reduces unnecessary exposure to child sexual abuse imagery for our analysts, protecting their welfare and well-being.

Our crawler also allows us to provide an additional service to our Members in the domain sector registry sector.

Our services

We provide a unique range of services to help our Members defend their customers, staff and children, wherever they are in the world.

Find out more
Our Members
Domain Alerts