FrCrawler

Crawler used by Afnic Labs.

Installation

Note: A messaging backend (e.g. Redis) is required, see the configuration example for more information.

python -m venv env
env/bin/pip install git+https://gitlab.rd.nic.fr/labs/frcrawler/frcrawler.git#egg=frcrawler

Usage

# Starts 50 workers to run the jobs.
frcrawler -c config.yml run --workers 50

# Schedule jobs. Input data is taken from the datasource named `jsonfile` in
# the configuration; results are saved the storage backend called `pg` in the
# configuration.
frcrawler -c config.yml schedule --datasource jsonfile --storage pg

# Start a process that will insert new results from jobs that are configured to
# be saved in storage backend `pg` into the said storage backend.
frcrawler -c config.yml process-results --storage pg

Storage

A basic example of a table schema can be found in the schema.example.*.sql, a more complete schema is avalable in the scripts repository.

License

Copyright (C) 2023 Afnic

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.