1.7 KiB
FrCrawler
Crawler used by Afnic Labs.
Installation
Note: A messaging backend (e.g. Redis) is required, see the configuration example for more information.
python -m venv env
env/bin/pip install git+https://gitlab.rd.nic.fr/labs/frcrawler/frcrawler.git#egg=frcrawler
Usage
# Starts 50 workers to run the jobs.
frcrawler -c config.yml run --workers 50
# Schedule jobs. Input data is taken from the datasource named `jsonfile` in
# the configuration; results are saved the storage backend called `pg` in the
# configuration.
frcrawler -c config.yml schedule --datasource jsonfile --storage pg
# Start a process that will insert new results from jobs that are configured to
# be saved in storage backend `pg` into the said storage backend.
frcrawler -c config.yml process-results --storage pg
Storage
A basic example of a table schema can be found in the schema.example.*.sql, a more complete schema is avalable in the scripts repository.
License
Copyright (C) 2023 Afnic
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.