frcrawler-siren/README.md

Siren Web Crawler

FrCrawler component to crawl the Siren number, the French company identification number, from websites.

Installation

python -m venv env
env/bin/pip install \
    git+https://gitlab.rd.nic.fr/labs/frcrawler/frcrawler.git#egg=frcrawler \
    git+https://gitlab.rd.nic.fr/labs/frcrawler/siren.git#egg=frcrawler-siren

Usage

The frcrawler_siren:ExtractSiren class is provided to be used as a job task for the FrCrawler. The frcrawler.tasks.http:HttpCrawler must have been executed before with the save_content option to yes.

The frcrawler.tasks.filters:IgnoreKeys filter can be used to remove the HTTP response content from the final object before saving.

Here is an example configuration:

jobs:
  polling: 5
  definitions:
    - name: crawl-siren
      tasks:
        - type: frcrawler.tasks.http:HttpCrawler
          config:
            save_content: yes

        - type: frcrawler_siren:ExtractSiren
          config:
            # UserAgent to use when sending HTTP queries
            useragent: frcrawler-siren
            # Max content to download from a page, in bytes
            max_content_length: 5120000

        - type: frcrawler.tasks.filters:IgnoreKeys
          config:
            ignore:
              - '/web/details/*/response/content'

License

Copyright (C) 2023 Afnic

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.