mirror of
https://gitlab.rd.nic.fr/labs/frcrawler/siren.git
synced 2025-04-11 23:05:12 +02:00
63 lines
1.9 KiB
Markdown
63 lines
1.9 KiB
Markdown
# Siren Web Crawler
|
|
|
|
FrCrawler component to crawl the Siren number, the French company identification number, from websites.
|
|
|
|
|
|
## Installation
|
|
|
|
```sh
|
|
python -m venv env
|
|
env/bin/pip install \
|
|
git+https://gitlab.rd.nic.fr/labs/frcrawler/frcrawler.git#egg=frcrawler \
|
|
git+https://gitlab.rd.nic.fr/labs/frcrawler/siren.git#egg=frcrawler-siren
|
|
```
|
|
|
|
## Usage
|
|
|
|
|
|
The `frcrawler_siren:ExtractSiren` class is provided to be used as a job task for the FrCrawler. The `frcrawler.tasks.http:HttpCrawler` must have been executed before with the `save_content` option to `yes`.
|
|
|
|
The `frcrawler.tasks.filters:IgnoreKeys` filter can be used to remove the HTTP response content from the final object before saving.
|
|
|
|
Here is an example configuration:
|
|
|
|
```yaml
|
|
jobs:
|
|
polling: 5
|
|
definitions:
|
|
- name: crawl-siren
|
|
tasks:
|
|
- type: frcrawler.tasks.http:HttpCrawler
|
|
config:
|
|
save_content: yes
|
|
|
|
- type: frcrawler_siren:ExtractSiren
|
|
config:
|
|
# UserAgent to use when sending HTTP queries
|
|
useragent: frcrawler-siren
|
|
# Max content to download from a page, in bytes
|
|
max_content_length: 5120000
|
|
|
|
- type: frcrawler.tasks.filters:IgnoreKeys
|
|
config:
|
|
ignore:
|
|
- '/web/details/*/response/content'
|
|
|
|
```
|
|
|
|
## License
|
|
|
|
Copyright (C) 2023 Afnic
|
|
|
|
This program is free software: you can redistribute it and/or modify
|
|
it under the terms of the GNU General Public License as published by
|
|
the Free Software Foundation, either version 3 of the License, or
|
|
(at your option) any later version.
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
GNU General Public License for more details.
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
along with this program. If not, see <https://www.gnu.org/licenses/>.
|