mirror of
https://gitlab.rd.nic.fr/labs/frcrawler/content-clustering.git
synced 2025-04-11 23:15:14 +02:00
87 lines
2.6 KiB
Markdown
87 lines
2.6 KiB
Markdown
# FrCrawler Content Clustering
|
|
|
|
Clustering module for the [FrCrawler](https://gitlab.rd.nic.fr/labs/frcrawler/frcrawler) project.
|
|
|
|
## Installation
|
|
|
|
A rust (nightly) development environemnt is required to build and install the project.
|
|
|
|
```sh
|
|
python -m venv env
|
|
env/bin/pip install \
|
|
git+https://gitlab.rd.nic.fr/labs/frcrawler/frcrawler.git#egg=frcrawler \
|
|
git+https://gitlab.rd.nic.fr/labs/frcrawler/content-clustering.git#egg=frcrawler-content-clustering
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Compute hashes
|
|
```py
|
|
>>> from frcrawler_content_clustering import ContentHash
|
|
|
|
>>> page_hash = ContentHash('toto')
|
|
|
|
>>> page_hash.lzjd
|
|
[-897065139, 152217691, 1748272243]
|
|
|
|
>>> page_hash.ssdeep
|
|
'3:wKn:wKn'
|
|
|
|
# Dict representation of the object.
|
|
# The LZJD hash is encoded in base64.
|
|
>>> page_hash.to_json()
|
|
{'lzjd': 'yoffTQkSqFtoNIRz', 'ssdeep': '3:wKn:wKn'}
|
|
```
|
|
|
|
### Clusterize content
|
|
|
|
Similarity pairs are saved in Clickhouse, see the `clustering_similiarities` tables in the [schema.dev.sql](./schema.dev.sql) file.
|
|
|
|
See the [examples](./examples/) folder for similarity computation and content clustering examples.
|
|
|
|
The [scripts repository](https://gitlab.rd.nic.fr/labs/frcrawler/scripts) provide a more complete use case.
|
|
|
|
|
|
### Crawler
|
|
|
|
The `frcrawler_content_clustering.crawler:ComputeHashes` class is provided to be used as a job task for the FrCrawler. The `frcrawler.tasks.http:HttpCrawler` must have been executed before with the `save_content` option to `yes`.
|
|
|
|
The `frcrawler.tasks.filters:IgnoreKeys` filter can be used to remove the HTTP response content from the final object before saving.
|
|
|
|
Here is an example configuration:
|
|
|
|
```yaml
|
|
jobs:
|
|
polling: 5
|
|
definitions:
|
|
- name: content-hash
|
|
tasks:
|
|
- type: frcrawler.tasks.http:HttpCrawler
|
|
config:
|
|
save_content: yes
|
|
|
|
- type: frcrawler_content_clustering.crawler:ComputeHashes
|
|
|
|
- type: frcrawler.tasks.filters:IgnoreKeys
|
|
config:
|
|
ignore:
|
|
- '/web/details/*/response/content'
|
|
|
|
```
|
|
|
|
## License
|
|
|
|
Copyright (C) 2023 Afnic
|
|
|
|
This program is free software: you can redistribute it and/or modify
|
|
it under the terms of the GNU General Public License as published by
|
|
the Free Software Foundation, either version 3 of the License, or
|
|
(at your option) any later version.
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
GNU General Public License for more details.
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
along with this program. If not, see <https://www.gnu.org/licenses/>.
|