FrCrawler Content Clustering

Clustering module for the FrCrawler project.

Installation

A rust (nightly) development environemnt is required to build and install the project.

python -m venv env
env/bin/pip install \
    git+https://gitlab.rd.nic.fr/labs/frcrawler/frcrawler.git#egg=frcrawler \
    git+https://gitlab.rd.nic.fr/labs/frcrawler/content-clustering.git#egg=frcrawler-content-clustering

Usage

Compute hashes

>>> from frcrawler_content_clustering import ContentHash

>>> page_hash = ContentHash('toto')

>>> page_hash.lzjd
[-897065139, 152217691, 1748272243]

>>> page_hash.ssdeep
'3:wKn:wKn'

# Dict representation of the object.
# The LZJD hash is encoded in base64.
>>> page_hash.to_json()
{'lzjd': 'yoffTQkSqFtoNIRz', 'ssdeep': '3:wKn:wKn'}

Clusterize content

Similarity pairs are saved in Clickhouse, see the clustering_similiarities tables in the schema.dev.sql file.

See the examples folder for similarity computation and content clustering examples.

The scripts repository provide a more complete use case.

Crawler

The frcrawler_content_clustering.crawler:ComputeHashes class is provided to be used as a job task for the FrCrawler. The frcrawler.tasks.http:HttpCrawler must have been executed before with the save_content option to yes.

The frcrawler.tasks.filters:IgnoreKeys filter can be used to remove the HTTP response content from the final object before saving.

Here is an example configuration:

jobs:
  polling: 5
  definitions:
    - name: content-hash
      tasks:
        - type: frcrawler.tasks.http:HttpCrawler
          config:
            save_content: yes

        - type: frcrawler_content_clustering.crawler:ComputeHashes

        - type: frcrawler.tasks.filters:IgnoreKeys
          config:
            ignore:
              - '/web/details/*/response/content'

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.

2.6 KiB Raw Permalink Blame History