scrapy/scrapy

Public scan — anyone with this URL can view this analysis. Sign up to track your own repos privately, run scheduled re-scans, and get AI fix prompts via your dashboard.

145 of your 223 findings came from Repobility's proprietary detections. ✓ Repobility tags below mark them.

Scan timing: clone 2.94s · analysis 51.7s · 4.6 MB · GitHub API rate-limit (preflight)

https://github.com/scrapy/scrapy · scanned 2026-06-05 09:27 UTC (5 days, 18 hours ago) · 10 languages

438 raw signals (208 security + 230 graph) 63rd percentile · Python · medium (20-100K LoC) System graph score 74 (lower by 2)

File as issue Image

UNIFIED Repobility · multi-layer engine · AI coders

Complete repo analysis

Last scanned 5 days, 18 hours ago · v2 · 182 actionable findings from 2 signal sources. 140 repeated signals grouped for readability. Security checks, system graph analysis, and verified AI-agent feedback are merged into one review queue.

JSON

88.9% cov · 87 findings

Score breakdown â 2026-05-18-v5

Component	Sub-score	Weight	Contribution
`structure_score`	60.0	0.15	9.00
`security_score`	46.0	0.25	11.50
`testing_score`	100.0	0.20	20.00
`documentation_score`	94.0	0.15	14.10
`practices_score`	70.0	0.15	10.50
`code_quality`	69.3	0.10	6.93
Overall		1.00	72.0

Severity distribution — click a segment to filter

Active filters: excluding tests × Reset all

Severity: Critical 5 High 16 Medium 11 Low 89 9-Layer: Software Security Quality Integrity Frontend Hardware Data Network Cicd Source: Security checks 95 System graph 87 Crowd 0 Layer: Software 70 Security 12 Quality 95 Cicd 3 Api 1 Frontend 1

Exclude dismissed / FP Exclude test files

All 6135 nodes from the latest scan, grouped by kind. Each node is a unit the engine identified (file, function, endpoint, table…). Most users won't need this view — it's primarily for debugging the engine's graph extraction or for AI agents that want to enumerate the project structure.

Label	Layer	Status	Path
`_py_files`	software	healthy	`conftest.py:20`
`pytest_addoption`	software	healthy	`conftest.py:59`
`mockserver`	software	healthy	`conftest.py:71`
`mitm_proxy_server`	software	healthy	`conftest.py:77`
`mitm_proxy_server_https`	software	healthy	`conftest.py:90`
`reactor_pytest`	software	healthy	`conftest.py:103`
`pytest_configure`	software	healthy	`conftest.py:107`
`pytest_runtest_setup`	software	healthy	`conftest.py:116`
`process_request`	software	healthy	`tests/test_request_cb_kwargs.py:15`
`process_response`	software	healthy	`tests/test_request_cb_kwargs.py:19`
`process_start`	software	healthy	`tests/test_request_cb_kwargs.py:30`
`process_spider_input`	software	healthy	`tests/test_request_cb_kwargs.py:36`
`process_spider_output`	software	healthy	`tests/test_request_cb_kwargs.py:41`
`start`	software	healthy	`tests/test_request_cb_kwargs.py:64`
`parse_first`	software	healthy	`tests/test_request_cb_kwargs.py:84`
`parse_second`	software	healthy	`tests/test_request_cb_kwargs.py:94`
`parse_general`	software	healthy	`tests/test_request_cb_kwargs.py:98`
`parse_no_kwargs`	software	healthy	`tests/test_request_cb_kwargs.py:110`
`parse_default`	software	healthy	`tests/test_request_cb_kwargs.py:114`
`parse_takes_less`	software	healthy	`tests/test_request_cb_kwargs.py:121`
`parse_takes_more`	software	healthy	`tests/test_request_cb_kwargs.py:127`
`parse_downloader_mw`	software	healthy	`tests/test_request_cb_kwargs.py:133`
`parse_spider_mw`	software	healthy	`tests/test_request_cb_kwargs.py:140`
`parse_spider_mw_2`	software	healthy	`tests/test_request_cb_kwargs.py:146`
`setup_class`	software	healthy	`tests/test_request_cb_kwargs.py:153`
`teardown_class`	software	healthy	`tests/test_request_cb_kwargs.py:158`
`test_callback_kwargs`	software	healthy	`tests/test_request_cb_kwargs.py:162`
`test_pinned_twisted_version`	software	healthy	`tests/test_dependencies.py:11`
`test_url_is_from_any_domain`	software	healthy	`tests/test_utils_url.py:16`
`test_url_is_from_spider`	software	healthy	`tests/test_utils_url.py:41`
`test_url_is_from_spider_class_attributes`	software	healthy	`tests/test_utils_url.py:51`
`test_url_is_from_spider_with_allowed_domains`	software	healthy	`tests/test_utils_url.py:61`
`test_url_has_any_extension`	software	healthy	`tests/test_utils_url.py:97`
`test_add_http_if_no_scheme`	software	healthy	`tests/test_utils_url.py:160`
`test_guess_scheme`	software	healthy	`tests/test_utils_url.py:190`
`test_guess_scheme_skipped`	software	healthy	`tests/test_utils_url.py:204`
`test_noop`	software	healthy	`tests/test_utils_url.py:216`
`test_fragments`	software	healthy	`tests/test_utils_url.py:219`
`test_path`	software	healthy	`tests/test_utils_url.py:236`
`test_credentials`	software	healthy	`tests/test_utils_url.py:271`
`test_default_ports_creds_off`	software	healthy	`tests/test_utils_url.py:311`
`test_default_ports`	software	healthy	`tests/test_utils_url.py:351`
`test_default_ports_keep`	software	healthy	`tests/test_utils_url.py:393`
`test_origin_only`	software	healthy	`tests/test_utils_url.py:420`
`test__is_filesystem_path`	software	healthy	`tests/test_utils_url.py:443`
`download_handler_cls`	software	healthy	`tests/test_downloader_handlers_http_base.py:1278`
`get_dh`	software	healthy	`tests/test_downloader_handlers_http_base.py:1296`
`test_unsupported_scheme`	software	healthy	`tests/test_downloader_handlers_http_base.py:94`
`test_download`	software	healthy	`tests/test_downloader_handlers_http_base.py:1308`
`test_download_head`	software	healthy	`tests/test_downloader_handlers_http_base.py:108`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`InjectArgumentsDownloaderMiddleware`	software	healthy	`tests/test_request_cb_kwargs.py:10`
`InjectArgumentsSpiderMiddleware`	software	healthy	`tests/test_request_cb_kwargs.py:25`
`KeywordArgumentsSpider`	software	healthy	`tests/test_request_cb_kwargs.py:51`
`TestCallbackKeywordArguments`	software	healthy	`tests/test_request_cb_kwargs.py:151`
`TestScrapyUtils`	software	healthy	`tests/test_dependencies.py:10`
`MySpider`	software	healthy	`tests/test_utils_url.py:62`
`MySpider2`	software	healthy	`tests/test_utils_url.py:73`
`MySpider3`	software	healthy	`tests/test_utils_url.py:79`
`TestStripUrl`	software	healthy	`tests/test_utils_url.py:208`
`TestHttpBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:61`
`TestHttpsBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:807`
`TestSimpleHttpsBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:867`
`TestHttpsWrongHostnameBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:914`
`TestHttpsInvalidDNSIdBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:924`
`TestHttpsInvalidDNSPatternBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:930`
`TestHttpsCustomCiphersBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:937`
`TestHttpWithCrawlerBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:941`
`TestHttpProxyBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:1080`
`TestMitmProxyBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:1160`
`TestRealWebsiteBase`	software	healthy	`tests/test_downloader_handlers_http_base.py:1275`
`ZeroDivisionErrorPipeline`	software	healthy	`tests/pipelines.py:6`
`ProcessWithZeroDivisionErrorPipeline`	software	healthy	`tests/pipelines.py:14`
`TestCoreStatsExtension`	software	healthy	`tests/test_stats.py:31`
`TestStatsCollector`	software	healthy	`tests/test_stats.py:70`
`CustomStatsCollector`	software	healthy	`tests/test_stats.py:159`
`TestSelector`	software	healthy	`tests/test_selector.py:14`
`TestJMESPath`	software	healthy	`tests/test_selector.py:108`
`TestRefererMiddleware`	software	healthy	`tests/test_spidermiddleware_referer.py:38`
`MixinDefault`	software	healthy	`tests/test_spidermiddleware_referer.py:65`
`MixinNoReferrer`	software	healthy	`tests/test_spidermiddleware_referer.py:92`
`MixinNoReferrerWhenDowngrade`	software	healthy	`tests/test_spidermiddleware_referer.py:102`
`MixinSameOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:184`
`MixinOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:253`
`MixinStrictOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:277`
`MixinOriginWhenCrossOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:305`
`MixinStrictOriginWhenCrossOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:412`
`MixinUnsafeUrl`	software	healthy	`tests/test_spidermiddleware_referer.py:524`
`TestRefererMiddlewareDefault`	software	healthy	`tests/test_spidermiddleware_referer.py:628`
`TestSettingsNoReferrer`	software	healthy	`tests/test_spidermiddleware_referer.py:633`
`TestSettingsNoReferrerWhenDowngrade`	software	healthy	`tests/test_spidermiddleware_referer.py:637`
`TestSettingsSameOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:645`
`TestSettingsOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:649`
`TestSettingsStrictOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:653`
`TestSettingsOriginWhenCrossOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:659`
`TestSettingsStrictOriginWhenCrossOrigin`	software	healthy	`tests/test_spidermiddleware_referer.py:667`
`TestSettingsUnsafeUrl`	software	healthy	`tests/test_spidermiddleware_referer.py:675`
`CustomPythonOrgPolicy`	software	healthy	`tests/test_spidermiddleware_referer.py:679`
`TestSettingsCustomPolicy`	software	healthy	`tests/test_spidermiddleware_referer.py:694`
`TestRequestMetaDefault`	software	healthy	`tests/test_spidermiddleware_referer.py:715`
`TestRequestMetaNoReferrer`	software	healthy	`tests/test_spidermiddleware_referer.py:719`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`conftest.py`	software	healthy	`conftest.py`
`.readthedocs.yml`	software	healthy	`.readthedocs.yml`
`CONTRIBUTING.md`	software	healthy	`CONTRIBUTING.md`
`.pre-commit-config.yaml`	software	healthy	`.pre-commit-config.yaml`
`INSTALL.md`	software	healthy	`INSTALL.md`
`pyproject.toml`	software	healthy	`pyproject.toml`
`CODE_OF_CONDUCT.md`	software	healthy	`CODE_OF_CONDUCT.md`
`SECURITY.md`	software	healthy	`SECURITY.md`
`codecov.yml`	software	healthy	`codecov.yml`
`test_request_cb_kwargs.py`	software	healthy	`tests/test_request_cb_kwargs.py`
`test_dependencies.py`	software	healthy	`tests/test_dependencies.py`
`test_utils_url.py`	software	healthy	`tests/test_utils_url.py`
`test_downloader_handlers_http_base.py`	software	healthy	`tests/test_downloader_handlers_http_base.py`
`pipelines.py`	software	healthy	`tests/pipelines.py`
`test_stats.py`	software	healthy	`tests/test_stats.py`
`test_selector.py`	software	healthy	`tests/test_selector.py`
`test_spidermiddleware_referer.py`	software	healthy	`tests/test_spidermiddleware_referer.py`
`test_downloadermiddleware_httpauth.py`	software	healthy	`tests/test_downloadermiddleware_httpauth.py`
`test_zz_resources.py`	software	healthy	`tests/test_zz_resources.py`
`test_http_cookies.py`	software	healthy	`tests/test_http_cookies.py`
`test_http_response_text.py`	software	healthy	`tests/test_http_response_text.py`
`test_request_left.py`	software	healthy	`tests/test_request_left.py`
`test_spiderstate.py`	software	healthy	`tests/test_spiderstate.py`
`test_addons.py`	software	healthy	`tests/test_addons.py`
`test_command_parse.py`	software	healthy	`tests/test_command_parse.py`
`test_crawler_subprocess.py`	software	healthy	`tests/test_crawler_subprocess.py`
`test_utils_template.py`	software	healthy	`tests/test_utils_template.py`
`test_link.py`	software	healthy	`tests/test_link.py`
`test_command_crawl.py`	software	healthy	`tests/test_command_crawl.py`
`test_extension_statsmailer.py`	software	healthy	`tests/test_extension_statsmailer.py`
`__init__.py`	software	healthy	`tests/__init__.py`
`test_engine_stop_download_headers.py`	software	healthy	`tests/test_engine_stop_download_headers.py`
`test_feedexport_storages.py`	software	healthy	`tests/test_feedexport_storages.py`
`test_downloader_handlers.py`	software	healthy	`tests/test_downloader_handlers.py`
`test_utils_serialize.py`	software	healthy	`tests/test_utils_serialize.py`
`test_utils_trackref.py`	software	healthy	`tests/test_utils_trackref.py`
`test_utils_python.py`	software	healthy	`tests/test_utils_python.py`
`test_command_version.py`	software	healthy	`tests/test_command_version.py`
`test_downloadermiddleware_retry.py`	software	healthy	`tests/test_downloadermiddleware_retry.py`
`test_downloadermiddleware.py`	software	healthy	`tests/test_downloadermiddleware.py`
`test_http2_client_protocol.py`	software	healthy	`tests/test_http2_client_protocol.py`
`test_utils_signal.py`	software	healthy	`tests/test_utils_signal.py`
`test_http_headers.py`	software	healthy	`tests/test_http_headers.py`
`test_squeues_request.py`	software	healthy	`tests/test_squeues_request.py`
`test_utils_deprecate.py`	software	healthy	`tests/test_utils_deprecate.py`
`test_loader.py`	software	healthy	`tests/test_loader.py`
`test_extension_periodic_log.py`	software	healthy	`tests/test_extension_periodic_log.py`
`test_downloadermiddleware_offsite.py`	software	healthy	`tests/test_downloadermiddleware_offsite.py`
`test_utils_datatypes.py`	software	healthy	`tests/test_utils_datatypes.py`
`test_downloadermiddleware_redirect.py`	software	healthy	`tests/test_downloadermiddleware_redirect.py`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`tests`	software	healthy	`tests`
`mocks`	software	healthy	`tests/mocks`
`test_settings`	software	healthy	`tests/test_settings`
`utils`	software	healthy	`tests/utils`
`AsyncCrawlerRunner`	software	healthy	`tests/AsyncCrawlerRunner`
`mockserver`	software	healthy	`tests/mockserver`
`keys`	software	healthy	`tests/keys`
`sample_data`	software	healthy	`tests/sample_data`
`link_extractor`	software	healthy	`tests/sample_data/link_extractor`
`test_site`	software	healthy	`tests/sample_data/test_site`
`CrawlerRunner`	software	healthy	`tests/CrawlerRunner`
`CrawlerProcess`	software	healthy	`tests/CrawlerProcess`
`test_cmdline`	software	healthy	`tests/test_cmdline`
`test_cmdline_crawl_with_pipeline`	software	healthy	`tests/test_cmdline_crawl_with_pipeline`
`test_spider`	software	healthy	`tests/test_cmdline_crawl_with_pipeline/test_spider`
`spiders`	software	healthy	`tests/test_cmdline_crawl_with_pipeline/test_spider/spiders`
`AsyncCrawlerProcess`	software	healthy	`tests/AsyncCrawlerProcess`
`test_utils_misc`	software	healthy	`tests/test_utils_misc`
`test_walk_modules`	software	healthy	`tests/test_utils_misc/test_walk_modules`
`mod`	software	healthy	`tests/test_utils_misc/test_walk_modules/mod`
`test_spiderloader`	software	healthy	`tests/test_spiderloader`
`test_spiders`	software	healthy	`tests/test_spiderloader/test_spiders`
`nested`	software	healthy	`tests/test_spiderloader/test_spiders/nested`
`spiders_from_addons`	software	healthy	`tests/test_spiderloader/spiders_from_addons`
`docs`	software	healthy	`docs`
`utils`	software	healthy	`docs/utils`
`_templates`	software	healthy	`docs/_templates`
`_static`	software	healthy	`docs/_static`
`_ext`	software	healthy	`docs/_ext`
`_tests`	software	healthy	`docs/_tests`
`extras`	software	healthy	`extras`
`scrapy`	software	healthy	`scrapy`
`settings`	software	healthy	`scrapy/settings`
`loader`	software	healthy	`scrapy/loader`
`commands`	software	healthy	`scrapy/commands`
`pipelines`	software	healthy	`scrapy/pipelines`
`utils`	software	healthy	`scrapy/utils`
`http`	software	healthy	`scrapy/http`
`response`	software	healthy	`scrapy/http/response`
`request`	software	healthy	`scrapy/http/request`
`selector`	software	healthy	`scrapy/selector`
`templates`	software	healthy	`scrapy/templates`
`project`	software	healthy	`scrapy/templates/project`
`module`	software	healthy	`scrapy/templates/project/module`
`spiders`	software	healthy	`scrapy/templates/project/module/spiders`
`spiders`	software	healthy	`scrapy/spiders`
`extensions`	software	healthy	`scrapy/extensions`
`downloadermiddlewares`	software	healthy	`scrapy/downloadermiddlewares`
`spidermiddlewares`	software	healthy	`scrapy/spidermiddlewares`
`core`	software	healthy	`scrapy/core`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`close-llm-pr`	cicd	healthy	`.github/workflows/auto-close-llm-pr.yml`
`checks`	cicd	healthy	`.github/workflows/checks.yml`
`pre-commit`	cicd	healthy	`.github/workflows/checks.yml`
`tests`	cicd	healthy	`.github/workflows/tests-windows.yml`
`tests`	cicd	healthy	`.github/workflows/tests-macos.yml`
`tests`	cicd	healthy	`.github/workflows/tests-ubuntu.yml`
`publish`	cicd	healthy	`.github/workflows/publish.yml`

Label	Layer	Status	Path
`192.169.0.15`	network	healthy	`tests/test_utils_url.py`
`127.0.0.2`	network	healthy	`tests/test_downloader_handlers_http_base.py`
`127.0.0.1`	network	healthy	`tests/test_downloader_handlers_http_base.py`
`109.187.217.200`	network	healthy	`tests/test_http_response_text.py`
`8.1.2.3`	network	healthy	`scrapy/core/http2/stream.py`
`8.1.2.6`	network	healthy	`scrapy/core/http2/stream.py`

Label	Layer	Status	Path
`gha::auto-close-llm-pr`	cicd	healthy	`.github/workflows/auto-close-llm-pr.yml`
`gha::checks`	cicd	healthy	`.github/workflows/checks.yml`
`gha::tests-windows`	cicd	healthy	`.github/workflows/tests-windows.yml`
`gha::tests-macos`	cicd	healthy	`.github/workflows/tests-macos.yml`
`gha::tests-ubuntu`	cicd	healthy	`.github/workflows/tests-ubuntu.yml`
`gha::publish`	cicd	healthy	`.github/workflows/publish.yml`

Label	Layer	Status	Path
`password_literal::tests/test_utils_url.py`	security	healthy	`tests/test_utils_url.py`
`password_literal::tests/test_downloader_handler_twisted_ftp…`	security	healthy	`tests/test_downloader_handler_twisted_ftp.py`

Label	Layer	Status	Path
`repobility-clone-v2ql_bag`	software	healthy	`/tmp/repobility-clone-v2ql_bag`

Label	Layer	Status	Path
`vps::aws`	hardware	healthy	`tests/test_downloader_handlers.py`

Label	Layer	Status	Path
`GITHUB_TOKEN`	cicd	healthy	`—`

For AI agents: Voting guide (TP/FP) MCP manifest Stdio wrapper SARIF Integrate Findings queue Vote TP/FP on findings to calibrate the engine.

For AI agents + API integrations

Email me when this repo regresses

Free. We re-scan periodically; new criticals → your inbox. No signup required for the scan itself.

API access

This page is publicly accessible at: https://repobility.com/scan/9db5f11e-57f3-477a-9bb0-dfae01c72bf5/

To check status programmatically (no auth required):

curl -s https://repobility.com/api/v1/public/scan/9db5f11e-57f3-477a-9bb0-dfae01c72bf5/

Important — please don't re-submit the same URL repeatedly. The submission endpoint is idempotent: re-submitting the same git URL returns this same scan_token, not a new one. To re-scan this repo, sign up free and use the dashboard.

scrapy/scrapy

Complete repo analysis

function 50+

class 50+

file 50+

directory 50+

job 7

service 6

pipeline 6

secret 2

repo 1

vps 1

env_var 1