Public scan — anyone with this URL can view this analysis. Sign up to track your own repos privately, run scheduled re-scans, and get AI fix prompts via your dashboard.

qiurui144/vlm-llm-benchmark

https://github.com/qiurui144/vlm-llm-benchmark · scanned 2026-06-15 23:43 UTC (2 months, 4 weeks ago)

58 raw signals (0 security + 58 graph)

File as issue Image

UNIFIED Repobility · multi-layer engine · AI coders

Complete repo analysis

Last scanned 2 months, 4 weeks ago · v1 · 44 actionable findings from 1 signal source. 14 repeated signals grouped for readability. Security checks, system graph analysis, and verified AI-agent feedback are merged into one review queue.

JSON

77.8% cov · 44 findings

Severity distribution — click a segment to filter

Active filters: excluding tests × Reset all

Severity: Critical 2 High 1 Medium 1 Low 34 9-Layer: Software Security Quality Integrity Frontend Hardware Data Network Cicd Source: Security checks 0 System graph 44 Crowd 0 Layer: Software 23 Frontend 1 Security 3 Cicd 1 Quality 16

Exclude dismissed / FP Exclude test files

All 2500 nodes from the latest scan, grouped by kind. Each node is a unit the engine identified (file, function, endpoint, table…). Most users won't need this view — it's primarily for debugging the engine's graph extraction or for AI agents that want to enumerate the project structure.

Label	Layer	Status	Path
`harness_version`	software	healthy	`run_benchmark.py:88`
`get_hardware_profile`	software	healthy	`run_benchmark.py:103`
`_default`	software	healthy	`run_benchmark.py:131`
`_model_hint`	software	healthy	`run_benchmark.py:142`
`_is_translation_capable`	software	healthy	`run_benchmark.py:163`
`_run_accuracy_dim`	software	healthy	`run_benchmark.py:170`
`_run_ttft_dim`	software	healthy	`run_benchmark.py:174`
`_run_throughput_dim`	software	healthy	`run_benchmark.py:178`
`_run_prefill_decode_dim`	software	healthy	`run_benchmark.py:182`
`_run_concurrency_dim`	software	healthy	`run_benchmark.py:186`
`_run_stability_dim`	software	healthy	`run_benchmark.py:190`
`_is_chat_capable`	software	healthy	`run_benchmark.py:194`
`_run_scenarios_dim`	software	healthy	`run_benchmark.py:206`
`_run_translation_dim`	software	healthy	`run_benchmark.py:211`
`_run_embedding_dim`	software	healthy	`run_benchmark.py:215`
`_run_rerank_dim`	software	healthy	`run_benchmark.py:219`
`_run_asr_dim`	software	healthy	`run_benchmark.py:223`
`_run_general_ability_dim`	software	healthy	`run_benchmark.py:227`
`_run_conditioned_dim`	software	healthy	`run_benchmark.py:231`
`_run_conversation_drift_dim`	software	healthy	`run_benchmark.py:235`
`run_all_for_model`	software	healthy	`run_benchmark.py:283`
`_resolve_judge`	software	healthy	`run_benchmark.py:325`
`aggregate_multi_seed`	software	healthy	`run_benchmark.py:354`
`render_markdown`	software	healthy	`run_benchmark.py:423`
`render_matrix`	software	healthy	`run_benchmark.py:460`
`main`	software	healthy	`run_benchmark.py:490`
`base_url`	software	healthy	`common.py:64`
`auth_header`	software	healthy	`common.py:80`
`effective_model_id`	software	healthy	`common.py:89`
`is_vlm`	software	healthy	`common.py:94`
`_derive_capabilities`	software	healthy	`common.py:105`
`load_models`	software	healthy	`common.py:116`
`load_benchmarks_config`	software	healthy	`common.py:131`
`encode_image_data_url`	software	healthy	`common.py:142`
`_post_with_retry`	software	healthy	`common.py:180`
`looks_truncated`	software	healthy	`common.py:225`
`infer_sync`	software	healthy	`common.py:233`
`infer_stream`	software	healthy	`common.py:342`
`infer_async`	software	healthy	`common.py:468`
`wait_model_ready`	software	healthy	`common.py:551`
`__post_init__`	software	healthy	`common.py:669`
`infer_embedding`	software	healthy	`common.py:597`
`infer_rerank`	software	healthy	`common.py:674`
`get_vram_info`	software	healthy	`common.py:742`
`percentile`	software	healthy	`common.py:763`
`summarize_latencies`	software	healthy	`common.py:780`
`proc_rss_mb`	software	healthy	`common.py:794`
`test_run_all_result_carries_schema_v1_fields`	software	healthy	`tests/test_report_schema.py:5`
`test_get_hardware_profile_degrades_to_unknown_not_crash`	software	healthy	`tests/test_report_schema.py:24`
`test_harness_version_is_short_string`	software	healthy	`tests/test_report_schema.py:30`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`RELEASE.md`	software	healthy	`RELEASE.md`
`README.md`	software	healthy	`README.md`
`run.sh`	software	healthy	`run.sh`
`CLAUDE.md`	software	healthy	`CLAUDE.md`
`models.yaml`	software	healthy	`models.yaml`
`run_benchmark.py`	software	healthy	`run_benchmark.py`
`DEVELOP.md`	software	healthy	`DEVELOP.md`
`LICENSE`	software	healthy	`LICENSE`
`common.py`	software	healthy	`common.py`
`requirements.txt`	software	healthy	`requirements.txt`
`start_all.sh`	software	healthy	`vllm_configs/start_all.sh`
`launch_helpers.sh`	software	healthy	`vllm_configs/launch_helpers.sh`
`README.md`	software	healthy	`datasets/asr/README.md`
`README.md`	software	healthy	`datasets/retrieval/README.md`
`README.md`	software	healthy	`datasets/translation/README.md`
`dialogs.json`	software	healthy	`datasets/scenarios/wechat_intent/dialogs.json`
`TESTING.md`	software	healthy	`tests/TESTING.md`
`conftest.py`	software	healthy	`tests/conftest.py`
`test_report_schema.py`	software	healthy	`tests/test_report_schema.py`
`test_provider_auth.py`	software	healthy	`tests/test_provider_auth.py`
`test_capabilities.py`	software	healthy	`tests/test_capabilities.py`
`test_registry.py`	software	healthy	`tests/test_registry.py`
`test_common_http.py`	software	healthy	`tests/test_common_http.py`
`test_accuracy.py`	software	healthy	`tests/test_accuracy.py`
`test_provider_compat.py`	software	healthy	`tests/test_provider_compat.py`
`test_common_stream.py`	software	healthy	`tests/test_common_stream.py`
`test_run_benchmark.py`	software	healthy	`tests/test_run_benchmark.py`
`test_compare.py`	software	healthy	`tests/test_compare.py`
`test_dataset_loading.py`	software	healthy	`tests/llama_benchmark/test_dataset_loading.py`
`__init__.py`	software	healthy	`tests/llama_benchmark/__init__.py`
`test_cli.py`	software	healthy	`tests/llama_benchmark/test_cli.py`
`test_synthetic_fallback_flag.py`	software	healthy	`tests/llama_benchmark/test_synthetic_fallback_flag.py`
`test_diarization_result.py`	software	healthy	`tests/llama_benchmark/test_diarization_result.py`
`__init__.py`	software	healthy	`tests/conditioned/__init__.py`
`test_runner.py`	software	healthy	`tests/conditioned/test_runner.py`
`test_context_corpus.py`	software	healthy	`tests/conditioned/test_context_corpus.py`
`__init__.py`	software	healthy	`tests/performance/__init__.py`
`test_stability_ttft.py`	software	healthy	`tests/performance/test_stability_ttft.py`
`test_prefill_decode.py`	software	healthy	`tests/performance/test_prefill_decode.py`
`test_throughput_stats.py`	software	healthy	`tests/performance/test_throughput_stats.py`
`__init__.py`	software	healthy	`tests/conversation_drift/__init__.py`
`test_conversation_drift_runner.py`	software	healthy	`tests/conversation_drift/test_conversation_drift_runner.py`
`__init__.py`	software	healthy	`tests/rag/__init__.py`
`test_reranker.py`	software	healthy	`tests/rag/test_reranker.py`
`test_offline_online_alignment.py`	software	healthy	`tests/rag/test_offline_online_alignment.py`
`test_regression_ci.py`	software	healthy	`tests/rag/test_regression_ci.py`
`test_judge_calibration.py`	software	healthy	`tests/rag/test_judge_calibration.py`
`test_retrieval_metrics.py`	software	healthy	`tests/rag/test_retrieval_metrics.py`
`test_judge_prompts.py`	software	healthy	`tests/rag/test_judge_prompts.py`
`test_groundedness.py`	software	healthy	`tests/rag/test_groundedness.py`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`ModelConfig`	software	healthy	`common.py:40`
`InferResult`	software	healthy	`common.py:208`
`EmbedResult`	software	healthy	`common.py:582`
`RerankResult`	software	healthy	`common.py:653`
`_M`	software	healthy	`tests/test_report_schema.py:11`
`TestBaseUrl`	software	healthy	`tests/test_provider_auth.py:23`
`TestAuthHeader`	software	healthy	`tests/test_provider_auth.py:67`
`TestEffectiveModelId`	software	healthy	`tests/test_provider_auth.py:99`
`TestLoadModelsCloud`	software	healthy	`tests/test_provider_auth.py:121`
`_Stub`	software	healthy	`tests/test_capabilities.py:31`
`_Model`	software	healthy	`tests/test_common_http.py:14`
`_BadJSONResponse`	software	healthy	`tests/test_common_http.py:30`
`_Client`	software	healthy	`tests/test_common_http.py:66`
`_OkResp`	software	healthy	`tests/test_common_http.py:79`
`_FakeResponse`	software	healthy	`tests/test_provider_compat.py:67`
`_Model`	software	healthy	`tests/test_common_stream.py:15`
`_FakeClock`	software	healthy	`tests/test_common_stream.py:68`
`_FakeStreamResponse`	software	healthy	`tests/test_common_stream.py:81`
`_FakeStreamCM`	software	healthy	`tests/test_common_stream.py:90`
`_M`	software	healthy	`tests/test_run_benchmark.py:17`
`_ToyDataset`	software	healthy	`tests/llama_benchmark/test_synthetic_fallback_flag.py:33`
`_StubDiarDataset`	software	healthy	`tests/llama_benchmark/test_synthetic_fallback_flag.py:112`
`_StubDiarDatasetFallback`	software	healthy	`tests/llama_benchmark/test_synthetic_fallback_flag.py:127`
`_StubDiarDatasetBroken`	software	healthy	`tests/llama_benchmark/test_synthetic_fallback_flag.py:133`
`_StubBackend`	software	healthy	`tests/llama_benchmark/test_diarization_result.py:33`
`_FailingBackend`	software	healthy	`tests/llama_benchmark/test_diarization_result.py:40`
`_StubDataset`	software	healthy	`tests/llama_benchmark/test_diarization_result.py:45`
`_StubDatasetFallback`	software	healthy	`tests/llama_benchmark/test_diarization_result.py:68`
`_Cfg`	software	healthy	`tests/conditioned/test_runner.py:17`
`_FakeTime`	software	healthy	`tests/performance/test_stability_ttft.py:30`
`_Model`	software	healthy	`tests/performance/test_stability_ttft.py:43`
`_FakeModel`	software	healthy	`tests/performance/test_prefill_decode.py:14`
`_FakeTime`	software	healthy	`tests/performance/test_throughput_stats.py:46`
`_Model`	software	healthy	`tests/performance/test_throughput_stats.py:58`
`_DummyClient`	software	healthy	`tests/performance/test_throughput_stats.py:90`
`_Model`	software	healthy	`tests/conversation_drift/test_conversation_drift_runner.py:9`
`_FakeModel`	software	healthy	`tests/asr/test_asr.py:82`
`_FakeModel`	software	healthy	`tests/rerank/test_rerank.py:46`
`_NativeModel`	software	healthy	`tests/rerank/test_rerank.py:93`
`_FakeHTTPResponse`	software	healthy	`tests/rerank/test_rerank.py:109`
`_FakeBackend`	software	healthy	`tests/general_ability/test_runner.py:12`
`_Cfg`	software	healthy	`tests/general_ability/test_runner.py:27`
`_DS`	software	healthy	`tests/general_ability/test_runner.py:39`
`_Cfg`	software	healthy	`tests/general_ability/test_backend_adapter.py:13`
`_Model`	software	healthy	`tests/translation/test_verdict_and_perf.py:30`
`_FakeTime`	software	healthy	`tests/translation/test_verdict_and_perf.py:334`
`_M`	software	healthy	`tests/translation/test_flores_loading.py:133`
`_VLM`	software	healthy	`tests/scenarios/test_runner.py:7`
`_Judge`	software	healthy	`tests/scenarios/test_runner.py:17`
`_TextModel`	software	healthy	`tests/scenarios/test_runner.py:21`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`vllm_configs`	software	healthy	`vllm_configs`
`datasets`	software	healthy	`datasets`
`asr`	software	healthy	`datasets/asr`
`retrieval`	software	healthy	`datasets/retrieval`
`translation`	software	healthy	`datasets/translation`
`scenarios`	software	healthy	`datasets/scenarios`
`wechat_intent`	software	healthy	`datasets/scenarios/wechat_intent`
`tests`	software	healthy	`tests`
`llama_benchmark`	software	healthy	`tests/llama_benchmark`
`conditioned`	software	healthy	`tests/conditioned`
`performance`	software	healthy	`tests/performance`
`conversation_drift`	software	healthy	`tests/conversation_drift`
`rag`	software	healthy	`tests/rag`
`asr`	software	healthy	`tests/asr`
`rerank`	software	healthy	`tests/rerank`
`general_ability`	software	healthy	`tests/general_ability`
`report`	software	healthy	`tests/report`
`translation`	software	healthy	`tests/translation`
`rigor`	software	healthy	`tests/rigor`
`scenarios`	software	healthy	`tests/scenarios`
`embedding`	software	healthy	`tests/embedding`
`fixtures`	software	healthy	`fixtures`
`docs`	software	healthy	`docs`
`case-studies`	software	healthy	`docs/case-studies`
`superpowers`	software	healthy	`docs/superpowers`
`specs`	software	healthy	`docs/superpowers/specs`
`golden`	software	healthy	`golden`
`scripts`	software	healthy	`scripts`
`benchmark`	software	healthy	`benchmark`
`llama_benchmark`	software	healthy	`benchmark/llama_benchmark`
`datasets`	software	healthy	`benchmark/llama_benchmark/datasets`
`reporters`	software	healthy	`benchmark/llama_benchmark/reporters`
`benchmarks`	software	healthy	`benchmark/llama_benchmark/benchmarks`
`docling`	software	healthy	`benchmark/llama_benchmark/benchmarks/docling`
`asr`	software	healthy	`benchmark/llama_benchmark/benchmarks/asr`
`rerank`	software	healthy	`benchmark/llama_benchmark/benchmarks/rerank`
`whisper`	software	healthy	`benchmark/llama_benchmark/benchmarks/whisper`
`speaker`	software	healthy	`benchmark/llama_benchmark/benchmarks/speaker`
`ocr`	software	healthy	`benchmark/llama_benchmark/benchmarks/ocr`
`embedding`	software	healthy	`benchmark/llama_benchmark/benchmarks/embedding`
`llm`	software	healthy	`benchmark/llama_benchmark/benchmarks/llm`
`utils`	software	healthy	`benchmark/llama_benchmark/utils`
`backends`	software	healthy	`benchmark/llama_benchmark/backends`
`core`	software	healthy	`benchmark/llama_benchmark/core`
`metrics`	software	healthy	`benchmark/llama_benchmark/metrics`
`conditioned`	software	healthy	`benchmark/conditioned`
`conversation_drift`	software	healthy	`benchmark/conversation_drift`
`rag`	software	healthy	`benchmark/rag`
`labs`	software	healthy	`benchmark/rag/labs`
`case_studies`	software	healthy	`benchmark/rag/case_studies`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`port:8001`	network	healthy	`models.yaml`
`port:8002`	network	healthy	`models.yaml`
`port:9001`	network	healthy	`models.yaml`
`port:9002`	network	healthy	`models.yaml`
`port:9101`	network	healthy	`models.yaml`
`port:9102`	network	healthy	`models.yaml`
`port:9201`	network	healthy	`models.yaml`
`port:9202`	network	healthy	`models.yaml`
`port:9203`	network	healthy	`models.yaml`
`port:8080`	network	healthy	`models.yaml`
`port:11434`	network	healthy	`models.yaml`
`port:100`	network	healthy	`models.yaml`
`port:8000`	network	healthy	`models.yaml`
`port:9999`	network	healthy	`tests/test_report_schema.py`
`port:11435`	network	healthy	`tests/test_provider_auth.py`
`port:8123`	network	healthy	`tests/conditioned/test_runner.py`
`port:8081`	network	healthy	`benchmark/llama_configs/models.yaml`
`port:80`	network	healthy	`benchmark/llama_configs/models.yaml`

Label	Layer	Status	Path
`lint`	cicd	healthy	`.github/workflows/ci.yml`
`syntax`	cicd	healthy	`.github/workflows/ci.yml`
`test`	cicd	healthy	`.github/workflows/ci.yml`
`shellcheck`	cicd	healthy	`.github/workflows/ci.yml`

Label	Layer	Status	Path
`192.168.1.100`	network	healthy	`models.yaml`
`192.168.1.10`	network	healthy	`tests/test_provider_auth.py`
`192.168.100.119`	network	healthy	`benchmark/llama_benchmark/utils/remote_device.py`

Label	Layer	Status	Path
`password_literal::benchmark/llama_benchmark/utils/remote_de…`	security	healthy	`benchmark/llama_benchmark/utils/remote_device.py`
`password_literal::benchmark/llama_configs/devices/k1.yaml`	security	healthy	`benchmark/llama_configs/devices/k1.yaml`

Label	Layer	Status	Path
`repobility-clone-kb00100v`	software	healthy	`/tmp/repobility-clone-kb00100v`

Label	Layer	Status	Path
`gpu (detected)`	hardware	healthy	`README.md`

Label	Layer	Status	Path
`gha::ci`	cicd	healthy	`.github/workflows/ci.yml`

For AI agents: Voting guide (TP/FP) MCP manifest Stdio wrapper SARIF Integrate Findings queue Vote TP/FP on findings to calibrate the engine.

For AI agents + API integrations

Email me when this repo regresses

Free. We re-scan periodically; new criticals → your inbox. No signup required for the scan itself.

API access

This page is publicly accessible at: https://repobility.com/scan/b4b78901-509d-4f5f-bab3-3e287948e99c/

To check status programmatically (no auth required):

curl -s https://repobility.com/api/v1/public/scan/b4b78901-509d-4f5f-bab3-3e287948e99c/

Important — please don't re-submit the same URL repeatedly. The submission endpoint is idempotent: re-submitting the same git URL returns this same scan_token, not a new one. To re-scan this repo, sign up free and use the dashboard.

qiurui144/vlm-llm-benchmark

Complete repo analysis

function 50+

file 50+

class 50+

directory 50+

port 18

job 4

service 3

secret 2

repo 1

gpu 1

pipeline 1