Public scan — anyone with this URL can view this analysis. Sign up to track your own repos privately, run scheduled re-scans, and get AI fix prompts via your dashboard.

ZeekrBaha/eval-hotel-bot-deepeval-eval-harness

https://github.com/ZeekrBaha/eval-hotel-bot-deepeval-eval-harness · scanned 2026-06-16 01:11 UTC (2 months, 1 week ago)

24 raw signals (0 security + 24 graph)

File as issue Image

UNIFIED Repobility · multi-layer engine · AI coders

Complete repo analysis

Last scanned 2 months, 1 week ago · v1 · 17 actionable findings from 1 signal source. 7 repeated signals grouped for readability. Security checks, system graph analysis, and verified AI-agent feedback are merged into one review queue.

JSON

77.8% cov · 17 findings

Severity distribution — click a segment to filter

Active filters: excluding tests × Reset all

Severity: Critical 0 High 0 Medium 2 Low 11 9-Layer: Software Security Quality Integrity Frontend Hardware Data Network Cicd Source: Security checks 0 System graph 17 Crowd 0 Layer: Software 5 Frontend 1 Security 1 Cicd 2 Quality 8

Exclude dismissed / FP Exclude test files

All 393 nodes from the latest scan, grouped by kind. Each node is a unit the engine identified (file, function, endpoint, table…). Most users won't need this view — it's primarily for debugging the engine's graph extraction or for AI agents that want to enumerate the project structure.

Label	Layer	Status	Path
`has_key`	software	healthy	`conftest.py:16`
`sut_variant`	software	healthy	`conftest.py:20`
`__init__`	software	healthy	`judge/deepseek_judge.py:27`
`load_model`	software	healthy	`judge/deepseek_judge.py:38`
`_chat`	software	healthy	`judge/deepseek_judge.py:47`
`generate`	software	healthy	`judge/deepseek_judge.py:59`
`a_generate`	software	dead	`judge/deepseek_judge.py:76`
`get_model_name`	software	healthy	`judge/deepseek_judge.py:79`
`test_loads_all_cases`	software	healthy	`tests/test_golden_loader.py:5`
`test_case_fields_parsed`	software	healthy	`tests/test_golden_loader.py:11`
`test_kinds_are_known`	software	healthy	`tests/test_golden_loader.py:19`
`test_perfect_agreement_is_one`	software	healthy	`tests/test_stats.py:5`
`test_total_disagreement_is_negative`	software	healthy	`tests/test_stats.py:9`
`test_all_same_label_returns_one`	software	healthy	`tests/test_stats.py:15`
`test_confusion_matrix_counts`	software	healthy	`tests/test_stats.py:20`
`test_wilson_zero_n_is_degenerate`	software	healthy	`tests/test_stats.py:27`
`test_wilson_interval_brackets_point_estimate`	software	healthy	`tests/test_stats.py:31`
`test_wilson_is_clamped_to_unit_interval`	software	healthy	`tests/test_stats.py:36`
`test_wilson_small_n_is_wider_than_large_n`	software	healthy	`tests/test_stats.py:41`
`__init__`	software	healthy	`tests/test_run_suite.py:42`
`run`	software	healthy	`tests/test_run_suite.py:35`
`measure`	software	healthy	`tests/test_run_suite.py:121`
`_goldens`	software	healthy	`tests/test_run_suite.py:51`
`_patch`	software	healthy	`tests/test_run_suite.py:78`
`test_grounding_only_for_non_booking_cases`	software	healthy	`tests/test_run_suite.py:86`
`test_deterministic_metrics_run_on_every_case`	software	healthy	`tests/test_run_suite.py:94`
`test_report_shape_has_expected_keys`	software	healthy	`tests/test_run_suite.py:101`
`test_bot_exception_becomes_error_row`	software	healthy	`tests/test_run_suite.py:109`
`test_judge_error_is_tracked_separately`	software	healthy	`tests/test_run_suite.py:125`
`test_report_shape_includes_judge_error_keys`	software	healthy	`tests/test_run_suite.py:137`
`test_check_env_vars_exits_on_missing_key`	software	healthy	`tests/test_run_suite.py:144`
`test_check_env_vars_passes_when_all_present`	software	healthy	`tests/test_run_suite.py:154`
`summary`	software	healthy	`tests/test_aggregate.py:76`
`test_overall_n`	software	healthy	`tests/test_aggregate.py:85`
`test_overall_passed`	software	healthy	`tests/test_aggregate.py:89`
`test_overall_failed`	software	healthy	`tests/test_aggregate.py:93`
`test_overall_pass_rate`	software	healthy	`tests/test_aggregate.py:97`
`test_by_kind_counts_detailed`	software	healthy	`tests/test_aggregate.py:106`
`test_by_kind_cancel`	software	healthy	`tests/test_aggregate.py:115`
`test_by_kind_faq`	software	healthy	`tests/test_aggregate.py:123`
`test_by_lang_en`	software	healthy	`tests/test_aggregate.py:135`
`test_by_lang_fr`	software	healthy	`tests/test_aggregate.py:144`
`test_overall_has_ci`	software	healthy	`tests/test_aggregate.py:157`
`test_by_lang_buckets_have_ci`	software	healthy	`tests/test_aggregate.py:162`
`test_by_metric_buckets_have_ci`	software	healthy	`tests/test_aggregate.py:167`
`test_markdown_shows_ci`	software	healthy	`tests/test_aggregate.py:172`
`test_by_metric_relevance`	software	healthy	`tests/test_aggregate.py:182`
`test_by_metric_relevance_avg_score`	software	healthy	`tests/test_aggregate.py:191`
`test_by_metric_faithfulness_avg_score`	software	healthy	`tests/test_aggregate.py:197`
`test_failures_count`	software	healthy	`tests/test_aggregate.py:208`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`conftest.py`	software	healthy	`conftest.py`
`README.md`	software	healthy	`README.md`
`REPORT.md`	software	healthy	`REPORT.md`
`.pre-commit-config.yaml`	software	healthy	`.pre-commit-config.yaml`
`.env.example`	software	healthy	`.env.example`
`pyproject.toml`	software	healthy	`pyproject.toml`
`Makefile`	software	healthy	`Makefile`
`requirements.txt`	software	healthy	`requirements.txt`
`__init__.py`	software	healthy	`judge/__init__.py`
`deepseek_judge.py`	software	healthy	`judge/deepseek_judge.py`
`test_golden_loader.py`	software	healthy	`tests/test_golden_loader.py`
`test_stats.py`	software	healthy	`tests/test_stats.py`
`test_run_suite.py`	software	healthy	`tests/test_run_suite.py`
`test_aggregate.py`	software	healthy	`tests/test_aggregate.py`
`test_bot_runner.py`	software	healthy	`tests/test_bot_runner.py`
`__init__.py`	software	healthy	`tests/__init__.py`
`test_system_prompt.py`	software	healthy	`tests/test_system_prompt.py`
`test_gate.py`	software	healthy	`tests/test_gate.py`
`test_grounding_failures.py`	software	healthy	`tests/test_grounding_failures.py`
`test_slot_extraction.py`	software	healthy	`tests/test_slot_extraction.py`
`test_regression.py`	software	healthy	`tests/test_regression.py`
`test_judge_retry.py`	software	healthy	`tests/test_judge_retry.py`
`test_schema_sync.py`	software	healthy	`tests/test_schema_sync.py`
`test_language_fidelity.py`	software	healthy	`tests/test_language_fidelity.py`
`test_payment_leak.py`	software	healthy	`tests/test_payment_leak.py`
`test_cost.py`	software	healthy	`tests/test_cost.py`
`test_synthesize.py`	software	healthy	`tests/test_synthesize.py`
`test_judge_validation.py`	software	healthy	`tests/test_judge_validation.py`
`kyrgyz-language-bug.md`	software	healthy	`docs/kyrgyz-language-bug.md`
`2026-06-12-analysis-and-improvement-plan.md`	software	healthy	`docs/2026-06-12-analysis-and-improvement-plan.md`
`2026-06-05-hotel-bot-deepeval-harness.md`	software	healthy	`docs/superpowers/plans/2026-06-05-hotel-bot-deepeval-harnes…`
`2026-06-12-p0-correctness-trust-hygiene.md`	software	healthy	`docs/superpowers/plans/2026-06-12-p0-correctness-trust-hygi…`
`2026-06-07-eval-harness-recommendations.md`	software	healthy	`docs/superpowers/plans/2026-06-07-eval-harness-recommendati…`
`2026-06-07-eval-harness-recommendations-design.md`	software	healthy	`docs/superpowers/specs/2026-06-07-eval-harness-recommendati…`
`__init__.py`	software	healthy	`golden/__init__.py`
`loader.py`	software	healthy	`golden/loader.py`
`cost.py`	software	healthy	`meta/cost.py`
`gate.py`	software	healthy	`meta/gate.py`
`__init__.py`	software	healthy	`meta/__init__.py`
`aggregate.py`	software	healthy	`meta/aggregate.py`
`grounding_failures.py`	software	healthy	`meta/grounding_failures.py`
`stats.py`	software	healthy	`meta/stats.py`
`judge_validation.py`	software	healthy	`meta/judge_validation.py`
`test_safety.py`	software	healthy	`evals/test_safety.py`
`__init__.py`	software	healthy	`evals/__init__.py`
`regression_check.py`	software	healthy	`evals/regression_check.py`
`test_quality.py`	software	healthy	`evals/test_quality.py`
`run_suite.py`	software	healthy	`evals/run_suite.py`
`test_language.py`	software	healthy	`evals/test_language.py`
`test_booking.py`	software	healthy	`evals/test_booking.py`

Showing first 50 of this kind. Full payload available via the JSON button at the top of the page.

Label	Layer	Status	Path
`JudgeError`	software	healthy	`judge/deepseek_judge.py:22`
`DeepSeekJudge`	software	healthy	`judge/deepseek_judge.py:26`
`_FakeRunner`	software	healthy	`tests/test_run_suite.py:15`
`_BoomRunner`	software	healthy	`tests/test_run_suite.py:32`
`_FakeGrounding`	software	healthy	`tests/test_run_suite.py:39`
`_JudgeErrorGrounding`	software	healthy	`tests/test_run_suite.py:118`
`_FakeResp`	software	healthy	`tests/test_bot_runner.py:32`
`_Completions`	software	healthy	`tests/test_bot_runner.py:226`
`_FakeOpenAI`	software	healthy	`tests/test_bot_runner.py:50`
`_Msg`	software	healthy	`tests/test_bot_runner.py:209`
`_Choice`	software	healthy	`tests/test_bot_runner.py:215`
`_Usage`	software	healthy	`tests/test_bot_runner.py:218`
`_Resp`	software	healthy	`tests/test_bot_runner.py:222`
`_Chat`	software	healthy	`tests/test_bot_runner.py:231`
`_FakeClient`	software	healthy	`tests/test_bot_runner.py:234`
`_Schema`	software	healthy	`tests/test_judge_retry.py:9`
`Golden`	software	healthy	`golden/loader.py:12`
`PaymentLeakMetric`	software	healthy	`metrics/payment_leak.py:107`
`LanguageFidelityMetric`	software	healthy	`metrics/language_fidelity.py:109`
`SlotExtractionMetric`	software	healthy	`metrics/slot_extraction.py:17`
`BotOutput`	software	healthy	`sut/bot_runner.py:26`
`BotRunner`	software	healthy	`sut/bot_runner.py:45`

Label	Layer	Status	Path
`judge`	software	healthy	`judge`
`tests`	software	healthy	`tests`
`docs`	software	healthy	`docs`
`superpowers`	software	healthy	`docs/superpowers`
`plans`	software	healthy	`docs/superpowers/plans`
`specs`	software	healthy	`docs/superpowers/specs`
`golden`	software	healthy	`golden`
`meta`	software	healthy	`meta`
`evals`	software	healthy	`evals`
`metrics`	software	healthy	`metrics`
`reports`	software	healthy	`reports`
`.github`	software	healthy	`.github`
`workflows`	software	healthy	`.github/workflows`
`sut`	software	healthy	`sut`
`hotel_bot`	software	healthy	`sut/hotel_bot`

Label	Layer	Status	Path
`gha::test`	cicd	healthy	`.github/workflows/test.yml`
`gha::live-fixed-regression`	cicd	healthy	`.github/workflows/live-fixed-regression.yml`
`gha::live-eval`	cicd	healthy	`.github/workflows/live-eval.yml`

Label	Layer	Status	Path
`test`	cicd	healthy	`.github/workflows/test.yml`
`live-fixed-regression`	cicd	healthy	`.github/workflows/live-fixed-regression.yml`
`live-eval`	cicd	healthy	`.github/workflows/live-eval.yml`

Label	Layer	Status	Path
`DEEPSEEK_API_KEY`	cicd	healthy	`—`
`OPENAI_API_KEY`	cicd	healthy	`—`

Label	Layer	Status	Path
`repobility-clone-mhfhd6f2`	software	healthy	`/tmp/repobility-clone-mhfhd6f2`

Label	Layer	Status	Path
`postgres`	data	healthy	`README.md`

Label	Layer	Status	Path
`port:06`	network	healthy	`.github/workflows/live-eval.yml`

For AI agents: Voting guide (TP/FP) MCP manifest Stdio wrapper SARIF Integrate Findings queue Vote TP/FP on findings to calibrate the engine.

For AI agents + API integrations

Email me when this repo regresses

Free. We re-scan periodically; new criticals → your inbox. No signup required for the scan itself.

API access

This page is publicly accessible at: https://repobility.com/scan/51f92302-74d7-41b6-ba6b-3094b3def475/

To check status programmatically (no auth required):

curl -s https://repobility.com/api/v1/public/scan/51f92302-74d7-41b6-ba6b-3094b3def475/

Important — please don't re-submit the same URL repeatedly. The submission endpoint is idempotent: re-submitting the same git URL returns this same scan_token, not a new one. To re-scan this repo, sign up free and use the dashboard.

ZeekrBaha/eval-hotel-bot-deepeval-eval-harness

Complete repo analysis

function 50+

file 50+

class 22

directory 15

pipeline 3

job 3

env_var 2

repo 1

database 1

port 1