Replaced Danswer to Darwin in frontend by ninja-shreyash · Pull Request #1 · UiPath/danswer

ninja-shreyash · 2024-07-04T12:28:47Z

No description provided.

vivekjhauipath

LGTM

The chat page calls GET /manage/indexing-status on every load to derive available source types. Its cost is a per-cc-pair document-count aggregation (~300ms on the live DB at a few hundred cc-pairs) — the dominant fan-out cost after the #1/#2 fixes. The result is identical for all users and changes only when a connector is added/removed or an indexing run completes, so front it with a short-TTL global Redis cache. - Split the DB build into `_build_basic_cc_pair_info` and wrap the endpoint with an inline fail-open cache (global key `danswer:cc_pair_basic_info`, default 60s TTL). - Pure TTL, no explicit invalidation: staleness is bounded by the TTL and harmless. Any Redis error falls straight through to a direct DB build — never an outage. - Default OFF via CC_PAIR_INFO_CACHE_ENABLED; prod overlay enables it, local leaves it opt-in. Documented in k8s/README.md key table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…, retention + connector reliability (#46) * docs: add Redis caching & scaling plan Plan for exposing chat to a few hundred users: P0 connection-pool/session fix, P1 Redis foundation + DynamicConfigStore read-through cache, P2 per-user request rate limiting, P3 per-chat-turn config caches. Plan only, no implementation yet. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * P1: Redis foundation + read-through KV cache Foundation for caching/rate-limiting work (see REDIS_CACHING_PLAN.md). This commit only ships the cache piece — no behavioural change unless REDIS_KV_CACHE_ENABLED=true is set. * requirements: pin redis==5.0.8. * configs/app_configs.py: REDIS_HOST/PORT/PASSWORD/DB_NUMBER/SSL, REDIS_POOL_MAX_CONNECTIONS, REDIS_HEALTH_CHECK_INTERVAL, REDIS_SOCKET_TIMEOUT_SECONDS; cache toggle REDIS_KV_CACHE_ENABLED (default OFF) and REDIS_KV_CACHE_TTL_SECONDS (1 day). * danswer/redis/redis_pool.py: lazy ConnectionPool singleton + get_redis_client() helper. Single-tenant — DANSWER_REDIS_KEY_PREFIX is the only namespace; upstream's TenantRedisClient is intentionally not ported. * dynamic_configs/store.py: RedisCachedDynamicConfigStore wraps any inner DynamicConfigStore with read-through / write-through caching. Inner store stays the source of truth (writes inner first), encrypted values are NEVER cached plaintext (just invalidated), every Redis call is fail-open so an outage degrades latency, not availability. * dynamic_configs/factory.py: when REDIS_KV_CACHE_ENABLED, transparently wraps the existing PostgresBackedDynamicConfigStore — call sites unchanged. * Deployment: redis service in docker-compose.dev.yml (cache-only: no AOF, no RDB snapshots, allkeys-lru @ 256mb so a runaway producer can't OOM the node). darwin-kubernetes/redis-statefulset.yaml mirrors that posture. REDIS_HOST etc. in env-configmap; REDIS_PASSWORD wired via optional secretKeyRef so the deployments still boot when Redis is unauth'd. NOT the Celery broker — that stays on Postgres by design. * backend/.gitignore: ignore stray pywikibot apicache/throttle.ctrl artifacts dropped by the existing mediawiki test. Tests (unittest, no real Redis required — mocks at the get_redis_client boundary): - tests/.../redis_layer/test_redis_pool.py: pool singleton, prefix constant, reset_pool_for_tests. - tests/.../dynamic_configs/test_redis_cached_store.py: read-through, write-through invalidation, TTL on SET, cached-None vs miss, not-found NOT cached, encrypted values not mirrored, corrupt entry treated as miss, fail-open on Redis errors. 13 cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * P2: per-user request rate limiter on chat/query endpoints Layered on top of P1's Redis client. Complements the existing token- budget limiter (token_limit.py) — that's a DB-backed COST cap, this is a Redis-backed REQUEST-COUNT cap that's correct across api_server replicas. Both run; this one runs first so a 429'd caller never even touches the DB-backed usage query. Default OFF. Enable per-environment via: REQUEST_RATE_LIMIT_ENABLED=true REQUEST_RATE_LIMIT_PER_MINUTE=<N> # 0 = disable that window REQUEST_RATE_LIMIT_PER_HOUR=<N> * server/middleware/request_rate_limit.py: fixed-window buckets keyed by floor(time/window). Atomic INCR + EXPIRE NX so the bucket boundary is fixed on first increment (without NX, every request would push expiry forward and the bucket would never reset — that bug is covered by an explicit test). Authenticated users keyed by uuid; anonymous keyed by the first X-Forwarded-For hop, falling back to the socket peer; if neither yields an IP we skip (better than bucketing every anonymous request under ""). * Fail-OPEN on any Redis error: a Redis blip lets requests through with a warning, never wedges the chat path. * 429 response carries a Retry-After header with seconds-until-bucket- rollover so well-behaved clients back off precisely. * Wired as a FastAPI Depends on: POST /chat/send-message POST /direct-qa/stream-answer-with-quote Both endpoints also keep the existing check_token_rate_limits. Tests (unittest, mocked Redis pipeline — no real Redis required): - default-OFF short-circuits before any Redis call (covers both REQUEST_RATE_LIMIT_ENABLED=false AND both windows = 0). - within-limit: N requests under cap all allowed. - over-limit raises 429 with Retry-After header. - per-user isolation: distinct users have independent counters. - bucket rollover resets count (time-mocked). - EXPIRE NX semantics — locks down the no-sliding-TTL invariant. - anonymous keyed by XFF first hop; no-IP skips silently. - fail-open: Redis error doesn't propagate. 9 cases total. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Persona list cache with explicit write-through invalidation GET /persona (Manage Assistants → "View available assistants") fires get_personas(user_id, ...) — a multi-OR permission-filtered query joining Persona, Persona__User, Persona__UserGroup, User__UserGroup. At hundreds of concurrent users opening the chat UI around the same time, the burst puts unnecessary pressure on the DB connection pool (which is the actual scaling ceiling for streaming chat — see REDIS_CACHING_PLAN.md). Design: global cache + per-user filter (not per-user response cache), so the multi-user-burst pattern collapses 200 queries into ~1: danswer:personas:all:not_deleted global, all PersonaSnapshot including is_public / users / groups (PersonaSnapshot already carries the permission inputs — no separate payload shape needed) danswer:personas:groups:{user_id} per-user, list[int] of group ids At request time the cached list is filtered in Python mirroring the SQL OR-block exactly: is_public OR user.id IN persona.users.id OR (user_group_ids ∩ persona.groups) The parity vs SQL is locked down by tests (one per branch + negative). Invalidation is explicit + write-through: - 9 mutation paths in db/persona.py call invalidate_personas_all() AFTER db_session.commit() (after-commit ordering avoids stale-fill race during open transactions). - 3 paths in ee/danswer/db/user_group.py (insert/update/prepare-delete) call invalidate_user_groups(uid) for each affected user. - 24h TTL is ONLY a safety net for missed busts; primary mechanism is explicit so persona/group edits are visible immediately. - Default OFF (PERSONA_CACHE_ENABLED=false); enable per environment. - Fail-OPEN on every Redis op: a Redis outage degrades latency, not availability, and a failed bust doesn't roll back the DB write. - include_deleted=True falls through to direct DB (uncommon shape; we deliberately don't cache it). Encrypted values: N/A — PersonaSnapshot has no encryption-at-rest guarantee to bypass (unlike the KV store layer from P1). Tests (17, mocked Redis + db boundary, no real services): - 6 filter-parity cases (one per SQL branch + mixed + zero-groups edge) - 2 user-group cache cases (miss/hit, TTL propagation) - 3 routing cases (disabled fallthrough, include_deleted bypass, admin user_id=None path skips group lookup) - 4 invalidation cases (right key for each side, disabled short-circuit, Redis-error-during-bust swallowed) - 2 fail-open-on-read cases (GET error → miss, corrupt entry → miss) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Manage Assistants page UX overhaul Replaces the prior "move up / move down inside a 3-dot popover" flow on /assistants/mine with eight coordinated changes. Backend unchanged — the existing PATCH /api/user/assistant-list endpoint already accepts the full chosen_assistants array, so every interaction lands as one optimistic local update + one PATCH + (on failure) a rollback. 1. Drag-and-drop reorder via @dnd-kit (already in package.json) with a grab handle on each visible row. Pointer activation distance of 6px so clicks on the handle don't accidentally start a drag. Keyboard reordering comes for free via dnd-kit's default activator focus. 2. Explicit "set as default" — pin/star icon on each visible row; filled when the row is the user's default (position 0 of chosen_assistants), with an accent border + "Default" chip on that row. Ordering and default are now orthogonal — reorder freely without accidentally changing your default. 3. Visibility as a row-level switch instead of a buried "Hide / Remove" popover item. One unified list with a "Hidden (N)" divider; hidden rows render at reduced opacity and have no drag handle (no position to drag to). The prior separate "Active Assistants" / "Your Hidden Assistants" sections collapse into this single list. Refuses to hide the last visible row (can't ship the user a broken picker). 4. Client-side search filter — matches name, description, or tool name. Applies to both visible and hidden sections so search-then-toggle for "where did I put X" is one motion. 5. Information density rebalanced. Description is now the primary signal (was the smallest text). Tools/sources collapse into compact "{n} tools" / "{n} sources" chips so the row scans for "should I pick this?" not "what are its internals?". Full tool list reveals on hover via title attribute. 6. Bulk select column + sticky action bar. Checkbox appears on hover or focus and stays visible when selected. Action bar shows Show / Hide / Remove + Clear when anything is selected. Refuses bulk-hide that would empty the visible list. 7. Header cleanup: title + 1-line subtitle + Create button top-right, "Browse all available" as a text link. The prior two giant nav tiles + paragraph of explanatory copy are gone — recovers vertical space on a page whose real content is the list. 8. Undo on every state-mutating toast (reorder / set-default / hide / show / bulk ops). PopupSpec gains an optional `undo: { onClick }` field; the popup stays on screen 6s instead of 4s when undoable so the user has time to react. Undo restores the prior chosen_assistants array via another PATCH — symmetric round-trip, no special endpoint. New helpers in lib/assistants/updateAssistantPreferences.ts: reorderAssistantList(newOrder) — full-array PATCH (drag, undo) setDefaultAssistant(id, list) — move id to position 0 bulkRemoveFromList(ids, list) — set difference bulkAddToList(ids, list) — set union, appended at end The pre-existing moveAssistantUp/Down helpers are kept (other callers may still import them) but no longer used on this page. Verified: npx tsc --noEmit clean across web/ (0 errors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Assistant Gallery page UX overhaul Sister rework to the Manage Assistants page. With 50+ accessible assistants and growing, the old flat 2-column grid had no hierarchy and no status signal — every card looked identical regardless of whether it was yours, shared, public, or already in your picker. Same conceptual fix as Manage: give the page structure so scanning answers "what's mine?", "what's new?", "what does this one do?". Backend unchanged — every interaction PATCHes chosen_assistants via the existing /api/user/assistant-list endpoint (same path the Manage page uses). All mutations are optimistic + undoable. Changes (numbers map to the design proposal): 1. Per-card "In your picker" badge + muted card style when added. Eye now finds the un-added ones in a glance. 2. Three implicit sections: Yours / Shared with you / Featured & Built-in. Empty sections hide; section headers carry counts. 3. Filter chip rows: availability (All / Available to add / Already added) with live counts, plus auto-generated per-tool chips for tools that appear in ≥2 assistants (avoids chip-bloat as the dataset grows). Tool filters use OR semantics. 4. Owner display: best-effort name from the email local-part (split on '@', dots/underscores→spaces) with a "Built-in" badge for default_persona assistants. Kills the fork-specific "Author: Darwin" magic string. 6. Responsive grid: 1 / 2 / 3 / 4 cols by breakpoint. 7. Header matches the Manage rebuild — title + subtitle + Create button top-right, "Back to my assistants" as a text link. Cut the giant centered nav button and the explanatory paragraph. 8. Sort dropdown: Featured (API order, respects admin display_priority) / A → Z / Newly added (id desc proxy for recency). 9. Search broadened to name + description + tool names + document-set names. Empty-result state with a real "Clear all filters" button. 10. Compact "{n} tools" / "{n} sources" chips with hover-reveal of the full tool list. Flat Add/Remove buttons replace Tremor's color="green/red" which was visually shoutier than the action. 11. Design tokens fixed — border-border / focus-ring-accent in place of hardcoded gray-300 / blue-500. Consistent with the rest of the app. Skipped (per proposal): - #5 detail drawer / modal — revisit after observing how users use the new grid; bigger feature. - Bulk select — adding 5 assistants at once isn't a real use case here (bulk hide on Manage was). The pre-existing addAssistantToList / removeAssistantFromList helpers are kept and used at the call sites. The shared reorderAssistantList helper added in the prior commit is reused for the undo paths. Verified: npx tsc --noEmit clean across web/ (0 errors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add backend/scripts/seed_assistants.py for local UX testing Dev-only seed script that creates N varied personas in the local DB for exercising the redesigned gallery / manage pages. Refuses to run when POSTGRES_HOST looks like a managed/prod database (azure.com, amazonaws.com, .cloud., "prod", etc.) — guard against pointing this at the wrong env by accident. Mix is designed to populate each section of the new gallery: ~30% "Yours" — owned by target user, private ~20% "Shared with you" — owned by another user, target user in users[] ~50% "Featured" — public, no specific owner Per row randomly attaches 0–3 tools and 0–2 document sets so the {n} tools / {n} sources chips render with variety. Half of "Yours" auto- land in chosen_assistants (and all "Shared with you" do), so the "Already added" vs "Available to add" filter chips have content on both sides without manual setup. 60 distinct names + 30 description templates so 50 rows feel populated and varied. Uses a fixed RNG seed by default (deterministic across runs). Name prefix "[seed] " makes rows easy to spot and to wipe via --clear. Usage: cd backend && source ../.venv/bin/activate python -m scripts.seed_assistants --email you@example.com python -m scripts.seed_assistants --clear # wipe and re-seed Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Assistants UX polish: toggle highlight + gallery declutter Two follow-ups on the assistants UX rework, both from user feedback. Manage (/assistants/mine): * Toggle now accepts a `highlight` prop that draws a transient ring + slight scale-up on the switch. Used by hidden rows so a click anywhere on the (faded) row body flashes the toggle for ~1.2s, pointing the eye at the action that brings the assistant back. Doesn't auto-enable on body click — surprising a user mid-read into enabling would be a worse outcome than the discoverability gap. * Restructured opacity: only the content column (icon + name + description + chips) fades when a row is hidden. The action zone (checkbox, drag-slot, pin, toggle, share, edit) stays at full opacity so the toggle is the bright, clickable target on a dim row. Previously the parent opacity-50 cascaded to every child, making the toggle the dimmest thing on the dimmest row. * stopPropagation on the action zone so clicks on buttons inside it don't trigger the row-body flash handler. Gallery (/assistants/gallery): * Removed all tool-related UI per user request — the page is for browsing assistants, and tool filter chips + per-card "{n} tools" pulled focus from the assistant itself. Gone: the auto-generated tool filter chip row, the per-card tools chip, the toolDisplayName / toolIcon helpers, the FiTool / FiImage / FiCheck imports, and the toolFilters state + commonTools memo. Search hay is now name + description + document-set names (no tool names). * Dropped the absolute top-right "In your picker" badge. The muted card style (border + opacity-75) plus the "Remove" button in the footer already signal "added"; the badge ate horizontal space (pr-24 on the header reserved 96px) and crowded the title at narrower widths. Removed the pr-24 reservation now that nothing overlays the header. * Grid capped at `1 / 2 / 3` cols — 1 on mobile, 2 on most laptops and standard desktops, 3 only at `2xl` (≥1536px). Previously 1/2/3/4 with the 4-col breakpoint making cards cramped and hard to read once descriptions hit their 3-line clamp. * Bumped card padding p-4 → p-5 and description line-height to leading-relaxed for breathing room. * Updated clearAllFilters / hasAnyFilter to drop the toolFilters references (now dead). Verified: npx tsc --noEmit clean across web/ (0 errors), zero stray references to the removed helpers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Remove tools chip from Manage Assistants page Mirrors the gallery treatment from the previous commit. The user reported tool execution isn't reliable yet, and surfacing "{n} tools" on assistant rows misleads users into picking an assistant for a capability that may not work in practice. Dropped: the {n} tools Bubble in the row's chip block, the toolCount derivation, and the FiTool import. The {n} sources chip stays — it's about the assistant's knowledge scope, which works fine. Verified: npx tsc --noEmit clean across web/ (0 errors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Parameterize gallery grid column count (default 3) AssistantsGallery now accepts an optional `columns` prop (default 3, supported 1-5). Responsive scaling below the widest breakpoint is fixed per row of GRID_CLASSES — each row is a complete static Tailwind class string so the purge step actually emits the classes (dynamic `md:grid-cols-${n}` would silently disappear at build). Unsupported values fall back to the default rather than rendering broken — a bad prop here shouldn't break the page. The single existing caller (page.tsx) doesn't pass columns, so it gets the default 3 — same layout as before. To switch to 4 columns on a wide-monitor deployment: `<AssistantsGallery columns={4} ... />`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Show document-set names on assistant cards (was: count only) A "{n} sources" chip told users how MUCH knowledge an assistant had access to but not WHICH knowledge — defeating the point of the chip for someone deciding "which assistant should I pick for this task?". Both the Manage page row and the Gallery card now render one Bubble per document-set name, capped at MAX_VISIBLE_DOC_SETS (3). When an assistant points at more than that, a "+N more" pill collects the overflow with the rest of the names exposed via the title tooltip, so we don't blow the card width or row layout at narrower column counts. Each name chip caps at a max-width with truncate + a hover title, so a single absurdly long document-set name can't push the actions off the row. Verified: npx tsc --noEmit clean across web/ (0 errors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Gallery: user-controllable column count (segmented control, persists) Adds a small "Columns 2 | 3 | 4" segmented control at the right end of the filter row (next to Sort). The pick persists to localStorage under "danswer:assistants-gallery:columns" so it survives reloads on the same device. State precedence: user choice (localStorage) wins ↓ prop `columns` from caller (default for new users / new device) ↓ DEFAULT_COLUMNS = 3 (final fallback) The localStorage read happens in a useEffect so SSR + first paint use the prop value — avoids a hydration mismatch the time the stored value disagrees with the prop. localStorage writes are wrapped in try/catch because some sandboxed contexts (private modes, restrictive iframes) throw on access — the control still works for the session, just doesn't persist there. Picker is hidden below md (768px) because the layout falls back to 1 column at that width regardless of the chosen value. Exposed options are 2/3/4 — 1 is mobile-only via responsive, 5 is too cramped for typical screens (GRID_CLASSES still supports 5 if a deployment wants to set it via prop). Verified: npx tsc --noEmit clean across web/ (0 errors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Gallery: column picker as dropdown to match Sort Reverts the segmented "2 | 3 | 4" button group to a single <select> that mirrors the existing Sort dropdown for visual consistency on the "view controls" cluster at the right end of the filter row. Behavior unchanged: pure client-side state + localStorage persistence, no fetch and no router.refresh() in the column path — the user's column choice never triggers a backend call. Verified: npx tsc --noEmit clean across web/ (0 errors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Scale indexing via remote Dask scheduler topology Replace the monolithic supervisord background pod with separate deployments for celery-worker, celery-beat, indexer-scheduler, dask-scheduler, and dask-worker. The indexer-scheduler now reads DASK_SCHEDULER_ADDRESS to dispatch run_indexing_entrypoint to a remote Dask cluster instead of an in-process LocalCluster, so indexing throughput scales horizontally with dask-worker replicas instead of being capped by one pod's RAM. Local dev keeps the LocalCluster path (no env var); a new scripts/dev_run_dask_distributed.py and docker-compose overlay reproduce the prod-shape topology without K8s. scripts/test_dask_distributed_e2e.py exercises the topology (parallelism, worker death, scheduler death) end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: add MIGRATION.md covering Redis / bg-scaling / UX Single migration doc covering all three slices on this branch: 1. Background indexing scaling (Dask topology) 2. Redis caching + rate limiting 3. Assistants UX rework Organized for an operator: TL;DR up top ("everything default OFF"), new deps/env/secrets summarized, deployment order, verification checklist BEFORE flipping any flags, per-feature enable steps, and the known footguns (k8s manifests missing REDIS_PASSWORD env wire-up in the bg-scaling path, seed script bypassing persona cache, CLAUDE.md update.py gate). Plus the recommended manual test list and the branch's commit map. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * darwin-kubernetes: port split-background manifests + lock convention in AGENTS.md The bg-scaling commit (03d1649f) added 5 new k8s manifests under `deployment/kubernetes/` that split the combined background pod into beat / celery / indexer-scheduler / dask-scheduler / dask-worker. But Darwin doesn't apply from `deployment/kubernetes/` — its prod manifests live under `darwin-kubernetes/`, and the two trees aren't kept in sync. Porting all five into `darwin-kubernetes/` with Darwin conventions: - Image registry sfbrdevhelmweacr.azurecr.io/danswer/danswer-backend - configMapRef env-configmap, secretKeyRef danswer-secrets - POSTGRES_USER / POSTGRES_PASSWORD wired everywhere that talks to PG - REDIS_PASSWORD wired as optional secretKeyRef (the latent footgun flagged in MIGRATION.md §10a is now closed for the Darwin path) - indexcpu nodeAffinity + darwin/indexing toleration on every indexing-side pod (celery, indexer-scheduler, dask-scheduler, dask-worker); beat stays on the default pool (lightweight) - dynamic-pvc + file-connector-pvc volume mounts where any task may stage files The existing `darwin-kubernetes/background-deployment.yaml` (combined beat+celery+indexer via supervisord) is intentionally LEFT IN PLACE — the split is an opt-in rollout, not a forced cutover. To switch: apply the new five, verify the new pods are healthy, scale the combined deployment to 0. Also lock the convention in AGENTS.md so this doesn't recur: - New divergence-table row noting darwin-kubernetes/ is source of truth for prod. - New "Critical facts that bite" §9 documenting the two-tree split, when to touch which, and the per-pod adaptation checklist (image registry, configmap, secrets, REDIS_PASSWORD, affinity, PVCs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(MIGRATION.md): reflect the darwin-kubernetes port §5b — Dask topology section now points at the actual ported darwin-kubernetes/*.yaml manifests with a concrete cutover script, not just "you'll need to port these later" boilerplate. §10a — Footgun is RESOLVED for the Darwin path (the 5 new Darwin manifests all wire REDIS_PASSWORD via optional secretKeyRef). Marks the entry as such rather than removing it, so the history of "why was this previously a concern" stays readable. §12 — Commit count, file count, and totals updated for the two new commits (MIGRATION.md itself + the darwin-kubernetes port). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(MIGRATION.md): update base reference after add-claude merge rajiv/add-claude was merged to feature/darwin upstream, so the doc's "on top of rajiv/add-claude (PR #45)" reference is stale. The branch is now rebased onto origin/feature/darwin directly — same diff, just a fresher base. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Introduce k8s/ — kustomize-based manifests replacing darwin-kubernetes/ Single source of truth for both production (Darwin AKS) and local dev, with image tags + env values + secrets externalised so a deploy is "edit one file, kubectl apply -k". No Helm. Replaces the flat darwin-kubernetes/ tree (which the operator will delete once they've verified the new structure against the live cluster). Layout: k8s/base/ Cleaned env-neutral manifests (one file per logical service; deployment+service merged where natural). Image refs use logical names (e.g. `danswer-backend`) which overlays rewrite to concrete registry+tag. k8s/overlays/prod/ Darwin AKS production: kustomization.yaml → images, replicas env.properties → non-secret config secrets.env.example → template (committed) secrets.env → real values (gitignored) k8s/overlays/local/ Same shape, local-dev defaults (host.docker.internal, latest tags, AUTH_TYPE=disabled, smaller replicas). k8s/optional/ Opt-in deployments not part of base: redis.yaml background-{beat,celery,indexer-scheduler}.yaml dask-{scheduler,worker}.yaml Apply with `kubectl apply -f <file>` when rolling out the corresponding feature. k8s/README.md Layout explanation + common workflows (image bump, env change, secret rotation, Redis rollout, migrating off darwin-kubernetes/). Built from the live-cluster dump in darwin-kubernetes/temp/ (gitignored, never committed). The cleaner script (intentionally not committed) strips status, uid, resourceVersion, generation, creationTimestamp, managedFields, last-applied-configuration annotation, restartedAt, progressDeadlineSeconds, revisionHistoryLimit, and the auto-assigned clusterIP/ipFamilies/sessionAffinity on Services. Image references in base/ are normalised to logical names so kustomize can rewrite them. SECURITY: the live env-configmap was discovered to contain real plaintext secrets — Slack tokens, GEN_AI client secret, Jira token, Opsgenie key. The new structure moves all of those to k8s/overlays/*/secrets.env (gitignored) which renders into a kustomize-generated Secret. api-server and background deployments gain `envFrom: secretRef: danswer-secrets` so the moved values continue to reach the app as env vars. Rotation of the leaked credentials is a separate operator task — every "REPLACE_ME" in secrets.env.example marked LEAKED is one of them. Validation: kubectl kustomize k8s/overlays/prod → 26 resources, clean render kubectl kustomize k8s/overlays/local → 26 resources, clean render Image substitution verified in both. .gitignore additions: darwin-kubernetes/temp/ Live cluster dumps k8s/overlays/*/secrets.env Real secret values per environment k8s/overlays/*/*.secrets.env Defensive (any *.secrets.env variant) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: opt-in Redis via kustomize component (local includes, prod doesn't) In-cluster Redis is now an opt-in kustomize Component at k8s/optional/redis/, included by the local overlay via `components: [../../optional/redis]` and NOT by prod (which uses managed Redis instead). Why Component instead of `resources: ../../optional/redis.yaml`: kustomize's load restrictor rejects file-resource refs that escape the overlay's directory tree. Components are explicitly designed for opt-in cross-tree refs and pass the security check; they also let us add patches later that only apply when the component is opted in. Layout change: before: k8s/optional/redis.yaml after: k8s/optional/redis/ kustomization.yaml (kind: Component) redis.yaml The plain `kubectl apply -f k8s/optional/redis/redis.yaml` or `kubectl apply -k k8s/optional/redis` workflows still work — the file just moved one level deeper. env.properties updates: local: REDIS_HOST=redis (the in-cluster Service name, matching the component's deployment) prod: REDIS_HOST=<your-managed-redis>.redis.cache.windows.net (placeholder for Azure Cache for Redis; rename + drop the access key into secrets.env as `redis_password` when you adopt managed Redis) Validated: kubectl kustomize k8s/overlays/prod → 26 resources (no Redis) kubectl kustomize k8s/overlays/local → 28 resources (+Service +StatefulSet) README updated with the components pattern and how to add more opt-in features the same way (split-background, Dask, etc.). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: prod now uses in-cluster Redis (was: managed Redis); bump memory to 512MB Reversal of the earlier "prod will use managed Redis" decision. Prod overlay now opts into the same in-cluster Redis component as local: k8s/overlays/prod/kustomization.yaml — adds: components: - ../../optional/redis k8s/overlays/prod/env.properties — REDIS_HOST back to `redis` (the in-cluster Service name) Redis StatefulSet bumped from 256MB to 512MB: --maxmemory 256mb → 512mb resources.requests.memory 128Mi → 256Mi resources.limits.memory 384Mi → 1Gi Limit set to ~2x maxmemory rather than 1.5x because the single-replica StatefulSet has no failover — OOM = cache outage. Redis uses extra RSS beyond --maxmemory for client output buffers, COW pages during BGSAVE (if we ever turn on persistence), and fragmentation; safer to over- provision the cgroup limit and let `maxmemory-policy: allkeys-lru` do its job inside Redis's own accounting. Validated: kubectl kustomize k8s/overlays/prod → 28 resources (now includes Redis) kubectl kustomize k8s/overlays/local → 28 resources (unchanged) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: move redis from optional component to base Both prod and local overlays opted into the in-cluster Redis component, so it's no longer optional — promoted to base/redis.yaml and added to base/kustomization.yaml. Removed the now-redundant `components:` blocks from both overlays and the optional/redis/ component dir. Net effect is identical (prod + local still render 28 resources each, both including Redis) — just less indirection now that Redis is universal rather than opt-in. README updated: optional/ table drops the redis row with a note that it moved to base; the components: "flag" explanation now points at the split-background deployments as the example opt-in. Validated: kubectl kustomize k8s/overlays/prod → 28 resources (redis in base) kubectl kustomize k8s/overlays/local → 28 resources (redis in base) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(k8s/README): drop migration narrative + darwin-kubernetes references The README is the doc for the k8s/ layout as it stands, not a record of how it came to be. Removed: - "Replaces the older darwin-kubernetes/ tree" subtitle - the whole "Migration plan (deleting darwin-kubernetes/)" section - the "darwin-kubernetes/ is being retired" + temp/ convention bullets Also fixed two bits left stale by moving Redis into base: - structure diagram listed Redis under optional/ → now correctly omits it (it's in base) - "Roll out Redis" workflow told you to `kubectl apply -f k8s/optional/redis.yaml` → rewritten as "Redis ships in base; flip the env flags to enable the cache/rate-limiter features" Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(k8s/README): add instructions for applying optional/ manifests optional/ holds plain manifests (no kustomization), so they need `kubectl apply -f` and aren't picked up by `apply -k overlays/...`. Added a workflow covering: - single-file and whole-folder apply - the dependency on the overlay being applied first (optional pods reference the overlay-generated env-configmap / danswer-secrets) - the full split-background + Dask cutover in dependency order (scheduler/workers → split bg pods → scale down combined), plus rollback and the dual-beat warning Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: make optional manifests parameterized like base (component + logical images) The optional/ manifests hardcoded the image tag (sfbrdevhelmweacr.azurecr.io/danswer/danswer-backend:vha-138) while base uses the logical name `danswer-backend` that the overlay's images: block rewrites. That inconsistency meant a tag bump had to be made in two places and the optional pods could drift from the rest of the cluster. Fix: grouped the five split-background + Dask manifests into a single kustomize Component at k8s/optional/background-scaling/, and changed their image refs to the logical `danswer-backend`. When an overlay opts in via `components: [../../optional/background-scaling]`, the overlay's existing `images:` entry for danswer-backend parameterizes them — same tag as api-server / background, set in one place. Verified: temporarily opting the component into the prod overlay renders all five bg-scaling pods with sfbrdevhelmweacr.azurecr.io/danswer/ danswer-backend:vha-138 (34 resources total), then reverted. Neither overlay opts in by default (prod/local still 28 resources each). Layout change: before: k8s/optional/{background-beat,background-celery, background-indexer-scheduler,dask-scheduler,dask-worker}.yaml (plain manifests, hardcoded image, applied via kubectl apply -f) after: k8s/optional/background-scaling/ kustomization.yaml (kind: Component) <same five manifests, logical image name> (opted into via the overlay's components: block) README updated: optional/ is now described as opt-in components with logical-image parameterization; the apply workflow switched from `kubectl apply -f` to the components: + replicas:0 overlay edits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: fully parameterize background-scaling component (replicas, env, env-neutral) Follow-up to the component conversion — three remaining inconsistencies vs base that were pointed out: 1. Replicas were hardcoded per-manifest. Removed them from the manifests and moved the counts into the component's kustomization.yaml `replicas:` block (one place; mirrors how the overlay parameterizes base replicas). dask-worker=3 is the indexing-throughput knob. 2. Secret/config loading differed: the component had an extra explicit REDIS_PASSWORD secretKeyRef that base doesn't. Dropped it so every pod's env block is byte-identical to base/background.yaml — explicit POSTGRES_USER/POSTGRES_PASSWORD via secretKeyRef + envFrom [configMapRef env-configmap, secretRef danswer-secrets]. (redis_password still reaches the app via the envFrom secretRef like every other secrets.env key; the explicit entry was redundant and base-divergent.) 3. Manifests carried Darwin-specific node affinity + darwin/indexing tolerations, which base does NOT (base is env-neutral; the live cluster runs without pool affinity). Stripped them so the component is environment-neutral and won't fail to schedule on a local cluster that lacks the indexcpu pool. The prod overlay re-adds indexcpu affinity + toleration via a patch when it opts in — documented in the README opt-in steps with a ready-to-paste patch block. Verified end-to-end: opting the component into prod renders 34 resources, all five bg-scaling pods get sfbrdevhelmweacr.azurecr.io/danswer/ danswer-backend:vha-138, replicas come from the component kustomization (beat=1, celery=2, worker=3), background-deployment scaled to 0. Default (not opted in): prod/local both 28 resources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(k8s/README): explicit apply/preview/verify commands for background-scaling The apply command was present but buried under the overlay-edit YAML block and read as the generic overlay apply. Made the deploy commands explicit and labeled: preview the rendered bg/dask pods, kubectl diff vs live, apply, and rollout-status watches. Also stated plainly why there's no standalone `kubectl apply -f` for the component (logical image name only resolves through the overlay's images: block). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: right-size background-scaling for a few-hundred-user deployment dask-worker 3 → 2, background-celery 2 → 1. The originals were inherited defaults from the cherry-picked feature/backgroundscaling commit, not sized to Darwin's load. - dask-worker=2: each pod runs one connector at a time (--nworkers=1 --nthreads=1), so this caps concurrent indexing at 2. Enough unless many connectors backlog in NOT_STARTED; raise then. Halves the worst-case indexcpu footprint (was 3×4Gi, now 2×4Gi). - background-celery=1: Celery here only runs maintenance tasks (prune, sync, deletion, analytics rollup) — NOT indexing. One pod already autoscales 3-10 threads (--autoscale=3,10), which easily covers the bursty maintenance queue at this scale. The 2nd replica was redundancy we don't need. Added inline comments noting which counts are singletons that must stay at 1 (beat, indexer-scheduler, dask-scheduler) vs the throughput knobs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: add slack-listener to background-scaling component The combined `background` supervisord pod ran 5 programs; the split component covered 4 (indexer-scheduler, celery, beat — via dask too) but NOT slack_bot_listener. Migrating to the split topology + scaling the combined pod to 0 would have killed the Slack bot, which Darwin uses. Adds slack-listener-deployment running `python danswer/danswerbot/slack/listener.py`, modeled on the celery manifest: logical danswer-backend image, env-neutral, env-configmap + danswer-secrets (the DANSWER_BOT_SLACK_* tokens arrive via the envFrom secretRef). SINGLETON (count: 1 in the component kustomization) — the listener holds a Slack Socket Mode websocket; a second replica would double-process every event. Added to the prod affinity patch's labelSelector in the README so it lands on the indexcpu pool with the other app pods. Validated: opting the component into prod now renders 35 resources (was 34), slack-listener gets the prod image tag + replicas=1; default (not opted in) unchanged at 28. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: consolidate background-scaling 6 deployments → 4 The split topology had three separate deployments for low-traffic singletons (background-celery, background-beat, slack-listener) that don't scale with indexing load — only dask-worker does. Collapsed them into one `background-lite` deployment running the three as separate containers in a single pod. This trims pod count + per-deployment resource reservations while keeping each container independently restartable. Now four deployments (was six): - background-lite celery-worker + celery-beat + slack-listener (3 containers, 1 pod, replicas:1 — contains beat + the Slack websocket, both singletons) - background-indexer-scheduler update.py polling loop (singleton) - dask-scheduler Dask scheduler Service + Deployment - dask-worker indexing executors (the actual scaling knob) Chose a multi-container pod over a supervisord-with-custom-conf approach: no ConfigMap to mount, no risk of the custom conf drifting from the image's baked-in supervisord.conf, and each container runs the exact command its former standalone deployment used. strategy: Recreate on the pod so celery-beat never overlaps during a rollout (dup beats double-fire). Validated: opting into prod renders 33 resources (was 35), background-lite shows containers [celery-worker, celery-beat, slack-listener] at replicas 1, dask-worker at 2, base background scaled to 0. Default (not opted in) unchanged at 28. README updated: component table, affinity-patch labelSelector (background-lite replaces the three), rollout-status command, and the dual-beat warning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: disable MULTILINGUAL_QUERY_EXPANSION in prod overlay The query-expansion path (secondary_llm_flows/query_expansion.py) builds its fast_llm with a HARDCODED 5s timeout. With no distinct fast model (FAST_GEN_AI_MODEL_VERSION empty), it routes to full gpt-4o via the UiPath gateway, which routinely takes >5s — causing repeated ReadTimeouts on Slack queries (observed in prod logs). The committed darwin-kubernetes/env-configmap.yaml already had this empty; the LIVE cluster had drifted to "English,Japanese" (set out-of-band, never committed), which is what triggered the timeouts. Setting it empty in the new prod overlay keeps the go-forward source of truth correct. App reads `os.environ.get(...) or None`, so empty = feature off. To re-enable later: wire FAST_GEN_AI_MODEL_VERSION to a genuinely fast model (gpt-4o-mini / gpt-4.1-mini) so the 5s budget is realistic, or make that timeout env-configurable first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: pin Vespa to 8.600.35 (was :latest → caused prod outage) INCIDENT: applying the kustomize overlay rendered vespaengine/vespa:latest against a cluster running 8.600.35. The bare→:latest image change rolled all Vespa StatefulSets onto 8.696.20 — a >30-release jump. Vespa's config server refuses an auto-upgrade that large (incompatible-upgrade guard, VersionState.verifyVersionIntervalForUpgrade) and crash-looped on ConfigServerBootstrap, taking the whole cluster down (config tier → no quorum → cluster-wide connection-refused 503s on every search + the api-server's ensure_indices_exist). FIX: pin both overlays to 8.600.35 — the version the content nodes' on-disk index is written in, so there is no upgrade and the version check passes. Recovery performed on the live cluster: set all 5 vespa StatefulSets back to 8.600.35, cleared the (now-irrelevant) wedged config-server ZK state, restarted. Content data on the 100Gi content PVCs was never touched. NEVER use :latest for Vespa. Upgrades must be STEPWISE (≤30 releases per hop) and done as a deliberate, ordered operation — not a bare tag bump. busybox also pinned (1.36.1) for the same drift hygiene. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: add readiness probes to Service-backed Vespa nodes Vespa nodes had NO probes, so a k8s Service added a pod to its endpoints the instant the container started — before Vespa was actually serving. That's why every Vespa restart re-opened a window of "upstream connect error / connection refused" 503s (the incident). Adds readinessProbe (httpGet /state/v1/health) to the three nodes that sit behind a query/deploy Service: - vespa-configserver (:19071) — gates the deploy + inter-node config path - vespa-query (:8080) — gates the search path the app hits - vespa-feed (:8080) — gates the feed/index path Deliberately NOT added to vespa-content / vespa-admin: they aren't behind a query-serving k8s Service (content is reached internally via Vespa's own distributor, admin runs cluster control), and a mis-tuned probe there could pull a healthy node from rotation and make things worse. Readiness ONLY — no liveness probe anywhere. An aggressive liveness check could kill a slow-but-healthy Vespa node mid-bootstrap and cause a restart loop, which is the failure mode we just spent the incident fighting. Generous initialDelay (45s configserver / 30s others) + failureThreshold 6 so normal slow startup doesn't flap nodes out of rotation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(AGENTS.md): add Critical fact §10 — never :latest for Vespa, pin the version Captures the prod-outage learning in the repo's shared operating notes (the section CLAUDE.md routes every agent to read first). Covers: why a big Vespa version jump takes the cluster down (config-server refuses >30-release auto-upgrade), the rule (pin to the running version, 8.600.35; upgrades stepwise; don't force SKIP_UPGRADE_CHECK on prod), and the recovery runbook (re-pin image, restart config servers, redeploy schema; ZK clear only if genuinely corrupt; readiness-only probes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: add guarded-apply.sh — block unsafe Vespa version jumps at apply time Turns the "never jump Vespa >30 minor releases" rule into an enforced guardrail instead of a thing to remember. guarded-apply.sh wraps kubectl apply -k and, before applying: - reads the LIVE running Vespa version (kubectl) + the version the overlay would deploy (kubectl kustomize) - REFUSES a >30-minor upgrade (Vespa's auto-upgrade limit — the thing that caused the outage) - REFUSES a major-version change (needs dedicated migration) - REFUSES a floating/unparseable tag (:latest) - WARNS + requires FORCE=1 on a large downgrade (legit only when recovering to the on-disk version — which is why downgrade isn't a hard block; our recovery was a 96-minor downgrade) - otherwise runs kubectl diff, then apply Checks against LIVE, not the repo's previous pin — config drifts out of git (we saw it), so the running version is the only truth that matters at apply time. FORCE=1 overrides with an explicit "I accept the risk". Wired into k8s/README.md (Quick start now uses guarded-apply.sh; new "Bump the Vespa version" + "Vespa version guard" sections) and referenced from AGENTS.md §10. Verified: parses live=8.600.35 vs overlay=8.600.35 (gap 0 → OK); bash -n clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: add KEDA indexing-autoscale component (opt-in) for dask-worker Autoscales dask-worker-deployment on real indexing demand instead of a fixed replica count, via a KEDA PostgreSQL scaler. Metric (validated against the live DB): the number of index attempts that can run CONCURRENTLY right now, respecting INDEXING_PER_SOURCE_CAP — SUM over source of LEAST(cap, pending_count_for_source) not a raw pending count (10 same-source attempts still run 1 at a time under cap=1, so we must not spin up 10 workers). Counting IN_PROGRESS in the metric keeps replicas >= running jobs, so KEDA never scales a busy worker away; scale-to-0 only when there's genuinely no work. Grounded in code + live DB, not guesses: - remote-Dask mode does NOT cap dispatch by NUM_INDEXING_WORKERS, so adding worker pods truly adds parallelism (the in-process LocalCluster path is the only one bounded by that env) - PER_SOURCE_CAP (default 1) is the real concurrency ceiling - index_attempt links directly to connector.connector_id (this fork), not connector_credential_pair_id - status is stored UPPERCASE (Enum native_enum=False) — confirmed live: NOT_STARTED / IN_PROGRESS / SUCCESS / FAILED Shipped as opt-in component k8s/optional/keda-indexing-autoscale/ (ScaledObject + TriggerAuthentication; password from danswer-secrets, no duplication). minReplica 0 (scale to zero when idle), maxReplica 4, cooldown 300s. README documents prerequisites (KEDA operator install; opt in after background-scaling; REMOVE the static dask-worker replicas entry so it doesn't fight the HPA), the scale-down safety reasoning, and a recommended dask-worker graceful-shutdown companion change. Validated: component renders standalone (2 resources) and opted into prod (namespaced to darwin). Not enabled by default. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: add KEDA operator install manifest (optional/keda, own namespace) Companion to the keda-indexing-autoscale component: the KEDA operator itself. No Helm — a standalone kustomization referencing KEDA's official release bundle PINNED to v2.14.0 (GitHub release assets are immutable, so the URL is a content pin; never a moving ref — same lesson as Vespa :latest, AGENTS.md §10). It's cluster-scoped infra (CRDs + RBAC + operator), installed ONCE per cluster independent of the danswer overlays — hence kind: Kustomization (apply on its own), NOT a Component layered into prod/local. KEDA's bundle creates and installs into its own `keda` namespace internally, so no `namespace:` override here (that would wrongly re-namespace the cluster-scoped CRDs). Install: kubectl apply --server-side -k k8s/optional/keda (--server-side required — KEDA CRDs exceed the client-side last-applied-configuration annotation limit) README updated: optional/ table now distinguishes Components (opt into an overlay) from Standalone installs (apply on their own), and the KEDA autoscale prereq points at `kubectl apply -k k8s/optional/keda`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(k8s/README): document mandatory pod restarts after a config change Fixes a misleading instruction and adds the missing operational step the user hit: after `kubectl apply -k`, ConfigMap changes do NOT auto-roll pods, because disableNameSuffixHash:true keeps the name stable (the hash-suffix is exactly what would trigger a rollout). envFrom reads env only at pod start, so running pods keep stale values until restarted. Changes: - "Add a new env var": corrected step that implied auto-pickup; now states you must restart consumers. - "Enable the Redis cache + per-user rate limiter": adds the explicit `kubectl rollout restart deploy/api-server-deployment deploy/background-deployment` after apply, clarifies the rate limit is PER-USER, and includes PERSONA_CACHE_ENABLED. - New "Which workloads to restart after a config change" table mapping changed vars → workloads (Redis flags → api-server + background; model vars → model servers; etc.), plus the split-background variant (background-lite / indexer-scheduler / dask-worker, no background-deployment). - disableNameSuffixHash footgun now spells out the manual-restart consequence. Also commits the prod env.properties with the Redis features enabled (REDIS_KV_CACHE_ENABLED / REQUEST_RATE_LIMIT_ENABLED 20-per-min,300-per-hr / PERSONA_CACHE_ENABLED) — the user turned these on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * k8s: add startup + readiness probes to api-server (own /health, not deps) api-server had no probes. Adds: - startupProbe on /health:8080 — the container runs `alembic upgrade heads` before uvicorn, so /health isn't up until migrations finish; this allows ~5min (30×10s) for migrations+boot before readiness/liveness count. Transitively gates on Postgres (no migrations → never Ready). - readinessProbe on /health:8080 — gates the api-server Service so it doesn't route to a still-booting pod. Deliberately: - checks the app's OWN /health, NOT Vespa/Redis. Those are partial/optional deps (Vespa retried-then-proceeds, Redis fail-open); coupling API availability to them would turn a partial outage into a total one — the Vespa incident is the proof (api-server stayed up serving auth/settings while search was down). - NO liveness probe — an aggressive liveness on a slow-migrating api-server could kill it mid-migration (same lesson as the Vespa probes). /health is in the auth_check public-endpoint allowlist, so the probe isn't 401'd. Postgres remains gated by the alembic step in the start command; Redis/Vespa intentionally NOT gated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(k8s/README): add "Verify Redis caching is working" runbook After enabling the Redis flags, document how to confirm the cache is actually populated + hit (vs silently failing open to Postgres): - key-namespace table (kv / personas / groups / ratelimit) - --scan for presence (the "is it on" check) - INFO stats keyspace hit/miss ratio - TTL/STRLEN on a specific entry - MONITOR to watch a live request hit the cache - DEL + reload to prove the read-through refill - rename-an-assistant to prove write-through invalidation (TTL -> -2) Plus the gotcha: the cache is silent on success (only logs on Redis error), so api-server logs won't show hits — Redis-side inspection is the only way to observe it; a "Redis GET/SET failed" warning means it's failing open to Postgres. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf: fix N+1 in basic indexing-status (chat page load + folder create) get_basic_connector_indexing_status reads cc_pair.connector.source for every cc-pair, but get_connector_credential_pairs didn't eager-load the connector relationship → one lazy query per cc-pair. At ~404 cc-pairs against a remote Azure Postgres that's 404 sequential round-trips, which dominated chat page load — and re-ran on every folder create (the chat page's router.refresh() re-fetches the whole fetchChatData bundle, and this is the slowest endpoint in it). Fix: add eager_load_connector to get_connector_credential_pairs (opt-in, joinedload on ConnectorCredentialPair.connector) and use it in the basic indexing-status endpoint. 405 queries -> 2. No API/contract change, no frontend change; speeds every chat page load, not just folder create. Verified the doc-count GROUP BY itself was already fast (7ms over 689k rows on the live DB) — the cost was the N+1, not the aggregation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(chat): optimistic folder create — no full refetch Creating a folder called router.refresh(), which re-runs the entire fetchChatData server bundle (chat sessions + doc sets + assistants + tags + llm providers + indexing-status + folders, uncached via noStore) just to show one new empty folder in the sidebar. The create POST itself is a single fast INSERT. Now: mirror the server `folders` prop into local state (re-synced when the prop changes) and, on create, append the returned folder to that state instead of refreshing. The folder appears as soon as the INSERT returns — no fan-out, no SSR re-render. Paired with the indexing-status N+1 fix, this removes both the trigger (the refetch) and the worst cost within it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(chat): Redis-cache the connector indexing-status read The chat page calls GET /manage/indexing-status on every load to derive available source types. Its cost is a per-cc-pair document-count aggregation (~300ms on the live DB at a few hundred cc-pairs) — the dominant fan-out cost after the #1/#2 fixes. The result is identical for all users and changes only when a connector is added/removed or an indexing run completes, so front it with a short-TTL global Redis cache. - Split the DB build into `_build_basic_cc_pair_info` and wrap the endpoint with an inline fail-open cache (global key `danswer:cc_pair_basic_info`, default 60s TTL). - Pure TTL, no explicit invalidation: staleness is bounded by the TTL and harmless. Any Redis error falls straight through to a direct DB build — never an outage. - Default OFF via CC_PAIR_INFO_CACHE_ENABLED; prod overlay enables it, local leaves it opt-in. Documented in k8s/README.md key table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(vespa): group into base/vespa/ + ordered, health-gated upgrade script Vespa is a version-stateful subsystem with a lifecycle unlike the rest of base (pinned version, the :latest outage history, multi-StatefulSet ordered upgrades). Group it and give the upgrade ordering a real home — which is a script, not the manifests, since kustomize is declarative and cannot sequence a health-gated multi-StatefulSet rollout. Structure: - Move the 7 vespa-*.yaml into k8s/base/vespa/ with its own kustomization.yaml (referenced from base as `- vespa/`). Rendered output is unchanged. - Split the single `vespa` logical image into per-role names (vespa-configserver/-admin/-content/-feed/-query); both overlays map all five to vespaengine/vespa:8.600.35. This lets the upgrade script move one role's version at a time. Safety prereqs (these change the content/admin pod template, so the next apply rolls those StatefulSets — safe, one pod at a time, data on retained PVCs): - Add readiness probes to content + admin on :19092 /state/v1/health (verified serving 200 live; node-agnostic, unlike the containers' 8080). - Set publishNotReadyAddresses: true on vespa-internal so peer discovery is never gated by readiness (a slow/booting node must stay resolvable). Upgrade tooling: - k8s/scripts/vespa-upgrade.sh <target> [ns]: ordered (configserver → admin → content one-ordinal-at-a-time via partition stepping → feed → query), health-gated between each (kubectl exec → localhost, Istio-aware), single hop, refuses >30-minor/major/downgrade (FORCE to override), DRY_RUN/YES flags. bash-3.2 compatible. Dry-run verified against live. Docs: README "Upgrade Vespa" rewritten around the script; base/ section describes the folder; guarded-apply clarified as the everyday-apply net, not the upgrade tool; AGENTS.md §10 updated with the script + per-role structure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chat): createFolder returns the new folder id (was undefined) POST /folder returns the new id as a bare integer, but createFolder read `data.folder_id` — always undefined. This was harmless while the create handler just called router.refresh() and ignored the return, but the optimistic-folder insertion (a84600b3) uses the id, so new folders rendered with folder_id=undefined and rename/delete PATCHed /api/folder/undefined → no-op. Parse the bare integer instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * polish(chat): tidy chat-row drag (compact ghost + no browser split-view) Two native-DnD annoyances when dragging a chat row to a folder: - The row is a <Link> (<a href>), so the browser auto-attaches the URL to the drag and some browsers (Arc/Edge/Safari) offer "open in split view" when dragging toward the edge. Clear the auto-added link payload and set effectAllowed=move so only the folder DnD remains. - The default drag image is a translucent clone of the full-width row that trails awkwardly across the sidebar. Replace it via setDragImage with a compact chip (chat name, ellipsized) built off-screen and removed on the next tick. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chat): pending spinner on "Manage Assistants" navigation Navigating to /assistants/mine awaits the heavy fetchChatData bundle, with no feedback — it felt frozen. (A route-level loading.tsx was wrong here: the app renders the sidebar inside each page, not a shared layout, so the fallback blanked the whole shell and read as a full reload.) Instead drive the navigation with useTransition + router.push: the current page (and sidebar) stays mounted and visible until the new page's server fetch completes, and isPending swaps the button's brain icon for a spinner + "Loading…" so the click clearly registers. Feels like an in-app transition, not a reload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(db): Celery broker on Redis + env-driven Postgres pool sizing Reduce and contain Postgres connection pressure (the real ceiling for chat at scale — sessions are held through the LLM stream): - Celery broker + result backend optionally on Redis via CELERY_BROKER_REDIS_ENABLED (default off; prod on). Uses a separate logical DB (CELERY_REDIS_DB_NUMBER, default 1) so Celery keys never collide with the cache/rate-limit DB 0. Removes Celery's queue polling/writes from Postgres. Task status is unaffected — this fork tracks it in its own task_queue_jobs table, not the Celery backend. Indexing stays on Dask. Falls back to the Postgres broker when off, so local dev without Redis still boots. Note: the broker (unlike the fail-open cache) is a hard dependency when enabled. - Postgres pool size/overflow are now env-driven (POSTGRES_POOL_SIZE / POSTGRES_POOL_OVERFLOW, defaults preserve the previous 40+10) so each deployment can size its pool to its replica count and stay under Azure Postgres max_connections. Applied to both the sync and async engines. Overlays: prod enables the Redis broker and sets explicit pool values (documented to lower as api-server replicas grow); local leaves both opt-in/empty. README gains a "Celery on Redis + pool sizing" section, a verify command, and restart-matrix rows; AGENTS.md divergence table notes the broker is now configurable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(config): black-format app_configs.py (line-length 130) Collapse the multi-line env-read statements added during the Redis/cache work to single lines, per the repo's pinned black==23.3.0 + pyproject line-length=130. Cosmetic only — no values or logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: black-format Redis persona cache + rate-limit middleware Collapse wrapped signatures/calls to single lines per the repo's black==23.3.0 + pyproject line-length=130. Cosmetic only — no logic change. Completes black compliance for the Redis-feature files on this branch (the remaining black-flagged files are unrelated connector/llm code, left for a separate cleanup). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(chat): per-user Redis cache for the document-set list /document-set is on the chat-page bundle (fired every load); the read is a multi-join plus, in EE, a per-user permission query. Cache it. Per-user (not global+filter like the persona cache) on purpose: the doc-set permission filter is edition-dependent — EE filters by is_public/users/groups, MIT base returns all — so memoizing the exact versioned result per user avoids replicating that branchy logic, where a parity bug would leak doc-set visibility. Trade-off: a cold burst of N distinct users still costs N first-loads, but a user's repeat loads (new-chat / nav / router.refresh) collapse to one DB build per TTL. - New db/d…

Strict 'directly supports a statement' verification dropped clearly-relevant authoritative docs whose content didn't literally restate the answer — e.g. the 'Elastic Robot Orchestration Setup For AWS' KB article was at prompt position #1 for an ERO/Automation-Suite question but got filtered out because the answer's key statement was a negative ('not available self-hosted') the setup guide doesn't literally assert. Loosen the verify prompt to 'is a RELEVANT authoritative reference for this answer (same subject/scenario; need not restate it)'. Surfaces the authoritative KB/docs link in these cases; tradeoff is occasionally a topically-related doc, acceptable for the support-engineer use case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e & indexing perf (#47) * feat(search): per-assistant + global cross-encoder reranking, unbiased candidates Incremental, A/B-comparable reranking instead of a single global switch. Two-level gate: rerank runs only when the global master switch RERANK_ENABLED (a GPU-backed model server is deployed) AND the per-assistant opt-in Persona.rerank_enabled are both on. Default off everywhere, so existing assistants and the GPU-free local setup are unchanged and need no GPU. - Persona.rerank_enabled column (+ migration f6a7b8c9d0e1, server_default false), threaded through upsert/create_update_persona and the persona API models, with a 'Rerank results (beta)' toggle in the assistant editor. - Single resolver _resolve_skip_rerank() in retrieval_preprocessing is now the one place both chat and Slack decide reranking (Slack passes skip_rerank=None to share it). Legacy ENABLE_RERANKING_* flags kept as a fallback. - RERANK_MODEL_NAME makes the cross-encoder env-selectable (prod can pick a stronger model, e.g. BAAI/bge-reranker-v2-m3); model server warms it when RERANK_ENABLED. - Retrieval split: when reranking is on, skip the two-query source-prioritization flow (it normalizes a narrow source-filtered set independently, inflating those scores and polluting the rerank candidate window) and run a single all-sources query; when off, the legacy prioritized flow is unchanged. Driven by prioritize_sources=query.skip_rerank. - Tests: global x per-assistant resolver matrix; single-vs-two-query split. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s: gpu-inference component (co-located rerank on a GPU node) Optional kustomize component that pins the inference-model-server onto a GPU node pool (nodeSelector agentpool=gpupool, toleration sku=gpu:NoSchedule, nvidia.com/gpu: 1) and serves the cross-encoder reranker (RERANK_MODEL_NAME, default BAAI/bge-reranker-v2-m3) alongside the embedding + intent models. Also sets real cpu/mem requests+limits (base leaves them empty -> eviction-prone). Opt in from the prod overlay's components: and set RERANK_ENABLED=true in env.properties. The existing model-server image already bundles CUDA torch, so no rebuild. Local omits the component and runs GPU-free. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: branch design doc for reranking/recency/retrieval quality Captures the verified query-path analysis (chat + Slack), the rerank flow and its corrected mental model, the recency decay math + levers (incl. the dead 'auto' auto-detect finding), the source-prioritization normalization bias and its two-path fix, the incremental per-assistant rollout, the GPU sizing/plan, the implementation map, and how to enable in prod. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(search): one-shot LLM relevance filter + source diversity + TEI reranker client - LLM relevance filter is now a single listwise call on the MAIN llm (llm_eval_chunks_listwise + LISTWISE_CHUNK_FILTER_PROMPT, fails open), gated independently of reranking by LLM_RELEVANCE_FILTER_ENABLED x per-assistant llm_relevance_filter (resolver _resolve_skip_llm_chunk_filter). No GPU. - Reranker served by Hugging Face TEI on CPU when RERANK_SERVER_URL is set (CrossEncoderEnsembleModel /rerank path); model server skips loading the cross-encoder in that case. Replaces the GPU plan. - _query_vespa simplified to a single all-sources query (removed the two-query source-prioritization union and its score-inflation bias). - Source diversity moved to final selection: ensure_source_diversity in doc_pruning reserves up to SOURCE_DIVERSITY_RESERVED_SLOTS slots for PROTECTED_SOURCES, so KB/web aren't crowded out. Always-on, global, no per-assistant knob. - Chat per-conversation toggles (use_reranking / use_relevance_filter) threaded via SearchTool -> SearchRequest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chat): per-conversation Rerank / Relevance toggles in the chat input bar Two per-conversation switches (default off) wired through sendMessage -> CreateChatMessageRequest (use_reranking / use_relevance_filter), independent of the assistant's own settings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s: serve the reranker via CPU TEI (drop the GPU plan) - optional/tei-rerank: Hugging Face TEI on CPU (bge-reranker-v2-m3, fp32), Deployment + Service, health probes, model-cache volume. - prod overlay includes it + sets RERANK_ENABLED / RERANK_SERVER_URL / LLM_RELEVANCE_FILTER_ENABLED. Local sets RERANK_ENABLED (no RERANK_SERVER_URL) so the model server loads the reranker in-process — no GPU anywhere. - Removed the optional/gpu-inference component. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test+docs: rerank/relevance/diversity tests + design-doc and dev-setup updates - Unit: relevance-filter gating matrix, listwise parser, source-diversity promotion/caps/disable (replaces the removed two-query test). - Integration: TEI rerank transport (mocked), real CPU cross-encoder reordering (MiniLM), filter_chunks with a stub LLM. - Docs updated to the final design (TEI-on-CPU, source diversity at selection, two assistant knobs); CONTRIBUTING documents local in-process reranking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s: bake bge-reranker-v2-m3 into our own ONNX TEI image The TEI CPU image is ONNX-Runtime-based and bge-reranker-v2-m3 ships no ONNX weights, so the stock image 404s on onnx/model.onnx. Our own image exports the model to ONNX at build time (HF Optimum; pinned optimum 1.23.3 + torch 2.2.2 + numpy<2 for the >2GB external-data path) and bakes it at /model — no runtime download, no re-download on restart, no HF dependency. Deployment uses the ACR image with --model-id /model (dropped the emptyDir that shadowed /data and the unneeded istio annotation). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-148, web vha-78 (rerank/relevance/diversity) Built + deployed from feature/improve-queries; api-server self-migrated persona.rerank_enabled on rollout. Validated live: TEI reranker healthy and scoring correctly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s: serve the reranker on GPU via the upstream TEI image CPU serving was validated as non-viable (24-98s for 15 chunks on prod; the long blocking inference also starved /health and got the pod liveness-killed). Move reranking to a T4 GPU node pool (Standard_NC4as_T4_v3, tainted gpu=true:NoSchedule) where the same bge-reranker-v2-m3 reranks in ~0.6-3.2s (45ms for short inputs). - Use the UPSTREAM TEI GPU image directly (ghcr turing-1.5) instead of building our own: TEI's GPU backend is Candle+safetensors, which the model ships, so the ONNX-export/custom-image dance (a CPU-runtime-only constraint) is gone. Deleted the custom Dockerfile. - Take the pod out of the istio mesh (sidecar.istio.io/inject: false): leaf service, PERMISSIVE mTLS so the api-server still reaches it, and an injected sidecar isn't up during the init phase (so the prefetch init container couldn't reach the network). - Prefetch the model into the PVC with the Python HF client in an init container: TEI's Rust hf-hub client fails on HF's redirect with 'relative URL without a base'; the Python client handles it. TEI loads offline from /data/model. Cluster-side (not in repo): added a gpu=true:NoSchedule toleration to the nvidia-device-plugin-daemonset so it schedules on the tainted GPU node and advertises nvidia.com/gpu. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): make source-diversity config explicit in env.properties PROTECTED_SOURCES and SOURCE_DIVERSITY_RESERVED_SLOTS were running on code defaults (web,sfkbarticles / 2), so the always-on diversity logic was active but invisible in the configmap. Pin them to the current defaults for visibility and tunability — no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(auth): authorize valid X-API-Key requests under enforced auth (OIDC) These api keys are service credentials for automation and intentionally don't map to a User. Enabling OIDC (AUTH_TYPE=oidc) flipped DISABLE_AUTH off, so current_user started 403'ing api-key requests (no session, and the keys don't resolve to a user) — bouncing automation into the SSO login flow. current_user now authorizes a request carrying a valid X-API-Key as an anonymous service caller (user=None, which endpoints already handle). Browser requests without a session still 403 (SSO gate intact), and a key alone does not grant admin (current_admin_user still requires an admin user). Adds request_has_valid_api_key() mirroring validate_api_key's lookup+cache, plus integration tests locking both the SSO and api-key flows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-149 (api-key auth under OIDC fix) Deployed + validated live: GET /api/persona with a valid x-api-key returns 200; without a key still 403s (SSO gate intact). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * web: app-wide dark mode (default) + chat UX overhaul + token theming - Make design-system color tokens CSS-variable-driven so they flip under a `.dark` ancestor (light values unchanged via fallbacks). Dark is now the app-wide default via a no-flash init script on <html>; users opt into light via the sidebar toggle (persisted app-wide to localStorage). - Typography: replace Inter with IBM Plex Sans (body) + Fraunces (display); empty-state headline uses the display font with a staggered entrance. - Chat polish: input bar elevated with an accent focus-glow (fixes the previously-broken focus ring); subtle atmosphere glow on the chat canvas; reusable da-fade-up motion (reduced-motion guarded). - Assistant picker ("Choose Assistant") gains a live search box (name + description) with an empty state. - Consistency sweep: map raw neutral colors to semantic tokens across shared components + search/admin so they flip correctly in dark. Followup judgment-call stragglers + per-page UX land in a separate commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * web(dark): tone down bright native controls in dark mode - Add color-scheme: dark to the .dark root so the browser renders native form controls (checkboxes, date pickers, default input backgrounds) and scrollbars in dark — fixes the glaring white checkboxes (assistant form Tools, knowledge- set list) and any browser-default white input backgrounds. - Brand the checkbox/radio accent-color to the app accent. - Give the chat Filters knowledge-set search input an explicit dark surface (bg-background + token text/placeholder) — it had no bg class and rendered a bright white box. Followup to 8c5f4f87 (kept separate so it can be reverted independently). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump web vha-79 (app-wide dark mode + chat UX) Deployed + verified live: web pod Running 2/2 on vha-79, build-deploy verify shows live==manifest, and /auth/login serves the dark-mode no-flash init. Build note: the local Mac amd64 web build SIGSEGVs (next build / musl under Rosetta), so vha-79 was built natively via 'az acr build' on darwinacr and copied into sfbrdevhelmweacr (docker pull/tag/push) — same digest d53f076e. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: web deploy runbook (az acr build path, RBAC/SP, local SIGSEGV) Captures the working web-image deploy flow: local Mac next build SIGSEGVs under amd64 emulation (even with Rosetta), so build natively via az acr build on darwinacr, copy into sfbrdevhelmweacr, bump tag, apply, verify. Includes the Contributor/PIM requirement and the optional service-principal path for CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * web: add light/dark toggle to the global user menu The toggle previously lived only in the chat sidebar, so admin/assistants and other pages had no way to switch themes. Add a Light/Dark item to UserDropdown (the avatar menu present app-wide), reusing the same darwin-theme localStorage + .dark-on-<html> mechanism. Verified: flips both ways, label reflects state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(web): move UserDropdown theme hooks above the early return next build's eslint (rules-of-hooks) failed the prod build because the darkMode useState/useEffect sat after the `if (!combinedSettings) return null` early return. Hooks must be unconditional — moved them above it. (Dev mode didn't catch this; only the production build's lint does.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump web vha-80 (global light/dark toggle in user menu) Deployed + verified: web pod Running 2/2 on vha-80, verify shows live==manifest. Built via az acr build on darwinacr -> transfer to sfbrdevhelmweacr (digest a1f3167). Note: Docker Hub anon pull rate-limit on the node:20-alpine base flaked several ACR runs; succeeded once the window freed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: make rerank + relevance cluster-config-driven; disable + drop GPU Single source of truth = RERANK_ENABLED / LLM_RELEVANCE_FILTER_ENABLED env: - Backend surfaces the effective global flags via /settings (load_settings), mirroring the existing chat_file_max_size_mb env-injection (relevance also respects the DISABLE_LLM_CHUNK_FILTER kill-switch). - Frontend hides the per-conversation (ChatInputBar) and per-assistant (AssistantEditor) rerank/relevance toggles when disabled cluster-wide. - prod overlay: RERANK_ENABLED=false, LLM_RELEVANCE_FILTER_ENABLED=false; drop the tei-rerank component so the GPU node pool can be removed. Code is intact — flip the env (+ re-add the tei-rerank component and a GPU node pool) to re-enable. No code change needed to toggle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-150 (/settings exposes rerank/relevance flags) Deployed + verified: load_settings() returns rerank_enabled=False, llm_relevance_filter_enabled=False — the cluster-level flags the chat + assistant UIs read to hide the toggles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump web vha-81 (hide rerank/relevance toggles when disabled) Deployed + verified: web Running 2/2 on vha-81; live /api/settings returns rerank_enabled=false / llm_relevance_filter_enabled=false, so the chat + assistant toggles are hidden. GPU pool (gput4) scaled to 0 — no GPU VM running. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s: self-managed Velero overlay for weekly Vespa backups Standalone prod overlay (k8s/overlays/prod-velero), applied separately from the app like prod-vespa. Backs up only the backup=vespa PVCs (Vespa index + configserver state) weekly, ttl 504h (last 3 retained). - reuses SP darwinvelero (155fae3c), which already holds Contributor on the backup/node/darwin RGs -> no Owner/UAA role assignment needed - BSL reuses the darwinaksbackup SA; disk snapshots go to the unlocked node RG - alerts via the existing robusta Prometheus stack: PrometheusRule + ServiceMonitor (release: robusta), incl. a no-recent-successful-backup staleness alert that catches silent failure - notifier CronJob posts per-run success/failure to #darwin-devs via the Slack bot token - SP secret + bot token sourced from gitignored files (templates committed) Replaces the prior self-managed Velero that silently failed for ~a year on an expired SP secret. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chat: disable LLM relevance filter on the default assistants Sets llm_relevance_filter=false for the 3 seeded personas (Darwin, GPT, Paraphrase) so the on-boot upsert_persona() reseed doesn't re-enable the filter. A/B eval over 120 questions showed the filter yields no measurable answer-quality gain (20/20/80 A/B/tie, p~1.0) and discarded all chunks in 14% of cases. Global LLM_RELEVANCE_FILTER_ENABLED=false already gates it off; this keeps the per-assistant seed config consistent and durable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: Azure-managed AKS Backup runbook (reference / alternative path) Runbook for the managed Azure Backup-for-AKS approach, kept for reference. The cluster uses self-managed Velero instead (k8s/overlays/prod-velero) because managed AKS Backup needs role assignments / Trusted Access that require Owner/UAA, which isn't available; this documents that path + the [OWNER] hand-off if it's ever pursued. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: local Redis via compose + shared danswer-stack network Document starting Postgres + Redis together under the -p danswer-stack project (shared danswer-stack_default network), run Vespa via the manual docker run on that same network (the compose `index` service is unreliable locally), add Redis ping/stop + no-auth notes, and set REDIS_HOST=localhost for the host-run backend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-151 (relevance-filter-off personas + settings gating) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deletion): re-drive connector deletions orphaned by lost broker messages Connector deletion is the event-driven cleanup_connector_credential_pair_task on the non-durable Redis broker. If Redis or the celery worker restarts while it's queued, the broker message is lost but the task_queue_jobs row stays PENDING forever — the connector shows "Deleting" indefinitely and nothing re-runs it (deletion, unlike sync/prune, is never periodically rescheduled), while the delete API's dedup guard blocks resubmission. We had 5 connectors stuck Deleting since Mar–May for exactly this reason. Adds a periodic celery-beat task (check_for_stuck_deletion_tasks, every 30m) that re-enqueues any cleanup task whose latest task_queue_jobs row has been non-terminal past JOB_TIMEOUT (db.tasks.get_stuck_deletion_cc_ids). The cleanup task's per-cc-pair advisory lock makes a re-enqueue a no-op if a deletion is genuinely still running, and the fresh row stays live for JOB_TIMEOUT, so this self-throttles to one re-drive per cc-pair per window. Also recovers the existing stuck connectors on first run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-152 (orphaned-deletion re-drive) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(web connector): opt-in latest-N version tracking for docs.uipath For docs.uipath.com versioned products (on-prem/standalone like Automation Suite, Robot standalone), index the latest N versions automatically instead of hard-coding each version URL — so new releases (~every 6 months) aren't missed. - get_uipath_docs_version_base_urls(): reads the product page's version selector (already newest-first, which correctly handles the calendar -> MAJOR.YYMM scheme change, e.g. 2.2510 newer than 2023.10), returns the latest N concrete version base URLs; falls back to [base_url] for non-docs.uipath, evergreen (only "latest"), or any fetch error. - OPT-IN via WebConnector(uipath_latest_versions=bool, max_versions=int=3), RECURSIVE only — existing connectors are untouched (auto-applying would make per-version connectors all crawl the same set). Re-evaluated each run, so new versions are picked up and the oldest drops off. - Recursion scope generalized to the N version prefixes (get_internal_links allowed_prefixes; WebConnector.recursive_prefixes). - Frontend: "Track latest versions" checkbox + "Number of latest versions" field on the Web connector form. - Fix: web poll fallback used is_polling=True (recursion off -> landing pages only) when the sitemap is unavailable; the product sitemap URL currently 404s, so this was the common path. Fall back to a full recursive crawl instead. - scripts/test_uipath_version_discovery.py: checks the agreed URL test set. Verified against the test set: robot/standalone/latest -> 3, apps/automation- suite -> 3, activities/other -> 1 (evergreen), automation-cloud -> as-is, learn.microsoft.com -> as-is (domain-gated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-153 (uipath latest-N web connector) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump web vha-82 (uipath latest-N connector form field) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(AGENTS): slow indexing = per-doc Vespa visit; content-hash dedup already exists New "Critical facts that bite" #11: the indexing bottleneck is the per-document Vespa Visit-API existence check (_get_vespa_chunks_by_document_id, selection scan ~11s/doc), not the source crawl, and Vespa isn't CPU-bound. Documents that the Postgres content-hash dedup (Document.indexed_content_hash + get_doc_ids_to_update) already provides should NOT be re-implemented; notes why it can still re-index everything (from_beginning bypass, NULL legacy hashes, web connector not setting doc_updated_at) and that it self-heals as hashes backfill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(web connector): never fall back to a product-base crawl on version-expand failure get_uipath_docs_version_base_urls previously returned [base_url] on any fetch/parse error. For a version-less product base (e.g. docs.uipath.com/automation-suite/automation-suite/) that means a recursive crawl spans EVERY version — the exact leak this feature prevents. Observed on connector 681: a transient init-time fetch failure made it index all 7 versions instead of the latest 3 (no cross-product leak, but cross-version). Now: retry the selector fetch (3x w/ backoff); on persistent failure RAISE (fail the run, retries cleanly) instead of crawling the product base. When no concrete versions are found, only crawl base_url as-is if it's already scoped to a version/'latest' (evergreen/deep pages); a version-less base RAISES. Test set (robot/standalone, apps/automation-suite, activities/other, automation-cloud deep, learn.microsoft) still passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-154 (uipath version-expand fallback hardening) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(web connector): scope POLL sitemap to recursive_prefixes (latest-N) The previous version-expansion only fixed the LOAD path and the 404-sitemap poll fallback. For products whose product sitemap EXISTS (e.g. automation-suite -> /products/automation-suite/sitemaps/en/sitemap.xml, 200, 4270 urls across all 7 versions), poll_source rebuilt to_visit_list from that sitemap filtered only by the product base, re-indexing EVERY version and bypassing the latest-N expansion (observed: connector 681 indexing 2021.10..2.2510). poll_source's RECURSIVE branch now also filters sitemap URLs by self.recursive_prefixes, so a uipath_latest_versions connector polls only the latest-N versions' changed pages (incremental preserved). For normal connectors recursive_prefixes == [base_url], so behavior is unchanged. Verified: 4270 -> 2592 urls, versions {2.2510, 2024.10, 2023.10} only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-155 (poll sitemap latest-N scoping) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(web connector): default uipath latest-N tracking to 2 versions Change the default from 3 to 2 (max_versions): backend get_uipath_docs_version_ base_urls + WebConnector.__init__ defaults, and the admin form initial value + subtext. New latest-version connectors now track the last 2 versions unless overridden. Existing connectors with an explicit max_versions are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-156 (default latest-N=2); web Dockerfile node-base arg - backend vha-156 carries the default-2 change. - web/Dockerfile: NODE_BASE build-arg (defaults to docker.io node:20-alpine for local builds) so ACR CI can pull the base from our mirror (darwinacr.azurecr.io/library/node:20-alpine) and avoid Docker Hub's unauthenticated pull-rate limit, which 429'd the web build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump web vha-83 (form default latest-N=2, ACR node base) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(indexing): per-source concurrency cap override (uncap web) Add INDEXING_PER_SOURCE_CAP_OVERRIDES on top of the global INDEXING_PER_SOURCE_CAP so a single DocumentSource can run at a different concurrency than the default without affecting others. Web is the motivating case: every web cc-pair has its own dummy credential and mostly distinct domains, so it parallelizes safely, while Slack/Jira/Confluence stay pinned at 1 (rate limits / shared creds). Set web=0 in prod (uncapped; bounded by NUM_INDEXING_WORKERS). Scheduler threads the override through _build_running_view and _evaluate_dispatch_for_attempt via a new cap_for_source() helper; both gained an optional trailing cap_overrides arg (backward compat). +5 unit tests (uncap one source, finite override, non-overridden source unaffected, per-cc-pair lock still holds, env parsing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(web): doc-set picker crash, EE 404 spam, indexing activity/failed shortcuts - getSourceMetadata: fall back to a generic tile for unmapped sources (e.g. ingestion_api) instead of throwing undefined.displayName — the document-set connector picker crashed the page on unmapped cc-pairs. - useUserGroups: pass a null SWR key when EE is off so it stops spamming /api/manage/admin/user-group 404s on every EE-gated page mount. - web connector page: alignTop on the long-subtext checkbox so it sits by its label instead of floating mid-paragraph. - Indexing status: composite 'Active (running + queued)' filter + read ?status= from the URL (Suspense-wrapped) so links land pre-filtered; new 'Indexing Activity' and 'Failed Indexing' sidebar shortcuts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-157, web vha-85 backend vha-157: per-source indexing cap override (web uncapped). web vha-85: doc-set crash fix, EE 404 fix, indexing activity/failed shortcuts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(web): smarter connector search in document-set picker The Pick-your-connectors dropdown matched only option.name with a single contiguous substring, so typing a URL fragment found nothing (web connectors carry their URL in config, not the display name) and pieces typed out of order didn't match. - Dropdown: add optional Option.searchableText (falls back to name) and switch the filter to token-based AND matching (split query on whitespace; every term must appear, any order, case-insensitive). - Document-set form: set searchableText to name + source + config summary (which holds base_url) so URL fragments and source names match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump web vha-86 (document-set connector search fix) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(web): rank connector search results best-match-first + cap rendering Builds on the searchableText/token-AND filter: matches are now scored and sorted so the closest result is at the top — exact name > name prefix > contiguous-in-name > contiguous-in-haystack > scattered tokens; ties break toward shorter names and earlier match position. Results are capped to the top 50 (with a 'showing top N of M' footer) so a broad query no longer renders hundreds of heavy rows. No query = original order, unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump web vha-87 (best-match-first connector search ranking) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db): pool_pre_ping + recycle on async engine The async engine (unlike the sync one) had no pool_pre_ping, so it handed out connections the server/proxy had already closed on idle -> asyncpg.InterfaceError: connection is closed -> intermittent 500s on async endpoints. This surfaced as frequent 'Application error' on /chat (the SSR data fetch hit a 500). Add pool_pre_ping=True + pool_recycle=3600 to match the sync engine and transparently reconnect stale connections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(api): user-friendly errors — stop leaking raw SQL/exceptions to the UI Add server/utils.py::user_facing_http_exception and route the document-set, folder, tool, and Highspot-connector write handlers through it: ValueError passes through (already friendly), IntegrityError becomes an actionable message, everything else is logged server-side and replaced with a generic message. No raw psycopg2/SQLAlchemy text reaches the browser. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(web): chat landing redesign + resilient SSR + dark-mode markdown fixes - Empty-state landing: centered 'Start your day with Darwin' + input, a focused glow behind the chat box (dark elsewhere), wider layout, and the conversation text aligned with the input; placeholder -> 'How can I help you today?'; removed the assistant/filter/model + knowledge/source tiles; docs sidebar collapsed by default and auto-opens when results return. - Resilient chat SSR: fetchChatData uses Promise.allSettled + guards the results[4] destructure, so one failed fetch degrades gracefully instead of 500-ing the whole /chat render ('object null is not iterable'). - Dark mode: prose <strong>/headings inherit the light text color so bold bullet labels are no longer dark-on-dark; fixed a -z-10 backdrop stacking bug via . Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-158, web vha-88 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(vespa): batch chunk-id visits + fix duplicate-append in bulk updates Two fixes to VespaIndex.update (used by doc-set sync, boost/ACL/hidden updates, indexing): 1. _get_vespa_chunks_by_document_id appended every chunk twice (a stray extend after the per-chunk loop), so every bulk update/delete issued 2x the Vespa writes. Removed the duplicate append. 2. Chunk-id lookup was one Vespa visit (a selection scan) PER document, so syncing N docs meant N scans. Added _get_vespa_chunk_ids_by_document_ids which selects up to _VESPA_VISIT_DOC_ID_BATCH (25) document_ids per visit (id1 or id2 or ...) -> one scan per batch. update() now uses it. Verified against live Vespa: identical chunk IDs to the per-doc path. Speeds up document-set sync (was ~285s/100 docs) and bulk Vespa updates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-159 (vespa bulk-update perf) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(celery): run prune check every 15m, not every 5s check_for_prune_task scans every cc-pair (444+ here) with lazy-loaded connector/credential (N+1, ~8s/run). At a 5s beat cadence the runs overlapped and accumulated until they occupied all 10 worker threads, starving sync_document_set_task — document sets sat in 'syncing' for hours with no thread to run on. Pruning is governed by each connector's prune_freq (~daily), so 15-min checks are more than enough. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-160 (prune-check cadence fix) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(celery): cap concurrent document-set syncs at 2 Each sync_document_set_task fans out _NUM_THREADS (32) concurrent Vespa requests, so with worker concurrency 10 up to ~320 simultaneous Vespa calls could run — uncontrolled load with no Vespa-side throttle. The scheduler now counts in-flight syncs and only kicks off enough new ones to reach _MAX_CONCURRENT_DOCUMENT_SET_SYNCS (2); the rest wait for a later tick. Keeps Vespa load predictable while still draining the backlog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-161 (cap doc-set sync concurrency at 2) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(vespa): skip documents with no chunks in bulk update (KeyError regression) The batched chunk-id lookup only creates a dict entry for documents that actually returned chunks, but update() iterated all_doc_chunk_ids[document_id] for every requested doc — so an orphaned doc with zero Vespa chunks (e.g. a Slack doc removed from Vespa but still in Postgres) raised KeyError and failed the whole document-set sync in a retry loop. Use .get(document_id, []) to skip chunkless docs, matching the prior per-document behavior. Verified batched == per-doc for real docs against live Vespa. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-162 (vespa bulk-update KeyError fix) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(celery): resumable document-set sync via persisted cursor sync_document_set_task now checkpoints its keyset cursor (last synced doc id) to key_value_store after each batch and resumes from it on the next run, instead of always restarting from cursor=None. Previously a set too large to finish within the 6h soft_time_limit (e.g. the ~216k-doc ServiceCatalog sets) re-did only its first batches every window and never reached the rest — an infinite non-progressing loop. Now it advances across restarts/timeouts and eventually completes. The cursor is cleared on successful completion, and a doc-set edit (membership change) drops it so the next sync re-tags from scratch. No schema migration (reuses key_value_store). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-163 (resumable doc-set sync) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(celery): raise document-set sync concurrency cap 2 -> 3 Vespa measured to have headroom at cap=2 (content nodes ~2-2.5 cores, feed/ query containers idle, no 429s/timeouts, stable ~25s/batch). 3 concurrent (~96 Vespa calls across 3 content nodes) drains the backlog faster while staying well under the old uncapped ~320. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend vha-164 (doc-set sync cap 3) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): chat history pagination, persona display_name, docs-version dedup, Highspot sync, slack blocklist, perf + tests - chat: paginated/date-bucketed sidebar history (get_chat_sessions_by_user limit/offset/time-window) + citation KeyError fix (translate_citations) - personas: display_name field (+migration a8b9c0d1e2f3) and N+1 eager-load fix on the admin list / edit endpoints (get_personas/get_persona_by_id) - retrieval: docs-version dedup in prune_documents (keep newest version per page; DOCS_VERSION_DEDUP_URL_SUBSTR) - highspot: spot->connector sync (connectors/highspot/sync.py) + endpoint + incremental delta-fetch - slackbot: response blocklist (+migration f7a8b9c0d1e2) - unit tests: translate_citations, clean_spot_name, indexing_concurrency, dedupe_doc_versions, persona_eager_load Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(web): chat readability + assistants UX + lazy-loaded sidebar history - chat history sidebar: Today loads by default; Previous 7/30 Days & Over 30 collapsed and lazy-load with a "Show more" button (ChatTab/ChatContext/ fetchChatData); removed redundant per-page theme toggle (ChatThemeToggle) - readability: AI response + ChatBanner/ChatPopup use dark:prose-invert; question text uses text-default; retrieved-docs keyword highlight removed - assistants: display_name field + hover tooltip + assistantDisplayName helper; My-Assistants drag fix for id=0; zero-document-set assistants flagged red; document-set picker split into Selected / Available - misc UI: intro heading/glow/logo, send-button states, doc-set creation connector-type filter + select-all, dark default theme, modal/border polish Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ops(prod): Apple-Silicon web cloud-build routing, dedup/Highspot env, tag bumps - build-deploy.sh: auto-route the web image to darwinacr cloud build + import on Apple Silicon (local next build SIGSEGVs under emulation); documented as AGENTS.md "Critical facts" #12 - env.properties: DOCS_VERSION_DEDUP_URL_SUBSTR + highspot in PROTECTED_SOURCES - kustomization: backend vha-170, web vha-100 (deployed) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(outsystems): inside.uipath.com connector with large-doc hardening One-time LoadConnector for the OutSystems-backed UiPath intranet: enumerates pages, fetches via the screenservice (interim session-cookie + x-csrftoken auth), extracts page HTML + attached files (PDF/DOCX/PPTX/XLSX, downloaded from the backing SharePoint via the W_Document metadata action), and indexes them. Auth swaps to a service account later with no change to the fetch/extract logic. Reliability for large/pathological docs (learned the hard way on prod): - HTTP (10,60) timeouts + skip-on-network-error - skip videos/images/oversized files (never download/parse) - long-token sanitizer (keeps tokenization linear) - 30K section-splitting (bounded chunk/tokenize units) - thread-bounded extraction; small batch size - skip list of ~20 giant table docs the CPU embedder can't handle Includes DocumentSource.OUTSYSTEMS + factory registration, the web admin page (cloned from sf-account) + source/credential types, ConnectorForm support for refreshFreq=null (one-time, no schedule), and unit tests for extraction, widget classification, the sanitizer, and section splitting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(web/admin): responsive indexing tabs + sidebar active state - AdminSidebar: highlight the active item by path+query, so the Existing Connectors / Indexing Activity / Failed Indexing tabs (same route, different ?status=) each light up. Previously every tab looked identical, so a click gave no feedback. Wrapped in Suspense in Layout (useSearchParams). - Indexing status table: sync the status filter with the ?status= deep-link. Navigating between the tabs is client-side and doesn't remount, so the filter (seeded once on mount) never updated -> the click appeared to do nothing. - Document Sets: stop the "Cannot update while syncing" tooltip from being clipped by the cell's overflow-hidden (column width is held by table-fixed, truncation by the inner span, so the cell didn't need overflow-hidden). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ops): verify image in registry before apply; bump prod tags build-deploy.sh now checks each target image actually exists in the registry (docker manifest inspect) immediately before `kubectl apply`. An interrupted or backgrounded run can bump the manifest tag without the build/push completing; applying that rolls deployments onto an unpullable tag (ImagePullBackOff), and for the dask scheduler it cascades the whole indexing pipeline down. The gate aborts before apply and leaves the cluster untouched. Also bumps prod image tags to the deployed values (backend vha-183). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump web=vha-102 (admin UX fixes: sidebar active-tab, status-filter sync, doc-set tooltip) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(assistants): opt-out visibility — new assistants appear for all users Flip per-user assistant visibility from opt-IN to opt-OUT. Previously chosen_assistants was both the visible allow-list AND the order, so any admin-created assistant was invisible to every user who had ever customized their list (hidden and never-seen were indistinguishable). Now visibility = "every accessible assistant MINUS hidden_assistants"; a new column hidden_assistants holds explicit per-user exclusions, and chosen_assistants is order/default only. A newly created (e.g. admin) assistant is in nobody's hidden list, so it shows up for all users automatically; a user can still disable one for themselves. - models.py: User.hidden_assistants (ARRAY(int), NOT NULL default {}) - migration b9c0d1e2f3a4: additive column, no backfill (chat not rolled out) - PATCH /user/hidden-assistants; UserPreferences + from_model serialize it - web: orderAssistantsForUser filters by hidden then orders by chosen; Manage Assistants + gallery hide/show now write hidden_assistants - tests: orderAssistants (7) + UserInfo.from_model serialization (3) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-184 web=vha-103 (opt-out assistant visibility) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ops(prod): add outsystems to PROTECTED_SOURCES (source-diversity reserved slots) inside.uipath.com OutSystems index (DocumentSource OUTSYSTEMS -> value 'outsystems') now gets reserved front slots at final doc selection, alongside web/sfkbarticles/highspot, so it isn't crowded out of the LLM prompt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(search): source-reserved retrieval — guarantee curated sources reach candidates PROTECTED_SOURCES / SOURCE_DIVERSITY_RESERVED_SLOTS only reserves FINAL-prompt slots among docs retrieval already surfaced. It can't help when a chatty source (e.g. the #help-benefits-amer Slack archive) saturates the entire top-N and a relevant curated doc (web/KB/OutSystems) never enters the candidate set — a recall problem, not a prioritisation one. Observed on AMERBenefits: an RSU/vesting question returned 50/50 Slack candidates, 0 OutSystems, while the inside.uipath.com index has highly relevant RSU guides (top score 0.87) that were simply crowded out. Fix: SearchPipeline.retrieved_chunks now runs ONE extra source-scoped retrieval (SOURCE_RESERVED_RETRIEVAL_SLOTS) reusing the SAME filters (ACL + persona doc-set fence) plus a source_type restriction, and merges up to N PROTECTED_SOURCES chunks into the candidate set. The existing diversity reservation then carries them into the prompt. Universal: both chat and Slack flows funnel through SearchPipeline, so it applies to every assistant with no per-persona gating. - chat_configs: SOURCE_RESERVED_RETRIEVAL_SLOTS (default 0 = off) - pipeline: pure protected_source_topup() + _supplement_protected_sources() - prod env: SOURCE_RESERVED_RETRIEVAL_SLOTS=3, SOURCE_DIVERSITY_RESERVED_SLOTS 2->3 - tests: 5 unit tests for the top-up/dedupe/scope logic Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-185 (source-reserved retrieval) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(search): build supplemental query via copy(update=) — SearchQuery is immutable Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-186 (source-reserved retrieval immutability fix) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(answering): cap docs-per-source in prompt so curated sources get cited Source-reserved retrieval + ensure_source_diversity already put curated docs (e.g. OutSystems RSU guides) at prompt positions [1][2][3], yet the LLM still cited only Slack — because the prompt held ~49 Slack docs vs 3 OutSystems, and the model grounds each claim in the dominant/conversational source. Citations are LLM-determined; the lever is prompt composition, not ranking/boosting. cap_docs_per_source (MAX_PROMPT_DOCS_PER_SOURCE) keeps the top-N docs per source (after diversity promotion) and drops the rest before the token cut, so no single source can monopolize grounding. Generic: lives in doc_pruning (universal chokepoint for both chat + Slack flows, all assistants), only binds when a source dominates, no per-persona config. - chat_configs: MAX_PROMPT_DOCS_PER_SOURCE (default 0 = off) - doc_pruning: cap_docs_per_source(), applied after ensure_source_diversity - prod env: MAX_PROMPT_DOCS_PER_SOURCE=8 - tests: 6 unit tests (cap/order/per-source/enum/disabled) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-187 (per-source doc cap) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(prompts): global authoritative-sources citation nudge (from PROTECTED_SOURCES) The per-source cap rebalances the prompt (3 OutSystems + 8 Slack) but the LLM still grounded in the conversational Slack threads. Add a generic, global nudge telling the model which sources are authoritative systems of record — derived from PROTECTED_SOURCES so it covers all of them and stays in sync — and to prefer citing them over chat discussions when they support the answer (and never cite an unsupporting source). Lives in build_task_prompt_reminders (rides the shared CITATION_REMINDER), so it applies to every assistant and both chat + Slack flows with no per-persona setup. Source names rendered via clean_up_source to match the "Source: X" doc labels. - prompt_utils: build_authoritative_sources_reminder(), appended when citations on - tests: 5 unit tests (names match doc labels, dedupe, gating, empty) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-188 (authoritative-sources citation nudge) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(prompts): strengthen authoritative-sources nudge to mandatory citation 'prefer' -> 'you MUST cite the authoritative document when it supports the statement; cite chat discussions only when no authoritative source does.' Aims to flip citation attribution away from near-duplicate Slack threads. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-189 (mandatory authoritative-sources nudge) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(prompts): grouped citations — 'Authoritative sources' vs 'Other sources' Mandatory 'you MUST cite authoritative' nudge failed to flip the LLM's attribution away from near-duplicate Slack threads. Reframe: let the model cite naturally, but require a trailing 'Sources' section with two labelled groups — authoritative systems of record vs the rest. Authoritative docs are promoted to prompt positions [1..3], so the model lists their [n] markers in the authoritative group and danswer parses them into real citations. Works with the model instead of against it; still global + prompt-only, derived from PROTECTED_SOURCES. Omits empty groups; only supporting docs listed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-190 (grouped authoritative citations) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(prompts): revert to soft authoritative-source suggestion (drop grouped output) The grouped 'Authoritative sources / Other sources' instruction caused the LLM to MISLABEL its Slack citations as authoritative — wrong and trust-damaging. Revert to a soft, honest suggestion: name the authoritative systems of record (from PROTECTED_SOURCES) and ask the model to prefer citing them over chat discussions when they support the point, citing only sources that actually support the answer. Benign and helps on queries where the authoritative doc is the real source; doesn't fabricate or relabel on the hard cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-191 (soft authoritative suggestion) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(answering): verify-then-retain authoritative citations (post-generation) Citations are the LLM's output and it inconsistently cites curated sources even when they're promoted to prompt position [1] (confirmed in chat + Slack across runs). Prompt nudging can't reliably fix this. Add a deterministic post-generation step in the shared Answer pipeline (covers chat + Slack, all assistants): - additive: the LLM's own inline citations are untouched - deduped: only authoritative (PROTECTED_SOURCES) docs the LLM did NOT cite, deduped by document_id (same page often appears as multiple chunks) - honest: one batched LLM call verifies each candidate actually SUPPORTS a statement in the answer (topic-match is not enough); fail-closed on any error - bounded: at most ONE extra call, and only when an uncited authoritative doc is in context (no call otherwise) Supporting docs are appended as an "Authoritative sources" markdown footer (renders in chat UI + Slack). Gated by AUTHORITATIVE_CITATION_RETENTION_ENABLED (default off; prod on). In this deployment fast_llm == main llm, so the verify call uses self.llm. - chat_configs: AUTHORITATIVE_CITATION_RETENTION_ENABLED - llm/answering/authoritative_retention.py: select/verify/footer + orchestrator - llm/answering/answer.py: accumulate answer+cited-ids in _process_stream, append verified footer after the citation stream - prod env: enabled - tests: 14 unit tests (selection/dedupe/parse/verify-fail-closed/footer) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-192 (verify-then-retain authoritative citations) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(answering): retry the authoritative-citation verify call once The verify step is a non-streaming gateway completion that occasionally times out (observed transiently; normally 1.6-5.5s). Retry once before failing closed so a single gateway hiccup doesn't drop the authoritative-sources footer. Still fail-closed after retries (never appends on a real error). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-193 (verify-call retry hardening) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(answering): merge retained authoritative citation into Sources + tighten gate Two changes per product feedback: 1. Single Sources section (no footer): inject the verified authoritative doc as a CitationInfo using its context position as citation_num. Since authoritative docs are promoted to positions 1-3 and the UI orders the Sources group by citation number (JS integer-key order / Slack list), it lands at the TOP of the existing single "Sources" section. Drops the separate markdown footer. 2. Tighter gate: only run when the answer cites NO authoritative source at all. If the LLM already cited any PROTECTED_SOURCES doc, do nothing (no verify call) — the answer is already authoritatively grounded. Net: at most one conditional verify call, only on answers missing authoritative citations; result merges into the one Sources section instead of a second block. - authoritative_retention.py: gate in select_authoritative_candidates; retained_authoritative_citations() returns CitationInfo (footer removed) - answer.py: yield the CitationInfo packets after the stream - tests: 18 (gate skip + no-LLM-call, context-position citation_num, verify reject) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-194 (authoritative citation merged into Sources) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * revert(answering): back to authoritative footer (merge-into-Sources collided) Merging the retained authoritative doc into the single Sources section assigned it a citation_num equal to its context position — which collides with the LLM's own citations (it owns the low numbers), and translate_citations de-dupes first-wins, so the injected citation was dropped (validated: 0 OutSystems shown). Putting it at the TOP of the numbered list would require renumbering the LLM's inline [[n]], which breaks the inline links. Revert to the 'Authoritative sources' footer block (reliably surfaces the link), but KEEP the tightened gate from the prior change: only run when the answer cites NO authoritative source at all (no verify call otherwise). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-195 (revert to authoritative footer) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(answering): loosen authoritative-source verify to relevance Strict 'directly supports a statement' verification dropped clearly-relevant authoritative docs whose content didn't literally restate the answer — e.g. the 'Elastic Robot Orchestration Setup For AWS' KB article was at prompt position #1 for an ERO/Automation-Suite question but got filtered out because the answer's key statement was a negative ('not available self-hosted') the setup guide doesn't literally assert. Loosen the verify prompt to 'is a RELEVANT authoritative reference for this answer (same subject/scenario; need not restate it)'. Surfaces the authoritative KB/docs link in these cases; tradeoff is occasionally a topically-related doc, acceptable for the support-engineer use case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-196 (loosen authoritative verify to relevance) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(answering): surface uncited authoritative docs even if another was cited The tightened gate ('skip if any authoritative source was cited') was too coarse: on an Orchestrator upgrade question the LLM cited a KB article (sfkbarticles) + Slack, which suppressed surfacing the relevant, uncited docs.uipath.com page (/2023.10/.../maintenance-considerations) that was at the front of the prompt — a regression vs the old 2-phase union. Loosen: candidates = any uncited relevant authoritative doc, regardless of whether some other authoritative source was cited. Verify call now fires whenever an uncited authoritative doc is present (still conditional/batched/retry-guarded). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-197 (surface uncited authoritative docs) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(answering): tighten authoritative-relevance verify (same-product != relevant) Relevance verify was too loose: for an 'AI in standalone Orchestrator' question it surfaced 'Does Standalone Orchestrator leverage Azure SignalR Service?' (a messaging/transport doc) as an authoritative reference. Tighten to require the doc be about the SAME SPECIFIC topic/feature, not merely the same product — with an explicit messaging/infra-vs-AI exclusion and 'when unsure, exclude'. Middle ground between strict 'supports-a-statement' (missed relevant docs) and loose 'same-subject' (false positives). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-198 (tighten authoritative-relevance verify) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(answering): generalize relevance verify (drop AI-specific example) The messaging-vs-AI example was overfit to one question. State the general rule: the doc must be about the SAME SPECIFIC topic/feature, not merely the same product; a different feature/component/service isn't a relevant reference. No domain example. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-199 (generalize relevance verify) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(answering): verify relevance against the matched chunk (reliable signal) The verify was judging relevance from the title + first 600 chars of the doc — usually boilerplate, not the passage that matched — causing oscillation (signalR false-positive when loose; 'Before you upgrade' false-negative when strict). Now pass each candidate's full MATCHED PASSAGE (LlmDoc.content = the retrieved chunk; capped at 4000 chars) and judge from that, with rebalanced wording (same topic as the answer = include; different feature/component = exclude). Candidate list is small (1-3), so the extra tokens are bounded. This makes precision come from the chunk rather than from overfit prompt strictness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-200 (verify relevance against matched chunk) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ops(prod): bump SOURCE_RESERVED_RETRIEVAL_SLOTS 3->6 The reserved pass guaranteed only the top-3 protected docs into the candidate set; relevant docs ranked #4 among protected sources (e.g. Orchestrator Maintenance Considerations on an upgrade question) fell off and never reached verify/footer. Bump to 6 so more of the top protected docs are guaranteed in; the per-source prompt cap still bounds how many reach the LLM. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(answering): query-time rewrite of versioned docs links to latest indexed The version dedup only collapses versions that were retrieved, so when retrieval surfaces a stale version (content is near-identical across versions, so which one ranks is ~arbitrary) the old link is shown even though newer versions are indexed (e.g. /2023.4/ surfaced while /2025.10/ exists). After pruning, rewrite each versioned docs.uipath.com link to the NEWEST version of that same page (same URL with the version segment stripped, slug preserved) found in the index. Fixes both inline docs citations and the authoritative footer (both use final_context_docs links). One PK-indexed prefix-scan per distinct page-prefix; no reindex. - doc_pruning: _versioned_url_parts + rewrite_docs_links_to_latest(db_session) - search_tool: rewrite final_context_documents after prune - tests: 6 (parse, rewrite-to-latest, noop-when-latest, slug-variants, non-docs, new-scheme) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-201 (rewrite docs links to latest indexed version) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(answering): version-aware docs link rewrite Blindly rewriting docs links to latest is wrong for version-specific questions ("is X supported in 23.10?" must resolve to the 23.10 doc, not latest). Make it version-aware: parse_question_doc_version() extracts a version from the question (23.10 -> 2023.10) only when EXACTLY one is named (multiple, e.g. 'upgrade 23.10 to 25.10', is ambiguous -> None -> latest). rewrite_docs_links() then resolves each docs page to that exact indexed version (even if older than retrieved); falls back to newest indexed when no version is specified. Not confused by '2.9 million'. - doc_pruning: parse_question_doc_version + rewrite_docs_links(target_version) - search_tool: parse from the query, pass through - tests: parse cases (single/multiple/none/2.9M), target-older, target-not-indexed Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-202 (version-aware docs link rewrite) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(answering): anchor authoritative relevance verify to the question Judging relevance only against the ANSWER let cloud-connectivity-adjacent KBs through non-deterministically: for 'does standalone Orchestrator have AI?' the answer says 'AI requires an Automation Cloud connection', so KBs sharing that theme (Azure SignalR, 'Automation Cloud cannot be accessed' Studio error) matched the answer even though they don't address AI capability. Pass the QUESTION to the verify and judge whether the doc helps answer THE QUESTION's specific subject — keyword/product/service overlap is explicitly not enough, and error/troubleshooting docs are excluded unless that's what's asked. When in doubt, exclude. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(answering): simplify verify prompt to relevance-vs-(question AND answer) Replace the carve-out-heavy wording with a clean rule: include a candidate only if its matched passage is genuinely relevant to BOTH the question and the answer. Requiring relevance to the question (not just the answer) is what excludes cloud-connectivity-adjacent KBs (Azure SignalR, 'Automation Cloud cannot be accessed' Studio error) that shared the answer's 'needs cloud connection' theme but don't address the question's AI-capability subject. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * k8s(prod): bump backend=vha-204 (simplified verify prompt) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(search-quality): document the source-prioritization + authoritative-citation pipeline Rewrite §5 from 'source diversity' into the full layered pipeline as built: 5.1 reserved retrieval (recall, SOURCE_RESERVED_RETRIEVAL_SLOTS=6) 5.2 diversity promotion (SOURCE_DIVERSITY_RESERVED_SLOTS=3) 5.3 per-source cap (MAX_PROMPT_DOCS_PER_SOURCE=8) 5.4 authoritative-sources nudge (soft, from PROTECTED_SOURCES) 5.5 verify-then-retain footer (chunk + question/answer relevance; why footer not merged) 5.6 version-aware docs link rewrite Plus the citation-attribution lesson (presence != citation), PROTECTED_SOURCES now web,sfkbarticles,highspot,outsystems, updated §6b/§8/§9, and §10 follow-ups (configmap externalization, inline-card version gap, indexed-content freshness, DB-backed prompts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replaced Danswer to Darwin in frontend

0bbb180

ninja-shreyash requested a review from vivekjhauipath July 4, 2024 12:29

vivekjhauipath reviewed Jul 4, 2024

View reviewed changes

Comment thread web/src/app/admin/bot/SlackBotConfigCreationForm.tsx Outdated

vivekjhauipath reviewed Jul 4, 2024

View reviewed changes

Comment thread web/src/app/admin/bot/SlackBotConfigCreationForm.tsx Outdated

vivekjhauipath approved these changes Jul 4, 2024

View reviewed changes

ninja-shreyash added 2 commits July 4, 2024 20:02

Updated logo

faa0667

Added logo.ico

9e62bf3

ninja-shreyash merged commit cc52648 into feature/darwin Jul 5, 2024

ninja-shreyash deleted the danswer-to-darwin branch July 8, 2024 08:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replaced Danswer to Darwin in frontend#1

Replaced Danswer to Darwin in frontend#1
ninja-shreyash merged 3 commits into
feature/darwinfrom
danswer-to-darwin

ninja-shreyash commented Jul 4, 2024

Uh oh!

Uh oh!

Uh oh!

vivekjhauipath left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ninja-shreyash commented Jul 4, 2024

Uh oh!

Uh oh!

Uh oh!

vivekjhauipath left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants