Senior LLM Systems Engineer at Risk Labs

Why This Role Exists: We are hiring a Senior LLM Systems Engineer to own and improve the LLM-driven components of our oracle automation stack. This person will focus on the accuracy, performance, resilience, and operational quality of the systems that use models to reason about wide ranging prediction market rules, evidence, and oracle outcomes. This is a production systems role, not a research-only or prompt-only role. You will build the evaluations, observability, tooling, fallbacks, and feedback loops that make LLM behavior measurable and dependable in real-world conditions. What You'll Own: LLM Accuracy: improve prompts, model selection, tool usage, structured outputs, retrieval, and evaluation coverage so the system gets more decisions right over time. System Performance: reduce latency, token usage, and cost while preserving decision quality and operational reliability. Resilience: design validation, retries, fallbacks, uncertainty handling, and human review paths for ambiguous, adversarial, incomplete, or conflicting inputs. Evaluation and Monitoring: build datasets, regression tests, dashboards, traces, and review loops that make model quality visible and prevent repeated failures. Agent and Tooling Architecture: Improve agent orchestration and tool use across internal services, APIs, search workflows, databases, and external data sources. Production Operations: help debug live issues, investigate regressions, improve runbooks, and reduce repeated operator friction. What Success Looks Like: The oracle automation system handles a wider range of market and resolution scenarios with higher measured accuracy. LLM quality is tracked through evaluations and regressions instead of judged only through manual spot checks. Engineers and operators can inspect model behavior, tool usage, reasoning paths, and uncertainty when investigating outcomes. Latency and cost improve without hiding quality regressions. The system fails more gracefully when data is missing, tools fail, sources disagree, or cases are genuinely ambiguous. Skills & Experience Required 3+ years of professional software engineering experience in Python, TypeScript, or similar production languages. Hands-on experience building production systems that use LLMs, agents, retrieval, structured outputs, or model-powered workflows. Experience designing evaluations, test datasets, regression checks, quality metrics, or manual review loops for AI systems. Strong debugging ability across APIs, databases, queues, logs, model outputs, and external data sources. Practical understanding of prompt engineering, tool calling, structured output validation, retrieval, and common LLM failure modes. Ability to reason carefully about correctness in uncertain or adversarial environments. High agency, strong ownership, and clear written communication. Nice to Have Experience with oracle systems, prediction markets, DeFi protocols, or other crypto infrastructure. Experience with UMA, optimistic oracle me