Senior LLM Systems Engineer at Risk Labs

<h2>Why This Role Exists:</h2><p style="min-height:1.5em">We are hiring a Senior LLM Systems Engineer to own and improve the LLM-driven components of our oracle automation stack. This person will focus on the accuracy, performance, resilience, and operational quality of the systems that use models to reason about wide ranging prediction market rules, evidence, and oracle outcomes.</p><p style="min-height:1.5em">This is a production systems role, not a research-only or prompt-only role. You will build the evaluations, observability, tooling, fallbacks, and feedback loops that make LLM behavior measurable and dependable in real-world conditions.</p><p style="min-height:1.5em"></p><h2>What You'll Own:<br /></h2><ul style="min-height:1.5em"><li><p style="min-height:1.5em">LLM Accuracy: improve prompts, model selection, tool usage, structured outputs, retrieval, and evaluation coverage so the system gets more decisions right over time.</p></li><li><p style="min-height:1.5em">System Performance: reduce latency, token usage, and cost while preserving decision quality and operational reliability.</p></li><li><p style="min-height:1.5em">Resilience: design validation, retries, fallbacks, uncertainty handling, and human review paths for ambiguous, adversarial, incomplete, or conflicting inputs.</p></li><li><p style="min-height:1.5em">Evaluation and Monitoring: build datasets, regression tests, dashboards, traces, and review loops that make model quality visible and prevent repeated failures.</p></li><li><p style="min-height:1.5em">Agent and Tooling Architecture: Improve agent orchestration and tool use across internal services, APIs, search workflows, databases, and external data sources.</p></li><li><p style="min-height:1.5em">Production Operations: help debug live issues, investigate regressions, improve runbooks, and reduce repeated operator friction.</p></li></ul><h2><br />What Success Looks Like:<br /></h2><ul style="min-height:1.5em"><li><p style="min-height:1.5em">The oracle automation system handles a wider range of market and resolution scenarios with higher measured accuracy.</p></li><li><p style="min-height:1.5em">LLM quality is tracked through evaluations and regressions instead of judged only through manual spot checks.</p></li><li><p style="min-height:1.5em">Engineers and operators can inspect model behavior, tool usage, reasoning paths, and uncertainty when investigating outcomes.</p></li><li><p style="min-height:1.5em">Latency and cost improve without hiding quality regressions.</p></li><li><p style="min-height:1.5em">The system fails more gracefully when data is missing, tools fail, sources disagree, or cases are genuinely ambiguous.</p></li></ul><h2><br />Skills &amp; Experience</h2><h3><br />Required</h3><ul style="min-height:1.5em"><li><p style="min-height:1.5em">3+ years of professional software engineering experience in Python, TypeScript, or similar production languages.</p></li><li><p style="min-height:1.5em">Hands-on experience buil