Conference Notes

NVIDIA GTC (San Jose): Overview, Session Recap, and Team Follow-Ups

This post reorganizes field notes from NVIDIA GTC into a durable internal record and an executive-friendly summary for colleagues.

Overview

Core theme: accelerated computing, where GPU-first infrastructure is increasingly required for modern generative and agentic workloads.
Economic signal: repeated emphasis on inference efficiency, token economics, and order-book scale as market demand indicators.
Operational signal: many talks converged on the same tension—agent utility requires broad access, while safety requires least privilege and strong isolation.
Practical takeaway: teams that win will pair model progress with evaluation loops, security-by-default sandboxing, and clear governance from day one.

Shift from retrieval-centric computing to generative computing is changing hardware, infrastructure, and product design assumptions.
Inference performance and cost were positioned as direct revenue drivers; token-per-watt efficiency remains a strategic metric.
Rubin-class claims focused on significant inference gains, especially in advanced reasoning scenarios.
Open questions captured: what an "open model strategy" should look like for enterprises and who will host best-in-class models over the next 12 months.

Strong consensus that governance cannot be bolted on at runtime; it needs to be designed into the sandbox and control plane from the start.
Recurring principle: useful agents seek broad capabilities ("god mode"), but secure systems require strict least-privilege boundaries.
Threat vectors discussed included prompt injection, malicious package supply chain attacks, fake MCP servers, and mutable agent settings.
Practical controls highlighted: immutable configuration, domain/network allowlisting, OS-level sandboxing, and explicit secrets/config isolation.

Rare-disease diagnosis remains a high-impact use case due to extreme long-tail complexity and specialist scarcity.
Positioning: specialized models can surface weak early signals in patient records and reduce multi-year time-to-diagnosis delays.
Challenge raised: balancing broad screening with false-positive burden and practical testing workflows.

Multiple sessions reinforced an iterative loop: Build → Improve → Scale, anchored by stable eval datasets and repeatable benchmark runs.
NVIDIA AI-Q and NeMo Agent Toolkit discussions emphasized profiling, routing, and optimization primitives for throughput and cost control.
Queue design insight: separate shallow and deep request paths so long-running reasoning jobs do not block fast user interactions.

AI-assisted code review tools focused on high-signal feedback, multi-repo context, and explanation-rich comments over raw issue lists.
A recurring pattern was generator-reflector-curator loops that feed reflections back into prompts and systems over time.
Organizational takeaway: audit trails, historical PR context, and prompt/version governance are becoming core engineering assets.

Policy panel themes centered on compliance with existing law, emerging industry standards, and growing workplace/child-safety regulation.
Execution concern: legal guidance is increasing, but implementation responsibility still lands on product, security, and engineering teams.
Near-term implication: teams need operational policy playbooks, red-flag triage paths, and documented review procedures.

The platform race is no longer just model quality. It is model quality + inference economics + secure orchestration.
Agent deployment risk is now mostly a systems problem (access control, sandboxing, config integrity), not just a prompt problem.
Evaluation discipline appears to be the dividing line between demos and production-grade agent platforms.
There is meaningful opportunity to standardize our internal stack around routing, policy, and measurement before tool sprawl grows.

Define our company "open model strategy": model mix, hosting assumptions, and data-boundary rules by workload type.
Evaluate NeMo Agent Toolkit and LangChain deep agent harness against current internal agent framework.
Review NVIDIA Dynamo design for queueing/scheduling fit with shallow-vs-deep request routing.
Prototype policy router behavior for local-vs-external inference on sensitive tasks.
Document a baseline agent sandbox standard: immutable control plane, network allowlisting, secrets isolation, and host-level controls.
Assess whether agent building ownership should shift further toward core engineering with MLOps as an enablement layer.
Run a security assessment on coding-agent workflows (package risk, prompt injection paths, hook visibility, and config protections).
Validate code-review AI options and test whether assumption-aware review prompts reduce noise while keeping defect detection high.
Investigate prompt/version management service options for dynamic persona- or team-specific system prompts.
Follow up on open questions: Cursor CLI enterprise control parity, Bedrock model compatibility, and private-repo marketplace install constraints.