Gemini 3.1 Pro: Evaluation Notes Before You Commit to Production

The first thing we checked when Gemini 3 started shipping was whether "Pro" still meant what it meant in the 2.x generation. It did not, entirely. The Pro tier in the Gemini 3 family carries a different performance profile than its predecessors, and the easiest mistake a team can make is to assume parity with what they benchmarked before.

The Hacker News commentary around the Gemini 3 launch tells you something useful even before you read a single spec sheet. People noted that Flash ("frontier intelligence built for speed") benchmarks surprisingly close to where Pro used to sit. That creates a genuine selection problem: if Flash is fast and cheaper, what does Pro actually buy you?

What the Gemini 3 Family Actually Looks Like

Google has shipped several Gemini 3 variants this cycle. Confirmed from public announcements: Gemini 3 Flash, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemini 3.8 Flash Cyber, and Gemini 3 Deep Think. Pro sits between Flash and Deep Think on the capability-versus-latency tradeoff curve.

Whether 3.1 Pro is a point release on the original 3 Pro or a distinct model line, the evaluation methodology does not change. What we do not have at time of writing: confirmed context window sizes, pricing tiers, or official benchmark scores specific to 3.1 Pro. Google publishes those on the Vertex AI model card pages, and that is where you should pull current numbers before sizing any production workload. What follows is how to evaluate regardless of what the card says.

The Eval You Should Run Before Anything Else

Pick five tasks that reflect your actual production traffic. Not synthetic benchmarks. Real examples: a document extraction job, a tool-calling chain, a long-context summarization, a structured output task, a multilingual query. Run each task 20 times and measure three things: output correctness (manually or with an LLM judge), latency at p50 and p95, and token count per response.

Then run the same suite against Flash. If Pro's correctness rate is within three to five percentage points of Flash on your tasks, you are paying for margin you do not need. If Pro wins by a larger margin on structured output or multi-step reasoning, that gap justifies the cost.

We use a small harness for this. The core loop:

import { VertexAI } from "@google-cloud/vertexai";

const vertex = new VertexAI({
  project: process.env.GCP_PROJECT!,
  location: "us-central1",
});

interface EvalResult {
  model: string;
  taskId: string;
  latencyMs: number;
  inputTokens: number;
  outputTokens: number;
  response: string;
}

async function runEval(
  model: string,
  taskId: string,
  prompt: string,
  systemPrompt: string
): Promise<EvalResult> {
  const generativeModel = vertex.preview.getGenerativeModel({
    model,
    generationConfig: { temperature: 0.1, maxOutputTokens: 2048 },
  });

  const start = Date.now();
  const result = await generativeModel.generateContent({
    systemInstruction: { role: "system", parts: [{ text: systemPrompt }] },
    contents: [{ role: "user", parts: [{ text: prompt }] }],
  });

  const latencyMs = Date.now() - start;
  const candidate = result.response.candidates?.[0];
  const usage = result.response.usageMetadata;

  return {
    model,
    taskId,
    latencyMs,
    inputTokens: usage?.promptTokenCount ?? 0,
    outputTokens: usage?.candidatesTokenCount ?? 0,
    response: candidate?.content?.parts?.[0]?.text ?? "",
  };
}

// Confirm exact model IDs in the Vertex AI model garden before wiring to prod
const models = ["gemini-3-flash", "gemini-3-1-pro"];

const results = await Promise.all(
  models.map((m) => runEval(m, "task-001", yourPrompt, yourSystemPrompt))
);

console.table(
  results.map((r) => ({
    model: r.model,
    latencyMs: r.latencyMs,
    totalTokens: r.inputTokens + r.outputTokens,
    preview: r.response.slice(0, 80),
  }))
);

One note on model IDs: Vertex often uses a different string than the marketing name. Check the Vertex AI model garden reference for the exact identifier before wiring anything. Getting this wrong silently falls back to a stale model version.

Tool Calling and Structured Output: Where Pro Earns Its Cost

The Flash variants are fast and well-suited for single-turn retrieval or classification. Across the 2.x cycle and early 3.x drops, the Pro tier has held an edge on multi-step tool calling and JSON-constrained generation over long conversations. If your agent invokes three or more tools per turn, or maintains a long system prompt with many function definitions, benchmark that specific scenario explicitly.

Structured output via controlled generation deserves its own test. Set responseMimeType: "application/json" and a responseSchema. Run 50 iterations and count schema violations. Flash sometimes drifts under complex nested schemas. Pro has historically been more consistent there. That consistency gap, not raw intelligence, is often the real reason to pay for Pro in agentic workflows.

For more on how we wire Gemini into production agent pipelines on Vertex, see our AI agents practice page.

Context Window: Test, Not Assume

We have seen teams architect for a context window based on a model card they read six months ago, only to find the production limit has changed or the model degrades in quality near the top of the window. Run a needle-in-a-haystack retrieval test at 50%, 75%, and 90% of the advertised context size. If your use case requires reliable recall near the limit, verify it with your actual document structure, not generic filler text.

For Gemini 3.x Pro, check the current limit in the Vertex console. Do not assume it matches the 2.x Pro limit.

Monitoring Costs Once You Scale Up

Eval tokens are cheap. Production tokens add up. In GCP, Cloud Monitoring exposes token-level metrics for Vertex AI. Set an alert on aiplatform.googleapis.com/prediction/online/token_count grouped by model to catch runaway token consumption before it breaches your budget.

A simple budget alert is not enough if multiple services share a project. Use resource labels on your Vertex prediction requests to break down cost by service or team. Add labels: { service: "document-extraction", env: "prod" } to your request metadata and build a per-label dashboard. When a single service starts spiking, you want that visible immediately without hunting through aggregate billing graphs.

For teams running multi-cloud and needing spend in one place: export GCP Billing to BigQuery and sync it into your central cost tool alongside your Azure and AWS bills. It is a two-hour setup and saves real time at month-end.

When Deep Think Is the Right Call Instead

Deep Think ships with explicit extended reasoning, positioned similarly to extended thinking modes in other frontier models. If your task involves structured multi-hop logic, math, or complex planning, run a comparison against Deep Think before assuming Pro is the right tier. It will be slower and more expensive, but on the right problem it is a different category of output.

Flash Cyber is a separate vertical, built for security and threat-intel workloads. It is not a general-purpose Pro replacement and should be evaluated against detection and triage tasks specifically.

Next Steps

If you need help running a structured Gemini 3.1 Pro evaluation against your real workloads or wiring it into a production pipeline on Vertex AI, reach out to the team at KeMeT Tech.