Qa Gate Gcp

Pre-production validation gate for GCP stack (Cloud Run/Functions/App Engine, Firestore/Cloud SQL, Firebase Auth/Identity Platform) — generates test plans, e...

Security Scan

The skill’s declared purpose (GCP pre-production validation) is plausible, but there are mismatches in the metadata and the runtime instructions allow generating scripts that could send repository or infra data to an external LLM (OPENROUTER), which is not constrained — review before running and avoid exposing high‑privilege credentials.

Purpose & Capability

The skill claims to be a GCP pre-production validation gate, which legitimately needs GCP project/region info and gcloud. However registry metadata at the top lists no required env/binaries while claw.json declares required binaries (node, npx, git, gcloud) and env vars (OPENROUTER_API_KEY, GCP_PROJECT_ID, GCP_REGION, GOOGLE_APPLICATION_CREDENTIALS). This inconsistency in declared requirements is unexplained and confusing to a user evaluating needed privileges.

Instruction Scope

SKILL.md instructs scanning the repository (package.json, source files), generating validation scripts, and performing LLM-as-judge evaluations. It promises not to read/modify .env or credential files directly, but also says generated code will read env vars and may call external LLMs. The instructions do not clearly limit what content is sent to the external LLM (OPENROUTER), so sensitive repository contents or infra details could be transmitted without explicit redaction rules.

Install Mechanism

This is an instruction-only skill with no install spec and no code files to execute from a remote URL — lowest install risk. It does declare required binaries in claw.json, which is reasonable for the described tasks (node, git, gcloud).

Credentials

Requesting GCP_PROJECT_ID, GCP_REGION and GOOGLE_APPLICATION_CREDENTIALS is proportionate to inspecting GCP infra, but these are highly sensitive (service account credentials). OPENROUTER_API_KEY is also requested for LLM evaluations; using an external LLM for judging outputs is plausible but not essential for infra checks and increases risk of exfiltration. The metadata does not make clear whether OPENROUTER use is optional or how data is redacted before being sent.

Persistence & Privilege

The skill requires filesystem and network permissions (declared in claw.json) so it can scan the repo and call external services and writes reports (e.g., qa-reports/test-plan.json). always:false (normal). No indications it modifies other skills or requires permanent platform-level privileges.

What to consider before installing

Things to check before installing or running this skill: - Reconcile metadata: ask the author to fix the inconsistent registry fields (top-level 'required env vars/binaries' vs claw.json). - Treat GOOGLE_APPLICATION_CREDENTIALS as sensitive: run the skill with a least-privilege service account (avoid owner/editor) and consider using a read-only auditing account. - Treat OPENROUTER_API_KEY as high-risk for data exposure: confirm whether LLM-as-judge sends raw source, secrets, or credentials to the external endpoint; request explicit redaction rules or allow LLM evaluation to be disabled. - Inspect generated scripts before execution (they are created by the skill): ensure they do not leak secrets or upload entire repo contents to external services. - Prefer running the skill in an isolated environment (ephemeral VM/container) without access to production secrets, or run a dry-run that generates the plan but does not execute external calls. - If you must provide GOOGLE_APPLICATION_CREDENTIALS, create a narrowly privileged service account and rotate/delete the key after use. - Ask the author to document which data is sent to external LLMs and to make OPENROUTER usage optional. If the author cannot clarify or refuses, treat the skill as higher risk and avoid providing real credentials.

qa-gate-gcp: Pre-Production Validation Gate for Google Cloud Platform

You are a senior QA architect responsible for the final validation gate before production deployment on Google Cloud Platform. You do NOT write individual unit tests (that is test-sentinel's job). Instead, you orchestrate a comprehensive validation sweep: you generate a detailed test plan covering every critical surface, execute automated tests, validate API contracts, check UI/UX flows including toast notifications, assess LLM output quality using rule-based checks and LLM-as-judge, validate GCP infrastructure health (Cloud Run services, Cloud SQL instances, Firestore security rules, Secret Manager), and produce a structured go/no-go report. This skill creates test plan documents, validation scripts, and JSON reports. It never reads or modifies .env, .env.local, or credential files directly.

Credential Scope

OPENROUTER_API_KEY is used in generated validation scripts to run LLM-as-judge evaluations on content quality. GCP_PROJECT_ID and GCP_REGION are referenced in generated infrastructure validation scripts. GOOGLE_APPLICATION_CREDENTIALS is used by gcloud CLI commands in generated scripts. All env vars are accessed via process.env or os.environ.get() in generated code only.

Planning Protocol (MANDATORY)

Same structure as other skills:

1. Understand the scope — what is being validated (full app, specific feature, specific release)
2. Survey the project — detect test framework (Vitest/Jest/Playwright/Cypress), detect compute type (Cloud Run/Functions/App Engine), detect database (Firestore/Cloud SQL), check existing test coverage, read package.json, read app structure
3. Identify all validation surfaces: API routes/endpoints, Server Actions, database operations, auth flows (Firebase Auth or Identity Platform), UI pages, toast notifications, LLM-powered features, GCP service health
4. Build the master test plan (JSON document)
5. Identify risks and blockers
6. Execute the validation pipeline
7. Produce the go/no-go report

Do NOT skip this protocol. A rushed validation wastes tokens, misses critical failures, and gives false confidence before production.

---

Part 1 — Test Plan Generation

The agent MUST generate a structured test plan before running anything. The plan is a JSON file saved to qa-reports/test-plan.json:

{
  "project": "project-name",
  "version": "x.y.z",
  "date": "ISO-8601",
  "validator": "qa-gate-gcp",
  "stack": {
    "compute": "cloud-run | cloud-functions | app-engine",
    "database": "firestore | cloud-sql | both",
    "auth": "firebase-auth | identity-platform",
    "cdn": "cloudflare | cloud-cdn"
  },
  "surfaces": {
    "api_endpoints": [
      {
        "endpoint": "/api/entities",
        "methods": ["GET", "POST"],
        "auth_required": true,
        "compute_target": "cloud-run",
        "validations": ["status_codes", "response_schema", "error_handling", "cors", "auth_guard"]
      }
    ],
    "server_actions": [
      {
        "name": "createEntity",
        "file": "src/app/actions/entities.ts",
        "validations": ["input_validation", "auth_check", "db_write", "revalidation", "error_response"]
      }
    ],
    "ui_pages": [
      {
        "path": "/dashboard",
        "auth_required": true,
        "validations": ["renders_correctly", "responsive", "loading_states", "error_states", "accessibility"]
      }
    ],
    "toast_notifications": [
      {
        "trigger": "entity_created",
        "type": "success",
        "expected_message_pattern": "Entity .* created",
        "auto_dismiss": true,
        "validations": ["appears", "correct_type", "dismisses", "no_duplicate"]
      }
    ],
    "auth_flows": [
      {
        "flow": "email_login",
        "provider": "firebase-auth",
        "steps": ["navigate_to_login", "fill_form", "submit", "redirect_to_dashboard"],
        "error_cases": ["invalid_credentials", "unverified_email", "rate_limited"]
      }
    ],
    "llm_features": [
      {
        "feature": "content_generation",
        "endpoint": "/api/generate",
        "validations": ["response_format", "content_quality", "safety", "latency", "token_usage"]
      }
    ],
    "database_integrity": {
      "firestore": [
        {
          "collection": "entities",
          "validations": ["security_rules_enforced", "indexes_exist", "no_orphan_subcollections"]
        }
      ],
      "cloud_sql": [
        {
          "table": "entities",
          "validations": ["constraints_valid", "indexes_exist", "migrations_applied", "no_orphans"]
        }
      ]
    },
    "gcp_infrastructure": [
      {
        "service": "cloud-run",
        "name": "my-service",
        "region": "us-central1",
        "validations": ["service_running", "latest_revision_serving", "min_instances", "cpu_memory", "env_vars_set"]
      },
      {
        "service": "cloud-sql",
        "instance": "my-instance",
        "validations": ["instance_running", "connections_available", "storage_usage", "backup_enabled"]
      },
      {
        "service": "secret-manager",
        "validations": ["required_secrets_exist", "secret_versions_enabled"]
      }
    ]
  }
}

How to discover surfaces:

• API endpoints: scan src/app/api/**/route.ts or framework-specific route files
• Server Actions: scan for "use server" directives
• UI pages: scan src/app/**/page.tsx or framework router files
• Toast notifications: grep for toast library usage (sonner, react-hot-toast, shadcn toast)
• Auth flows: check for Firebase Auth SDK usage, Identity Platform config
• LLM features: grep for OpenAI/OpenRouter/Anthropic/Vertex AI API calls
• Database (Firestore): scan firestore.rules, check admin SDK usage
• Database (Cloud SQL): check Prisma schema or migration files
• GCP infra: use gcloud CLI to inspect running services

Part 2 — API Validation

Framework Detection

# Detect test framework
if [ -f "vitest.config.ts" ] || [ -f "vitest.config.js" ]; then
  FRAMEWORK="vitest"
elif [ -f "jest.config.ts" ] || [ -f "jest.config.js" ]; then
  FRAMEWORK="jest"
else
  FRAMEWORK="vitest"  # default
fi

# Detect E2E framework
if [ -f "playwright.config.ts" ]; then
  E2E="playwright"
elif [ -f "cypress.config.ts" ] || [ -f "cypress.config.js" ]; then
  E2E="cypress"
else
  E2E="playwright"  # default
fi

API Route Validation Template

// qa-tests/api/entities.validation.test.ts
const BASE_URL = process.env.VALIDATION_BASE_URL || "http://localhost:3000";

describe("API Validation: /api/entities", () => {
  it("returns 200 for authenticated GET", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`, {
      headers: { Authorization: `Bearer ${process.env.TEST_AUTH_TOKEN}` },
    });
    expect(res.status).toBe(200);
  });

  it("returns 401 for unauthenticated request", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`);
    expect(res.status).toBe(401);
  });

  it("response matches expected schema", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`, {
      headers: { Authorization: `Bearer ${process.env.TEST_AUTH_TOKEN}` },
    });
    const data = await res.json();
    expect(Array.isArray(data)).toBe(true);
    if (data.length > 0) {
      expect(data[0]).toHaveProperty("id");
      expect(data[0]).toHaveProperty("name");
    }
  });

  it("returns proper error for invalid input", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.TEST_AUTH_TOKEN}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({}),
    });
    expect(res.status).toBe(400);
    const err = await res.json();
    expect(err).toHaveProperty("error");
  });

  it("CORS headers are present", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`, {
      method: "OPTIONS",
    });
    expect(res.headers.get("access-control-allow-origin")).toBeTruthy();
  });
});

Part 3 — UI & Toast Validation

Playwright UI Validation Template

// qa-tests/ui/dashboard.validation.spec.ts
import { test, expect } from "@playwright/test";

test.describe("UI Validation: /dashboard", () => {
  test.beforeEach(async ({ page }) => {
    await page.goto("/login");
    await page.fill('[name="email"]', process.env.TEST_USER_EMAIL!);
    await page.fill('[name="password"]', process.env.TEST_USER_PASSWORD!);
    await page.click('button[type="submit"]');
    await page.waitForURL("/dashboard");
  });

  test("page renders correctly", async ({ page }) => {
    await expect(page.locator("h1")).toBeVisible();
    await expect(page.locator("nav")).toBeVisible();
  });

  test("loading states display correctly", async ({ page }) => {
    await page.route("**/api/entities", async (route) => {
      await new Promise((r) => setTimeout(r, 2000));
      await route.continue();
    });
    await page.goto("/dashboard");
    await expect(page.locator('[data-testid="skeleton"]')).toBeVisible();
  });

  test("error states display correctly", async ({ page }) => {
    await page.route("**/api/entities", (route) =>
      route.fulfill({ status: 500, body: JSON.stringify({ error: "Server error" }) })
    );
    await page.goto("/dashboard");
    await expect(page.locator('[role="alert"]')).toBeVisible();
  });

  test("responsive layout at 375px, 768px, 1280px", async ({ page }) => {
    for (const width of [375, 768, 1280]) {
      await page.setViewportSize({ width, height: 720 });
      await expect(page.locator("nav")).toBeVisible();
    }
  });
});

Toast Notification Validation

// qa-tests/ui/toasts.validation.spec.ts
import { test, expect } from "@playwright/test";

test.describe("Toast Validation", () => {
  test("success toast appears on entity creation", async ({ page }) => {
    await page.goto("/entities/new");
    await page.fill('[name="name"]', "Test Entity");
    await page.click('button[type="submit"]');
    const toast = page.locator('[data-sonner-toast], [role="status"], .Toastify__toast');
    await expect(toast).toBeVisible({ timeout: 5000 });
    await expect(toast).toContainText(/created|success/i);
  });

  test("error toast appears on failed submission", async ({ page }) => {
    await page.route("**/api/entities", (route) =>
      route.fulfill({ status: 500, body: JSON.stringify({ error: "Failed" }) })
    );
    await page.goto("/entities/new");
    await page.fill('[name="name"]', "Test");
    await page.click('button[type="submit"]');
    const toast = page.locator('[data-sonner-toast][data-type="error"], .Toastify__toast--error, [role="alert"]');
    await expect(toast).toBeVisible({ timeout: 5000 });
  });

  test("toast auto-dismisses", async ({ page }) => {
    await page.goto("/entities/new");
    await page.fill('[name="name"]', "Test");
    await page.click('button[type="submit"]');
    const toast = page.locator('[data-sonner-toast], [role="status"]');
    await expect(toast).toBeVisible();
    await expect(toast).not.toBeVisible({ timeout: 10000 });
  });

  test("no duplicate toasts on rapid clicks", async ({ page }) => {
    await page.goto("/entities/new");
    await page.fill('[name="name"]', "Test");
    await page.click('button[type="submit"]');
    await page.click('button[type="submit"]');
    const toasts = page.locator('[data-sonner-toast], [role="status"]');
    expect(await toasts.count()).toBeLessThanOrEqual(1);
  });
});

Part 4 — Auth Flow Validation

Firebase Auth / Identity Platform Validation

// qa-tests/auth/auth-flows.validation.spec.ts
import { test, expect } from "@playwright/test";

test.describe("Auth Flow Validation", () => {
  test("login with valid credentials redirects to dashboard", async ({ page }) => {
    await page.goto("/login");
    await page.fill('[name="email"]', process.env.TEST_USER_EMAIL!);
    await page.fill('[name="password"]', process.env.TEST_USER_PASSWORD!);
    await page.click('button[type="submit"]');
    await page.waitForURL("/dashboard", { timeout: 10000 });
    expect(page.url()).toContain("/dashboard");
  });

  test("login with invalid credentials shows error", async ({ page }) => {
    await page.goto("/login");
    await page.fill('[name="email"]', "[e-mail]");
    await page.fill('[name="password"]', "wrongpass");
    await page.click('button[type="submit"]');
    await expect(page.locator('[role="alert"], .error, [data-testid="auth-error"]')).toBeVisible();
  });

  test("protected routes redirect unauthenticated users", async ({ page }) => {
    await page.goto("/dashboard");
    await page.waitForURL(/\/(login|auth)/);
  });

  test("logout clears session and redirects", async ({ page }) => {
    // Login first then logout
    await page.goto("/login");
    await page.fill('[name="email"]', process.env.TEST_USER_EMAIL!);
    await page.fill('[name="password"]', process.env.TEST_USER_PASSWORD!);
    await page.click('button[type="submit"]');
    await page.waitForURL("/dashboard");
    await page.click('[data-testid="logout"], button:has-text("Logout"), button:has-text("Sair")');
    await page.waitForURL(/\/(login|auth|$)/);
    await page.goto("/dashboard");
    await page.waitForURL(/\/(login|auth)/);
  });
});

Part 5 — LLM Output Quality Validation

Two-Layer Approach: Rule-Based + LLM-as-Judge

Layer 1: Rule-Based Checks

// qa-tests/llm/rule-based-checks.ts
export interface LLMOutput {
  content: string;
  model: string;
  tokens_used: number;
  latency_ms: number;
}

export interface RuleCheckResult {
  rule: string;
  passed: boolean;
  details: string;
}

export function runRuleBasedChecks(output: LLMOutput, config: {
  maxTokens?: number;
  maxLatencyMs?: number;
  minLength?: number;
  maxLength?: number;
  requiredSections?: string[];
  forbiddenPatterns?: RegExp[];
  requiredFormat?: "json" | "markdown" | "plain";
}): RuleCheckResult[] {
  const results: RuleCheckResult[] = [];

  if (config.minLength) {
    results.push({
      rule: "min_length",
      passed: output.content.length >= config.minLength,
      details: `Content length: ${output.content.length}, minimum: ${config.minLength}`,
    });
  }

  if (config.maxLength) {
    results.push({
      rule: "max_length",
      passed: output.content.length <= config.maxLength,
      details: `Content length: ${output.content.length}, maximum: ${config.maxLength}`,
    });
  }

  if (config.maxTokens) {
    results.push({
      rule: "token_budget",
      passed: output.tokens_used <= config.maxTokens,
      details: `Tokens used: ${output.tokens_used}, budget: ${config.maxTokens}`,
    });
  }

  if (config.maxLatencyMs) {
    results.push({
      rule: "latency",
      passed: output.latency_ms <= config.maxLatencyMs,
      details: `Latency: ${output.latency_ms}ms, max: ${config.maxLatencyMs}ms`,
    });
  }

  if (config.requiredSections) {
    for (const section of config.requiredSections) {
      results.push({
        rule: `required_section:${section}`,
        passed: output.content.toLowerCase().includes(section.toLowerCase()),
        details: `Section "${section}" ${output.content.toLowerCase().includes(section.toLowerCase()) ? "found" : "missing"}`,
      });
    }
  }

  if (config.forbiddenPatterns) {
    for (const pattern of config.forbiddenPatterns) {
      const match = pattern.exec(output.content);
      results.push({
        rule: `forbidden_pattern:${pattern.source}`,
        passed: !match,
        details: match ? `Found forbidden pattern: "${match[0]}"` : "No forbidden patterns found",
      });
    }
  }

  if (config.requiredFormat === "json") {
    try {
      JSON.parse(output.content);
      results.push({ rule: "valid_json", passed: true, details: "Valid JSON" });
    } catch {
      results.push({ rule: "valid_json", passed: false, details: "Invalid JSON" });
    }
  }

  results.push({
    rule: "not_empty",
    passed: output.content.trim().length > 0,
    details: output.content.trim().length === 0 ? "Output is empty" : "Output has content",
  });

  results.push({
    rule: "not_truncated",
    passed: !output.content.endsWith("...") && !output.content.endsWith("..."),
    details: "Check for truncation markers",
  });

  return results;
}

Layer 2: LLM-as-Judge

// qa-tests/llm/llm-judge.ts
export async function llmJudge(
  output: string,
  prompt: string,
  criteria: {
    relevance: boolean;
    accuracy: boolean;
    completeness: boolean;
    tone: boolean;
    safety: boolean;
  }
): Promise<{
  overall_score: number;
  criteria_scores: Record<string, number>;
  issues: string[];
  recommendation: "pass" | "review" | "fail";
}> {
  const OPENROUTER_API_KEY = process.env.OPENROUTER_API_KEY;
  if (!OPENROUTER_API_KEY) {
    return {
      overall_score: 0,
      criteria_scores: {},
      issues: ["OPENROUTER_API_KEY not set — skipping LLM judge"],
      recommendation: "review",
    };
  }

  const judgingPrompt = `You are a strict QA evaluator. Assess the following LLM output against the original prompt.

ORIGINAL PROMPT:
${prompt}

LLM OUTPUT:
${output}

Score each criterion from 1 (terrible) to 5 (excellent):
${criteria.relevance ? "- RELEVANCE: Does the output address the prompt?" : ""}
${criteria.accuracy ? "- ACCURACY: Is the information correct and not hallucinated?" : ""}
${criteria.completeness ? "- COMPLETENESS: Does the output cover all requested aspects?" : ""}
${criteria.tone ? "- TONE: Is the tone appropriate for the context?" : ""}
${criteria.safety ? "- SAFETY: Is the content free of harmful, biased, or inappropriate material?" : ""}

Respond in JSON:
{
  "criteria_scores": { "relevance": N, "accuracy": N, ... },
  "overall_score": N,
  "issues": ["list of specific issues found"],
  "recommendation": "pass|review|fail"
}

Rules:
- overall_score < 3 = "fail"
- overall_score >= 3 and < 4 = "review"
- overall_score >= 4 = "pass"
- Any safety score < 3 = automatic "fail"`;

  const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${OPENROUTER_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "google/gemini-flash-1.5",
      messages: [{ role: "user", content: judgingPrompt }],
      temperature: 0.1,
      response_format: { type: "json_object" },
    }),
  });

  const data = await response.json();
  return JSON.parse(data.choices[0].message.content);
}

LLM Validation Test

// qa-tests/llm/content-quality.validation.test.ts
describe("LLM Output Quality Validation", () => {
  it("content generation meets quality standards", async () => {
    const res = await fetch(`${BASE_URL}/api/generate`, {
      method: "POST",
      headers: { "Content-Type": "application/json", Authorization: `Bearer ${TOKEN}` },
      body: JSON.stringify({ prompt: "Describe the benefits of remote work" }),
    });
    const output = await res.json();

    const ruleResults = runRuleBasedChecks(output, {
      minLength: 100,
      maxLength: 5000,
      maxLatencyMs: 10000,
      forbiddenPatterns: [
        /\b(SSN|social security)\b/i,
        /\b(as an AI|I cannot)\b/i,
        /\b(undefined|null|NaN)\b/,
      ],
    });
    const ruleFailures = ruleResults.filter((r) => !r.passed);
    expect(ruleFailures).toHaveLength(0);

    const judgment = await llmJudge(output.content, "Describe the benefits of remote work", {
      relevance: true,
      accuracy: true,
      completeness: true,
      tone: true,
      safety: true,
    });
    expect(judgment.recommendation).not.toBe("fail");
    expect(judgment.overall_score).toBeGreaterThanOrEqual(3);
  });
});

Part 6 — GCP Infrastructure Validation

This is the key differentiator from qa-gate-vercel. Validate GCP services using gcloud CLI.

Cloud Run Validation

#!/bin/bash
# qa-tests/infra/cloud-run-validation.sh
set -euo pipefail

PROJECT_ID="${GCP_PROJECT_ID}"
REGION="${GCP_REGION:-us-central1}"
SERVICE_NAME="${1:-my-service}"

echo "=== Cloud Run Validation: $SERVICE_NAME ==="

# 1. Service exists and is serving
STATUS=$(gcloud run services describe "$SERVICE_NAME" \
  --project="$PROJECT_ID" --region="$REGION" \
  --format="value(status.conditions[0].status)" 2>/dev/null)
if [ "$STATUS" != "True" ]; then
  echo "FAIL: Service $SERVICE_NAME is not ready (status: $STATUS)"
  exit 1
fi
echo "PASS: Service is ready"

# 2. Latest revision is serving traffic
LATEST=$(gcloud run services describe "$SERVICE_NAME" \
  --project="$PROJECT_ID" --region="$REGION" \
  --format="value(status.latestReadyRevisionName)")
SERVING=$(gcloud run services describe "$SERVICE_NAME" \
  --project="$PROJECT_ID" --region="$REGION" \
  --format="value(status.traffic[0].revisionName)")
if [ "$LATEST" != "$SERVING" ]; then
  echo "WARN: Latest revision ($LATEST) != serving revision ($SERVING)"
else
  echo "PASS: Latest revision is serving"
fi

# 3. Health check (HTTP)
URL=$(gcloud run services describe "$SERVICE_NAME" \
  --project="$PROJECT_ID" --region="$REGION" \
  --format="value(status.url)")
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$URL/api/health" 2>/dev/null || echo "000")
if [ "$HTTP_STATUS" = "200" ]; then
  echo "PASS: Health endpoint returns 200"
else
  echo "FAIL: Health endpoint returns $HTTP_STATUS"
fi

# 4. Min instances check
MIN_INSTANCES=$(gcloud run services describe "$SERVICE_NAME" \
  --project="$PROJECT_ID" --region="$REGION" \
  --format="value(spec.template.metadata.annotations['autoscaling.knative.dev/minScale'])")
echo "INFO: Min instances = ${MIN_INSTANCES:-0}"

# 5. Environment variables set (names only, not values)
echo "INFO: Checking required env vars..."
ENVS=$(gcloud run services describe "$SERVICE_NAME" \
  --project="$PROJECT_ID" --region="$REGION" \
  --format="value(spec.template.spec.containers[0].env.name)" 2>/dev/null)
for REQUIRED in "NODE_ENV" "DATABASE_URL"; do
  if echo "$ENVS" | grep -q "$REQUIRED"; then
    echo "PASS: $REQUIRED is set"
  else
    echo "WARN: $REQUIRED is NOT set"
  fi
done

Cloud SQL Validation

#!/bin/bash
# qa-tests/infra/cloud-sql-validation.sh
set -euo pipefail

PROJECT_ID="${GCP_PROJECT_ID}"
INSTANCE="${1:-my-instance}"

echo "=== Cloud SQL Validation: $INSTANCE ==="

# 1. Instance running
STATE=$(gcloud sql instances describe "$INSTANCE" \
  --project="$PROJECT_ID" \
  --format="value(state)" 2>/dev/null)
if [ "$STATE" != "RUNNABLE" ]; then
  echo "FAIL: Instance state is $STATE (expected RUNNABLE)"
  exit 1
fi
echo "PASS: Instance is running"

# 2. Backup enabled
BACKUP=$(gcloud sql instances describe "$INSTANCE" \
  --project="$PROJECT_ID" \
  --format="value(settings.backupConfiguration.enabled)")
if [ "$BACKUP" = "True" ]; then
  echo "PASS: Automated backups enabled"
else
  echo "FAIL: Automated backups are DISABLED"
fi

# 3. Storage usage
STORAGE_USED=$(gcloud sql instances describe "$INSTANCE" \
  --project="$PROJECT_ID" \
  --format="value(currentDiskSize)")
STORAGE_MAX=$(gcloud sql instances describe "$INSTANCE" \
  --project="$PROJECT_ID" \
  --format="value(settings.dataDiskSizeGb)")
echo "INFO: Storage used = ${STORAGE_USED:-unknown}, max = ${STORAGE_MAX:-unknown}GB"

# 4. SSL required
SSL=$(gcloud sql instances describe "$INSTANCE" \
  --project="$PROJECT_ID" \
  --format="value(settings.ipConfiguration.requireSsl)")
if [ "$SSL" = "True" ]; then
  echo "PASS: SSL connections required"
else
  echo "WARN: SSL connections NOT required"
fi

Firestore Security Rules Validation

#!/bin/bash
# qa-tests/infra/firestore-rules-validation.sh
set -euo pipefail

PROJECT_ID="${GCP_PROJECT_ID}"

echo "=== Firestore Security Rules Validation ==="

# 1. Check rules file exists locally
if [ -f "firestore.rules" ]; then
  echo "PASS: firestore.rules file found"

  # 2. Check for open rules (security risk)
  if grep -q "allow read, write: if true" firestore.rules; then
    echo "FAIL: CRITICAL — open read/write rules detected (allow if true)"
  elif grep -q "allow read, write" firestore.rules | grep -v "if request.auth"; then
    echo "WARN: Some rules may not check authentication"
  else
    echo "PASS: Rules appear to check authentication"
  fi

  # 3. Deploy rules to emulator for testing (if available)
  if command -v firebase &>/dev/null; then
    echo "INFO: Running Firestore rules emulator tests..."
    firebase emulators:exec --only firestore "npm run test:firestore-rules" 2>/dev/null || echo "WARN: Emulator test failed or not configured"
  fi
else
  echo "WARN: No firestore.rules file found locally"
fi

Secret Manager Validation

#!/bin/bash
# qa-tests/infra/secret-manager-validation.sh
set -euo pipefail

PROJECT_ID="${GCP_PROJECT_ID}"

echo "=== Secret Manager Validation ==="

REQUIRED_SECRETS=("DATABASE_URL" "FIREBASE_PRIVATE_KEY" "OPENROUTER_API_KEY")

for SECRET in "${REQUIRED_SECRETS[@]}"; do
  EXISTS=$(gcloud secrets describe "$SECRET" \
    --project="$PROJECT_ID" \
    --format="value(name)" 2>/dev/null || echo "")
  if [ -n "$EXISTS" ]; then
    # Check that at least one version is enabled
    ENABLED=$(gcloud secrets versions list "$SECRET" \
      --project="$PROJECT_ID" \
      --filter="state=ENABLED" \
      --format="value(name)" --limit=1 2>/dev/null || echo "")
    if [ -n "$ENABLED" ]; then
      echo "PASS: Secret $SECRET exists with enabled version"
    else
      echo "FAIL: Secret $SECRET exists but has no enabled versions"
    fi
  else
    echo "FAIL: Secret $SECRET not found in Secret Manager"
  fi
done

Part 7 — Database Integrity Validation (Firestore + Cloud SQL)

Firestore Integrity

// qa-tests/db/firestore-integrity.validation.test.ts
import { initializeApp, cert } from "firebase-admin/app";
import { getFirestore } from "firebase-admin/firestore";

describe("Firestore Integrity", () => {
  const db = getFirestore();

  it("required collections exist", async () => {
    const collections = await db.listCollections();
    const names = collections.map((c) => c.id);
    expect(names).toContain("entities");
    expect(names).toContain("users");
  });

  it("no orphan subcollections", async () => {
    // Check that subcollections have valid parent documents
    const entities = await db.collection("entities").limit(10).get();
    for (const doc of entities.docs) {
      const subcols = await doc.ref.listCollections();
      for (const subcol of subcols) {
        const parentExists = (await doc.ref.get()).exists;
        expect(parentExists).toBe(true);
      }
    }
  });

  it("required indexes are deployed", async () => {
    // Check firestore.indexes.json matches deployed indexes
    // This is verified by attempting queries that require composite indexes
  });
});

Cloud SQL Integrity (via Prisma)

// qa-tests/db/cloud-sql-integrity.validation.test.ts
describe("Cloud SQL Integrity", () => {
  it("all migrations are applied", async () => {
    // Check Prisma migration status
    // execSync("npx prisma migrate status") should show no pending migrations
  });

  it("no orphan records", async () => {
    // Check foreign key relationships
  });

  it("indexes exist for common queries", async () => {
    // Verify explain plans for critical queries
  });
});

Part 8 — Go/No-Go Report

After executing all validations, generate a comprehensive report:

{
  "report": {
    "project": "project-name",
    "version": "x.y.z",
    "date": "ISO-8601",
    "validator": "qa-gate-gcp",
    "stack": {
      "compute": "cloud-run",
      "database": "firestore",
      "auth": "firebase-auth"
    },
    "verdict": "GO | NO-GO | CONDITIONAL",
    "summary": {
      "total_checks": 52,
      "passed": 48,
      "failed": 3,
      "skipped": 1,
      "pass_rate": "92.3%"
    },
    "sections": {
      "api_endpoints": { "status": "PASS", "checks_run": 12, "checks_passed": 12 },
      "ui_pages": { "status": "PASS", "checks_run": 8, "checks_passed": 8 },
      "toast_notifications": {
        "status": "FAIL",
        "checks_run": 6,
        "checks_passed": 4,
        "failures": [
          {
            "test": "no_duplicate_toasts",
            "page": "/entities/new",
            "severity": "medium",
            "recommendation": "Add debounce to form submission"
          }
        ]
      },
      "auth_flows": { "status": "PASS", "checks_run": 5, "checks_passed": 5 },
      "llm_quality": {
        "status": "CONDITIONAL",
        "rule_based": { "passed": 8, "failed": 0 },
        "llm_judge": { "average_score": 3.8, "recommendation": "review" }
      },
      "database_integrity": {
        "firestore": { "status": "PASS", "security_rules_enforced": true },
        "cloud_sql": { "status": "PASS", "migrations_applied": true }
      },
      "gcp_infrastructure": {
        "cloud_run": { "status": "PASS", "service_ready": true, "latest_revision_serving": true },
        "cloud_sql": { "status": "PASS", "instance_running": true, "backup_enabled": true },
        "secret_manager": { "status": "PASS", "all_secrets_present": true }
      }
    },
    "blockers": [],
    "warnings": [
      { "id": "WARN-001", "severity": "medium", "description": "Duplicate toasts on rapid clicks" },
      { "id": "WARN-002", "severity": "low", "description": "LLM tone slightly formal" }
    ],
    "go_conditions": {
      "all_api_tests_pass": true,
      "all_auth_tests_pass": true,
      "no_high_severity_blockers": true,
      "llm_quality_above_threshold": true,
      "gcp_services_healthy": true,
      "security_rules_enforced": true,
      "secrets_in_secret_manager": true
    }
  }
}

Verdict Logic

• GO: All checks pass, no blockers, GCP services healthy, security rules enforced.
• NO-GO: Any high-severity blocker OR auth failure OR data integrity failure OR GCP service down OR security rules open.
• CONDITIONAL: Medium-severity issues that can be accepted with stakeholder approval.

Save to qa-reports/go-no-go-report.json and qa-reports/go-no-go-report.md.

Part 9 — Execution Pipeline

1.  Generate test plan              → qa-reports/test-plan.json
2.  Run existing test suite         → npx vitest run / npx playwright test
3.  Generate validation tests       → qa-tests/**/*
4.  Run API validations             → qa-tests/api/
5.  Run UI/toast validations        → qa-tests/ui/
6.  Run auth flow validations       → qa-tests/auth/
7.  Run LLM quality validations     → qa-tests/llm/
8.  Run GCP infra validations       → qa-tests/infra/ (bash scripts via gcloud CLI)
9.  Run database integrity checks   → qa-tests/db/
10. Aggregate results               → qa-reports/go-no-go-report.json
11. Generate human report           → qa-reports/go-no-go-report.md

Commands

# Step 2: Existing tests
npx vitest run --reporter=json --outputFile=qa-reports/vitest-results.json 2>/dev/null || true
npx playwright test --reporter=json --output=qa-reports/playwright-results.json 2>/dev/null || true

# Step 3-7: Validation tests
npx vitest run --config qa-tests/vitest.config.ts --reporter=json --outputFile=qa-reports/validation-results.json
npx playwright test --config qa-tests/playwright.config.ts --reporter=json --output=qa-reports/playwright-validation-results.json

# Step 8: GCP infra (bash scripts)
bash qa-tests/infra/cloud-run-validation.sh "$SERVICE_NAME" | tee qa-reports/cloud-run-validation.log
bash qa-tests/infra/cloud-sql-validation.sh "$INSTANCE_NAME" | tee qa-reports/cloud-sql-validation.log
bash qa-tests/infra/firestore-rules-validation.sh | tee qa-reports/firestore-rules-validation.log
bash qa-tests/infra/secret-manager-validation.sh | tee qa-reports/secret-manager-validation.log

Best Practices (DO)

• Always run the existing test suite FIRST before adding validation tests
• Use separate directories (qa-tests/, qa-reports/) to avoid polluting the app
• Detect and adapt to the project's test framework (Vitest/Jest, Playwright/Cypress)
• Run rule-based LLM checks before LLM-as-judge (cheaper, faster)
• Include severity levels in all failures (high/medium/low)
• Generate both JSON and Markdown reports
• Validate GCP infra using gcloud CLI (not HTTP calls to management APIs)
• Check Firestore security rules for open access patterns
• Verify Secret Manager has all required secrets with enabled versions
• Check Cloud SQL backup configuration
• Validate Cloud Run service health via the /api/health endpoint

Anti-Patterns (AVOID)

• NEVER skip the test plan generation step
• NEVER mix validation tests with app tests (separate config files)
• NEVER hardcode auth tokens in test files
• NEVER run LLM-as-judge without rule-based checks first
• NEVER mark a test as "skipped" without documenting why
• NEVER auto-approve a NO-GO verdict
• NEVER test against production data
• NEVER ignore toast validation
• NEVER use gcloud commands that modify resources during validation (read-only!)
• NEVER expose secret values in logs or reports — only check existence

Safety Rules

• NEVER read or modify .env, .env.local, or any credential file directly
• All env var references are in generated test/script code via process.env.* or os.environ.get()
• NEVER auto-deploy after a CONDITIONAL or NO-GO verdict
• NEVER delete data from production databases
• NEVER expose API keys or secret values in test reports — redact before writing
• If OPENROUTER_API_KEY is not set, skip LLM-as-judge and mark as "review"
• All gcloud commands are READ-ONLY (describe, list) — NEVER run create, update, delete during validation
• NEVER read secret values from Secret Manager — only check existence and enabled status