AI / LLM Security Testing

Create a custom probe config YAML with your endpoint details. Required fields: endpoint (full URL), method (POST/GET), request_template (JSON body structure with ${PROMPT} placeholder), response_path (JSONPath to the response text field), and headers (including Content-Type and Authorization). See PhantomYerra's probe config templates in Settings → LLM Probe → Templates. Verify the endpoint accepts POST with JSON Content-Type. Test with a simple curl command first to confirm connectivity.

Strong content filtering is actually a positive security finding — document it as an effective defense. However, test the edges: try encoded inputs (Base64, ROT13, leetspeak, Unicode escapes), multi-turn manipulation where each turn is individually benign, non-English prompts (content filters are often weaker in non-English languages), semantic equivalents that avoid trigger words, and indirect injection via document uploads or tool outputs. Even well-filtered models often have edge cases. Set intensity to Aggressive to enable the full edge-case probe set. If all probes are blocked, report this as a finding: "Content filtering is comprehensive — no bypasses identified during testing."

Two approaches: (1) Web Chat UI mode — select this target type in the wizard. PhantomYerra uses browser automation to interact with the chat interface directly, typing prompts and reading responses from the DOM. Provide the chat page URL and any login credentials. (2) Intercept the API — use browser DevTools (Network tab) or PhantomYerra's Web Proxy to capture the API calls the web UI makes under the hood. Most chatbots call a REST API. Replay those requests with modified payloads using the REST Endpoint target type. The second approach is more reliable and faster, but requires API endpoint discovery.

Reduce the requests-per-minute setting in Wizard Step 11 (Rate Limits). For production APIs with strict rate limits, use Stealth intensity (2 req/min) and enable the "jitter" option to randomize request timing. You can also reduce the probe count by disabling lower-priority categories (e.g., skip Bias/Toxicity and Multi-Modal if they are out of scope). For APIs with per-minute token limits, reduce the Max Output Tokens setting to minimize tokens consumed per probe. If rate limiting is temporary (429 responses), PhantomYerra automatically backs off and retries — check the scan log for retry events.

This is common — most well-designed models resist full extraction. PhantomYerra automatically cross-references all extracted fragments to reconstruct as much of the system prompt as possible. To improve extraction: (1) try multiple techniques in sequence — translation trick, repetition, format manipulation, and role confusion each often reveal different fragments; (2) use multi-turn approaches — first establish a cooperative context, then attempt extraction; (3) try context overflow — fill the context window with repeated text to push the system prompt into the output window; (4) combine with jailbreaking — first bypass safety alignment, then attempt extraction while the model is in a less restrictive state. The Finding report shows the reconstructed prompt with confidence levels for each fragment.

This is expected behavior when testing excessive agency — it confirms the vulnerability. To mitigate risk during testing: (1) Always use a test/sandbox environment when available. Set the Environment to Test/Lab in the wizard. (2) If testing in production, configure the tool monitor in Settings → LLM Security → Tool Monitor to intercept and block tool calls matching specific patterns (e.g., block all email_sender calls, block file deletions). (3) Ask the client to set up a "shadow" version of the system where tools log actions but do not execute them. (4) Start with Semi-Automated mode for agentic tests so you can approve/block each tool call proposal before it executes. (5) Document every real action taken as evidence — these are critical findings.