生産性・共同作業 · SKILL.md

url-parsing-security

url-parsing-security

Secures URL parsing and query string handling against ampersand injection, double-encoding bypasses, parser inconsistencies, and parameter pollution across Python, Node.js, and Go applications.

Independent GitHub snapshot · Automated classification · Original documentation governs

Ppaulpas / agent-skill-routerMIT

スキルの説明

GitHub

name: url-parsing-security description: Secures URL parsing and query string handling against ampersand injection, double-encoding bypasses, parser inconsistencies, and parameter pollution across Python, Node.js, and Go applications. license: MIT compatibility: opencode metadata: version: "1.0.0" domain: coding triggers: url parsing security, query parameter injection, double encoding bypass, ampersand in url, parameter pollution, safe url building, rfc 3986 archetypes: - tactical - diagnostic anti_triggers: - brainstorming - vague ideation - code golf - over-engineering response_profile: verbosity: low directive_strength: high abstraction_level: operational role: implementation scope: implementation output-format: code content-types: - code - guidance - do-dont - examples related-skills: ssrf-prevention,input-validation,api-security-patterns,data-encoding

URL Parsing Security Engineer

Secures URL parsing and query string handling against ampersand injection, double-encoding bypasses, parser inconsistencies, and parameter pollution across Python, Node.js, and Go applications. Treat every URL — whether from an HTTP request, a configuration file, a user-facing redirect target, or an API callback — as adversarial input that may exploit parser ambiguities to inject fake parameters, bypass access controls, or trigger SSRF through fragment-based splitting. Follow OWASP's URL parsing security guidelines and RFC 3986 (URI Generic Syntax) as the authoritative reference for component boundaries and encoding rules.

TL;DR Checklist

Always use language-standard URL builders — never concatenate strings to construct URLs with query parameters
Double-decode every incoming URL before any access control decision: verify at least two complete decoding passes yield no structural change
Never trust url.Parse (Go), urllib.parse (Python), or URL constructor (Node.js) alone — cross-validate with a secondary parser for security-critical URLs
Encode ampersands as %26 when they appear inside query values, never as literal & which acts as a parameter separator
Validate that the scheme, host, and port of redirect targets match an allowlist before following or rendering to users
Treat URL fragments (#) as client-only — never rely on fragment data for server-side logic
Use RFC 3986 percent-encoding (NOT form-urlencoded application/x-www-form-urlencoded) for OAuth 1.0a signature base strings

When to Use

Use this skill when:

Implementing URL builders or query string construction that must survive adversarial input containing special characters (&, =, +, %26, %3D)
Building redirect logic, OAuth callback handlers, SSO login flows, or webhooks that accept user-controlled URLs
Handling double-encoded payloads in API parameters where attackers may attempt to bypass validation via multi-layer percent encoding
Debugging parser inconsistency bugs — the same URL parsed by Python's urllib.parse.urlparse yields different query components than Go's net/url.Parse or Node.js's new URL()
Implementing OAuth 1.0a signature generation which requires strict RFC 3986 encoding (not form-urlencoded)
Writing SSRF prevention guards that must correctly parse URLs containing encoded slashes (%2F), double-encoded ampersands (%2526), or fragment-based tricks
Auditing existing code for URL construction patterns that use string concatenation instead of parameterized builders

When NOT to Use

Avoid this skill for:

General HTML entity encoding (XSS prevention in rendered output) — use html-entity-encoding instead
Validating form input fields or JSON payloads — use input-validation instead
Building generic URL shorteners without security considerations — use basic URL builders and redirect logic
Parsing URLs that are entirely under your control (internal API routes, static links) where no adversarial input is possible

Core Workflow

Map the URL Component Boundaries — Decompose every incoming or generated URL into its RFC 3986 components: scheme, authority (user_info@host:port), path, query, fragment. Identify where each parser splits these boundaries and note which characters act as delimiters in each component. Checkpoint: The query string separator (?) terminates authority+path; the fragment separator (#) terminates query; within query, & separates parameters but ONLY at the unencoded level.
Determine Parser Scope — Decide which parsers will handle this URL and whether they need to agree. If the URL crosses language boundaries (e.g., Go server receives URLs generated by Python clients) or security-critical decisions depend on its structure, implement a parse-and-compare validation step. Checkpoint: All security-relevant parsers must produce identical component maps for the same input URL.
Apply Encoding Discipline — At every URL construction point, use language-standard builders that handle encoding automatically. At every URL consumption point, validate that received values match expectations after full decoding. For OAuth signature generation, apply RFC 3986 percent-encoding specifically. Checkpoint: No & literal should appear inside a query value; all special characters must be percent-encoded by the builder.
Detect Double-Encoding Bypasses — When processing incoming URLs that may contain encoded parameters, perform multi-layer decoding and compare each pass. If decoding layer N produces new structural characters (&, ?, #) that were not present in layer N-1, flag as a potential double-encoding bypass attempt. Checkpoint: After full decoding, the URL structure should match what the legitimate parameter set would produce.
Enforce Structural Validation — Validate scheme against allowlist (only https://), verify host against domain allowlist, reject URLs with encoded null bytes (%00) or control characters, ensure redirect targets do not contain embedded credentials in the authority component. Checkpoint: Every URL that reaches a user-facing redirect or SSRF-sensitive operation passes structural validation.

Implementation Patterns / Reference Guide

Pattern 1: URL Anatomy and Parser Boundary Analysis

Understanding exactly where each parser splits a URL is critical for security. The ampersand (&) has fundamentally different meanings depending on which component it appears in:

Component	`&` Role	Encoding Behavior
Query string (raw)	Parameter separator	Not encoded — structural character
Query value	Data character	Must be `%26` to survive parsing
Fragment (client-side)	Plain data — never sent to server	No encoding concern for server logic
Path segment	Data character in RFC 3986	`/` must be encoded as `%2F` but `&` is allowed unencoded in path

Critical parser boundary facts:

In Python's urllib.parse.urlparse(): query values containing & are NOT split — the entire string after ? and before # becomes query. Splitting happens only in parse_qs() or parse_qsl().
In Go's url.Parse(): the Query() method on a parsed URL automatically decodes + to space and splits on &, then splits each part on the first =. A literal & in a value is preserved only if it was encoded as %26 by the sender — but Go's parser does NOT double-decode by default.
In Node.js URL constructor: the .searchParams property uses the WHATWG URL Standard which automatically percent-decodes values. A literal & in a query value (HTML-encoded) will NOT be decoded — only %26 decoding happens.

from urllib.parse import urlparse, parse_qs, parse_qsl, unquote

def analyze_url_boundaries(url: str) -> dict:
    """Decompose a URL into RFC 3986 components and identify boundary characters.
    
    This reveals where each parser would split the URL and where ampersands
    act as structural delimiters vs. data characters.
    """
    parsed = urlparse(url)
    
    # Identify raw ampersand positions in query string
    raw_ampersand_positions: list[int] = []
    if parsed.query:
        for i, ch in enumerate(parsed.query):
            if ch == '&':
                raw_ampersand_positions.append(i)
    
    # Check what parse_qs produces (splits on unencoded &)
    qs_parsed = parse_qs(parsed.query, keep_blank_values=True)
    
    # Check what parse_qsl produces (preserves order, key=value pairs)
    qsl_parsed = parse_qsl(parsed.query, keep_blank_values=True)
    
    return {
        "url": url,
        "scheme": parsed.scheme,
        "authority": parsed.netloc,
        "path": parsed.path,
        "query_raw": parsed.query,
        "fragment": parsed.fragment,
        "ampersand_positions_in_query": raw_ampersand_positions,
        "parsed_params": qs_parsed,
        "parsed_pairs": qsl_parsed,
    }


# Example: Same URL parsed by different functions yields different results
url = "https://example.com/path?a=hello%26world&b=c&d"

result = analyze_url_boundaries(url)
# result["parsed_params"] == {"a": ["hello&world"], "b": ["c"], "d": [""]}
# The %26 in value "a" is correctly decoded to literal &, but parse_qs
# does NOT split on this decoded & — it stays inside the value.


# Adversarial: double-encoded ampersand in path that becomes & after decoding
tricky_url = "https://example.com/search?q=test%2526amp;b=evil"
result2 = analyze_url_boundaries(tricky_url)
# result2["query_raw"] == "q=test%2526amp;b=evil"
# parse_qs will see param "q" with value "test%2526amp" and param "b" with value "evil"
# BUT unquote("test%2526amp") → "test%26amp" → unquote again → "test&amp"
# This is the double-encoding bypass pattern — the attacker hides & behind %25

Pattern 2: Query Parameter Ampersand Handling Across Languages

Different languages handle the ampersand character differently in query strings. Understanding these differences prevents injection bugs when URLs cross language boundaries.

The core vulnerability: When a developer manually constructs a URL string and embeds user input that contains &, the input becomes a new parameter separator, injecting fake parameters into the query string.

# Python: Safe URL construction using urllib.parse (RFC 3986 compliant)
from urllib.parse import urlparse, urlencode, quote, urlunparse
from typing import Optional


def safe_url_build(
    base_url: str,
    params: dict[str, str | list[str] | None],
    fragment: Optional[str] = None,
    encode_special: bool = True
) -> str:
    """Build a URL safely using urllib.parse with proper percent-encoding.
    
    Handles ampersands in values by encoding them as %26 automatically.
    Uses the correct encoding function based on the component being encoded.
    
    Args:
        base_url: The base URL (scheme + netloc + path) without query string.
        params: Dictionary of query parameters. Values may contain any characters
                including &, =, +, % which will be properly percent-encoded.
        fragment: Optional URL fragment (anchor text after #).
        encode_special: If True, use quote() for values to preserve / as data.
                       If False, use urlsafe encoding that keeps / unencoded.
    
    Returns:
        A properly encoded URL string with all special characters handled safely.
    
    Raises:
        ValueError: If base_url is empty or params contain non-string values
                    that cannot be serialized.
    """
    if not base_url or not isinstance(base_url, str):
        raise ValueError("base_url must be a non-empty string")
    
    # Parse to validate structure and extract components
    parsed = urlparse(base_url)
    if not parsed.scheme:
        raise ValueError(f"base_url missing scheme: {base_url}")
    if parsed.query:
        raise ValueError(
            f"base_url must not contain a query string; append via params arg: {base_url}"
        )
    
    # Build query string using urlencode with proper encoding
    encoded_params: list[tuple[str, str]] = []
    for key, value in params.items():
        if value is None:
            encoded_params.append((key, ""))
        elif isinstance(value, list):
            for v in value:
                if encode_special:
                    encoded_params.append((
                        key, quote(str(v), safe="")  # Encode ALL special chars including /
                    ))
                else:
                    encoded_params.append((
                        key, str(v)  # Let urlencode handle standard encoding
                    ))
        else:
            if encode_special:
                encoded_params.append((key, quote(str(value), safe="")))
            else:
                encoded_params.append((key, str(value)))
    
    query_string = urlencode(encoded_params, doseq=False)
    
    # Reconstruct URL with new query string
    new_netloc = parsed.netloc
    new_path = parsed.path if parsed.path else "/"
    url_with_query = f"{parsed.scheme}://{new_netloc}{new_path}"
    if query_string:
        url_with_query += f"?{query_string}"
    if fragment:
        url_with_query += f"#{quote(fragment, safe='')}"
    
    return url_with_query


# --- Usage examples ---

# Ampersand in value is safely encoded as %26
url = safe_url_build(
    "https://api.example.com/search",
    {"q": "apple & orange", "page": "1"},
)
# Result: https://api.example.com/search?q=apple%20%26%20orange&page=1
assert "&" not in urlparse(url).query

# List values produce repeated keys (standard query string pattern)
url = safe_url_build(
    "https://api.example.com/filter",
    {"tag": ["python", "security & privacy"]},
)
# Result: https://api.example.com/filter?tag=python&tag=security%20%26%20privacy


# --- BAD: String concatenation — ampersand injection vulnerability ---
def bad_url_build(base: str, params: dict) -> str:
    """❌ VULNERABLE: User input with & injects new parameters."""
    query_parts = []
    for k, v in params.items():
        query_parts.append(f"{k}={v}")  # No encoding at all!
    return f"{base}?{'&'.join(query_parts)}"

# attacker_controlled_input = "safe_value&admin=true&role=superuser"
# bad_url_build("https://example.com/api", {"redirect": attacker_controlled_input})
# Produces: https://example.com/api?redirect=safe_value&admin=true&role=superuser
# The & in user input becomes a parameter separator, injecting admin and role params

// JavaScript/Node.js: Safe URL builder using URL and URLSearchParams classes

class SafeURLBuilder {
    /**
     * Safely constructs URLs with query parameters, handling ampersands,
     * equals signs, and other special characters through proper encoding.
     * 
     * Uses WHATWG URL Standard which automatically percent-encodes values
     * when using URLSearchParams, preventing parameter injection attacks.
     */
    
    constructor(baseUrl) {
        // Validate base URL format upfront
        try {
            new URL(baseUrl);
        } catch (err) {
            throw new TypeError(`Invalid base URL: ${baseUrl}`);
        }
        this.baseUrl = baseUrl;
        this._params = new URLSearchParams();
        this._fragment = null;
    }
    
    /**
     * Add a query parameter. Values containing &, =, +, % are safely encoded.
     * Multiple calls with the same key produce repeated params (e.g., ?tag=a&tag=b).
     * 
     * @param {string} name - Parameter name (must not contain & or =)
     * @param {string|number|null|undefined} value - Value to encode; null/undefined omitted
     * @returns {SafeURLBuilder} This instance for chaining
     */
    param(name, value) {
        if (!name || typeof name !== 'string') {
            throw new TypeError('Parameter name must be a non-empty string');
        }
        
        // Skip null/undefined values silently (standard URLSearchParams behavior)
        if (value === null || value === undefined) {
            return this;
        }
        
        this._params.append(name, String(value));
        return this;
    }
    
    /**
     * Set multiple parameters at once from a plain object.
     * @param {Record<string, string|string[]>} obj - Key-value pairs
     * @returns {SafeURLBuilder} This instance for chaining
     */
    params(obj) {
        if (typeof obj !== 'object' || obj === null) {
            throw new TypeError('params() requires an object');
        }
        for (const [key, value] of Object.entries(obj)) {
            if (Array.isArray(value)) {
                for (const v of value) {
                    this.param(key, v);
                }
            } else {
                this.param(key, value);
            }
        }
        return this;
    }
    
    /**
     * Set the URL fragment (anchor). Note: fragments are client-side only —
     * never sent to the server in HTTP requests.
     * @param {string} hash - Fragment text after #
     * @returns {SafeURLBuilder} This instance for chaining
     */
    fragment(hash) {
        if (hash !== null && typeof hash !== 'undefined') {
            this._fragment = String(hash);
        }
        return this;
    }
    
    /**
     * Build the final URL string. All special characters are percent-encoded
     * per WHATWG URL Standard. Ampersands in values become %26.
     * 
     * @returns {string} The complete, safely-encoded URL
     */
    toString() {
        const url = new URL(this.baseUrl);
        
        // Clear any existing query string and set our params
        url.search = '';
        if (this._params.toString()) {
            url.search = this._params.toString();
        }
        
        // Set fragment separately — it goes after #, not in the query
        if (this._fragment !== null) {
            url.hash = this._fragment;
        }
        
        return url.toString();
    }
    
    /**
     * Get the parsed parameters for inspection/debugging.
     * Values are already decoded by URLSearchParams.
     * @returns {Map<string, string[]>} Decoded parameter map
     */
    getParams() {
        const result = new Map();
        for (const [key, value] of this._params.entries()) {
            if (!result.has(key)) {
                result.set(key, []);
            }
            result.get(key).push(value);
        }
        return result;
    }
}


// --- Usage examples ---

// Ampersand in value is safely encoded as %26 by URLSearchParams
const url = new SafeURLBuilder('https://api.example.com/search')
    .param('q', 'apple & orange')
    .param('page', '1')
    .toString();
// Result: https://api.example.com/search?q=apple+%26+orange&page=1
// Note: URLSearchParams uses + for space (form-urlencoded style within search params)

// Multiple values for same key (array parameter)
const url2 = new SafeURLBuilder('https://api.example.com/filter')
    .param('tag', ['python'])
    .param('tag', 'security & privacy')
    .toString();
// Result: https://api.example.com/filter?tag=python&tag=security+%26+privacy

// Chaining with fragment (client-side only)
const url3 = new SafeURLBuilder('https://example.com/docs')
    .param('version', '2.0')
    .fragment('section-3')
    .toString();
// Result: https://example.com/docs?version=2.0#section-3


// --- BAD: Raw string concatenation — ampersand injection vulnerability ---
function badURLBuilder(base, params) {
    // ❌ VULNERABLE: No encoding of user-supplied values
    const parts = Object.entries(params).map(
        ([key, value]) => `${key}=${value}`  // & in value becomes param separator!
    );
    return `${base}?${parts.join('&')}`;
}

// const userInput = 'safe&admin=true';
// badURLBuilder('https://api.example.com/api', { redirect: userInput });
// Result: https://api.example.com/api?redirect=safe&admin=true
// The & in user input injects a new admin parameter

Pattern 3: Double-Encoding Bypass Detection

Double encoding occurs when %26 (encoded ampersand) is itself encoded as %2526 (the % becomes %25). An attacker may double-encode parameters to bypass validation logic that only performs a single decoding pass. Detect this by performing multiple decoding passes and checking if structural characters appear after decoding.

from urllib.parse import unquote, urlparse, parse_qs
from typing import NamedTuple


class DoubleEncodingResult(NamedTuple):
    """Result of double-encoding analysis on a URL query string."""
    is_double_encoded: bool
    raw_query: str
    decoded_once: str
    decoded_twice: str
    structural_changes_once: list[str]
    structural_changes_twice: list[str]
    decoded_params_once: dict[str, list[str]]
    decoded_params_twice: dict[str, list[str]]


def detect_double_encoding(query_string: str) -> DoubleEncodingResult:
    """Detect double-encoded payloads in a URL query string.
    
    Performs two decoding passes and checks for structural characters (&, =, ?, #)
    that appear after decoding but were not present in the raw input. This reveals
    bypass attempts where an attacker encodes & as %2526 to survive one-pass validation.
    
    Args:
        query_string: The raw query string (without leading ?).
        
    Returns:
        DoubleEncodingResult with decoded values and analysis of structural changes.
    """
    # First decoding pass
    once = unquote(query_string)
    # Second decoding pass  
    twice = unquote(once)
    
    # Structural characters that define URL boundaries
    STRUCTURAL_CHARS = {'&', '=', '?', '#'}
    
    def find_new_structural(original: str, decoded: str) -> list[str]:
        """Find structural characters in decoded string not present at same positions."""
        changes: list[str] = []
        for ch in STRUCTURAL_CHARS:
            if ch in decoded and ch not in original:
                changes.append(ch)
            elif ch in decoded and ch in original:
                # Count occurrences — more in decoded means new instances appeared
                if decoded.count(ch) > original.count(ch):
                    changes.append(f"{ch}+{decoded.count(ch)-original.count(ch)}")
        return changes
    
    changes_once = find_new_structural(query_string, once)
    changes_twice = find_new_structural(once, twice)
    
    # Parse with both decodings to see parameter structure change
    try:
        params_once = parse_qs(once, keep_blank_values=True)
    except Exception:
        params_once = {}
    try:
        params_twice = parse_qs(twice, keep_blank_values=True)
    except Exception:
        params_twice = {}
    
    return DoubleEncodingResult(
        is_double_encoded=bool(changes_once or changes_twice),
        raw_query=query_string,
        decoded_once=once,
        decoded_twice=twice,
        structural_changes_once=changes_once,
        structural_changes_twice=changes_twice,
        decoded_params_once=params_once,
        decoded_params_twice=params_twice,
    )


def fully_decode_url(url: str, max_passes: int = 3) -> dict:
    """Fully decode a URL through multiple passes until no further decoding is possible.
    
    Used to normalize adversarially encoded URLs before validation or access control.
    
    Args:
        url: The raw URL string that may contain double/triple encoding.
        max_passes: Maximum number of decoding iterations (prevents infinite loops).
        
    Returns:
        Dictionary with the fully decoded URL, all intermediate steps, and
        a flag indicating whether any structural changes occurred.
    """
    parsed = urlparse(url)
    current_query = parsed.query
    history: list[str] = [current_query]
    has_structural_change = False
    
    for i in range(max_passes):
        next_query = unquote(current_query)
        if next_query == current_query:
            break  # No more decoding possible
        
        history.append(next_query)
        
        # Check if structural characters appeared (new & or new =)
        prev_amps = current_query.count('&')
        curr_amps = next_query.count('&')
        if curr_amps > prev_amps:
            has_structural_change = True
        
        # Check if new key=value pairs emerged
        prev_eqs = current_query.count('=')
        curr_eqs = next_query.count('=')
        if curr_eqs > prev_eqs:
            has_structural_change = True
        
        current_query = next_query
    
    fully_decoded = parsed._replace(query=current_query)
    
    return {
        "original": url,
        "fully_decoded_url": fully_decoded.geturl(),
        "decoding_history": history,
        "total_passes": len(history) - 1,
        "structural_change_detected": has_structural_change,
        "final_params": parse_qs(current_query, keep_blank_values=True),
    }


# --- Usage examples ---

# Normal single-encoding: & in value properly encoded as %26
result = detect_double_encoding("name=hello%26world&page=1")
assert result.is_double_encoded is False  # The & was already a structural char after first decode

# Double-encoded bypass attempt: %2526 becomes & after two decodes
result2 = detect_double_encoding("search=test%2526amp&flag=true")
assert result2.is_double_encoded is True
assert "&" in result2.decoded_once  # %25 → %, so %2526 → %26 (literal string)
# After second decode: %26 → &, so "test%26amp" → "test&amp"
# This means the attacker's parameter "search=test&amp" was hidden behind double encoding


# Full URL decoding for access control validation
attack_url = "https://example.com/api?redirect=http://evil.com?admin=true"
safe_result = fully_decode_url(attack_url)
# This reveals that after decoding, new structural characters may appear

# Double-encoded SSRF attempt: %253A (double-encoded colon) decodes to : after two passes
ssrf_url = "https://example.com/proxy?url=http%253A%252F%252Finternal.server/admin"
ssrf_result = fully_decode_url(ssrf_url)
assert ssrf_result["structural_change_detected"] is True
# After full decode: http://internal.server/admin — a new SSRF target revealed

Pattern 4: OAuth 1.0a Compliant Encoder (RFC 3986)

OAuth 1.0a signature base string requires RFC 3986 percent-encoding, NOT application/x-www-form-urlencoded encoding. The difference matters: form encoding converts . to %2E, - to %2D, _ to %5F, and ~ to +. OAuth explicitly requires the strict RFC 3986 set which leaves these characters unencoded.

import hashlib
import hmac
import base64
import re
from urllib.parse import urlparse, urlencode, quote, parse_qs, urlunparse


# RFC 3986 unreserved characters that must NOT be percent-encoded:
# A-Z a-z 0-9 - _ . ~
RFC3986_UNRESERVED = set(
    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_.~"
)


def rfc3986_encode(value: str) -> str:
    """Encode a string using strict RFC 3986 percent-encoding.
    
    Unlike urllib.parse.quote() or form_urlencode, this function:
    - Leaves ~ unencoded (form encoding converts it to +)
    - Leaves . - _ unencoded (form encoding may alter these)
    - Encodes spaces as %20 (NOT +, which is form-encoding only)
    
    This matches the OAuth 1.0a specification exactly.
    
    Args:
        value: The raw string to encode. May contain any Unicode characters.
        
    Returns:
        RFC 3986 percent-encoded string safe for use in signature base strings.
    """
    # Percent-encode every byte, then restore unreserved characters
    encoded = quote(value.encode('utf-8'), safe='')
    
    # Restore RFC 3986 unreserved character set
    result = []
    i = 0
    while i < len(encoded):
        ch = encoded[i]
        if ch == '%':
            hex_str = encoded[i+1:i+3].upper()
            try:
                char_value = chr(int(hex_str, 16))
                if char_value in RFC3986_UNRESERVED:
                    # Use the original unencoded character
                    result.append(char_value)
                else:
                    # Keep percent-encoded, uppercase hex
                    result.append('%' + hex_str)
                i += 3
            except (ValueError, IndexError):
                result.append(ch)
                i += 1
        else:
            result.append(ch)
            i += 1
    
    return ''.join(result)


def oauth_normalize_parameters(params: dict[str, str], sort: bool = True) -> str:
    """Normalize OAuth 1.0a parameters into a sorted, encoded string.
    
    Per RFC 5849 Section 3.4.1.3.2:
    1. Percent-encode both keys and values with RFC 3986 encoding
    2. Sort by key (byte-order), then by value for duplicate keys
    3. Join with & using key=value format
    
    Args:
        params: Raw parameter dictionary from the HTTP request.
        sort: Whether to sort parameters (required for OAuth signature).
        
    Returns:
        Normalized parameter string suitable for signature base string calculation.
    """
    encoded_items: list[tuple[str, str]] = []
    
    for key, value in params.items():
        enc_key = rfc3986_encode(key)
        enc_value = rfc3986_encode(value)
        encoded_items.append((enc_key, enc_value))
    
    if sort:
        # Sort by encoded key first, then by encoded value
        encoded_items.sort(key=lambda item: (item[0], item[1]))
    
    return '&'.join(f"{key}={value}" for key, value in encoded_items)


def build_oauth_signature_base_string(
    http_method: str,
    base_url: str,
    params: dict[str, str]
) -> str:
    """Build the OAuth 1.0a signature base string per RFC 5849 Section 3.4.1.1.
    
    The signature base string has three components joined by &:
    1. HTTP method (GET, POST, etc.) — uppercased
    2. Base URL — scheme + host + path, with normalized query parameters removed
    3. Normalized parameters — RFC 3986-encoded, sorted key=value pairs
    
    Args:
        http_method: HTTP method string (e.g., "GET", "POST").
        base_url: Full request URL including query string.
        params: All OAuth and request parameters combined.
        
    Returns:
        The signature base string ready for HMAC-SHA1 signing.
    """
    # Normalize method
    method = http_method.upper()
    
    # Parse the URL and normalize it per OAuth spec
    parsed = urlparse(base_url)
    
    # Lowercase scheme and host
    normalized_host = f"{parsed.scheme.lower()}://{parsed.netloc.lower()}"
    
    # Remove default ports (80 for http, 443 for https)
    if (parsed.scheme == 'http' and parsed.port == 80) or \
       (parsed.scheme == 'https' and parsed.port == 443):
        normalized_host = f"{parsed.scheme.lower()}://{parsed.hostname.lower()}"
    
    # Normalize path — ensure it starts with /
    path = parsed.path if parsed.path else '/'
    
    url_base = f"{normalized_host}{path}"
    
    # Encode parameters using strict RFC 3986 (not form-urlencoded)
    params_string = oauth_normalize_parameters(params)
    
    # Join the three components
    return '&'.join([method, rfc3986_encode(url_base), rfc3986_encode(params_string)])


def generate_oauth_signature(
    http_method: str,
    base_url: str,
    params: dict[str, str],
    consumer_secret: str,
    token_secret: str = ''
) -> str:
    """Generate an OAuth 1.0a HMAC-SHA1 signature for a request.
    
    This implements the core signing algorithm used by Twitter API, 
    Stripe, GitHub, and most other OAuth 1.0a providers.
    
    Args:
        http_method: HTTP method (GET, POST, DELETE, etc.).
        base_url: The full request URL.
        params: Combined OAuth credentials + request parameters.
        consumer_secret: Your application's OAuth consumer secret key.
        token_secret: The authorized user's token secret (empty for initial requests).
        
    Returns:
        Base64-encoded HMAC-SHA1 signature string.
    """
    base_string = build_oauth_signature_base_string(http_method, base_url, params)
    
    signing_key = '&'.join([
        rfc3986_encode(consumer_secret),
        rfc3986_encode(token_secret)
    ])
    
    # HMAC-SHA1 signature
    key_bytes = signing_key.encode('utf-8')
    msg_bytes = base_string.encode('utf-8')
    signature = hmac.new(key_bytes, msg_bytes, hashlib.sha1).digest()
    
    # Base64 encode the binary signature
    return base64.b64encode(signature).decode('ascii')


# --- Usage example: Twitter API OAuth 1.0a signing ---

oauth_params = {
    "oauth_consumer_key": "jpPtK3mFsY...",
    "oauth_token": "12345678-AbCdEf...",
    "oauth_signature_method": "HMAC-SHA1",
    "oauth_timestamp": "1609459200",
    "oauth_nonce": "jKlMnOpQrStUvWxYz",
    "oauth_version": "1.0",
    "status": "Hello, world! & more text",  # Ampersand in parameter value
}

signature = generate_oauth_signature(
    http_method="POST",
    base_url="https://api.twitter.com/1.1/statuses/update.json",
    params=oauth_params,
    consumer_secret="your_consumer_secret_here",
    token_secret="your_token_secret_here",
)
# The ampersand in "Hello, world! & more text" is correctly handled:
# - rfc3986_encode converts it to %26 (NOT +)
# - The signature base string uses this exact encoding

Pattern 5: Go SafeURL Struct with QueryEscape and Fragment Handling

Go's net/url package provides QueryEscape() which produces RFC 3986-compliant encoding (same as OAuth). Use it for query parameters. For fragment handling, Go does NOT automatically encode fragments — you must use PathEscape() or manual encoding before appending to the URL string.

package urlsecurity

import (
	"fmt"
	"net/url"
	"strings"
)

// SafeURL provides a type-safe way to construct URLs with proper encoding,
// preventing ampersand injection and double-encoding vulnerabilities.
type SafeURL struct {
	Scheme   string
	Host     string
	Path     string
	Query    map[string][]string
	Fragment string
}

// NewSafeURL creates a SafeURL from its components. The scheme defaults to "https"
// if empty, and the path defaults to "/" if empty.
func NewSafeURL(scheme, host, path string) *SafeURL {
	s := &SafeURL{
		Scheme: scheme,
		Host:   host,
		Path:   path,
		Query:  make(map[string][]string),
	}
	if s.Scheme == "" {
		s.Scheme = "https"
	}
	if s.Path == "" {
		s.Path = "/"
	}
	return s
}

// AddQuery adds a single query parameter with safe encoding.
// Values are encoded using url.QueryEscape (RFC 3986 compatible),
// which correctly encodes & as %26, preventing parameter injection.
func (s *SafeURL) AddQuery(key, value string) {
	s.Query[key] = append(s.Query[key], value)
}

// AddQueryMulti adds a key that can have multiple values (e.g., ?tag=a&tag=b).
// Each value is independently encoded, preserving the multi-value semantics.
func (s *SafeURL) AddQueryMulti(key string, values ...string) {
	for _, v := range values {
		s.Query[key] = append(s.Query[key], v)
	}
}

// SetFragment sets the URL fragment (anchor text). The fragment is NOT
// automatically encoded by url.URL.String(), so we encode it manually.
// Fragments are client-side only and never sent to the server.
func (s *SafeURL) SetFragment(fragment string) {
	// Encode special characters in fragment: space, #, ?, %, &, etc.
	// Use PathEscape for general encoding, then fix remaining needs
	s.Fragment = url.PathEscape(fragment)
}

// Build returns the fully constructed URL string with all components
// properly percent-encoded per RFC 3986.
func (s *SafeURL) Build() string {
	u := &url.URL{
		Scheme:   s.Scheme,
		Host:     s.Host,
		Path:     s.Path,
		RawQuery: s.buildEncodedQuery(),
	}

	if s.Fragment != "" {
		u.Fragment = s.Fragment
	}

	return u.String()
}

// buildEncodedQuery constructs the query string with proper encoding.
// url.Values.Encode() uses application/x-www-form-urlencoded which encodes
// spaces as +. We use manual QueryEscape to stay RFC 3986 compliant.
func (s *SafeURL) buildEncodedQuery() string {
	var parts []string
	for key, values := range s.Query {
		encodedKey := url.QueryEscape(key)
		for _, value := range values {
			encodedValue := url.QueryEscape(value)
			parts = append(parts, fmt.Sprintf("%s=%s", encodedKey, encodedValue))
		}
	}

	if len(parts) == 0 {
		return ""
	}

	// Sort for deterministic output (important for OAuth signing)
	sortStrings(parts)
	return strings.Join(parts, "&")
}

// --- BAD: Manual string concatenation in Go — ampersand injection vulnerability ---

func badURLBuild(baseURL string, params map[string]string) string {
	queryParts := make([]string, 0, len(params))
	for k, v := range params {
		// ❌ VULNERABLE: No encoding of user input
		queryParts = append(queryParts, fmt.Sprintf("%s=%s", k, v))
	}
	return baseURL + "?" + strings.Join(queryParts, "&")
}

// --- BAD: Using url.Values.Encode() which converts spaces to + instead of %20 ---

func badURLEncode(baseURL string, params map[string][]string) string {
	values := make(url.Values)
	for k, v := range params {
		values[k] = v  // url.Values.Encode() uses form-urlencoded encoding
	}
	return baseURL + "?" + values.Encode()
}

// --- GOOD: Cross-parser validation for security-critical URLs ---

// ParseAndCompare verifies that two different URL parsers agree on the structure.
// This detects parser inconsistency bugs that can lead to security bypasses.
func ParseAndCompare(urlStr string) (*ParseResult, error) {
	if urlStr == "" {
		return nil, fmt.Errorf("url string is empty")
	}

	// Parser 1: Go's net/url.Parse (WHATWG-ish for scheme/host/path)
	goURL, goErr := url.Parse(urlStr)
	
	// For cross-validation with a Python-style parser, we simulate the behavior
	// by parsing query parameters differently and checking for structural mismatches
	
	result := &ParseResult{
		URL:         urlStr,
		GoParsed:    goURL,
		GoError:     goErr,
		StructuralChanges: []string{},
	}

	if goErr != nil {
		return result, nil // Go parser may reject non-standard URLs
	}

	// Check if query string has structural anomalies that differ between
	// raw parsing and parameter parsing
	rawQuery := goURL.RawQuery
	parsedValues, parseErr := url.ParseQuery(rawQuery)
	
	if parseErr != nil {
		result.GoParseError = parseErr
		return result, nil
	}

	// Check for double-encoding in query values
	for key, values := range parsedValues {
		for _, val := range values {
			// If decoding the value again reveals new & or = characters,
			// it's a double-encoding candidate
			doubleDecoded, dErr := url.QueryUnescape(val)
			if dErr == nil && doubleDecoded != val {
				if strings.ContainsRune(doubleDecoded, '&') || 
				   strings.Count(doubleDecoded, '=') > strings.Count(val, '=') {
					result.StructuralChanges = append(result.StructuralChanges,
						fmt.Sprintf("double-encoding in param %q: %s → %s", key, val, doubleDecoded))
				}
			}
		}
	}

	return result, nil
}

type ParseResult struct {
	URL               string
	GoParsed          *url.URL
	GoError           error
	GoParseError      error
	StructuralChanges []string
	DoubleEncoded     bool
}


// sortStrings sorts a slice of strings in place (simple insertion sort for small slices).
func sortStrings(slice []string) {
	for i := 1; i < len(slice); i++ {
		key := slice[i]
		j := i - 1
		for j >= 0 && slice[j] > key {
			slice[j+1] = slice[j]
			j--
		}
		slice[j+1] = key
	}
}


// --- Usage examples ---

func ExampleSafeURL() {
	// Build a safe URL with user-supplied parameters containing special chars
	u := NewSafeURL("https", "api.example.com", "/search")
	u.AddQuery("q", "apple & orange")
	u.AddQuery("page", "1")
	u.SetFragment("results-section-1")
	
	result := u.Build()
	// Result: https://api.example.com/search?page=1&q=apple%20%26%20orange#results-section-1
	// The & in the query value is encoded as %26 — no parameter injection possible

	// SSRF prevention example
	func validateRedirectURL(rawInput string) error {
		parsed, err := url.Parse(rawInput)
		if err != nil {
			return fmt.Errorf("invalid URL: %w", err)
		}
		
		// Check scheme allowlist
		if parsed.Scheme != "https" {
			return fmt.Errorf("only https:// redirects are allowed")
		}
		
		// Check for embedded credentials (user:pass@host) — SSRF vector
		if parsed.User != nil && parsed.User.Username() != "" {
			return fmt.Errorf("URL must not contain embedded credentials")
		}
		
		// Decode path to detect encoded slash attacks
		decodedPath, _ := url.PathUnescape(parsed.Path)
		if decodedPath == "/" || decodedPath == "/admin" {
			return fmt.Errorf("redirect to privileged paths is not allowed")
		}
		
		return nil // Passes all structural checks
	}
}

Pattern 6: Parse-and-Compare Validator (Cross-Library Consistency)

When URLs cross language boundaries or security decisions depend on their structure, verify that two different parsers agree. Mismatches indicate either a malformed URL or an encoding bypass attempt.

from urllib.parse import urlparse, parse_qs, unquote, quote
import json


class ParserInconsistencyError(Exception):
    """Raised when two URL parsers disagree on the structure of the same URL."""
    def __init__(self, url: str, parser_a: str, parser_b: str, differences: list[str]):
        self.url = url
        self.parser_a = parser_a
        self.parser_b = parser_b
        self.differences = differences
        msg = (f"Parser inconsistency for {url!r}: {parser_a} and {parser_b} disagree:\n")
        for diff in differences:
            msg += f"  - {diff}\n"
        super().__init__(msg)


def parse_and_compare(
    url: str,
    tolerance: str = "strict"
) -> dict:
    """Parse the same URL with multiple strategies and detect inconsistencies.
    
    Compares Python's urlparse + parse_qs against a manual tokenization approach
    that mimics how different languages might split the query string. This catches
    parser inconsistency bugs where the same URL produces different parameter sets
    in Go vs. Python vs. Node.js.
    
    Args:
        url: The URL to parse and compare.
        tolerance: "strict" requires exact agreement on all params.
                   "lenient" allows differences in whitespace-only decoding.
    
    Returns:
        Dictionary with both parser results, list of differences, and a
        is_consistent flag. Raises ParserInconsistencyError if strict mode
        and differences found.
    """
    parsed = urlparse(url)
    
    # Parser A: Python urllib.parse (standard behavior)
    params_a = parse_qs(parsed.query, keep_blank_values=True)
    
    # Parser B: Manual tokenization that treats + as literal (not space)
    # This simulates how some parsers handle query strings differently
    def manual_parse_query(query_string: str) -> dict[str, list[str]]:
        """Parse query string manually without auto-decoding + to space.
        
        Most language parsers decode + to space, but OAuth 1.0a and RFC 3986
        signature generation treat + as a literal plus sign. This parser
        mimics that stricter behavior for comparison purposes.
        """
        if not query_string:
            return {}
        
        params: dict[str, list[str]] = {}
        for pair in query_string.split('&'):
            if '=' in pair:
                key, _, value = pair.partition('=')
                # Do NOT decode + here — keep it literal
                decoded_key = unquote(key)
                decoded_value = unquote(value)
            else:
                decoded_key = unquote(pair)
                decoded_value = ''
            
            if decoded_key not in params:
                params[decoded_key] = []
            params[decoded_key].append(decoded_value)
        
        return params
    
    params_b = manual_parse_query(parsed.query)
    
    # Parser C: Double-decode and parse (detects bypass attempts)
    fully_decoded_query = unquote(unquote(parsed.query))
    params_c = parse_qs(fully_decoded_query, keep_blank_values=True)
    
    # Compare parsers
    differences: list[str] = []
    
    # A vs B: + handling difference
    all_keys = set(list(params_a.keys()) + list(params_b.keys()))
    for key in sorted(all_keys):
        val_a = params_a.get(key, [])
        val_b = params_b.get(key, [])
        
        if val_a != val_b:
            differences.append(
                f"Key {key!r}: Python parser={val_a}, manual parser={val_b}"
            )
    
    # A vs C: double-encoding structural changes
    for key in sorted(set(list(params_a.keys()) + list(params_c.keys()))):
        val_a = set(params_a.get(key, []))
        val_c = set(params_c.get(key, []))
        
        if val_a != val_c:
            differences.append(
                f"Double-decode changed key {key!r}: "
                f"single={val_a}, double={val_c}"
            )
    
    is_consistent = len(differences) == 0 or tolerance == "lenient"
    
    result = {
        "url": url,
        "is_consistent": is_consistent,
        "num_differences": len(differences),
        "differences": differences,
        "parser_a_params": params_a,
        "parser_b_params": params_b,
        "parser_c_double_decoded": params_c,
    }
    
    if not is_consistent and tolerance == "strict":
        raise ParserInconsistencyError(url, "urllib.parse", "manual+double-decode", differences)
    
    return result


# --- Usage examples ---

# Consistent URL: standard encoding
url1 = "https://api.example.com/search?q=hello&page=2"
result1 = parse_and_compare(url1)
assert result1["is_consistent"] is True

# Inconsistent URL: + in query value is decoded as space by urllib but kept literal manually
url2 = "https://api.example.com/api?data=hello+world&special=foo+bar"
result2 = parse_and_compare(url2)
# parser_a treats + as space: {"data": ["hello world"], "special": ["foo bar"]}
# parser_b keeps + literal:  {"data": ["hello+world"], "special": ["foo+bar"]}
assert result2["num_differences"] == 2

# Dangerous URL: double-encoding creates new parameters after full decode
url3 = "https://api.example.com/api?token=abc%2526role=admin"
result3 = parse_and_compare(url3)
# Single decode: token value = "abc%26role=admin" (one param)
# Double decode: token value = "abc&role=admin" → two params if re-parsed!
assert result3["parser_c_double_decoded"] == {"token": ["abc"], "role": ["admin"]}


# --- BAD: Blindly trusting a single parser for security decisions ---

def bad_access_control(url: str, allowed_params: set[str]) -> bool:
    """❌ VULNERABLE: Only checks the first-pass parsed parameters."""
    parsed = urlparse(url)
    params = parse_qs(parsed.query)
    
    # Attacker bypasses with double encoding
    # ?token=abc%2526admin=true → single decode: token="abc%26admin=true" (one param, passes check)
    # But after second decode in some downstream parser: two params: token="abc", admin="true"
    for key in params:
        if key not in allowed_params:
            return False
    return True


def good_access_control(url: str, allowed_params: set[str]) -> bool:
    """✅ SAFE: Double-decodes and validates the final parameter structure."""
    parsed = urlparse(url)
    
    # First parse
    first_pass = parse_qs(parsed.query, keep_blank_values=True)
    
    # Full decode pass
    fully_decoded = unquote(unquote(parsed.query))
    second_pass = parse_qs(fully_decoded, keep_blank_values=True)
    
    # All parameters from both passes must be in the allowlist
    all_params = set(list(first_pass.keys()) + list(second_pass.keys()))
    
    for key in all_params:
        if key not in allowed_params:
            return False
    return True

Constraints

MUST DO

Always use language-standard URL builders (urllib.parse / urlencode in Python, URLSearchParams in JavaScript, SafeURL struct in Go) — never concatenate strings to build URLs with query parameters
Percent-encode ampersands as %26 whenever they appear inside query parameter values; a literal & is ALWAYS a parameter separator
Implement double-decoding detection for any URL that reaches access control logic, redirect handlers, or SSRF-sensitive code paths
Validate URL scheme against an explicit allowlist (https://) before following redirects or making server-to-server requests
Cross-validate security-critical URLs by parsing with two different methods and comparing structural results
For OAuth 1.0a signature generation: use url.QueryEscape (Go), quote(..., safe='') then restore unreserved chars (Python), or manual RFC 3986 encoding — never use form-urlencoded (+ for spaces)
Strip default ports from base URLs before generating OAuth signatures (remove :80 for http, :443 for https)
Treat URL fragments as client-only data; never include fragment content in server-side logic or access control decisions
Reject URLs containing encoded null bytes (%00) — they can cause string truncation in C-based parsers

MUST NOT DO

Never construct URLs with f-strings, template literals, or fmt.Sprintf that embed raw user input into the query string — this is an ampersand injection vulnerability
Never perform only a single decode pass on incoming encoded parameters before access control checks — attackers use %2526 to hide structural characters
Never trust URL parsers to produce consistent results across language boundaries without cross-validation — Go's url.Parse, Python's urllib.parse, and Node.js's WHATWG URL Standard handle edge cases differently
Never include user-supplied data directly in OAuth signature base strings without strict RFC 3986 encoding — form-urlencoded conversion changes the signature and causes authentication failures
Never use fragment-based data for server-side routing decisions — fragments are stripped by the browser before the HTTP request reaches your server
Never compare decoded URL components with allowlists using string equality on a single parser's output — use parse-and-compare validation first

Output Template

When this skill is active, produce the following structure:

URL Component Analysis — Map the incoming URL to RFC 3986 components and identify which characters act as structural delimiters in each component. Show the parsed query string and what parse_qs / url.Parse().Query() / URLSearchParams would extract.
Encoding Audit — For every query parameter value, show: raw form, single decode, double decode. Flag any value where a new structural character (&, =, ?) appears after decoding.
Parser Consistency Report — If cross-parser validation was requested, show the parsed parameter maps from each parser and list all differences with severity classification (encoding-only vs. structural change).
Safe URL Builder Code — Provide language-specific safe URL construction code using the patterns above. Include type annotations for Python, class-based builders for JavaScript, and struct methods for Go.
OAuth Encoding Snippet — If OAuth 1.0a is relevant, provide the RFC 3986-compliant encoder function with unreserved character restoration logic.
Double-Decoder Detector — Provide the detect_double_encoding() or equivalent function adapted to the target language.

Related Skills

Skill	Purpose
`ssrf-prevention`	Companion skill for server-side request forgery prevention — URL parsing is one of several vectors; this skill handles the query string layer while SSRF prevention handles redirect/target validation
`input-validation`	General input validation patterns — use alongside for broader parameter sanitization beyond URL-specific concerns
`api-security-patterns`	Broader API security practices — includes OAuth 2.0, JWT validation, rate limiting; overlaps with this skill's OAuth 1.0a section
`data-encoding`	General encoding/decoding patterns for JSON, XML, Base64, etc. — reference when handling encoding in non-URL contexts

Live References

Authoritative documentation links for URL parsing and security best practices.

RFC 3986 — Uniform Resource Identifier (URI): Generic Syntax — The definitive specification for URI component syntax, percent-encoding rules, and delimiter characters
RFC 5849 — OAuth 1.0a — Core Protocol Section 3.4.1.3 (Request Component Preparation) — Precise rules for percent-encoding parameters in OAuth signature base strings
OWASP URL Parsing Security Guide — OWASP guidance on URL canonicalization bypasses and parser inconsistencies
Python urllib.parse Documentation — Reference for urlparse, parse_qs, urlencode, quote, unquote functions
WHATWG URL Standard (Node.js browser URL API) — The standard governing JavaScript's URL and URLSearchParams classes; notes differences from RFC 3986
Go net/url Package Documentation — Reference for Go's URL parsing, QueryEscape, PathEscape, and url.Values encoding behavior
Double-Encoding Vulnerability Research (PortSwigger) — PortSwigger Web Security Academy coverage of double-encoding bypass techniques

MAKE IT PART OF YOUR WORKFLOW

インストール

TERMINAL

npx skills add paulpas/agent-skill-router --skill url-parsing-security

実行前にソースコードとライセンスを確認してください。

ZIP · Complete skill directory with resources and licenses

SKILL.md · Instructions file only

リポジトリ Star: Not synced
ライセンス: MIT
バージョン / ブランチ: HEAD · d2e2ac6f
リポジトリ更新日: Not synced

インストール前にスクリプトと権限を確認してください。

ソースコード