qwen35-rp

README

qwen35-rp

Qwen 3.5 Reverse Proxy is a lightweight HTTP reverse proxy that automatically adjusts sampling parameters (temperature, top_p, etc.) and thinking mode based on one of four predefined profiles. It sits between your application and the backend LLM server serving Qwen 3.5 (e.g., vLLM). It also provides a /tokenize endpoint that replaces virtual model names with the backend model name before forwarding to vLLM.

Core Functionality

This proxy's primary purpose is to:

1. Accept requests for four virtual model names (configured via -thinking-general, -thinking-coding, -instruct-general, and -instruct-reasoning), rejecting all other model names with HTTP 400
2. Set appropriate sampling parameters automatically based on one of four profiles (official Qwen-recommended values from Hugging Face):
   • Thinking mode for general tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
   • Thinking mode for precise coding tasks: temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
   • Instruct mode for general tasks: temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
   • Instruct mode for reasoning tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
3. Configure thinking mode by setting chat_template_kwargs.enable_thinking:
   • enable_thinking=true for thinking modes (general and coding)
   • enable_thinking=false for instruct modes (general and reasoning)
4. Rewrite the model name to the actual backend model name (e.g., Qwen/Qwen3.5-397B-A17B-FP8) before forwarding to vLLM
5. Fix vLLM response bugs where non-thinking, non-streaming responses incorrectly place content in reasoning_content or reasoning fields instead of content
6. Enrich /v1/models endpoint by fetching backend models and exposing 4 virtual models with the same metadata (permissions, max_model_len, etc.)
7. Provide a /tokenize endpoint that replaces virtual model names with the backend model name before forwarding to vLLM's /tokenize

Installation

Requirements: Go 1.24.2 or later

go build -o qwen35-rp .

Usage

./qwen35-rp \
  -target "http://127.0.0.1:8000" \
  -served-model "Qwen/Qwen3.5-397B-A17B-FP8" \
  -thinking-general "qwen-thinking-general" \
  -thinking-coding "qwen-thinking-coding" \
  -instruct-general "qwen-instruct-general" \
  -instruct-reasoning "qwen-instruct-reasoning"

Or using environment variables:

export QWEN35RP_TARGET="http://127.0.0.1:8000"
export QWEN35RP_SERVED_MODEL_NAME="Qwen/Qwen3.5-397B-A17B-FP8"
export QWEN35RP_THINKING_GENERAL_MODEL="qwen-thinking-general"
export QWEN35RP_THINKING_CODING_MODEL="qwen-thinking-coding"
export QWEN35RP_INSTRUCT_GENERAL_MODEL="qwen-instruct-general"
export QWEN35RP_INSTRUCT_REASONING_MODEL="qwen-instruct-reasoning"
./qwen35-rp

Configuration

Configure the proxy using command-line flags or environment variables:

Flag	Environment Variable	Default	Description
-listen	QWEN35RP_LISTEN	0.0.0.0	IP address to listen on
-port	QWEN35RP_PORT	9000	Port to listen on
-target	QWEN35RP_TARGET	http://127.0.0.1:8000	Backend target URL
-loglevel	QWEN35RP_LOGLEVEL	INFO	Log level (COMPLETE, DEBUG, INFO, WARN, ERROR)
-served-model	QWEN35RP_SERVED_MODEL_NAME	(required)	Backend model name to use in outgoing requests
-thinking-general	QWEN35RP_THINKING_GENERAL_MODEL	qwen3.5-thinking-general	Name of the thinking-general model (incoming request identifier)
-thinking-coding	QWEN35RP_THINKING_CODING_MODEL	qwen3.5-thinking-coding	Name of the thinking-coding model (incoming request identifier)
-instruct-general	QWEN35RP_INSTRUCT_GENERAL_MODEL	qwen3.5-instruct-general	Name of the instruct-general model (incoming request identifier)
-instruct-reasoning	QWEN35RP_INSTRUCT_REASONING_MODEL	qwen3.5-instruct-reasoning	Name of the instruct-reasoning model (incoming request identifier)
-enforce-sampling-params	QWEN35RP_ENFORCE_SAMPLING_PARAMS	false	Enforce sampling parameters, overriding client-provided values

Enforce Sampling Parameters

By default, the proxy only sets sampling parameters if they are not already present in the request. When -enforce-sampling-params is enabled, the proxy will always override client-provided sampling parameters with the predefined values for the detected mode.

Request Routing

• GET /v1/models: Enriched (fetches backend models, validates served model, exposes 4 virtual models)
• POST /v1/responses: Returns HTTP 501 Not Implemented by design (vLLM's Responses API doesn't support chat_template_kwargs needed to activate thinking mode)
• POST /v1/chat/completions: Transformed (sampling params + thinking mode applied)
• POST /v1/completions: Model name validated and swapped (no sampling params or thinking mode — raw prompt completions bypass the chat template)
• POST /tokenize: Replaces virtual model names with backend model name and forwards to vLLM's /tokenize
• All other paths: Passed through unchanged to the backend

Responses API

The /v1/responses endpoint returns HTTP 501 Not Implemented by design. vLLM's Responses API endpoint does not support chat_template_kwargs, which is required to control Qwen's thinking mode (enable_thinking=true or false). Without this, the proxy cannot apply the four predefined profiles that adjust sampling parameters and thinking behavior. Use the Chat Completions API (/v1/chat/completions) instead.

vLLM Backend Requirements

For full functionality with thinking mode and tool calls using the Chat Completions API, the vLLM backend should be started with the following flags:

--reasoning-parser=qwen3                                  # Required for thinking/reasoning mode
--enable-auto-tool-choice --tool-call-parser=qwen3_coder  # Required for tool/function calls

Note: These flags are only needed when using Chat Completions API (/v1/chat/completions). The Responses API (/v1/responses) is not yet supported.

Tokenize API

The proxy provides a /tokenize endpoint that forwards tokenization requests to vLLM's /tokenize. The proxy replaces virtual model names with the backend served model name, then forwards the request body unchanged. Two modes:

• {"prompt": "..."} — raw text tokenization, forwarded as-is. No chat template is applied.
• {"messages": [...], "tools": [...]} — vLLM applies the model's chat template (apply_chat_template) then tokenizes the result. Messages and tools must be in Chat Completions API format (same as transformers apply_chat_template): a list of dictionaries with role and content keys.

Health Check

• GET /health: Returns {"status":"healthy"} for Docker health checks

Log Levels

The proxy supports the following log levels:

Level	Description
COMPLETE	Most verbose - includes full HTTP request/response dumps
DEBUG	Debug information including parameter application details
INFO	General operational information
WARN	Warning messages
ERROR	Error messages only

When set to COMPLETE, the proxy will log full HTTP request and response bodies, which is useful for debugging but very verbose.

⚠️ Privacy Warning: LLM requests often contain sensitive or personal data (conversation history, personal information, confidential content). The COMPLETE log level will expose all this data in plaintext. Only enable it in secure, non-production environments or ensure logs are properly secured and retained temporarily.

systemd Integration

The proxy includes native systemd support for production deployments:

• Type: notify - The proxy signals readiness to systemd automatically
• Status Updates: Sends periodic status updates to systemd showing processed request counts
• Graceful Shutdown: Properly signals systemd when stopping
• Journald Logging: Structured logging output is compatible with journald

Example systemd unit file:

[Unit]
Description=Qwen 3.5 Reverse Proxy
After=network.target

[Service]
Type=notify
User=qwen35-rp
Group=qwen35-rp
ExecStart=/usr/local/bin/qwen35-rp -served-model "Qwen/Qwen3.5-397B-A17B-FP8" -thinking-general "qwen-thinking-general" -thinking-coding "qwen-thinking-coding" -instruct-general "qwen-instruct-general" -instruct-reasoning "qwen-instruct-reasoning"
Restart=on-failure
Environment=QWEN35RP_LOGLEVEL=INFO

[Install]
WantedBy=multi-user.target

⚠️ Security Best Practice: Always run the proxy under a dedicated, unprivileged user account (e.g., qwen35-rp). Never run as root. Create the user with:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin qwen35-rp
sudo chown qwen35-rp:qwen35-rp /usr/local/bin/qwen35-rp

Graceful Shutdown

The server supports graceful shutdown with a 3-minute timeout to allow in-flight requests to complete. Send SIGINT or SIGTERM to initiate shutdown. When running under systemd, the proxy will automatically signal the service manager when ready and during shutdown.

License

MIT License - see LICENSE file for details.