ncmrecognize

README

NeteaseCloudMusic-Audio-Recognize-Reverse

A pure-Go reimplementation of the NetEase Cloud Music browser plugin's afp.wasm audio-fingerprint algorithm, plus the reverse-engineering notes.

The repository ships no WebAssembly file and has no WebAssembly runtime dependency: the fingerprint algorithm is implemented entirely in Go. The original afp.wasm was only used as the reference during reverse engineering.

import ncm "github.com/XiaoMengXinX/NeteaseCloudMusic-Audio-Recognize-Reverse"

c := new(ncm.Client) // Duration 0 = whole recording, adapted to 2-16 s
resp, err := c.RecognizeFile(ctx, "song.ogg") // wav / mp3 / flac / ogg (vorbis, opus)
fmt.Println(string(resp.Raw))                  // raw API JSON; resp.Code, resp.Data too

Install / CLI

go install github.com/XiaoMengXinX/NeteaseCloudMusic-Audio-Recognize-Reverse/cmd/ncmrecognize@latest

ncmrecognize song.mp3                                # whole recording, adaptive
ncmrecognize -duration 6 song.mp3                    # requested 6-second window
ncmrecognize -offset 30 -duration 7.5 song.mp3       # custom offset/length

The CLI prints the chosen fingerprint length and form duration to stderr:

fingerprint: 8.000s audio, form duration 9s (pad)

Adaptive clip length (2–16 s)

The NetEase endpoint only accepts certain duration values. The client adapts automatically: it keeps the full clip when possible, otherwise moves it to the nearest usable length (on a tie it pads with silence rather than truncating) and sends the matching duration.

Client.Duration controls the requested window:

• 0 (default) – all available audio, capped/adapted to 2–16 s;
• 6 – requested 6-second window (clamped to the available audio);
• any other value – that many seconds, adapted to a supported length.

Client.Offset sets the start offset for Recognize / RecognizeFile; RecognizePCM takes the offset as its fourth argument. Lower-level APIs:

// Whole recording. 0 = all available.
resp, err := new(ncm.Client).RecognizeAdaptive(ctx, pcm, 48000, 0, 0)

// Explicit length / offset:
resp, err = new(ncm.Client).RecognizeFileAdaptiveAt(ctx, "recording.mp3", 30, 7.5)

// Inspect the chosen duration:
fp, info, err := ncm.FingerprintAdaptive(ctx, pcm, 48000, 0, 7.5)
// info = {Duration: 9, Used: 8.0, Requested: 7.5, Action: "pad"}
resp, err = new(ncm.Client).MatchDuration(ctx, fp, info.Duration)

Empirical server rules (see REVERSE_ENGINEERING.md):

• floor(duration) must be one of 2, 3, 4, 6, 9, 12, 15.
• The fingerprint's actual/embedded duration must be within ±1 s of the form duration.
• Practical range is about 1.5–16 s; shorter clips may have too little content, and >16 s is truncated to 16 s. 6 s remains a safe default.

The adaptive chooser keeps the full audio whenever a legal form duration is within ±1 s; only the gaps (7–8, 10–11, 13–14 s) are adjusted, using the smallest change. On a tie it pads with silence instead of truncating, since truncation can remove the only distinctive part of a short clip.

Supported input

Pure Go decoders, detected by content: WAV (PCM), MP3, FLAC, Ogg Vorbis, Ogg Opus (via pion/opus). AAC/M4A/WMA are not supported; decode them yourself and pass PCM to FingerprintAdaptive / Client.RecognizePCM.

Client.Recognize / RecognizeFile* decode only the audio the fingerprint needs (offset + 16 s + 1 s), so memory stays bounded however long the input is. Decode decodes everything; DecodeLimit(data, maxSeconds) stops after maxSeconds.

Fingerprint / Match keep the fixed 6-second behaviour; use FingerprintAdaptive / MatchDuration for custom lengths.

How it works

The algorithm was recovered from the original Emscripten/embind afp.wasm build of the "hyai" fingerprint SDK (hyai_1.2.0_client_1.0.0). This repository contains only the pure-Go pipeline:

1. Input – channel 0, resampled to 8 kHz. The web client only accepts 48 kHz and point-samples every 6th sample; other rates are an extension and use linear interpolation. Resampling uses the same non-fused float32 arithmetic as the wasm pipeline, so fingerprints are identical on arm64 and amd64. (This deliberately does not reproduce the old Go wrapper's architecture-dependent FMA result for non-48 kHz input.)
2. Spectrogram – Hamming window 0.54 - 0.46*cos(2*pi*n/2047), FFT length 2048, frame length 2048, hop 160 (20 ms); KISS FFT radix 4/2 in float32; stored value is |X|² for bins 0…1024.
3. Log spectrum – ln(sqrt(power)) using musl's logf, then subtract the global mean.
4. Peak picking – 3×3 local maxima on bins 25…1023 (100–4000 Hz); after a peak the scan skips one time frame (a quirk of the original).
5. Interference filter – keep a peak if its ±10 bin / ±5 frame neighborhood mean is > 2, or if peak - mean > 1; drop peaks with value ≤ 0.
6. Rival suppression – drop a peak if any value in its ±30 bin / ±8 frame neighborhood is strictly larger.
7. Serialization – header "hyai_1.2.0_client_1.0.0", u32 0, f32 0, f32 duration, "FPVER" + version, "Peak" + count, then 12-byte records {i32 bin, i32 frame, f32 logLevel} sorted by (frame, bin).
8. Encryption – zlib level 6, PKCS#7 padding, AES-128-ECB with the fixed key 4B97221F27F02907, then base64.

Go's compress/zlib produces a valid but not byte-identical deflate stream compared to the wasm's zlib 1.2.11. The decrypted/decompressed fingerprint data matches the original implementation and the API accepts the Go-produced stream.

See REVERSE_ENGINEERING.md for the recovered wasm addresses, function indices, parameter values and validation methodology (historical analysis; the wasm itself is not shipped here).

Tests

go test ./...

testdata/ is git-ignored because it contains optional recorded test clips, except for the small synthetic sine.b64 golden fixture. The wasm golden fingerprint test therefore always runs; the format/smoke tests still skip without the optional local clips. No WebAssembly runtime is involved.

Credits

• akinazuki/NeteaseCloudMusic-Audio-Recognize
• MicroCBer/BeatSaverSongData
• The hyai audio-fingerprint SDK embedded in the NetEase plugin.

License

GPL-3.0, see LICENSE.

Documentation

Overview

Package ncmrecognize recognizes songs through NetEase Cloud Music's audio-match API. It is a pure-Go reimplementation of the browser plugin's afp.wasm fingerprint algorithm: no cgo, no WebAssembly runtime, no Node.js.

Index

• Constants
• func Fingerprint(ctx context.Context, pcm []float32, sampleRate int, offsetSec float64) (string, error)
• type Audio
  • func Decode(data []byte) (*Audio, error)
  • func DecodeLimit(data []byte, maxSeconds float64) (*Audio, error)
• type Client
  • func (c *Client) Match(ctx context.Context, fingerprint string) (*Response, error)
  • func (c *Client) MatchDuration(ctx context.Context, fingerprint string, duration int) (*Response, error)
  • func (c *Client) Recognize(ctx context.Context, audio []byte) (*Response, error)
  • func (c *Client) RecognizeAdaptive(ctx context.Context, pcm []float32, sampleRate int, offsetSec, seconds float64) (*Response, error)
  • func (c *Client) RecognizeFile(ctx context.Context, path string) (*Response, error)
  • func (c *Client) RecognizeFileAdaptive(ctx context.Context, path string, seconds float64) (*Response, error)
  • func (c *Client) RecognizeFileAdaptiveAt(ctx context.Context, path string, offsetSec, seconds float64) (*Response, error)
  • func (c *Client) RecognizePCM(ctx context.Context, pcm []float32, sampleRate int, offsetSec float64) (*Response, error)
• type Engine
  • func NewEngine(ctx context.Context) (*Engine, error)
  • func (e *Engine) Close(ctx context.Context) error
  • func (e *Engine) ExtractQueryFP(ctx context.Context, pcm []float32) ([]byte, error)
  • func (e *Engine) Fingerprint(ctx context.Context, pcm []float32, sampleRate int, offsetSec float64) (string, error)
  • func (e *Engine) FingerprintAdaptive(ctx context.Context, pcm []float32, sampleRate int, offsetSec, seconds float64) (string, FingerprintInfo, error)
• type FingerprintInfo
  • func FingerprintAdaptive(ctx context.Context, pcm []float32, sampleRate int, offsetSec, seconds float64) (string, FingerprintInfo, error)
• type Response

Constants

const (
	// AdaptiveKeep means all requested audio was used unchanged.
	AdaptiveKeep = "keep"
	// AdaptivePad means the window was extended to a longer usable length.
	// If the source ends before that length, the remainder is silence.
	AdaptivePad = "pad"
	// AdaptiveTruncate means the tail was dropped to reach a usable length.
	AdaptiveTruncate = "truncate"
)

const (
	// FingerprintRate is the sample rate expected by the fingerprint algorithm.
	FingerprintRate = 8000
	// FingerprintSeconds is the clip length used by the NetEase web client.
	FingerprintSeconds = 6
)

Functions

func Fingerprint

func Fingerprint(ctx context.Context, pcm []float32, sampleRate int, offsetSec float64) (string, error)

Fingerprint returns the base64 fingerprint ("rawdata") for the given mono PCM samples, starting offsetSec seconds in. It takes FingerprintSeconds seconds of audio, resampled to 8 kHz like the web client: 48 kHz input is point-sampled every 6th sample; other rates use linear interpolation.

Types

type Audio

type Audio struct {
	PCM        []float32
	SampleRate int
}

Audio is decoded audio: the first channel as float32 in [-1, 1].

func Decode

func Decode(data []byte) (*Audio, error)

Decode decodes audio data, detecting the format by content.

Supported: WAV (PCM), MP3, FLAC, Ogg Vorbis, Ogg Opus. Anything else (AAC/M4A, WMA, ...) must be decoded by the caller and passed as PCM to Fingerprint / Client.RecognizePCM.

func DecodeLimit

func DecodeLimit(data []byte, maxSeconds float64) (*Audio, error)

DecodeLimit is Decode that stops after the first maxSeconds of audio, so memory stays bounded for long inputs. maxSeconds <= 0 decodes everything.

type Client

type Client struct {
	HTTP   *http.Client // defaults to http.DefaultClient
	Engine *Engine      // defaults to a shared engine

	// Duration is the requested clip length in seconds used by Recognize,
	// RecognizeFile and RecognizePCM. Zero (the default) means "all available
	// audio", adapted to the endpoint's supported lengths (2-16 s). Set it to
	// 6 for a 6-second window (clamped to the available audio), or to any
	// other value for a custom length.
	Duration float64
	// Offset is the start offset in seconds used by Recognize and
	// RecognizeFile. RecognizePCM takes its offset as an argument.
	Offset float64
}

Client calls the NetEase audio match API.

func (*Client) Match

func (c *Client) Match(ctx context.Context, fingerprint string) (*Response, error)

Match sends an already computed 6-second fingerprint to the API.

func (*Client) MatchDuration

func (c *Client) MatchDuration(ctx context.Context, fingerprint string, duration int) (*Response, error)

MatchDuration sends an already computed fingerprint with an explicit "duration" form value. Use FingerprintAdaptive to obtain a compatible duration for clips that are not 6 seconds long.

func (*Client) Recognize

func (c *Client) Recognize(ctx context.Context, audio []byte) (*Response, error)

Recognize fingerprints the audio (wav/mp3/flac/ogg bytes) and queries the API.

func (*Client) RecognizeAdaptive

func (c *Client) RecognizeAdaptive(ctx context.Context, pcm []float32, sampleRate int, offsetSec, seconds float64) (*Response, error)

RecognizeAdaptive decodes nothing: it fingerprints the given PCM with an adaptive clip length and sends the matching duration to the API. seconds is the clip length starting at offsetSec; <= 0 means all available audio (capped at 16 s).

func (*Client) RecognizeFile

func (c *Client) RecognizeFile(ctx context.Context, path string) (*Response, error)

RecognizeFile is Recognize for a file path.

func (*Client) RecognizeFileAdaptive

func (c *Client) RecognizeFileAdaptive(ctx context.Context, path string, seconds float64) (*Response, error)

RecognizeFileAdaptive is RecognizeAdaptive for a file path. seconds <= 0 uses the whole file (bounded by the endpoint's maximum query length).

func (*Client) RecognizeFileAdaptiveAt

func (c *Client) RecognizeFileAdaptiveAt(ctx context.Context, path string, offsetSec, seconds float64) (*Response, error)

RecognizeFileAdaptiveAt is RecognizeFileAdaptive with a start offset.

func (*Client) RecognizePCM

func (c *Client) RecognizePCM(ctx context.Context, pcm []float32, sampleRate int, offsetSec float64) (*Response, error)

RecognizePCM queries the API with mono PCM samples (first channel) starting at offsetSec. The clip length is c.Duration (0 = all available audio, 6 = a 6-second window clamped to the available audio).

type Engine

type Engine struct{}

Engine computes NetEase audio fingerprints. It is stateless and safe for concurrent use.

func NewEngine

func NewEngine(ctx context.Context) (*Engine, error)

NewEngine returns a fingerprint engine. The context is accepted for compatibility; no resources are allocated.

func (*Engine) Close

func (e *Engine) Close(ctx context.Context) error

Close is a no-op kept for API compatibility.

func (*Engine) ExtractQueryFP

func (e *Engine) ExtractQueryFP(ctx context.Context, pcm []float32) ([]byte, error)

ExtractQueryFP computes the raw (encrypted) fingerprint for exactly FingerprintRate*FingerprintSeconds samples of 8 kHz mono PCM.

func (*Engine) Fingerprint

func (e *Engine) Fingerprint(ctx context.Context, pcm []float32, sampleRate int, offsetSec float64) (string, error)

Fingerprint is like the package-level Fingerprint but uses this engine.

func (*Engine) FingerprintAdaptive

func (e *Engine) FingerprintAdaptive(ctx context.Context, pcm []float32, sampleRate int, offsetSec, seconds float64) (string, FingerprintInfo, error)

FingerprintAdaptive fingerprints a clip of the given length (seconds) at offsetSec and adapts it to a duration accepted by the match endpoint. Pass seconds <= 0 to use all audio available after offsetSec (capped at 16 s).

The returned FingerprintInfo.Duration must be sent as the API "duration" field; Client.MatchDuration / Client.RecognizeAdaptive do this automatically.

type FingerprintInfo

type FingerprintInfo struct {
	// Duration is the value to send as the API's "duration" form field.
	Duration int
	// Used is the length of audio actually fingerprinted, in seconds. It is
	// also the duration embedded in the fingerprint itself.
	Used float64
	// Requested is the requested length after clamping to the available audio.
	Requested float64
	// Action is AdaptiveKeep, AdaptivePad or AdaptiveTruncate.
	Action string
}

FingerprintInfo describes how an adaptive fingerprint was produced.

func FingerprintAdaptive

func FingerprintAdaptive(ctx context.Context, pcm []float32, sampleRate int, offsetSec, seconds float64) (string, FingerprintInfo, error)

FingerprintAdaptive is the package-level version of (*Engine).FingerprintAdaptive.

type Response

type Response struct {
	Code int             `json:"code"`
	Data json.RawMessage `json:"data"`
	Raw  json.RawMessage `json:"-"`
}

Response is the API response. Raw always holds the unmodified JSON.