sisimai

README

[!NOTE] Sisimai is a library provided as a Go package that decodes various formats of bounce emails and outputs structured data necessary for analysis, such as destination addresses and bounce reasons. Besides Go, it is very useful for grasping the occurrence status of bounces by obtaining decoded results in any environment as long as it is a language that can read JSON, such as PHP, Java, Python, and Rust.

• README-JA(日本語)
• What is Sisimai
  • The key features of Sisimai
  • Command line demo
• Setting Up Sisimai
  • System requirements
  • Install and Build
• Usage
  • Basic usage
  • Convert to JSON
  • Callback feature
  • Output example
• Differences between Go vs. Others
• Contributing
  • Bug report
  • Emails could not be decoded
• Other Information
  • Related sites
  • See also
• Author
• Copyright
• License

What is Sisimai

Sisimai is a Go package, is a library that decodes complex and diverse bounce emails and outputs the results of the delivery failure, such as the reason for the bounce and the recipient email address, in structured data. It is also possible to output in JSON format.

The key features of Sisimai

• Decode email bounces to structured data
  • Sisimai provides detailed insights into bounce emails by extracting 26 key data points.[^1]
    • Essential information: Timestamp, Origin
    • Sender information: Addresser, SenderDomain,
    • Recipient information: Recipient, Destination, Alias
    • Delivery information: Action, ReplyCode, DeliveryStatus, Command
    • Bounce details: Reason, DiagnosticCode, DiagnosticType, FeedbackType, FeedbackID, HardBounce
    • Message details: Subject, MessageID, ListID,
    • Additional information: DecodedBy, TimezoneOffset, Lhost, Rhost, Token, Catch
  • Output formats
    • struct (sisimai/sis.Fact)
    • JSON (by using encoding/json)
• Easy to Install, Use.
  • $ go get -u libsisimai.org/sisimai/v5@latest
  • import "libsisimai.org/sisimai/v5"
• High Precision of Analysis
  • Support 59 MTAs/MDAs/ESPs
  • Support Feedback Loop Message(ARF)
  • Can detect 36 bounce reasons

[^1]: The callback function allows you to add your own data under the Catch field.

Command line demo

The following screen shows a demonstration of Dump function of libsimai.org/sisimai package at the command line using Go(go-sisimai) and jq command.

Setting Up Sisimai

System requirements

More details about system requirements are available at Sisimai | Getting Started page.

• Go 1.17.0 or later
• No external dependencies (excluding standard library) since v5.2.1

Install and Build

Install

$ mkdir ./sisimai
$ cd ./sisimai
$ go mod init example.com/sisimaicli
go: creating new go.mod: module example.com/sisimaicli

$ go get -u libsisimai.org/sisimai/v5@latest
go: added libsisimai.org/sisimai/v5 v5.3.0

$ cat ./go.mod
module example.com/sisimaicli

go 1.20

require (
	libsisimai.org/sisimai/v5 v5.3.0 // indirect
)

Build

For example, the following sisid.go: sisimai decoder is a minimal program that decodes bounce emails and outputs them in JSON format.

$ vi ./sisid.go

// sisimai decoder program
package main
import "os"
import "fmt"
import "libsisimai.org/sisimai/v5"

func main() {
    path := os.Args[1]
    args := sisimai.Args()

    sisi, nyaan := sisimai.Rise(path, args)
    for _, e := range *sisi {
        cv, _ := e.Dump()
        fmt.Printf("%s\n",cv)
    }
    if len(*nyaan) > 0 { fmt.Frpintf(os.Stderr, "%v\n", *nyaan) }
}

Once you have written sisid.go, build an executable binary with go build command.

$ CGO_ENABLED=0 go build -o ./sisid ./sisid.go

Specifying the path to a bounce email (or Maildir/) as the first argument will output the decoded results as a JSON string.

$ ./sisid ./path/to/bounce-mail.eml | jq
{
  "addresser": "michistuna@example.org",
  "recipient": "kijitora@google.example.com",
  "timestamp": 1650119685,
  "action": "failed",
  "alias": "contact@example.co.jp",
  "catch": null,
  "decodedby": "Postfix",
  "deliverystatus": "5.7.26",
  "destination": "google.example.com",
  "diagnosticcode": "host gmail-smtp-in.l.google.com[64.233.187.27] said: This mail has been blocked because the sender is unauthenticated. Gmail requires all senders to authenticate with either SPF or DKIM. Authentication results: DKIM = did not pass SPF [relay3.example.com] with ip: [192.0.2.22] = did not pass For instructions on setting up authentication, go to https://support.google.com/mail/answer/81126#authentication c2-202200202020202020222222cat.127 - gsmtp (in reply to end of DATA command)",
  "diagnostictype": "SMTP",
  "feedbackid": "",
  "feedbacktype": "",
  "hardbounce": false,
  "lhost": "relay3.example.com",
  "listid": "",
  "messageid": "hwK7pzjzJtz0RF9Y@relay3.example.com",
  "origin": "./path/to/bounce-mail.eml",
  "reason": "authfailure",
  "rhost": "gmail-smtp-in.l.google.com",
  "replycode": "550",
  "command": "DATA",
  "senderdomain": "google.example.com",
  "subject": "Nyaan",
  "timezoneoffset": "+0900",
  "token": "5253e9da9dd67573851b057a89cbcf41293e99bf"
}

Usage

Basic usage

libsisimai.org/sisimai.Rise() function provides the feature for getting decoded data as *[]sis.Fact struct, occurred errors as *[]sis.NotDecoded from bounced email messages as the following.

package main
import "os"
import "fmt"
import "libsisimai.org/sisimai/v5"

func main() {
    path := os.Args[1]     // go run ./sisid /path/to/mailbox or maildir/
    args := sisimai.Args() // sis.DecodingArgs{}

    // If you also need analysis results that are "delivered" (successfully delivered),
    // set `true` into the "Delivered" option for the Rise() function as shown below.
    args.Delivered = true

    // If you also need analysis results that show a "vacation" reason, set `true` into
    // the "Vacation" option for the Rise() function as shown in the following code.
    args.Vacation  = true

    // sisi is a pointer to []sis.Fact
    sisi, nyaan := sisimai.Rise(path, args)
    if len(*sisi) > 0 {
        for _, e := range *sisi {
            // e is a sis.Fact struct
            fmt.Printf("- Sender is %s\n", e.Addresser.Address)
            fmt.Printf("- Recipient is %s\n", e.Recipient.Address)
            fmt.Printf("- Bounced due to %s\n", e.Reason)
            fmt.Printf("- With the error message: %s\n", e.DiagnosticCode)

            cv, _ := e.Dump()     // Convert the decoded data to JSON string
            fmt.Printf("%s\n",cv) // JSON formatted string the jq command can read
        }
    }
    // nyaan is a pointer to []sis.NotDecoded
    if len(*nyaan) > 0 { fmt.Fprintf(os.Stderr, "%v\n", *nyaan) }
}

Convert to JSON

The following code snippet illustrates the use of the libsisimai.org/sisimai.Dump() function to obtain decoded bounce email data in JSON array format.

package main
import "os"
import "fmt"
import "libsisimai.org/sisimai/v5"

func main() {
    path := os.Args[1]
    args := sisimai.Args()

    json, nyaan := sisimai.Dump(path, args)
    if json != nil && *json != "" { fmt.Printf("%s\n", *json)   }
    if len(*nyaan) > 0 { fmt.Fprintf(os.Stderr, "%v\n", *nyaan) }
}

Callback feature

sis.DecodingArgs have the Callback0 and Callback1 fields for keeping callback functions. The former is called at message.sift() for dealitng email headers and entire message body. The latter is called at the end of each email file processing inside of sisimai.Rise().

The results generated by the callback functions are accessible via Catch field defined in sis.Fact.

Callback0: For email headers and the body

The function set in args.Callback0 is called at message.sift().

package main
import "os"
import "fmt"
import "strings"
import "libsisimai.org/sisimai/v5"

func main() {
    path := os.Args[1]     // go run ./sisid /path/to/mailbox or maildir/
    args := sisimai.Args() // sis.DecodingArgs{}

    args.Callback0 = func(arg *sisimai.CallbackArg0) (map[string]interface{}, error) {
        // - This function allows users to add custom processing to the email before parsing.
        // - For example, you can extract the delivery ID from the "X-Delivery-App-ID:" header
        //   and store it in the data map like this: data["x-delivery-app-id"] = "neko22-2".
        // - The library executes this function and assigns the return value to the Catch field of the Fact struct.
        // - Users can then retrieve and access the data from Catch by type assertion in the caller.
        name := "X-Delivery-App-ID"
        data := make(map[string]interface{})
        data[strings.ToLower(name)] = ""

        if arg.Payload != nil && len(*arg.Payload) > 0 {
            mesg := *arg.Payload
            if p0 := strings.Index(mesg, "\n" + name + ":"); p0 > 0 {
                cw := p0 + len(name) + 2
                p1 := strings.Index(mesg[cw:], "\n")
                if p1 > 0 {
                    data[strings.ToLower(name)] = mesg[cw + 1:cw + p1]
                }
            }
        }
        return data, nil
    }

    sisi, _ := sisimai.Rise(path, args)
    if len(*sisi) > 0 {
        for _, e := range *sisi {
            // e is a sis.Fact struct
            re, as := e.Catch.(map[string]interface{})
            if as == false { continue }
            if ca, ok := re["x-delivery-app-id"].(string); ok {
                fmt.Printf("- Catch[X-Delivery-App-ID] = %s\n", ca)
            }
        }
    }
}

Callback1: For each email file

The function set in args.Callback1 is called at sisimai.Rise() function for dealing each email file after decoding each bounce message.

package main
import "os"
import "io/ioutil"
import "libsisimai.org/sisimai"

func main() {
    path := os.Args[1]     // go run ./sisid /path/to/mailbox or maildir/
    args := sisimai.Args() // sis.DecodingArgs{}

    args.Callback1 = func(arg *sisimai.CallbackArg1) (bool, error) {
        // - This function defines custom operations that the user wants to perform on the parsed email file.
        // - For example, you can write the contents of the parsed bounce email to a file in /tmp/.
        if nyaan := ioutil.WriteFile("/tmp/copy.eml", []byte(*arg.Mail), 0400); nyaan != nil {
            return false, nyaan
        }
        return true, nil
    }

    // sisi is a pointer to []sis.Fact
    sisi, nyaan := sisimai.Rise(path, args)
    if len(*sisi) > 0 {
        for _, e := range *sisi {
            // e is a sis.Fact struct
            ...
        }
    }
}

More information about the callback feature is available at Sisimai | How To Parse - Callback Page.

Output example

[
  {
    "addresser": "michistuna@example.org",
    "recipient": "kijitora@google.example.com",
    "timestamp": 1650119685,
    "action": "failed",
    "alias": "contact@example.co.jp",
    "catch": null,
    "decodedby": "Postfix",
    "deliverystatus": "5.7.26",
    "destination": "google.example.com",
    "diagnosticcode": "host gmail-smtp-in.l.google.com[64.233.187.27] said: This mail has been blocked because the sender is unauthenticated. Gmail requires all senders to authenticate with either SPF or DKIM. Authentication results: DKIM = did not pass SPF [relay3.example.com] with ip: [192.0.2.22] = did not pass For instructions on setting up authentication, go to https://support.google.com/mail/answer/81126#authentication c2-202200202020202020222222cat.127 - gsmtp (in reply to end of DATA command)",
    "diagnostictype": "SMTP",
    "feedbackid": "",
    "feedbacktype": "",
    "hardbounce": false,
    "lhost": "relay3.example.com",
    "listid": "",
    "messageid": "hwK7pzjzJtz0RF9Y@relay3.example.com",
    "origin": "./path/to/bounce-mail.eml",
    "reason": "authfailure",
    "rhost": "gmail-smtp-in.l.google.com",
    "replycode": "550",
    "command": "DATA",
    "senderdomain": "google.example.com",
    "subject": "Nyaan",
    "timezoneoffset": "+0900",
    "token": "5253e9da9dd67573851b057a89cbcf41293e99bf"
  }
]

Differences between Go and Others

The following table show the differences between the Go version of Sisimai and the other language versions: p5-sisimai and rb-sisimai.

Features

Features	Go	Perl	Ruby / JRuby
System requirements	1.17 -	5.26 -	2.4 - / 9.2 -
Dependencies (Except standard libs)	0 packages	2 modules	1 gem
Supported character sets	UTF-8 only	UTF-8,etc. [^2]	UTF-8,etc.[^3]
Source lines of code	9,000 lines	9,900 lines	9,800 lines
The number of tests	144,000 tests	320,000 tests	410,000 tests
The number of bounce emails decoded/sec [^4]	2900 emails	750 emails	620 emails
License	2 Clause BSD	2 Caluse BSD	2 Clause BSD
Commercial support	Available	Available	Available

[^2]: Character sets supported by Encode and Encode::Guess modules [^3]: Character sets supported by String#encode method [^4]: macOS Monterey/1.6GHz Dual-Core Intel Core i5/16GB-RAM/Go 1.22/Perl 5.30/Ruby 2.6.4

Contributing

Bug report

Please use the issue tracker to report any bugs.

Emails could not be decoded

Bounce emails that couldn't be decoded by the latest version of sisimai are saved in the repository set-of-emails/to-be-debugged-because/sisimai-cannot-parse-yet. If you have found any bounce email cannot be decoded using sisimai, please add the email into the directory and send Pull-Request to this repository.

Other Information

Related sites

• @libsisimai | Sisimai on Twitter (@libsisimai)
• LIBSISIMAI.ORG | SISIMAI | MAIL ANALYZING INTERFACE | DECODING BOUNCES, BETTER AND FASTER.
• Sisimai Blog | blog.libsisimai.org
• Facebook Page | facebook.com/libsisimai
• GitHub | github.com/sisimai/go-sisimai
• Perl verison | Perl version of Sisimai
• Ruby version | Ruby version of Sisimai
• Fixtures | set-of-emails - Sample emails for "make test"

See also

• README-JA.md - README.md in Japanese(日本語)
• RFC3463 - Enhanced Mail System Status Codes
• RFC3464 - An Extensible Message Format for Delivery Status Notifications
• RFC3834 - Recommendations for Automatic Responses to Electronic Mail
• RFC5321 - Simple Mail Transfer Protocol
• RFC5322 - Internet Message Format

Author

@azumakuniyuki and sisimai development team

Copyright

Copyright (C) 2014-2025 azumakuniyuki and sisimai development team, All Rights Reserved.

License

This software is distributed under The BSD 2-Clause License.

Documentation

Overview

sisimai is a library that decodes complex and diverse bounce emails and outputs the results of the delivery failure, such as the reason for the bounce and the recipient email address, in structured data. It is also possible to output in JSON format. More information are available at https://libsisimai.org

Index

• func Args() *sis.DecodingArgs
• func Dump(path string, args *sis.DecodingArgs) (*string, *[]sis.NotDecoded)
• func Factor() *sis.Fact
• func Reason() map[string]string
• func Rise(path string, args *sis.DecodingArgs) (*[]sis.Fact, *[]sis.NotDecoded)
• func Version() string
• type CallbackArg0
• type CallbackArg1
• type CfParameter0
• type CfParameter1

Functions

func Args

func Args() *sis.DecodingArgs

Args returns a pointer to sis.DecodingArgs as the 2nd argument of the Rise function.

func Dump

func Dump(path string, args *sis.DecodingArgs) (*string, *[]sis.NotDecoded)

Dump returns decoded data as a JSON string.

Arguments:
  - path (string):            Path to an mbox, Maildir/, or "STDIN" for standard input.
  - args (*sis.DecodingArgs): Options and callback functions for decoding bounce messages
Returns:
  - (*string):                Decoded data as a JSON string array
  - (*[]sis.NotDecoded):      List of occurred errors

func Factor

func Factor() *sis.Fact

Factor retuns empty (initialized with zero values) sis.Fact instance.

func Reason

func Reason() map[string]string

Reason returns the list of bounce reasons sisimai can detect.

func Rise

func Rise(path string, args *sis.DecodingArgs) (*[]sis.Fact, *[]sis.NotDecoded)

Rise is a function for decoding bounce mails in a mailbox or a Maildir/.

Arguments:
  - path (string):            Path to an UNIX mbox, Maildir/, or "STDIN" for standard input.
  - args (*sis.DecodingArgs): Options and callback functions for decoding bounce messages
Returns:
  - (*[]sis.Fact):            List of successfully decoded bounce messages
  - (*[]sis.NotDecoded):      List of occurred errors

func Version

func Version() string

Version returns the version number of sisimai such as "v5.2.0" or "v5.2.1p22".

Types

type CallbackArg0

type CallbackArg0 = sis.CallbackArg0

type CallbackArg1

type CallbackArg1 = sis.CallbackArg1

type CfParameter0

type CfParameter0 = sis.CfParameter0

type CfParameter1

type CfParameter1 = sis.CfParameter1