similar

README

Similar Manga Recommendations

This repo has both the scraping and matching utilities to find mangas which are close in content to others. The idea is to create a recommendation system outside MangaDex since there isn't one and thus allow for users to discover other content. Right now it is pretty rudimentary, but seems to give ok results most of the time. If you want to use it, please download Neko and enable recommendations inside the app or use the Chrome Extension.

Setup / Dependencies

Client was generated using swagger. You will need to setup a golang workspace, and then run the following commands. Only manga need to be downloaded / scraped from mangadex to be able to perform similar manga identification.

go get golang.org/x/oauth2
go get github.com/antihax/optional
go get github.com/james-bowman/nlp
go get github.com/caneroj1/stemmer
go get github.com/PuerkitoBio/goquery

Runtime Instructions

1. Run ./1_cache_mangas to get local list of all mangas on the site (28 hours for 73k manga)
2. Run ./2_calc_similar to generate similarity index between manga (6 hours)
3. Run ./3_calc_mappings to export external ID to mangadex UUID mapping files (2 minutes)
4. Run ./4_calc_mu_ids convert MangaUpdate ids into uniform "new" ids for their new api (27 hours)
5. Run python utils/create_sqlite.py to generate mapping file for Neko

Manga Links Data

Key	Related site	URL	URL details
al	anilist	https://anilist.co/manga/`{id}`	Stored as id
ap	animeplanet	https://www.anime-planet.com/manga/`{slug}`	Stored as slug
bw	bookwalker.jp	https://bookwalker.jp/`{slug}`	Stored has "series/{id}"
mu	mangaupdates	https://www.mangaupdates.com/series.html?id=`{id}`	Stored has id
nu	novelupdates	https://www.novelupdates.com/series/`{slug}`	Stored has slug
kt	kitsu.io	https://kitsu.io/api/edge/manga/`{id}` or https://kitsu.io/api/edge/manga?filter[slug]={slug}	If integer, use id version of the URL, otherwise use slug one
amz	amazon	N/A	Stored as full URL
ebj	ebookjapan	N/A	Stored as full URL
mal	myanimelist	https://myanimelist.net/manga/{id}	Store as id
raw	N/A	N/A	Stored as full URL, untranslated stuff URL (original language)
engtl	N/A	N/A	Stored as full URL, official english licenced URL