twitter

README

YouTube Feeds

The feed plugin is complicated because we use the user's uploads playlist, and that is sorted by actual upload date instead of publish date.

So say you have this channels videos:

1. vid2 - uploaded 10pm - published 10pm
2. vid1 - uploaded 6pm - published 6pm

Then a video was published (but uploaded a long time ago):

1. vid2 - uploaded 10pm - published 10pm
2. vid1 - uploaded 6pm - published 6pm
3. vid3 - uploaded 5pm - published 11pm

vid3 was published after the latest vidoe but still appears at the bottom. this causes issues as we have no idea when to stop looking now. Currently YAGPDB handles this fine as long as it's not uploaded longer than 50 videos ago, in which case it may or may not catch it.

In the future I'll do a hybrid mode with search. Those super late published videos however will show up in Discord super late. I cannot use search for 100% either because it costs 100 times for api quota to use, meaning it could be up to hours behind.

The feed plugin is complicated because we use the user's uploads playlist, and that is sorted by actual upload date instead of publish date

so say you have this channels videos:

1. vid2 - uploaded 10pm - published 10pm
2. vid1 - uploaded 6pm - published 6pm

Then a video was published (but uploaded a long time ago)

1. vid2 - uploaded 10pm - published 10pm
2. vid1 - uploaded 6pm - published 6pm
3. vid3 - uploaded 5pm - published 11pm

vid3 was published after the latest vidoe but still appears at the bottom. this causes issues as we have no idea when to stop looking now. Currently YAGPDB handles this fine as long as its not uploaded longer than 50 videos ago, in which case it may or may not catch it.

In the future i'll do a hybrid mode with search, those super late published videos however will show up in Discord super late, i cannot use search for 100% either because it costs 100 times for api quota to use, meaning it could be up to hours behind.

Storage layout:

Postgres tables: youtube_guild_subs - postgres - guild_id - channel_id - youtube_channel

youtube_playlist_ids - channel_name PRIMARY - playlist_id

Redis keys:

youtube_subbed_channels - sorted set

key is the channel name score is unix time in seconds when it was last checked

youtube_registered_websub_channels - sorted set

key is the channel name score is unix time in seconds when it expires

At the start of a poll, it uses zrange/zrevrange to grab an amount of entries to process and if they do get processed it updates the score to the current unix time.

youtube_last_video_time:{channel} - string

Holds the time of the last video in that channel we processed, all videos before this will be ignored.

youtube_last_video_id:{channel} - string

Holds the last video id for a channel, it will stop processing videos when it hits this video.

youtube_push_registrations - sorted set

Key is the channel id, value is the time it expires

youtube_currently_adding:{channelid} - set, set when this channel is being added

Documentation

Index

• Constants
• Variables
• func BaseEditHandler(inner web.ControllerHandlerFunc) web.ControllerHandlerFunc
• func RegisterPlugin()
• type ContextKey
• type CtxKey
• type EditForm
• type Form
• type Plugin
  • func (p *Plugin) CheckCredentials()
  • func (p *Plugin) DisableFeed(elem *mqueue.QueuedElement, err error)
  • func (p *Plugin) HandleEdit(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)
  • func (p *Plugin) HandleNew(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)
  • func (p *Plugin) HandleRemove(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)
  • func (p *Plugin) HandleTwitter(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)
  • func (p *Plugin) InitWeb()
  • func (p *Plugin) LoadServerHomeWidget(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)
  • func (p *Plugin) PluginInfo() *common.PluginInfo
  • func (p *Plugin) StartFeed()
  • func (p *Plugin) Status() (string, string)
  • func (p *Plugin) StopFeed(wg *sync.WaitGroup)
  • func (p *Plugin) WebhookAvatar() string

Constants

const TwitterIcon = `` /* 2546-byte string literal not displayed */

Variables

var DBSchemas = []string{`
CREATE TABLE IF NOT EXISTS twitter_feeds (
	id BIGSERIAL PRIMARY KEY,
	guild_id BIGINT NOT NULL,
	created_at TIMESTAMP WITH TIME ZONE NOT NULL,

	twitter_username TEXT NOT NULL,
	twitter_user_id BIGINT NOT NULL,
	channel_id BIGINT NOT NULL,
	text_filter TEXT NOT NULL,
	enabled BOOLEAN NOT NULL
);
`, `
CREATE INDEX IF NOT EXISTS twitter_user_id_idx ON twitter_feeds(twitter_user_id);
`, `
ALTER TABLE twitter_feeds ADD COLUMN IF NOT EXISTS text_filter TEXT NOT NULL DEFAULT '';
`, `
ALTER TABLE twitter_feeds ADD COLUMN IF NOT EXISTS include_replies BOOLEAN NOT NULL DEFAULT false;
`, `
ALTER TABLE twitter_feeds ADD COLUMN IF NOT EXISTS include_rt BOOLEAN NOT NULL DEFAULT true;
`,
}

var PageHTML string

Functions

func BaseEditHandler

func BaseEditHandler(inner web.ControllerHandlerFunc) web.ControllerHandlerFunc

func RegisterPlugin

func RegisterPlugin()

Types

type ContextKey

type ContextKey int

const (
	ContextKeySub ContextKey = iota
)

type CtxKey

type CtxKey int

const (
	CurrentConfig CtxKey = iota
)

type EditForm

type EditForm struct {
	DiscordChannel  int64  `valid:"channel,false"`
	TextFilter      string `valid:"regex,300"`
	IncludeReplies  bool
	IncludeRetweets bool
}

type Form

type Form struct {
	TwitterUser    string `valid:",1,256"`
	DiscordChannel int64  `valid:"channel,false"`
	TextFilter     string `valid:"regex,300"`
	ID             int64
}

type Plugin

type Plugin struct {
	Stop chan *sync.WaitGroup
	// contains filtered or unexported fields
}

func (*Plugin) CheckCredentials

func (p *Plugin) CheckCredentials()

func (*Plugin) DisableFeed

func (p *Plugin) DisableFeed(elem *mqueue.QueuedElement, err error)

func (*Plugin) HandleEdit

func (p *Plugin) HandleEdit(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)

func (*Plugin) HandleNew

func (p *Plugin) HandleNew(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)

func (*Plugin) HandleRemove

func (p *Plugin) HandleRemove(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)

func (*Plugin) HandleTwitter

func (p *Plugin) HandleTwitter(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)

func (*Plugin) InitWeb

func (p *Plugin) InitWeb()

func (*Plugin) LoadServerHomeWidget

func (p *Plugin) LoadServerHomeWidget(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)

func (*Plugin) PluginInfo

func (p *Plugin) PluginInfo() *common.PluginInfo

func (*Plugin) StartFeed

func (p *Plugin) StartFeed()

func (*Plugin) Status

func (p *Plugin) Status() (string, string)

func (*Plugin) StopFeed

func (p *Plugin) StopFeed(wg *sync.WaitGroup)

func (*Plugin) WebhookAvatar

func (p *Plugin) WebhookAvatar() string