IRC log for #sourcefu, 2018-05-08

http://sourcefu.com

All times shown according to UTC.

Time	Nick	Message
02:59	prologic	anyone around?
03:00	prologic	anyone interested in discussing pub/sub designs? specifically as a rest/websocket service
03:00	prologic	https://github.com/prologic/msgbus
03:00	prologic	I need to (eventually; but could do it now) solve for the case of "queue depth" and "leaking queues"
03:01	prologic	meaning that right now; as long as there are no explicit clients using the GET /<topic> API; queues for <topic> will grow unbounded
03:01	prologic	as messages are delivered to topics; they are first published to all subscribers (if any) and enqueued in per-topic queues
03:02	prologic	without some kind of "ack" from subscribers there is no way to delete already processed items from the queue
03:02	prologic	assuming that should be the design at all
03:02	prologic	1) ack messages from subscribers
03:03	prologic	2) messages have ttls (per topic configured) with garbage collection
03:03	prologic	3) explicit deletion of old items in queue
14:13		bear joined #sourcefu
14:27	dotplus_	if the system has "exactly once" or "at least once" semantics, you have to accept that (1) ack-from-subscriber) is the only possibility and deal with the possibility of unbounded queue-growth elsewhere in the overall architecture. If you can accept that some messages will not be delivered (either because they don't matter or because they will be retried), then you can accept a ttl/gc approach.
15:30	prologic	dotplus_ that's precisely what I'm tossing up between
15:30	prologic	so far for my use-case this is working pretty nicely
15:30	prologic	a) as a generic webhook (using it to receive docker hub notifications)
15:31	prologic	and b) as an aggregator of prometheus alerts which you can then do interesting things with (for now I just alert on irc)
15:31	prologic	and 3rd and upcoming use-case is actually going to be the queue for a "job scheduler"
15:45	prologic	I could actually support both options Via configuration
15:45	prologic	But of course that would be mutually exclusive I think
15:46	prologic	Or maybe not come to think of it
15:47	prologic	A —enable-ttl option
17:47	bear	this isn't always a popular opinion... but I find anything that is a "Job" system ends up abusing queue systems to make things work
17:48	bear	IMO it's a more robust design to have the job scheduler be a user of the queue but ensure that you don't try to use the queue to maintain state
17:48	bear	maintain state of the job that is
17:48	bear	use the queue all you want to deliver tasks and jobs to who ever is listening, but then use a data store for state
17:49	bear	(yes, being explicit and maybe a bit judgy, but a lot of job systems I see in the wild make this mistake
17:49	bear	)
18:36		codex joined #sourcefu
19:54	prologic	I agree
19:54	prologic	What do you think about what I said above though in terms of design choices?
19:59	prologic	bear: And what do you mean by storing state in a queue system? I haven’t seen too many examples of abuse and curious like that myself
20:05	bear	I'm liking what you had above
20:05	bear	the biggest I have with storing state is either the consumer or the producer of a message being able to know what the message state is from it's view of the queue
20:06	bear	reality and that rarely agree - consumers of a queue may hold a message but be unaware that it's ttl died and it was requeued, producer of a message may be unaware that a message was dropped or handled but the ack was dropped
20:06	bear	especially when people think they can apply CAP to message queues
20:16	prologic	Right
20:17	prologic	In that case I think I’ll support both
20:17	prologic	Configurable TTLs with auto GC
20:17	prologic	API to purge queues
20:18	prologic	Which are themselves like an explicit ack
20:18	prologic	Make sense ?
20:24	bear	that makes sense to have for queue management - yes
20:24	bear	then it allows your queue consumer/producers to establish the behaviour they want
20:25	bear	one thing I've wanted a queue environment to have is a callback/notice to happen when a TTL hits - so that both sides get an event that something has changed

http://sourcefu.com