greptilian logo

IRC log for #openknot, 2015-07-27

Open Knot Communications Platform - https://github.com/openknot

| Channels | #openknot index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
00:27 prologic where are out thoughts at now?
00:27 prologic Are we still heading in the right direction? :P
01:31 pdurbin prologic: good job with the smtp thing
01:43 prologic well it's only the first step
01:43 prologic baby steps
01:44 prologic it'll serve as a way to get email delivery into something else
01:44 prologic now whilst we could just use out-of-the-box email servers and inboxes
01:44 prologic I don't really feel like having to deal with that
01:44 prologic let's just accept the email(s) directly
01:44 prologic and shove them into Redis
01:44 prologic but on that I'm thinking the next "thing" should be an API
01:44 prologic smtp -> API -> Store
01:44 prologic same with IRC
01:44 prologic IRC -> API -> Store
01:45 prologic and clients access the same API
12:32 prologic (broker)
12:32 prologic prologic@daisy
12:32 prologic Mon Jul 27 22:36:07
12:32 prologic ~/openknot/broker
12:32 prologic $ docker-compose ps
12:32 prologic Name        Command    State         Ports
12:32 prologic ---------------------------​---------------------------
12:32 prologic broker_smtpd_1   smtpd.py   Up      0.0.0.0:25->25/tcp
12:58 prologic so if anyone is awake/interested/etc
12:58 prologic I have a fully functioning mail server
12:58 prologic that will accept email @circuitsframework.com
13:03 prologic fantastic
13:03 prologic even gmail (tested from my google apps account) will deliver it email
13:03 prologic anyone else feel like testing?
13:30 prologic I'm actually rethinking (no phun intended) the choice of data store here
13:30 prologic I'm thinking the kind of thing we're trying to build *is* actually more realtime
13:30 prologic and rethinkdb may actually fit that bill a lot better
13:30 prologic data comes in; gets processes very quickly
13:31 prologic and gets sent out to multitudes of clients/brokers in realtime
13:31 prologic which means I also need to build an smtp client as well
13:32 prologic fortunately there is an official docker image of this at https://registry.hub.docker.com/_/rethinkdb/
13:32 prologic so it's quite trivial to spin up and do some testing around the kinds of data we expect to store and delivery in realtime
13:33 pdurbin I've been meaning to try rethinkdb.
13:43 prologic well what do you think so far?
13:43 prologic this prototype was never going to take a night or two :)
13:44 prologic let's not kid ourselves :)
13:44 prologic feel free to send an email to <anything>@circuitsframework.com
13:44 prologic I'll let you know if it reached the test instance
13:44 prologic I'll even paste the log if you like :)
13:44 dotplus I don't think that the datastore needs to be chosen for strength in realtime. At least not yet. The message router is the logical centre not the datastore. The message router will be receiving messages either directly (i.e. where the message content is the correct JSON) or via an API server (to convert the original format into appropriate JSON).
13:45 dotplus I would suggest using whatever is easiest and just making sure that it's well enough documented/defined that it's easy to swap out later if we feel the need.
13:45 prologic the logs so far: https://gist.github.com/prologic/eb76bde6d992e62072a4
13:46 prologic :)
13:46 prologic ahh right
13:46 prologic I see what you're saying dotplus
13:46 prologic and I *think* I agree
13:47 prologic and you've just helped solidify the next thing I need to build
13:47 prologic the "router"
13:47 dotplus and then the datastore (service) is just one more client to the messsage router. It's only special in the sense that it subscribes to *every* message queue
13:47 prologic the other things are brokers
13:47 prologic they take messages as input/output
13:47 dotplus dingdingding
13:47 prologic routers decide where they should go and possibly process them
13:47 prologic well
13:48 prologic in this case I don't need to use a data store at all really
13:48 prologic if I take full advantage of what circuits can do here
13:48 prologic I can completely make this distributed and decentarlized
13:48 prologic a "data store" can be plugged in later for persistence
13:48 prologic by interesting the "router" messages
13:48 prologic err
13:48 prologic intercepting*
13:49 dotplus that's fine. But the persistence is required for the web archive component.
13:49 prologic precisely
13:49 prologic but as you just said
13:49 prologic I think we're saying the same thing
13:49 prologic the "data store" can be a client
13:49 dotplus and any other component that will provide search functionality (which might be a null set. TBD)
13:49 prologic the "web client" depends on the "data store"
13:49 dotplus definitely
13:49 prologic but by the same token
13:50 prologic a web client can be buitl witout persistence as such
13:50 prologic by just listening/reacting to the realtime messages
13:50 prologic the store is useful for archiveal purposes and history
13:50 prologic I guess every part of this *should* be pluggable
13:50 prologic with well defined interfaces
13:50 dotplus the data store is a client. it is a destination. it will also be a source, but that can come later.
13:50 dotplus I agree in spades
13:51 prologic wonderful :)
13:51 prologic so I still need to:
13:51 prologic define a "message" event
13:51 prologic a standard form that brokers push to routers
13:52 prologic i.e: the interface between brokers/rouers/clients
13:52 prologic then start work on a "router"
13:52 dotplus a) because that kind of modularity is the right way to build stuff in order to tame the complexity b) because that eases the development of any individual component c) because it will make it easier to rebuild a component or change the tech/stack that we've chosen to use for that particular functionality
13:52 prologic right now my only broker (smtp) just dumps incoming messages (emails) to stdout
13:53 prologic oh yeah yeah
13:53 prologic I completely and wholeheartedly agree
13:53 prologic d'uh :P
13:53 prologic I'm the author of a library/framework that really strongly facilitates such architecture and design of any app/system :)
13:54 prologic so far we have three componetns I guess
13:54 prologic and one subsystem
13:54 prologic let's quick hash out what defines a message?
13:55 prologic baring in mind it has to encompass lots of different sources/types of messages
13:55 prologic email/smtp and irc arleady look quite similar in fact
13:56 prologic SMTP/Email: peer, mailfrom, rcpttos, message
13:56 prologic IRC: peer, source, target, message
13:56 dotplus yes. in my mind the 2 next steps are 1) define a basic json schema for messages 2) build a message router that can accept messages in that schema, has a (configurable) list of destinations, and (for the mvp a no-op) that is able to decide which messages get punted to which destinations.
13:57 dotplus If we make target plural then we can say targets == rcpttos and mailfrom == source
13:58 prologic destinations in this case being gateways and clients?
13:58 prologic so a gateway here can be an irc server
13:58 prologic whilst a client can be an irc client
13:58 prologic yeah yeah great I like it; targets
13:58 dotplus no, destination in that sentence means a queue to which various gateways/clients can subscribe
13:58 prologic I guess we need a timestamp too
13:59 prologic ahh right
13:59 prologic so broker -> router -> dest/queue/store
14:00 prologic I guess this'll just be en event sink for now
14:00 prologic with room to put something in-between
14:02 dotplus not sure we're completely on the same page here. The persistence archive is just another endclient (it's special in the sense that it's subscribed to *every* message queue).
14:03 dotplus I think the structure is broker <-> router <-> broker.
14:03 dotplus some of those might be unidirectional
14:05 dotplus but the basic idea is that a smtp broker for exmaple is able to receive a message from MTAs or perhaps MUAs, transform it into JSON, and send it to a message queue. And in other direction which is simpler, it's subscribed to certain message queues from where it collects messages and does JSON->SMTP (i.e. sends mail)
14:09 dotplus an irc broker is similar. it is an irc client (could be a bot, could be a special irc daemon) that receives message from irc clients (or irc networks). it converts those messages to appropriate JSON and sends them to messages queues in the router. it is also subscribed to (other) messages queues to go in the other direction.
14:11 dotplus the persistence broker(s) is/are similar, but unidirectional. Each persistence broker is subscribed to message queues and knows how to store the JSON messages that it receives in rethinkdb or redis or whatever.
14:15 dotplus It's becoming clearer to me know and I think that a big part of the complexity is going to lie in deciding how we want to create/categorize the queues. Does a message queue map to an email address? a mailing list? or an individual email thread? to an IRC channel? to a twitter #tag?
14:19 dotplus this is part of why I think that rabbitmq would be suitable for the message router. its major strength is being a collection of queues; it's really simple, very robust, very performant and already written. the only downside is that it's in erlang and therefore pulls in a ton of dependencies.
14:27 dotplus also, that would make that particular interface extremely simple and easy to (re)implement. The interface/interaction becomes merely "be an AMQP client publishing to X set of queues, subscribing to Y set of queues. You will read/write from/to those queues according to the following JSON Schema".
14:32 prologic hmm
14:32 prologic I believe we *are* on the same page as such
14:33 prologic https://gist.github.com/prologic/4a455d8f1d11339a7d51
14:33 prologic I think though that RabbitmQ only solves the queue side of things nicely
14:33 dotplus I'm pretty certain that the basic architecture or broker <-> router <-> broker is correct. I think that whether the router is rabbitmq and/or the transport is amqp is still up for grabs. I think it's probably the best way to go.
14:34 prologic yeah sure it has a daemon/servers and you can talk it oit via libraires and what not
14:34 dotplus yes that's all it's for.
14:34 prologic but if all we need to be to swap that parciaulr part out is AMQP compliant we can look at that later?
14:34 dotplus correct
14:34 prologic so really we still need to build that "smart" router
14:34 prologic that does all the decision making
14:34 prologic RabbitMQ won't help there
14:35 dotplus that's not a router. that's just a collection of rules.
14:36 dotplus and I think the rules probably going to be protocol dependent, so those rules belong in the brokers
14:38 dotplus the smarts are to answer 2 questions "which queues should I publish this (original, incoming, from a user's smtp/irc/xmpp/whatever client) message to?" and "which queues do I subscribe to in order to receive messages?"
14:41 dotplus btw, can I change the name of your repo to openknot/smtpbroker?
15:01 dotplus should we be building this for python3?
15:02 dotplus or 2.7+?
15:16 bear in my thoughts it's always broker <--> router
15:16 bear with both sending and receiving being brokers
15:16 bear so yes, it's a huge message sink
15:17 bear which is why I like redis for the router part - it's the simplest way to get queues that can be subscribed to
15:17 bear brokers will need a set of filters or rules that are defined to let them work - the second part of this is deciding if said rules engine lives in it's own component or is a part of each broker
15:20 dotplus I think redis for the router will be fine to start with. we can and probably still should use AMQP to talk to those queues.
15:21 dotplus that way we don't come up with something unnecessarily custom
15:22 prologic sorry was afk for a bit
15:22 prologic I think there's something even better (perhaps) than rabbitMQ
15:22 prologic MQTT
15:22 prologic but I think I agree with the whole broker <-> router <-> broker arch
15:27 dotplus yeah, I'm somewhat aware of mqtt. it's not an alternative to rabbitmq, it's an alternative to AMQP. an MQTT broker (might) be an alternative to rabbitmq, although rabbitmq actually has a mqtt plugin.
15:27 prologic interesting
15:28 prologic I've actually used mqtt a bit myself too
15:28 prologic it has queues and pub/sub
15:28 prologic and afaik is much (probably) better suited to this
15:28 prologic lightweight deps, lightweight transport
15:28 prologic designed for flakey networks
15:28 prologic i.e: mobile
15:29 bear I haven't looked much at it - my biggest concern is ease of setup believe it or not
15:29 prologic really?
15:29 dotplus Overall, I think the ubiquity/maturity of AMQP is more important than anything else here at the beginning.
15:29 prologic hah
15:29 bear something that is complicated to setup/run will fail hard as the key router part
15:30 prologic to be clear; when we say "router"
15:30 prologic we're talking about a daemon/service that'll accept messages of our MessageSchema over some transport
15:30 prologic and shove it into a queue
15:30 prologic for other brokers to subscribe to
15:31 dotplus yes
15:31 bear router == "rules engine"
15:31 dotplus no
15:31 bear because it's more than just a pub/sub queue
15:31 bear it has to decide on the incoming message what secondary queues an item needs
15:31 prologic okay
15:31 bear or are you going to hang a rules engine off of the main message bus?
15:31 prologic we seem to differ on this a bit then
15:31 prologic also: https://registry.hub.docker.com/u/prologic/mosquitto/
15:32 prologic in regards to your concern of setting up mqtt
15:32 prologic I'm kind of on dotplus's side here with the way he's thinking about broker/rules
15:33 prologic where say we write (or I) a plugin for charla
15:33 bear to be frank about my bias - I once supported and ran an amqp java stack 10 yrs ago and it failed horribly for many many reasons
15:33 prologic that irc daemon I started a while back
15:33 prologic to plugin/talk-to openknot's router
15:33 bear i'm flexible where the rules engine lives
15:33 prologic when an email/message/tweet/irc/whatever comes in it uses that information to create channels and move users around, etc
15:34 prologic well it's just that I think the "rules" are likely tightly "coupled" with the "broker"
15:34 bear but is it the presence of the data on the bus that causes a new channel to be formed or the broker
15:34 dotplus the pub/sub queue service that prologic described is the router. rules are "merely" configuration that dictate where (that is, which queues) messages get read from/written to.
15:34 prologic but it could also just redeliver new messages into other queues
15:35 prologic the broker decides
15:35 bear and how do other consumers learn of them
15:35 prologic what's on the bus is irrlevant I guess
15:35 dotplus some of that config (i.e. the rules) will be controlled by brokers, some by the auth/auth service.
15:35 prologic consumers as in user clients?
15:36 bear other brokers when they are reading from the bus  are consumers
15:36 bear (or proxies for them)
15:36 bear the irc broker will be a proxy for any target that "lives" in the irc address space
15:38 bear basically - as long as the bus carries any payload with a given set of topics/targets then we can work out many combinations of brokers, consumers, rules
15:39 prologic http://blogs.vmware.com/vfabric/2013/02/choosing-your-messaging-protocol-amqp-mqtt-or-stomp.html
15:39 prologic I kind of tend to think we should use MQTT :)
15:40 dotplus right. We don't need to settle on all the details to make progress. At least not the progress that we've identified as coming next. We *MUST* decide on a protocol, with the understanding that if we want to change our minds later we can, but that it will require redoing a certain amount of development work.
15:41 bear I wasn't even thinking about that yet - it's why I said redis - until some of the things are worked out it should be as simple as possible of a json delivery mechanism as we can get
15:41 bear see, i'm going the complete opposite way - choosing a protocol so soon locks us in to a lot of small details
15:41 dotplus it doens't make sense to push forward on dev until we agree on a protocol. which at the moment seems to be AMQP vs MQTT.
15:42 prologic how about this
15:42 prologic I'll push forward with prototying this out anyway
15:42 prologic and leave it possible to change/swap-out
15:42 prologic :)
15:43 dotplus if prologic starts writing an smtp broker and an irc broker and I start writing the router and/or a persistence broker, these components need to agree on how to talk before we can test anything:)
15:43 bear if asked to pick amqp or mqtt, i'll pick mqtt every day :)
15:43 prologic as will I
15:44 prologic so we may as well decide and move on then
15:44 prologic mqtt + jsonschema?
15:44 bear but if the concern is the wiring of things to this bus, then all that means is that everything goes thru a shim send and a shim receive helper item
15:44 bear because the more "interesting" part isn't the bus, but the structure of the payload that will be shoved onto the bus
15:47 prologic okay so quickly then
15:47 prologic https://gist.github.com/prologic/9faa5d62435b5125bcfb
15:47 * bear will pick his preferred in order to stop wrangling about plumbing
15:47 prologic here's a draft schema
15:47 bear python + json + mqtt
15:47 dotplus I'm not sure yet. I have 3 concerns about mqtt: does it have the flexibility we'll need for queues/groups of queues/etc? are the implementations (client libraries and server daemons) solid, mature, still maintained (or are we going to be stuck with betamax)? Do the implementations of the servers properly support distributedness?
15:48 dotplus If you guys think the answer to those 3 questions is "yes" or "good enough", then I'm ok with MQTT
15:48 prologic well as you know I'm quite biased towards circuits
15:48 prologic which is unto itself it's own pub/sub if you will
15:48 bear given that this is going to be more of a personal client/server/daemon/proxy - distributedness is a lower level concern
15:48 prologic so it's not hard for my to shim/stub out the parts necessary for replacing with mqtt/amqp later
15:49 prologic in fact; if everything that makes up the imporatnt componetns of this sytem are written in Python and circuits
15:49 prologic the interfaces will be much easier to manage
15:49 prologic between systems that is
15:49 bear yea, in my mind it's going to be a small helper object for both of us - you with circuits and my proclivity to using subprocess queue()'s
15:49 prologic yeah
15:49 prologic if we want to use mqtt/amqp later we can
15:50 prologic just replace that component
15:50 prologic that was acting as the bus/queue
15:51 prologic dotplus, yes; yes and yes
15:51 dotplus ok, then mqtt is fine by me.
15:52 prologic it's a very solid paltform, support for hierarchical queues/topics and solid client libraires for many langauges/frameworks/platforms
15:52 prologic it's actually used by facebook for their mobile app :)
15:52 prologic then it's settled
15:52 prologic I can still do this without mqtt and swap that in later
15:53 prologic thoughts on the schema I preseneted?
15:53 prologic I generated/created it with http://jsonschema.net/#/
15:53 bear my concern is that message needs to be down one level
15:53 prologic some of the types need to be a bit more concrete  though
15:54 dotplus I think your schema looks good. Are you assuming that thread/inreplyto information is part of the definition of a target?
15:54 prologic I'm pretty sure JSON Schema has types and meta types
15:54 bear because messages will need to be routed - so source/target will apply to a message which may have it's own source/target
15:54 prologic dotplus, yes
15:54 bear threading will imply that an irc/email message has a parent of another message's id
15:54 prologic trying to encapsulate as many types of messages in a single payload I guess
15:55 prologic within reason
15:55 prologic there will be some communications protocols/platforms that may be "too hard" to bother with
15:55 prologic ahh
15:55 prologic message id
15:55 prologic we shoudl add that in?
15:55 bear agreed, those will need a gateway to translate
15:56 bear as soon as a message is created on inbound processing it should be assigned an id
15:56 dotplus prologic: to help us understand where you're going, what do you see as valid values for the strings that "message/type"?
15:56 bear are you thinking that type == gateway ?
15:56 bear i.e. IRC, SMTP, XMPP, ...
15:57 dotplus so "type" is the kind of broker that the message came in on?
15:57 prologic agreed
15:57 prologic just added id btw
15:57 bear I would suggest s/type/gateway/
15:57 bear since IMO type is always so overloaded of a phrase
15:57 prologic yeah
15:57 bear or broker or ///
15:57 bear err ???
15:58 prologic type; enum of IRC, SMTP, Twitter, SMTP, etc
15:58 prologic any kind of broker we've implemented and defined
15:58 bear +1
15:58 prologic hmm
15:59 prologic what about Protocol
15:59 prologic as opposed to Type
15:59 dotplus I prefer protocol
15:59 prologic okay done
15:59 bear :)
16:00 dotplus protocol is about how communication is done, broker sounds like the communicator itself
16:00 bear I can see some of the differences in style between the three of us in how things are named and in what part each of us focuses on first :)
16:00 dotplus yep me too. and from diversity arises Greatness.
16:01 bear protocol is about the message; gateway/broker is about who acts on the message
16:01 prologic https://gist.github.com/prologic/9faa5d62435b5125bcfb
16:02 prologic haha
16:02 prologic bear, care to share? :)
16:02 prologic interesting to see other's pov of one's self :P
16:02 prologic so I think we have an honest to goodness schema
16:02 bear I would not even define a schema right now - and just pass json of (id, timestamp, source, target, data)
16:03 prologic hmm
16:03 bear but i've also come to learn how to be extremely flexible in the early days of new systems
16:03 prologic why not?
16:04 bear because each piece is so unknown as to what is needed and IMO having a schema assigns an authority to the schema more than it really needs/has currently
16:04 dotplus we can change the schema later on, a definition now just gives us something to work together with
16:04 bear but I've also learned that schema's are a communication tool for the devs in this stage more so the tech
16:04 bear so it's just a different style - i'm very much an adhoc kind of coder :)
16:05 dotplus prologic: :%s/puropose/purpose/
16:05 dotplus Can you commit that to the repo and make a link to it from https://github.com/openknot/openknot/blob/master/related.md#implementation-notes
16:05 bear yea, +1 to the schema as is
16:05 dotplus link & explanation preferably
16:06 prologic haha
16:06 prologic okay
16:06 prologic let's use this as a guide then
16:06 bear I may be suggesting later an envelope schema
16:06 prologic I'll commit it to a repo somewhere soon
16:06 dotplus bear: quite possibly
16:06 dotplus a repo? no, into openknot/openknot
16:06 prologic yeah there
16:07 prologic I think I'll head to bed shortly
16:07 prologic it's ~2am :)
16:07 dotplus also, can you change openknot/broker to openknot/smtpbroker
16:07 prologic I can
16:07 dotplus oops. yeah, go to bed. And I should be concentrating on $paid_work:)
16:08 dotplus But I'm glad we've made some more progress here.
16:08 prologic haha
16:08 prologic haha
16:08 prologic I'm just glad my ~300 loc smtp server works
16:08 prologic amazing what you can do with borrowed code!
16:08 prologic yoink!
16:08 bear yea, $paid_work is making itself known over here also
16:09 prologic haha
16:09 bear a little bit of progress or decisions every day ... a Good Thing(tm)
16:09 prologic if I don't get some sleep I won't be going to my own paid work in the morning
16:09 prologic oh wait it's already morning
16:09 prologic :/
16:09 prologic g'night :)
16:09 bear o/
16:09 prologic I'll start writing a router tomorrow
16:09 prologic with mqtt/json in mind
16:11 prologic https://pypi.python.org/pypi/paho-mqtt
16:11 prologic cool!
16:11 prologic looks like integrating this is going to be both easy and nice
16:11 prologic also dotplus re Python 3; no
16:11 prologic writing against Python 2.7 as a minimum for most things
16:12 prologic it's still the defacto standard as far as I'm concenred
16:12 dotplus and hope 2to3 is good enough when we need it?
16:12 dotplus ok, you're the python guy. whatever you say:)
16:12 bear v2.7 can be pointed to v3 easily
16:12 bear v3 cannot be backported to v2.7
16:12 bear (without some pain)
16:13 dotplus given that so many projects aren't supporting 3 yet, it can't be that easy to port 2 -> 3:)
16:13 bear it is now with the very latest changes to 2.7
16:13 bear as long as you target the more "pure" 2.7 features
16:14 bear as before, the real trouble comes from what libraries you use
16:24 prologic dotplus, 2to3 is "good enough"
16:24 prologic I suspect 2.7 will be around a while longer: )
16:24 prologic we *can* write for Python 3 if you really insist
16:24 prologic circuits supports 2.6 thorugh to 3.5 and PyPy
16:25 prologic :)
16:25 dotplus I don't insist.
16:25 prologic it's not that it's hard to support 2/3
16:25 prologic in fact many codebases do
16:25 prologic and 2 -> 3 is not hard at all
16:25 prologic in fact quite easy
16:25 prologic 2to3 does a 98% job of the conversion
16:26 prologic the reason many don't port to Python 3
16:26 prologic or support 2/3
16:26 prologic is because of either large legacy codebases
16:26 prologic or in case of 2/3 having to continue to support 2 anyway
16:26 prologic as is the case of many libraries/frameworks
16:26 dotplus maybe so, but *I* don't remember the differences
16:26 prologic difference?
16:27 dotplus between 2 & 3
16:27 dotplus go to bed
16:27 prologic yes sir :)
16:27 prologic oh
16:27 prologic 2.7/3.1/3.2/3.3 aren't that different
16:27 prologic but 3.4/3.5 is very different
16:28 prologic anyway will definately code with 2.7+ with a view in mind of porting to 3 down the track
16:28 prologic that basically means being strict with your bytes vs unicode
16:28 prologic and getting it right :P

| Channels | #openknot index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Open Knot Communications Platform - https://github.com/openknot