| Time |
S |
Nick |
Message |
| 06:20 |
|
|
abba joined ##friendlyjava |
| 07:36 |
|
|
aditsu joined ##friendlyjava |
| 07:42 |
|
|
aditsu joined ##friendlyjava |
| 09:16 |
|
pulsar |
guys. i am evaluating low latency high throughtput cache backends to basically pass an byte[] stream across jvm instnaces without buffering too much. |
| 09:18 |
|
pulsar |
i was thinking about redis and implementing a kind of a a blocking byte array output stream (unblocking when the consumer has read enough data to make room for another chunck) |
| 09:18 |
|
pulsar |
any other ideas, perhaps something less hacky? |
| 09:20 |
|
pulsar |
i am not so eager to implement chunking myself and would rather use an existing, well tested backend to handle the stream multiplexing for me. |
| 09:20 |
|
aditsu |
pulsar: there are many options.. are you sure you want a cache and not just a queue of messages? |
| 09:21 |
|
pulsar |
amqp / mqtt / stop - on my radar right now, did not look very deep into it yet. |
| 09:21 |
|
pulsar |
and yes, forget i mentioned the cache word |
| 09:21 |
|
pulsar |
conceptually it is about streams |
| 09:22 |
|
pulsar |
and does not need to be cached. perhaps for debugging but not as a requirement for the system i need to implement. |
| 09:22 |
|
aditsu |
well, you can check out things like Chronicle or Mappedbus |
| 09:22 |
|
pulsar |
zeromq ... i worked with that once. might be also a viable solution. |
| 09:23 |
|
aditsu |
there are various mq's too |
| 09:23 |
|
pulsar |
thanks, i will take a look at these. |
| 09:23 |
|
pulsar |
i like zeromq since it is a very basic, low level mq almost ideal for my needs... now that i look at it :) |
| 09:23 |
|
aditsu |
I used ActiveMQ some years ago |
| 09:23 |
|
pulsar |
at least from what i remember from a couple of years ago |
| 09:24 |
|
pulsar |
i used / use rabbit. did not measure the latency though |
| 09:24 |
|
aditsu |
oh, does it need to work over the network? or just on the same machine? |
| 09:24 |
|
pulsar |
clustered, so network. |
| 09:25 |
|
pulsar |
so no pipes or shm |
| 09:25 |
|
aditsu |
ok.. Mappedbus might not be suitable then |
| 09:25 |
|
pulsar |
and i would rather skip the manual chunking if possible |
| 09:26 |
|
pulsar |
if i was going to use a mq - i would have to deal with that most likely |
| 09:26 |
|
pulsar |
and somehow impelment the buffer caps / blocking |
| 09:26 |
|
aditsu |
you can also look into JGroups |
| 09:26 |
|
pulsar |
either on the protocl level or by a secondard control channel. |
| 09:26 |
|
pulsar |
jgroups... wow. that has been a while |
| 09:26 |
|
pulsar |
like 10 years :) |
| 09:27 |
|
pulsar |
and from what i remember: PITA to configure |
| 09:27 |
|
pulsar |
due to multicast / network realted foo |
| 09:27 |
|
aditsu |
it's been updated, but I also haven't used it recently |
| 09:28 |
|
pulsar |
i think i will take a look at mappedbus and then maybe evaluate zeromq |
| 09:28 |
|
pulsar |
thanks! |
| 09:28 |
|
pulsar |
oh, mappedbus seems to be shm |
| 09:29 |
|
pulsar |
so single machine |
| 09:29 |
|
aditsu |
I'm afraid Mappedbus doesn't do network, try Chronicle |
| 09:29 |
|
pulsar |
yeah, it doesnt |
| 09:29 |
|
aditsu |
or the other things I mentioned |
| 09:30 |
|
pulsar |
https://github.com/OpenHFT/Chronicle-Queue this one? |
| 09:31 |
|
aditsu |
yeah.. don't know much about it, it also does memory-mapped stuff, but seems to support network too |
| 09:31 |
|
aditsu |
I guess there's also stuff like Netty and MINA |
| 09:34 |
|
pulsar |
i am familiar ... soft of with netty |
| 09:34 |
|
aditsu |
btw I was considering Mappedbus or Chronicle recently for IPC on the same machine.. but in my case simplicity (and light weight) is much more important than performance |
| 09:35 |
|
aditsu |
and just now I finished coding something with file locks and WatchService |
| 09:35 |
|
aditsu |
(I don't want sockets) |
| 09:46 |
|
pulsar |
file locks... no trouble with that? |
| 09:46 |
|
pulsar |
depending on the filesystem you might run into issues, been there, done that :) |
| 09:47 |
|
pulsar |
and switched to zookeper |
| 09:47 |
|
aditsu |
what kind of trouble? |
| 09:47 |
|
pulsar |
the deployment target was running on nfs |
| 09:47 |
|
pulsar |
and that had some issues with locks |
| 09:48 |
|
pulsar |
like... delayed or something |
| 09:48 |
|
pulsar |
cant remember exaclty, it is a couple of years ago. |
| 09:48 |
|
pulsar |
basically i ran into race conditions because of that |
| 09:48 |
|
pulsar |
and switching to zookeeper (even a single-instance-quorum) was quite easy to implement |
| 09:49 |
|
pulsar |
and actually meant to be used for cluster-locks |
| 09:49 |
|
aditsu |
nfs? I wouldn't touch that with a 20-meter pole |
| 09:49 |
|
pulsar |
yeah, tell that to the guys managing the systems i had to deploy my stuff on :) |
| 09:50 |
|
pulsar |
"it is just a filesystem, you should not care". |
| 09:50 |
|
pulsar |
right. |
| 09:51 |
|
aditsu |
my program is for end users to run on their computers, and I'm only locking files in the temp dir |
| 09:51 |
|
aditsu |
if they have their tmp on nfs... well, they deserve it :p |
| 09:51 |
|
pulsar |
or smb |
| 09:51 |
|
pulsar |
or dokan/fuse |
| 09:51 |
|
pulsar |
there is so much crap out there |
| 10:06 |
|
aditsu |
anyway, in my case there's only one machine doing the locking |
| 10:37 |
|
pdurbin |
pulsar: What problem are you trying to solve? The customer says your app is slow? |
| 11:00 |
|
pulsar |
pdurbin: not really. i am thinking ahead. |
| 11:01 |
|
pulsar |
implementing an prototype as we speak and try to keep the architecutre in the back of my head if want to push it live |
| 11:01 |
|
pulsar |
and that would mean 3.000.000 clients |
| 11:01 |
|
pulsar |
request comes in, server delivers a a promise with the url containing the result |
| 11:02 |
|
pulsar |
client comes back and fetches the responses |
| 11:02 |
|
pulsar |
the tricky part: it has to be streamed all the way through. i cannot precache the response and serve that to the client |
| 11:03 |
|
pulsar |
but rather pass the result output stream to the client directly. |
| 11:03 |
|
pulsar |
to be scalabe / fit into the deployment platform, i need my (micro) services to be stateless |
| 11:05 |
|
pulsar |
so i cannot store references to the streams in the jvm but rather have to figure a way how to pipe it between instances |
| 11:06 |
|
pulsar |
does that make any sense? |
| 11:07 |
|
pulsar |
for the prototype i am using only one jvm, so i can/will store the input stream somewhere inside a map connected to the worker thread |
| 11:07 |
|
pulsar |
and wait for client's stream.read() to do the actuall response reading inside the worker thread. |
| 11:08 |
|
pulsar |
either using a stream proxy / wrapper or by passing the output stream directly to the consumer |
| 11:10 |
|
pulsar |
... implementation details. anway. this will obviously only work when the initial request firing of the processing thread and the request asking for the result are bound to the same jvm. i could use some kind of sticky sessoins / distributed hashtables / whatnot to make every client hit the same server over and over again based on the user id or something |
| 11:10 |
|
pulsar |
but this will have implications on the scalability and will not be as easy as firing up a few more instances behind the ha proxy. |
| 11:21 |
|
aditsu |
damn, java still doesn't have a (cross-platform) way to get the application data location |
| 11:22 |
|
pulsar |
yeah, not natively. |
| 11:22 |
|
pulsar |
there is a whole ecosystem evolved around this question. |
| 11:26 |
|
aditsu |
pulsar: do you have a recommended solution? |
| 11:28 |
|
pulsar |
by the deployment architecture? |
| 11:28 |
|
pulsar |
sort of, they have an actor model |
| 11:28 |
|
pulsar |
but that would require my application to live within an preconfigured jvm environment |
| 11:28 |
|
pulsar |
which is not an option right now |
| 11:29 |
|
pulsar |
since i have to use http/2 |
| 11:29 |
|
pulsar |
and this requires a modified boot classpath as of jdk8 |
| 11:29 |
|
aditsu |
um.. I'm talking about the application data folder |
| 11:29 |
|
pdurbin |
It's good to think ahead. |
| 11:30 |
|
pulsar |
aditsu: come again? |
| 11:30 |
|
pulsar |
pdurbin: thats what i am being payed for :) |
| 11:31 |
|
aditsu |
pulsar: I need a way to get the application data location, to write data files in there, you said there's a whole ecosystem? or were you talking about your stuff? |
| 11:31 |
|
pulsar |
oh, right. yeah, i was talking about my stuff |
| 11:32 |
|
pulsar |
but i see you point now |
| 11:32 |
|
pulsar |
actually i did experience that pain aswell |
| 11:32 |
|
pulsar |
ended up putting everything into the jar and using class.getResource* |
| 11:32 |
|
pulsar |
which is fine... for static stuff. |
| 11:33 |
|
aditsu |
no go, I need to write stuff at runtime |
| 11:34 |
|
aditsu |
most people seem to do some OS sniffing, env. variable juggling and sometimes JNA calls |
| 11:34 |
|
aditsu |
unless they just dump things in user.home :p |
| 11:34 |
|
pulsar |
sounds like a perfectly stable soltuion.... so, why not bite the bullet and pass it via -Dfoo.location=/home/foo/bar ? |
| 11:35 |
|
aditsu |
the end user might not have any concept of command line arguments |
| 11:37 |
|
aditsu |
(and no clue whatsoever about where applications are supposed to store their data) |
| 11:57 |
|
pulsar |
enduser, right. i keep on forgetting. |
| 12:05 |
|
aditsu |
windoze also has this concept of a "roaming profile".. |
| 14:57 |
|
|
df0 joined ##friendlyjava |
| 16:03 |
|
aditsu |
damn, I got pretty much everything working, now hit a java bug >_< |
| 16:17 |
|
aditsu |
hm, that bug is benign, I had another bug in my code.. |
