greptilian logo

IRC log for #friendlyjava, 2017-08-10

##friendlyjava on freenode

| Channels | #friendlyjava index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
05:34 df0 joined ##friendlyjava
06:28 abba joined ##friendlyjava
07:21 mr_lou joined ##friendlyjava
08:19 kirua joined ##friendlyjava
08:19 kirua hi
08:19 kirua here is the situation i'm confronted to : i need to develop a webservice thatneeds to retrieve a list  of elements from a database and send it back via a webservice.  For example the webservice' client asks for the list of commands made between date A and date B. The list can contain several millions elements, which creates two problems,  1) execution speed (the whole process takes a long time to perfom) 2) memory issue (the app runs on a jvm
08:19 kirua with 1gb RAM). When the query returns 100 000 elements, it works fine but when it has to return several millions, it runs out of memory..     I tried for testing purposes to increase the memory allowed to the JVM in the dev envoironment, and when the client asks for the list of commands for a whole year (it's a list of around 10 millions rows ), it takes up to 20 minutes between the moment he sends the request and receives a response. I'm using
08:19 kirua Ibatis to perform a queryforlist (which is not streamed like a JDBC' queryforlist and retrieves the full db response in one time) and send the list back via  a JAX-WS  webservice. I tried to set a fetch size in the query but it didnt make any difference. So I though about another possibility, which is to store the list (of commands) in a file on the webcontainer (WebLogic in this case) so instead of making a DB query each time a client makes a
08:19 kirua request and so on. I can just read the file and stream out the content to the client, what do you think of that solution and will it solve the problem ? or are there better solutions and ways to solve this issue ?
09:58 foojs joined ##friendlyjava
10:00 kirua help^please
10:00 foojs kirua
10:00 kirua hi foojs
10:01 foojs hello
10:01 kirua i'm going to repost my question
10:01 kirua maybe you know better
10:01 kirua i'm sure you are a competent java dev
10:01 kirua here is the situation i'm confronted to : i need to develop a webservice thatneeds to retrieve a list  of elements from a database and send it back via a webservice.  For example the webservice' client asks for the list of commands made between date A and date B. The list can contain several millions elements, which creates two problems,  1) execution speed (the whole process takes a long time to perfom) 2) memory issue (the app runs on a jvm
10:01 kirua with 1gb RAM). When the query returns 100 000 elements, it works fine but when it has to return several millions, it runs out of memory..     I tried for testing purposes to increase the memory allowed to the JVM in the dev envoironment, and when the client asks for the list of commands for a whole year (it's a list of around 10 millions rows ), it takes up to 20 minutes between the moment he sends the request and receives a response. I'm using
10:01 kirua Ibatis to perform a queryforlist (which is not streamed like a JDBC' queryforlist and retrieves the full db response in one time) and send the list back via  a JAX-WS  webservice. I tried to set a fetch size in the query but it didnt make any difference. So I though about another possibility, which is to store the list (of commands) in a file on the webcontainer (WebLogic in this case) so instead of making a DB query each time a client makes a
10:01 kirua request and so on. I can just read the file and stream out the content to the client, what do you think of that solution and will it solve the problem ? or are there better solutions and ways to solve this issue ?
10:03 foojs k i have worked on JAX-WS and not JAX-RS , coming back to your question
10:04 foojs 20 min is heck lot of time
10:06 foojs I tried to set a fetch size in the query but it didnt make any difference ???
10:06 foojs there is a better solution to this i believe
10:11 foojs why do you need to return that amount of data kirua ?
10:12 foojs i have used pagination most of the time when data is dynamic
10:39 kirua because the client needs the whole resultset and not to get it by chunk
10:39 kirua do you know if it's possible to stream out a list with jax-ws ?
15:22 abba_ joined ##friendlyjava
16:50 df0 joined ##friendlyjava

| Channels | #friendlyjava index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

##friendlyjava on freenode