| Time |
S |
Nick |
Message |
| 05:34 |
|
|
df0 joined ##friendlyjava |
| 06:28 |
|
|
abba joined ##friendlyjava |
| 07:21 |
|
|
mr_lou joined ##friendlyjava |
| 08:19 |
|
|
kirua joined ##friendlyjava |
| 08:19 |
|
kirua |
hi |
| 08:19 |
|
kirua |
here is the situation i'm confronted to : i need to develop a webservice thatneeds to retrieve a list of elements from a database and send it back via a webservice. For example the webservice' client asks for the list of commands made between date A and date B. The list can contain several millions elements, which creates two problems, 1) execution speed (the whole process takes a long time to perfom) 2) memory issue (the app runs on a jvm |
| 08:19 |
|
kirua |
with 1gb RAM). When the query returns 100 000 elements, it works fine but when it has to return several millions, it runs out of memory.. I tried for testing purposes to increase the memory allowed to the JVM in the dev envoironment, and when the client asks for the list of commands for a whole year (it's a list of around 10 millions rows ), it takes up to 20 minutes between the moment he sends the request and receives a response. I'm using |
| 08:19 |
|
kirua |
Ibatis to perform a queryforlist (which is not streamed like a JDBC' queryforlist and retrieves the full db response in one time) and send the list back via a JAX-WS webservice. I tried to set a fetch size in the query but it didnt make any difference. So I though about another possibility, which is to store the list (of commands) in a file on the webcontainer (WebLogic in this case) so instead of making a DB query each time a client makes a |
| 08:19 |
|
kirua |
request and so on. I can just read the file and stream out the content to the client, what do you think of that solution and will it solve the problem ? or are there better solutions and ways to solve this issue ? |
| 09:58 |
|
|
foojs joined ##friendlyjava |
| 10:00 |
|
kirua |
help^please |
| 10:00 |
|
foojs |
kirua |
| 10:00 |
|
kirua |
hi foojs |
| 10:01 |
|
foojs |
hello |
| 10:01 |
|
kirua |
i'm going to repost my question |
| 10:01 |
|
kirua |
maybe you know better |
| 10:01 |
|
kirua |
i'm sure you are a competent java dev |
| 10:01 |
|
kirua |
here is the situation i'm confronted to : i need to develop a webservice thatneeds to retrieve a list of elements from a database and send it back via a webservice. For example the webservice' client asks for the list of commands made between date A and date B. The list can contain several millions elements, which creates two problems, 1) execution speed (the whole process takes a long time to perfom) 2) memory issue (the app runs on a jvm |
| 10:01 |
|
kirua |
with 1gb RAM). When the query returns 100 000 elements, it works fine but when it has to return several millions, it runs out of memory.. I tried for testing purposes to increase the memory allowed to the JVM in the dev envoironment, and when the client asks for the list of commands for a whole year (it's a list of around 10 millions rows ), it takes up to 20 minutes between the moment he sends the request and receives a response. I'm using |
| 10:01 |
|
kirua |
Ibatis to perform a queryforlist (which is not streamed like a JDBC' queryforlist and retrieves the full db response in one time) and send the list back via a JAX-WS webservice. I tried to set a fetch size in the query but it didnt make any difference. So I though about another possibility, which is to store the list (of commands) in a file on the webcontainer (WebLogic in this case) so instead of making a DB query each time a client makes a |
| 10:01 |
|
kirua |
request and so on. I can just read the file and stream out the content to the client, what do you think of that solution and will it solve the problem ? or are there better solutions and ways to solve this issue ? |
| 10:03 |
|
foojs |
k i have worked on JAX-WS and not JAX-RS , coming back to your question |
| 10:04 |
|
foojs |
20 min is heck lot of time |
| 10:06 |
|
foojs |
I tried to set a fetch size in the query but it didnt make any difference ??? |
| 10:06 |
|
foojs |
there is a better solution to this i believe |
| 10:11 |
|
foojs |
why do you need to return that amount of data kirua ? |
| 10:12 |
|
foojs |
i have used pagination most of the time when data is dynamic |
| 10:39 |
|
kirua |
because the client needs the whole resultset and not to get it by chunk |
| 10:39 |
|
kirua |
do you know if it's possible to stream out a list with jax-ws ? |
| 15:22 |
|
|
abba_ joined ##friendlyjava |
| 16:50 |
|
|
df0 joined ##friendlyjava |
