Using Web Services

[BigR]

I'm currently working on a project that uses a web service for retrieving/modifying data. The web service is written by another team in our company, and we are writing a web site that is a client of the web service. This is my first real exposure to web services, so I'm not sure of the "correct" way of designing them.

Say the web service's back-end database has two tables: Books and Authors. For simplicity, assume one book has just one author, and one author can have many books. The authors are identified by a unique identifier (integer), and the books have the author id as a foreign key.

Now say we want, as the web site client, to display a list of all the books in the system; each item we display should also have the author's name. What we would like to do is submit one request to the web service, and have that return an XML document where each book element also contains the necessary author information. The team developing the web service, however, would rather we make one call to the web service to get a list of all the authors, hold that list in memory, then make another call to the web service to get a list of all the books, then manually join the authors to the books.

Our feeling is that the clients should not have to perform what are essentially database operations manually. In addition to performance problems, there is also the problem of making sure our in-memory authors data in current, and possible resource limitations of having dozens of concurrent users each keeping an in-memory copy of the authors table.

The web service team feels that since they are only serving up the data, it is the client's responsibility to combine the data in any ways we need, and are worried about performance when sending larger XML responses.

Since we are new to web services, our team isn't sure what the "standard" way of using them is. We are very reluctant to mix the data layers and the presentation layers, especially since we would have to do this for multiple clients (the web site only being one).

Has anyone here used web services enough to have a good idea on which approach is better?

[RecoveringYuppy]

Is the web service site supporting other applications? Is you client a browser application?

Generally, I'm with your team, simplify the client, complicate the server.

A single XML document can easily represent a multi column table. And, as you seem to realize, multiple calls to retrieve each column rasises the possibility that an update will occur between your calls and you will get inconsistent results.

[PogoPedant]

I'm currently working on a project that uses a web service for retrieving/modifying data. The web service is written by another team in our company, and we are writing a web site that is a client of the web service. This is my first real exposure to web services, so I'm not sure of the "correct" way of designing them.

Say the web service's back-end database has two tables: Books and Authors. For simplicity, assume one book has just one author, and one author can have many books. The authors are identified by a unique identifier (integer), and the books have the author id as a foreign key.

Now say we want, as the web site client, to display a list of all the books in the system; each item we display should also have the author's name. What we would like to do is submit one request to the web service, and have that return an XML document where each book element also contains the necessary author information. The team developing the web service, however, would rather we make one call to the web service to get a list of all the authors, hold that list in memory, then make another call to the web service to get a list of all the books, then manually join the authors to the books.

Our feeling is that the clients should not have to perform what are essentially database operations manually. In addition to performance problems, there is also the problem of making sure our in-memory authors data in current, and possible resource limitations of having dozens of concurrent users each keeping an in-memory copy of the authors table.

The web service team feels that since they are only serving up the data, it is the client's responsibility to combine the data in any ways we need, and are worried about performance when sending larger XML responses.

Since we are new to web services, our team isn't sure what the "standard" way of using them is. We are very reluctant to mix the data layers and the presentation layers, especially since we would have to do this for multiple clients (the web site only being one).

Has anyone here used web services enough to have a good idea on which approach is better?

Heh! Typical back-end developers, always wanting to push responsibility away from themselves. I'm in the exact same type of disagreement myself.

It seems like the web service team only sees the web service as a database access thing; something you use to read a database. In reality, a web service is the thing that lies between the database and the client that is responsible for enforcing business rules and for massaging any incoming or outgoing data. In other words, if the client needs a list of books with authors, the web service should give them a list of books with authors.

All in my opinion, of course.

[BigR]

Heh! Typical back-end developers, always wanting to push responsibility away from themselves. I'm in the exact same type of disagreement myself.

It seems like the web service team only sees the web service as a database access thing; something you use to read a database. In reality, a web service is the thing that lies between the database and the client that is responsible for enforcing business rules and for massaging any incoming or outgoing data. In other words, if the client needs a list of books with authors, the web service should give them a list of books with authors.

All in my opinion, of course.

That is our opinion as well. As clients, we don't want to be responsible for manipulating lots of data in a way that the back-end could do far more quickly and efficiently. Plus mixing presentation layers with data integrity is usually not a good thing.

[BigR]

Is the web service site supporting other applications? Is you client a browser application?

Generally, I'm with your team, simplify the client, complicate the server.

A single XML document can easily represent a multi column table. And, as you seem to realize, multiple calls to retrieve each column rasises the possibility that an update will occur between your calls and you will get inconsistent results.

Right now, we are the only ones using the web service. In the future, the hope is that other clients will, even ones external to our company. I think we're making headway in convincing the other team, but since I don't have much experience with the web service world, it's good to get other opinions.

Thanks for the feedback everyone!

[Wudang]

I like developerworks
http://www-128.ibm.com/developerworks/webservices

[shadron]

In general, the app could go either way. The question is, where is the greatest bottleneck? Is the server power required going to slow down all the users? Is the client power going to be wasted always waiting for results? What are the possibilities of scaling for either, or is the cost going to be high? And finally...what of the interface between them? Generally there is some fixed amount of bandwidth available - can the processes be designed so the data crossing the interface is minimized? Are there other constraints? Perhaps you need to implement a dozen different clients for different platforms - that would mitigate for making the clients as thin as possible so as to minimize development time and effort. Are you thinking of using COTS for some of the hardware/software?

There is no single way to do it. Did you hack a prototype so you could measure some of the cases? If you did no analysis before you started design, then your first try is going to be your prototype, willy-nilly.

[RecoveringYuppy]

The question is, where is the greatest bottleneck? Is the server power required going to slow down all the users?
IMO the architecture BigR is going for would help the server. It would save the server redundant evaluations of selection criteria.

[van_dutch]

Why is the data stored in two structures? Why not use one table that has two fields, author and title? You could even expand this to multiple fields like year, subject etc etc. This way, one could query for a specific item and get back a portion of the table (the authors that match this string or the year that match this number etc). This would remove the problem of combining and simplify the situation. Each book that is being represented has multiple fields so it makes the most sense to represent the multiple fields as part of one object. This framework would also make it easy to represent different media types such as movies.

[Wowbagger]

Why is the data stored in two structures? Why not use one table that has two fields, author and title? I imagine there is a many-to-many relationship, here: One author can have multiple books, and one book could have multiple authors.

A View, with the right joins, could combine the tables into one, and send a consistent result through the web service. That is what I recommend, in most situations.

If performance is going to be an issue when sending all that data out, at once, then perhaps a paging system might be a good idea: Break up the table horizontally, and send the chunks as "pages", one at a time. But, again, only if necessary. That is more standard than doing it vertically: with titles and authors sent separately.

[logical muse]

It might be a good idea to take a look at what others have done in this regard.

For example, here's the digg service API documentation:

http://apidoc.digg.com/

It's more like your preferred option than what is being proposed by your back-end service team. For example, you can request a list of comments on a particular story. You don't get a list of all stories and all comments to match up in the client.

[logical muse]

Why is the data stored in two structures? Why not use one table that has two fields, author and title? You could even expand this to multiple fields like year, subject etc etc.

Because that's not normalised.