e-Commerce with Hybris: 10 Million Product Catalog?

Last month a prospective customer called our office.
“We know that you have a team of software developers that build e-commerce applications with Hybris software. Can you help us with developing our online store?”
“Sure we can – we have a solid expertise in developing e-commerce applications with Hybris software”.
“But our store will have pretty large catalog: ten million products”.
“We didn’t have a chance to develop online stores that have more than a million products. We know how to approach such a project to minimize your risk, but so far we didn’t deploy such an application in production.”

After this conversation we’ve never heard from this person again. We know why: he was looking for another answer like, “Sure, we did it before and will do it again!” This is not to say that creating a responsive online store with 10M products is not possible with Hybris, but we didn’t do it. In any online store product catalogs have to be indexed from time to time. For example, a store needs to add a new line of products. How much time is required to re-index a store with 10M with Hybris software? How about importing of the CSV product data and synchronization? We simply don’t know.
Hybris does several passes over these IMPEX to resolve referential integrity, in fact – many passes. In may take a day and finish or a day and break/start over. Then either SOLR (Lucene) or Endeca full text search index will need to be built. Then, there is a task of synchronizing the staging catalog with its production version. Indexing of itself is not Hybris specific, but the import and synch are. Hybris has yet to show that it’s a high performing solution and publish the appropriate case studies.

In fact, if you do not provide indexing information, importing of a mere half million records may never finish on Hybris server. We do not have any metrics of the synchronization process yet, but the good news is that Hybris is build on Java servers and careful clusterization, cacheing, and fine-tuning of the database will produce acceptable solution to large inventories. Wall Street application written im Java handle huge amounts of data in timely fashion. But Wall Street managers understand that they have to hire the right people. Large-scale e-Commerce projects have the same level of complexity.

We’d be happy to set up a lab (parallel merging while loading, finding deltas, proper Java clustering, stress tests) with the appropriate hardware and create a pilot to answer these questions and to optimize the process to get rid of the bottle necks, but this prospect customer is gone.

What will happen next? I’ll tell you: the deja vu of IT consulting. Some brave salesman of another consulting company will explain to this customer that working with large product catalogs is their bread and butter and will get this project. Six months down the road the customer will see a lot of hours billed to the project and detailed explanations of “unforeseen circumstances” followed by new promises.

We’ve seen this scenario several times – it doesn’t depend on the technology in use. Today it’s e-commerce, but six years ago I wrote a blog about a similar scenario, but that time it was about redesigning a portal for a major publisher. Software developers hate working on such projects. They don’t know that the customer has these unrealistic expectations because of some over-promising salesman.

What’s the moral of this story? We value a moral dimension to consulting, which costs us dearly.