Towards a Scalable Social Recommender Engine for Online Marketplaces: The Case of Apache Solr

What problem does this address?

Deploying real-time recommender systems at scale is messy. Especially if you want to combine traditional signals (purchase or click data) with social interactions. Classic recommender libraries give you algorithms, but not a production-ready, modular architecture. Meanwhile, performance details—model updates, hardware requirements, server stress handling—are missing from most published work. The result: anyone deploying social recommenders in online marketplaces is left stitching together bespoke solutions with little guidance on practical tradeoffs.

How was the engine built?

We designed a modular recommender framework on top of Apache Solr, not just for its popularity and scalability in search, but because its text-analytics stack conveniently matches the needs of memory-based recommendation and hybrid models. The architecture cleanly separates algorithms (MostPopular, Collaborative Filtering, Content-based, and Hybrids) from workflow orchestration and data storage, so new recommenders or workflows can be swapped in with minimal pain. We used both social and marketplace features, accessed via flexible, schema-based Solr cores for user profiles, purchases, social actions, etc. REST APIs handle both recommendation requests and live updates. Horizontal scaling—sharding and replication—comes essentially for free via Solr. Implementation was in Java and designed for extension across connectors (e.g., plugging in other social or commerce data).

Was it actually tested at scale?

Yes, and that’s what makes this more than a conceptual paper. Using the SecondLife virtual world as a sandbox (chosen for having both marketplace and social activity with user overlap), we ran through a battery of offline and online evaluations. Traditional metrics—nDCG, MAP, MRR, F1, Coverage, and Diversity—are reported for multiple algorithm and data combinations. Real server stress and latency benchmarks are included (IBM x3550, Solr 4.3.1, 128GB RAM): mean response times for even the hybrid models stayed under 0.2 seconds per recommendation, with graceful scaling across simulated loads.

Takeaways? Anything surprising?

• Social features do help accuracy, but their coverage is limited (e.g., using ‘interests’ directly did not work well—too messy, not standardized).
• Content-based diversity is strong, but only when good item meta-data exists.
• Hybrid models—combining social, marketplace, and meta-data features—beat any single-source model, both for accuracy and for covering cold-start users.
• Solr is a viable base for production recommenders, not just search, with fast updates and real-time serving possible with open-source stacks.

If you’re looking to understand the engineering and empirical tradeoffs in social recommender deployments—down to the code interfaces and stress test numbers—this paper offers detail you rarely see in the literature.

@inproceedings{lacic2014towards,
  title = {Towards a scalable social recommender engine for online marketplaces: The case of apache solr},
  author = {Laci\'{c}, E. and Kowald, D. and Parra, D. and Kahr, M. and Trattner, C.},
  year = {2014},
  booktitle = {Proceedings of the 23rd International Conference on World Wide Web},
  pages = {817-822},
  publisher = {ACM},
}