Revamping College Listing Pages

How we moved off from Solr to Elasticsearch and challenges faced in migration

In the world of web development, efficiency and speed are key. As user demands evolve, so must the technology powering our websites. One of the major transitions we recently undertook was revamping our college listing page. Formerly powered by Solr, we migrated it to Elasticsearch, a move that has significantly enhanced performance and functionality.

The Solr Era

In the earlier days, our college listing page relied on the Solr search platform, which indeed served us effectively. However, in the dynamic realm of technology, one thing is certain: change is constant. As our platform continued to evolve, so did the expectations of our users. It became evident that we needed a solution that not only matched but surpassed our existing capabilities in terms of power and flexibility.

The Power of Elasticsearch

Elasticsearch, with its distributed and RESTful nature, emerged as the ideal candidate for our revamped listing page. This open-source, full-text search and analytics engine is built on top of Apache Lucene, providing robust features and scalability. Its ability to handle large volumes of data and complex queries made it a perfect fit for our needs.

Why Elasticsearch?

• Speed: Elasticsearch is known for its exceptional speed when it comes to searching and retrieving data. This means that our users can now access the information they need faster than ever before.
• Scalability: With Elasticsearch's distributed architecture, we can effortlessly scale our infrastructure to handle increased loads. Whether it's a small user base or a rapidly expanding one, Elasticsearch grows with us.
• Flexibility: Elasticsearch offers flexible querying capabilities. This means we can fine-tune our search queries to deliver more accurate results to our users, improving their overall experience.
• Near Real-Time Search: Elasticsearch provides near real-time search capabilities. This ensures that the data presented to our users is always up-to-date.

The Migration Process

Migrating from Solr to Elasticsearch was a meticulous process. It involved rethinking our data structure, optimising our queries, and ensuring a seamless transition for our users as well as providing them with new functionality. Our dedicated team worked tirelessly to make this migration as smooth as possible.

Migration Steps

• Data Indexing: The first step was to index our existing data into Elasticsearch. Elasticsearch uses JSON over HTTP, making it relatively easy to insert, update, and query data.
• Schema Mapping: Elasticsearch uses dynamic mapping, which means you don't need to define a schema explicitly. However, we fine-tuned the schema to ensure it aligned with our specific data structure.
• Query Rewriting: Our existing search queries were rewritten to match Elasticsearch's query structure. This involved adjusting filter criteria, aggregations, and sorting.
• Relevance Tuning: Elasticsearch's relevance scoring system differs from Solr's. We fine-tuned the scoring algorithm to ensure the most relevant results appeared at the top of search results.

Certainly, here are some of the challenges we encountered during the migration

• Data Integrity: Ensuring that data in the Elasticsearch index is consistently updated and aligned with changes made in our CMS was a significant challenge. We had to develop a robust data synchronisation mechanism to address this.
• Complex Queries: The implementation of multi-select filters introduced complexity into our search queries. Constructing queries that efficiently handled these filters while delivering accurate results was a considerable challenge.
• URL Combinations: Creating new URL combinations for the listings page, especially with multi-select filters, required careful handling to ensure a seamless user experience.
• Elasticsearch Maintenance: Keeping Elasticsearch updated and responsive was a crucial challenge. We needed to make sure that any updates or changes in college data were immediately reflected in our Elasticsearch index.
• Query Performance: With the growing dataset, maintaining query performance was vital. We had to continuously optimise our queries and Elasticsearch settings to ensure quick response times.
• Scaling: As our data continued to grow, we needed to consider how Elasticsearch would scale to handle the increased load effectively.
• Error Handling: Developing a robust error-handling mechanism to catch and resolve issues promptly was essential to maintaining a stable system.

New Capabilities

In addition to the migration, we've introduced several new capabilities to enhance the user experience.

Search Within College List : With this feature, users can now search for any college by its name. This search functionality is seamlessly integrated with the existing filters, allowing users to narrow down their results effectively.

Search in Filters Option : In this update, we've provided users with the ability to search for specific filters by their names. This feature simplifies the process of finding and selecting filters, ensuring a more user-friendly experience.

Understanding user behaviour and preferences, we implemented multi-select filters. This allows users to choose multiple options within a filter category, offering them the flexibility they desire.

Inline Filters : We've also introduced inline filters within the college list. Each filter displays the top 10 options, prioritising those with a higher number of colleges. This makes it easier for users to quickly access the most relevant options. Clicking on any of these filter options redirects the user to the respective URL

The Results

The results of this transition have been nothing short of remarkable. Our listing page now works faster. Searches are more accurate, and users can navigate through vast amounts of data effortlessly. Elasticsearch's robustness has enabled us to provide a better experience for our users.

In Conclusion

In the ever-evolving landscape of web development, staying ahead of the curve is essential. Our migration to Elasticsearch is a testament to our commitment to delivering the best possible experience to our users. As technology continues to advance, so will we, ensuring that our platform remains fast, efficient, and user-friendly. The revamp of our college listing page is just one step in our ongoing journey of improvement and innovation. Stay tuned for more exciting developments!