What we’re about
This is a group for anyone interested in search relevance, relevance engineering, natural language search, Learning to Rank, machine learning for search, neural search, Apache Lucene/Solr, Elasticsearch, Vespa AI, vector search, ecommerce search and all the topics covered by the Haystack conference series (www.haystackconf.com)
Our events are both:
- online as part of the Haystack LIVE! series (usually on Thursdays at 4pm GMT, every 4-6 weeks). Have an idea for a talk? We're always looking for speakers!
- in-person in the EU & USA, such as our Haystack On Tour events.
This Youtube channel contains recordings of most of our event talks.
Upcoming events (1)
See all- Pre-Haystack US MeetupThe Residence Inn Charlottesville Downtown, Charlottesville, VA
Join us the evening before Haystack for this in-person meetup in Charlottesville! We've got a couple of talks planned by Eric Pugh and Daniel Wrigley from OpenSource Connections.
Obviously, not every member of this group will be in town and we don't plan to stream or record this meetup, but you can still join the main Haystack conference remotely via: https://www.eventbee.com/t/292625631/meetup!
Agenda
18:00 - Doors open
18:30- First talk (Eric Pugh)
19:15 - Networking & pizza
19:45 - Second talk (Daniel Wrigley)
20:30 - Let's socialize ;-)Talks
Can I use a really Stoopid LLM as a Judge and STILL get some Value? Experiments with Ollama models running on the CPU
Eric PughIn this talk Eric will demonstrate how easy it is to play with three models in Quepid, starting with industry standard gpt-4o and then swapping to the 398 MB qwen-2.5:0.5b and llama3.2:1b models. We’ll look at some inter rater reliability statistics and then discuss if simplistic models you run on your laptop bring anything to the table!
Measuring Search Relevance with User Behavior Data in OpenSearch
Daniel WrigleyEnsuring high-quality search results is a complex and ongoing challenge for search engineers. As data, ranking algorithms, and search platforms evolve, measuring relevance effectively becomes even more critical—yet also increasingly difficult. Traditional evaluation methods often require significant resources, making it particularly challenging for small organizations or teams without dedicated search expertise.
In this talk, we will explore how user behavior data can be leveraged to measure search relevance within OpenSearch. We'll discuss key challenges, such as position bias (users preferring higher-ranked results) and data sparsity, and how these factors impact implicit judgments. We will show how User Behavior Insights (UBI) can be leveraged as a tool for collecting and analyzing behavioral signals, and how the search quality evaluation app can transform these raw interaction signals into meaningful search relevance metrics.
The session will include a live demo showcasing how to collect user behavior data with UBI, derive implicit judgments, and use these insights to quantify search quality over time. Attendees will leave with practical strategies to improve search relevance measurement, even in resource-constrained environments.
