When configuring your servers, there are some aspects to consider. Depending on your needs, you will need to implement the most beneficial scalability model. We will be highlighting some options that can guide you along. The first aspect to consider is the amount of documents. You see, as the number of documents in your index increases, hosting the index on a single hard disk can often lead to size and performance limitations. In this scenario, consider adding a slice. The slice is in effect, a separate physical storage location for a section of the master index and distributes the index content, hence increasing available space. It’s possible that it can potentially speed indexing once you have passed a certain threshold, so add a slice when it’s needed. A slice can potentially contain up to 40million documents, and since one Coveo server can typically host up to two slices, it would then contain up to 80 million documents evenly distributed between the two. Potential problem averted!
Following some strong reactions about my last post (that were caused, I believe, by a poor use of the word lock on my part), I decided to write a little follow up post to remedy the situation.
Small update: Having noticed comments about this post on Twitter, I think it’s important to specify that the word “lock” might have been badly chosen here. It’s more of a best-effort thing to reduce the frequency of situations where certain operations would need to be retried. It’s perfectly fine if for any reason those still occur concurrently (indeed before there was no locking whatsoever). The only adverse effect would be slightly decreased performance for the querying user. So, kids, do not use this approach if you need real resource exclusion.
As Coveo’s cloud usage analytics product matures, more and more events are logged every seconds and a lot more people are using the analytics dashboards from the cloud admin. That increased load is great and the usage analytics handles it easily, but there was one thing we did not see coming: transaction cycles in the database. They did not happen often, but this was still problematic as continuous increase in the load on the service only meant more transaction cycles. These cycles were the result of scheduled jobs running when insertion of new events and reporting queries occurred at the same time.
It all began…
It is in September of 2010 that the idea of doing a programming contest came to life. Coveo was a startup with highly passionate employees, ready to grow and bring more passionate people on board.
Did I say passionate?
Our primary source for hiring talented and passionate - I know, but it’s really important to us - software developers was (and still is) universities. We bring students on board during an internship and show them what it’s like to work with us - the challenges, the environment, the people, … - it usually works really well and we end up hiring almost 33% of the interns.
As the UX design department grows at Coveo we start to get enough manpower to actually create stuff before it’s implemented (yep…). One of these concepts is for an online community.
The idea for this project came from an internal initiative to unify the search on our various websites. You’re probably familiar with how great Coveo is at this kind of problem. Basically, it’s what we do. I was tasked to create mockups for this initiative. As I was working on this, another idea came up. We have Coveo staff who are communicating directly with developers. We have online resources (a lot) that are frequently updated. Maybe we could bring all of this together. We talked about it in our meetings with the design team and soon it started consuming me.
In our awesome cloud Usage Analytics API, there is a call that returns the analytics data in data points format (these are meant to be used to build a graph). Recently, we added a feature allowing the user to choose the time period (initially, only days was available). Problem is, the code was strongly coupled with the day period…
Our days are packed. We work on bug fixes, we work on features that have been sold to clients, on pieces required before the next rollout or it may be on the new architecture. We work on things that have a due date or that somebody is waiting for. We don’t have much time to ponder and wonder about all those little things we would like to change or those ideas we have; for all those itches we lack the time to scratch.
How do we encourage people to scratch the itch even if they are not part of our planning? Enter the Hackathon.