What does your search look like? How far back, how large is the result set when it finally completes?
Depending on how you form the search query, it changes how hard LEM has to search for it. For example, the Refine Fields you see are faster than a bare text search.