Well, it's not completing so I can't say how large the result set is. I know the appliance takes in about 7 million events per day and my last failed search was for a 48 hour period of time so it would be searching roughly 14 million events.
In the specific case that came up today I am searching events from mid-end of December of last year. I am specifically looking for user logon events where source machine is equal to Bad IP's. Bad IP's is a user defined group that contains a few dozen IP's.
I guess the point of my frustration is that this doesn't seem like an unreasonable use case and the specifics shouldn't matter too much. I have logs and I need to search them, that shouldn't be as much of a problem as it seems to be. I would be more understanding of the specifics if I was trying to search a 6 month period of time from 3 years ago but that isn't the case, what i am doing is pretty specific for a pretty short (shorter than I would like) period of time.
I am looking into the possibility of a storage IOP's/Latency issue causing the problem, I haven't yet confirmed anything on that front.