I think the biggest issue to tackle is the connector bit. When a customer asks "can you support logs for <insert item here>", you need to be able to say YES in nearly all cases. Every time I tell somebody NO, I get Splunk thrown in my face . Based on your responses HERE, HERE, and HERE, I get to go back to a customer and relay that information, they have already said they will go with Splunk. This is bad for both us and SolarWinds as it makes it more difficult for us to sell our Log Management service that is based off LEM thus we will be purchasing less licensing from SolarWinds.
I think a generic connector is EXACTLY what you need so lets think on that...
The fact that it wont' apply to any of the default rules is not a problem so long as you make that clear. I think that its always better to error on the side of giving your users more capabilities than not. Flag these connectors as being for advanced users and specify why in a pop-up or something creative like that; you have an awesome UX team so I am sure they can help there.
The volume issue seems like it could also be addressed in several different possible ways, a few that quickly come to mind are as follows...
- As part of the generic connector add on a filter that users can build as part of the connector configuration, this would offload the filtering to the end-nodes and reduce what comes into LEM
- Add the ability to expire the logs that are pulled by a specific connector so that they are not kept as long in the database
- Provide the ability to dump logs from the database on a routine database via some form of archive process; I think we have discussed this in a different thread as well
These are just a few ideas that come to mind, I am sure if you posted this problem out to the community you would get all sorts of creative ideas, you have a great group of people here willing to help solve problems (and the best part is we work for free).
On the reporting side you definitely need some serious reform (as I have pointed out in a different thread). The filtering using the reporting tools is AWFUL! Move the reporting into the WebUI using a better model more conducive to customized reporting and filtering.
Thanks for responding and hopefully we can keep these types of discussions moving for future improvements of the product and to help me sell more of it!
P.S. I realize that it's easy for me to sit here and suggest changes without understanding what would be necessary to make those things work, I realize that I don't fully understand the back-end of the product and the implications of what I ask for. I am just trying to provide feedback to help make a product I love better and to help better position the product and myself to be successful out there in the world.