If you have a first-generation Mac Mini (as I do) and you haven't dished out money for an extra (faster) hard drive, you're probably suffering with terrible performance. They put some kind of slow laptop hard drive in there to keep the thing small, or whatever.
This is a good tip on how to disable Spotlight (the post by jjccgg, NOT the original post), which should in theory boost performance by stopping the constant hard drive indexing. I've done this and I can indeed immediately see a performance boost. I haven't benchmarked it, but it's noticeable enough that I don't think I need to.
I find Spotlight to be nearly worthless. Searching is not the most efficient way to find files, in my opinion. It's good in the average case, but terrible in the worst case. I define "worst case" here to mean that you don't know the NAME of a file. If I organize things by folder, I only need to know the LOCATION of a file. It's more likely that I forget a file name or description than that I forget a file's general location, in the worst case.
For example, I can think of a bunch of wallpapers that I downloaded, and they likely have names like 1.jpg, 03984.png etc. I know where to get them (some subdir of the wallpaper directory), but searching would not find them easily. I can think of documents I've written, but I don't know the name of them or their titles. The solution to this (and what Spotlight expects, apparently) is for you to tag all your files so the search can find them easier. But in that case, we've fallen back to a pseudo-folder structure, only with greater room for error and lots more fuzzy areas.
That's my problem with desktop search, I guess: it's too fuzzy. You have to rely upon the strength of your search algorithm, and I don't have a heck of a lot of faith in search algorithms.