Show Posts
1
Site Suggestions & Support / Re: Improving search
« on: July 11, 2015, 01:54:43 AM »Yes, I have to build a page and load up all the overhead that comes with it, but in terms of server load I'd argue this is ultimately no different than me browsing the actual threads/posts with my web browser. Thus even though the amount of of work I'm putting on the server is larger than if I'd used an indexed search, would me creating such a bot really be that problematic?
Remember that I'd be setting it to read one thread per hour. That way it's not pulling up the webpages any faster than a normal person who wanted to read the entire contents of the reference section which is 117 threads in all. We're talking about reading in the entire reference section once over the course of a week. Would that really be that much of an imposition or even noticable over the background noise?
Yep, you are correct - if you slow the bot to one request per hour then you spread the server hit to a minimum - assuming that Iago pays for thresholds from his hosting service would mean that it wouldn't directly impact his pocket.
Your ultimate goal is to create an offsite search engine - If I would help you do that I would have you create the queries to get the data you would be accessing - export them to a csv or other shared data source type and simply let you have them. Which is what I posted previously.
Since it's only 117 threads that you are after which contain what 20 to 30 replies on average? that's a lot of hours. I liked your Dresden Game KNN. Just offering some hints or tips for success. Fu.