Feature suggestions - Multi-page feeds and maximum jobs imported
An observation and suggestion for improving the system - many feeds have variables that can be modified to access further pages of their feeds.
For example
- Monster: &page=2
- Indeed: &start=25, 50, 75, etc...
- US.jobs: &pg=2
- Craigslist: &s=100 <= this one seems to have some strange pattern I can't quite figure out, but the feed definitely changes, and "s" is the paginate key for their standard web search
So a feature for your plugin could be
1) optionally perform deeper feed imports, if you know the modifier. Since you may not want to do deep regular scrapes on a cron basis (overhead on both ends) maybe just have it be an option on the import screen, to backfill and catch up, maybe up to 5 or 10 or n (although jobs get stale quickly up slower feeds)
2) note the maximum jobs set at default 20 in the plugin, does not save to template when changed. Since few feeds have more than 25 jobs, this is a promising feature that doesn't really work as well as it could, if it had the ability to do deeper imports as sketched out above
3) it would be nice if we could do cron imports on a more granular basis, say hourly. This would allow the job feed to constantly update with new jobs every time the users came back, throughout the day
4) along these lines, it would be good if we could control when the crons fire for the individual feeds. I haven't poked there in your plugin code, but I guess the crons all fire at the same time of day? It might be better for the appearance of updating and for load distribution if the imports could be set on an individual basis. Especially when certain jobs have much better looking listings. For example, Indeed fed description fields are very small - they are good to bulk up the database, but you want to show off the best looking listings for the "latest jobs added" which are viewed most. So running Indeed at midnight followed by Monster before sunup would optimize your listings as far as the viewer sees.
Fantastic plugin!!! Would love to see it get even fantasticer! 
)
Reply With Quote
You must be an AppThemes customer and logged in to view this response. 
