RubyURL 2.0 on the horizon 9
RubyURL was a project that I built about 2 1/2 years ago as a late night attempt to see what I could build and deploy with Ruby on Rails in a night. It’s nearing 50,000 unique website links, has a Ruby gem that you can use with it, and rbot plugins.
I’ve rewritten it about three times in the past six months, to try out some new approaches, but haven’t deployed with a new version as I’ve been waiting for someone to help me with a new design. Chris has offered to help out and once we integrate his new design with it, we’ll be launching it.
Everything is not great in RubyURL land though. It appears that it’s become an easy target for comment spammers to abuse the site to generate rubyurls and paste those links in their spam comments. Several pissed off bloggers, forum administrators, and system administrators have emailed me to complain that I’m spamming their site. Sadly, even with a basic disclaimer on the site, they still like to blame me for their spam. It’s gotten common enough, that I’ve written a template email that I respond with that explains how the site works and that I’m not accountable for people posting links to my URL redirect tool.
You can see that it’s popping up around the net via a google search.
So, I’ve been trying to think of ways to make it easier for people to flag URLs as being abusive of the site. I’ve not come up with any elegant solution that doesn’t force the good users of the site to have more steps in their process to create a basic RubyURL.
The ideal (and current) workflow:
- User navigates to http://rubyurl.com
- User pastes in long url into text box/area
- User submits form
- User is provided with new (shortened) rubyurl
- User copies the rubyurl and does what they want with it (generally… pastes into IM, IRC, Email, etc.)
Some people have suggested using a user system to do this, but I really don’t like that as a solution.
Another idea, which I built… and later removed from my new version, involved having the original url load in a frame, and then provide a way for users to flag it as ‘spam’, ‘nsfw’, or ‘dead’. Then, we could provide the user with a warning that the following URL was flagged before, are you sure you want to continue? I didn’t like this as a solution in this way as it felt very obtrusive to have a rubyurl frame at the top of the browser window.
One person suggested a captcha to try and verify that the user is human, but there are problems with this.
- I really dislike captchas. ;-)
- This doesn’t prevent spammers from using the ShortURL gem, which does everything via an API.
In regards to the API, this could be enhanced by requiring that everyone register an email address to get an API key, but only solves the API abusers.
I’m starting to brainstorm some solutions that specifically help the requests made through the web. I haven’t checked the logs enough yet to verify it, but I have a strong suspicion that much of the abuse is happening through a web-based bot, not through ShortURL… because Ruby developers are nicer than that. (I hope…)
So, I am curious… dear readers of my blog. How might you solve this problem without disrupting the user experience? Or, should I just stick with what I’ve got going and find a better way to respond to pissed off bloggers who think I’m spamming them?
Discuss…
Enjoying the content? Be sure to subscribe to my RSS feed.
I just had a thought,
Could use robots.txt to prevent google and other SEs from spidering all the URLs except the homepage so RubyURL would show up in the SERPS.
Problem with this is no rubyurl will get google juice. I don’t think people would care except spammers.
Here’s a thought,
Instead of using captcha’s, you could try a more “human” method. One example of this could be: Instead of using a submit button of the normal variety, you could display a sampling of three (or so) random thumbnail images, and display a message like “Please click the Red Circle to submit” or “Please click the picture of the moon to submit”.
Just a thought, Ben
I read somewhere that it is quite effective to have a hidden field in your form. It seems that bots fill this field in, so it’s easy to filter them out. I have no personal experience with it and I doubt it will solve everything :-)
Regards, Stijn
How about using a blacklist database on the urls or some sort of Akismet based system?
For the gem and API you could go the Amazon and Google route and require a unique key for use. Generating and providing the key could be completely automated, but at least it’s an additional step (and it would provide a way to track and group spammer generated URLS).
Just a thought,
— Mando
What value do shortened URLs provide?
Few people type URLs and fewer still type arbitrary alpha hashes. What if the rubyURL generated actually had something to do with the content of the destination?
Using Hpricot and Net:HTTP, you could make a rubyURL that generated something readable and concise, with the added advantage that if the destination were spammy, it would likely be apparent in the generated URL, something which is often obscured by the original URL.
As far as requiring humans to fill out a form, perhaps form-spam-protection http://code.google.com/p/form-spam-protection/ would help out?
I have to say that I’m with the others when they suggest blacklisting. I’ve been using Akismet on my girlfriends blog and it gets rid of most, but not all spam.
In addition to Aksimet I’ve been looking at Project Honey Pot which uses a dns lookup to get information on a particular ip address. I haven’t been using it for long yet so I don’t have any concrete figures but I think the premise of Project Honey Pot is a good one.
By the way I’ve written a simple ruby interface for the http blacklisting service.
Please pull the plug on this “service” that is being abused. My servers are getting hammered by BotNets using your domain to redirect traffic.
You can make lame excuses on how your not responsible, but the abusive spam is using your systems with your knowledge. If someone complains about your running an open relay do you just ignore them too.
I have thousands of post attempts that use your domain rubyurl.com in order to work.
I have already spoken to my legal counsel and your are liable for not taking action to correct this situation. I am already debating if I should spend the money to file a case since that seems to be the only way to get your attention.
This site is obviously a better looking version of tinyurl.com. Have you contacted anybody at tinyurl to see what they have done? I’m sure the creator(s) of that site have run into this same issue :)