Tag Archives: Programming

opml to csv converter

Programming, ruby, software, webdev, , , , , ,

This is a first step in being able to make all  my  planets    configurable from anywhere. The following ruby script parses the opml file specified on the command line and generates a comma separated file with XML feed URL and feed title. In case of nested outline elements, it just picks the elements which actually have xmlUrl attribute (this will flatten the opml hierarchy which is used by google reader - for implementing labels and bloglines - for implementing folders)

require 'csv'
require "rexml/document"
include REXML
if ARGV.length >=1
fname = ARGV[0]
else
fname = "opml.xml"
end

doc = Document.new File.new(fname)
CSV.open('csvfile.csv', 'w') do |writer|
doc.elements.each("//outline[@type='rss']") {|element|
writer < <  [element.attribute("xmlUrl").value, element.attribute("text").value]
}
end

django queryset weirdness

django, Python, ,

I needed to reset the django admin password and found this page which tells us how to do this. It did not work for me, however I tried to get the user object explicitly, reset the password and save it and it worked!
Here is a session describing the behavior:

> python manage.py shell
In [1]: from django.contrib.auth.models import User
In [2]: u=User.objects.all()
In [3]: u[0].password
Out[3]: 'sha1$0913d$6c5cfefb89b3c77dc8573e466a943c7acd177f6b'
In [9]: u[0].set_password('testme')
In [10]: u[0].save()
In [11]: u[0].password
Out[11]: 'sha1$0913d$6c5cfefb89b3c77dc8573e466a943c7acd177f6b'
In [12]: q=User.objects.get(id=1)
In [13]: q
Out[13]: <user : root>
In [14]: q.set_password('testme')
In [15]: q.save()
In [16]: q.password
Out[16]: 'sha1$24e9c$868ff39f08c3bde96397e33ea6a8847a658a16bc'

Maybe this is related to queryset caching, which would print the same password even after calling set_password. But it looks like in the first method, the password does not even get written to the database. This might be a bug. I will need to run more tests and report (or possibly patch) it...

Update: Perhaps I am not clear in the above post. The weirdness is that in the above two scenarios u[0] and q should essentially reference the same user object, but calling set_password (and later save) method on the u[0] reference does not seem to work (i.e. the password attribute remains unchanged and changed password cannot be used on the admin pages) but calling the same method on q reference seems to work!

Update2:I got it finally. Thanks all (Esp SmileyChris) ! every array slice is a new query! so in the first solution, I need this to make it work:

u=User.objects.all()
u[0].password # to display current hash (Query #1)
q=u[0] # (Query #2)
q.set_password('testme')
q.save()
q.password
u[0].password # (Query #3) this should print updated hash!

Dear Lazyweb

audio, Programming, software, webdev, , , , ,

I am looking for a flash media player which can play audio/video content in browser. I am currently using del.icio.us's playtagger for now, but it is limited to mp3 only (I had to change the code a little bit to prevent it from adding multiple inclusions - my changed code is here) and it does not allow rewind/forward.

I also looked at this post which suggests using google video player, which would be nice as it could then also play video files, but it consumes a lot of real estate on the pages. There is also this website mentioned which is nicer but that player is bigger than I want too... And it is probably not a fair use as it is undocumented use...

I also checked out the odeo player, which is nice but you need to host the files on odeo, which is a no-no.

Here are my requirements for this with a scorecard for Playtagger and Google:

  • Easy to use. No need to add tag soup for each file. (Del.icio.us + Google -)
  • Small size on the page (need to collapse the buttons until user clicks on play button) (Del.icio.us + Google -)
  • Ability to rewind/fast forward (Del.icio.us - Google +)
  • Play different media formats (mp3/wav/ogg/avi/mpeg video, possibly realmedia) (Del.icio.us - Google +)

Any suggestions ?

Thanks!

Fix maharashtratimes.com font problems on firefox

devanagari, India, Programming, Scripting, webdev, , , , , , ,

Maharshtratimes.com (or maharashtratimes.indiatimes.com) is a marathi news website using unicode fonts. But it does not display correctly on firefox browser. The problem is because of a single HTML div which uses justified font style. It displays correctly on IE (which is why it is not getting fixed) - this could be because of firefox's buggy implementation of "align: justify" or that IE simply ignores that style (likely).

Anyway, here is a javascript one liner that you can bookmark and once the page is loaded, click on it to fix your font problem.

Fix Ma. Ta. -- Drag this link to your bookmarks.

I tried to create a greasemonkey script to do this automatically, but it's not working for some reason...

Update: Apparently it *is* a mozilla/firefox bug open for 4+ years. See here and here.

Update2: Here is a greasemonkey script by Saravana Kumar to fix this issue. Caution: you might want to change the included domains carefully (it by default runs on all http and https sites!)

Update3: Here is my greasemonkey script specific for maharashtratimes. Enjoy!

Update4: (2009-02-16) The original site seems to have removed this style attribute now. So the above post is now only for posterity.

Using Ruby for integer format conversions

Programming, ruby, Scripting, ,

Here are some recipes for interpreting stream of bytes as different C types. I will keep adding more to this as I go...

Convert a byte stream (embedded in a string) into 1 byte signed integers:
"\xfc\xfd\xfe\xff".unpack("c*")
=> [-4, -3, -2, -1]
Convert a byte stream (embedded in a string) into 1 byte unsigned integers:
"\xfc\xfd\xfe\xff".unpack("C*")
=> [252, 253, 254, 255]

Check if a site is phishing site.

Programming, Scripting, Security, webdev, , , , , ,

Here is the updated bookmarklet: Phishy? (tested on firefox 2.0 only!)

1. Drag this link to your bookmark. This checks if the site you are currently on is a phishing site.
2. Drag this link to your bookmark. This prompts for a URL and checks if it is a phising site.

Uses phishtank's check URL API.

If this does not work try turning debug to true above if you want to see the encoding.

Update: This still uses the GET method for checking the URL. Phishtank recommends using the POST interface (which will remove limitations on URL length: base64 inflates the length by 33%). Implementing that would need some kind of xmlhttprequest hackery. Stay tuned...

Update2: I got the AJAX bookmarklet ready, (thanks!)but it hits the infamous "uncaught exception: Permission denied to call method XMLHttpRequest.open" bug. i.e. you cannot do cross-domain xmlhttprequests. To solve that I think I need to convince PhishTank to host the javascript code, so the bookmarklet will insert a hidden iframe into the current page which will load the javascript from phishtank page, which will eventually make xmlhttprequest to phistank and display the result back. Are you listening PhishTank ?

Update3: Thanks to "till" who commented below, here is the bookmarklet using the POST method so now the solution will also work for really long URLs. Till's solution is good, but it makes users trust his site (in addition to phishtank). So basically user has to trust that he is not trying to filter the results being presented..

I have also merged the two earlier bookmarklets so that the current site location will be autopopulated in the prompt, so that user can easily change it if he wants to check a URL different from the one he currently is on.

Fixing the geocoder gem.

Programming, ruby,

I wanted to use the ruby geocoder library on the windows machine, but the installation of the gem failed due to some weird error. I checked the rubyforge project page to see if someone else had a similar problems and someone actually had, but the bug was open for a long time. I decided to fix this issue and found that the problem was due to the fact that windows platform does not allow characters '?' and '&' in the filename with any escaping, period. The said files were used (in a very innovative way I must say!) to test the library by modifying http.rb to return the test datafile contents instead of fetching the URL from the net. (yay open classes in ruby!). The way I fixed the problem was to change the filenames to use '_' instead of '?' and '__' instead of '&'.

I wrote to the developer, but there was no response. Anyway I managed to create a new GEM with the changed files so that this should be installable on windows now. Here are the files if you want to try installing the gem. (Also including the tgz because... it got generated anyway!)
geocoder-0.1.1.gem
geocoder-0.1.1.tgz

Phish Tank

Internet, Programming, ruby, Security, software, , , ,

http://www.phishtank.com is a new service which aims to help weed out phishing URLs and email addresses using wisdom of the crowds. Users can submit emails/URLs which they suspect of fraud and others can vote if they really are fraudulent or not. I think it is a great concept. There is a REST API using which applications can embed this webservice within them. So for example, there could be a outlook plugin which will display "phishy" email addresses in a special way in order to alert the user immediately. Same for web browsers which can render phishing websites in a special stylesheet. The applications can also add interface for the user to submit suspect pages and email easily without using web browsers.

I checked out the API and it does not feel like it is fully baked! There are interfaces for authorization and checking email/url status and submitting new emails/urls. Some things that stand out immediately are:

  • Exclusive use of SSL for the API access.
  • Parameter authentication (i.e. including cryptographic digest of all the parameters to ensure that parameters are not changed using man-in-the-middle attack)
  • Choice of xml or php output.

The api calling sequence works like this:

  1. User registers on the web for API access and gets api key and shared secret
  2. Using the API, application gets a frob (what is behind the name ?) and authorization url using auth.frob.request
  3. User has to authorize the frob using the authorization url specified in the response. (optionally you can specify callback url which the server will call for authorization, I will need to check this from home when I have access to a server -- the docs are very thin about the mechanism)
  4. Once authorized, app uses the frob and gets a token for short time API access (30 minutes in my tests) (auth.token.request)
  5. App can check token status which tells remaining time on token.(auth.token.status)
  6. App can revoke the token when it is done using it. (auth.token.revoke)
  7. The APIs for check.url, check.email, submit.url, submit.email then use the token.

I did not understand why there is a need for FROB in this, why can't you just get the token from api key and shared secret ? What problem are they solving by this indirection ?

Anyway, here is the ruby script that I used for testing this... I am planning to turn this into a module, but providing it here for early access...
phishtank.rb
config.yml

P.S. the check_url interface is not working, I am getting invalid token error. and the same token can be revoked successfully.
P.P.S. The API uses SSL (no cleartext api available) and ruby's open-uri library insists on checking the server SSL certificate which always fails (probably because signer needs to be trusted by openssl), I had to change it locally to ignore ssl verification in order to proceed.

Update (Oct/12/06): the check.url interface is finally working. For this API, the signature needs to be calculated before escaping the url. I refactored the ruby script a bit to remove redundant code and moved the configuration to a seperate file. I still need to work with the response parser and make it general for all types of responses. XML parsing gets so ugly so fast, it's amazing!

Yahoo Geocoding in ruby…

Internet, Programming, ruby, Scripting, webdev, , , , , , , ,

Find lattitude and longitude of any address

Yahoo just released a new beta of their maps webservice. Here is a small ruby script (inspired by Rasmus's PHP code ) that I wrote that returns Lattitude, Longitude of the address provided...

require 'open-uri'
require "rexml/document"
include REXML
url='http://api.local.yahoo.com/MapsService/V1/geocode?appid=yahoomap.rb&location='
puts 'Enter Location: '
address=gets
address=URI.escape(address)
result=URI(url+address).read
doc = Document.new result
r=doc.elements["/ResultSet/Result"]
print "Precision: ", r.attributes["precision"],"\n"
r.children.each { |c| print c.name, " : ",c.text,"\n"}

Update: Here is a link to the script in github or to the .