I was given a task to scrape a site that uses ssl. So the first thing I thought of was using Mechanize. When going to the site that uses ssl, I was presented with the following error

OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

At first it was a real pain having to find the right tutorial to set the certificates that you'd use and how to tell mechanize to use them. To help those who might need this in the future, here it is.

First, download the list of certificates from http://curl.haxx.se/docs/caextract.html and save it to a filename. We'll use cacert.pem for this blog. Move cacert.pem inside the lib folder.

Next part is setting up the scraper agent.

cert_store = OpenSSL::X509::Store.new
cert_store.add_file = 'lib/cacert.pem'

agent = Mechanize.new
agent.cert_store = cert_store
agent.get 'https://github.com'

And that's it! Enjoy!