15 Feb 2017
Last year, when I lived in Utah, I slapped together a Google Map of ACA Locations. Back when I was working on it, I lamented my use of Google Maps. The licensing seemed too restricive for my purpose — to openly share what I felt was public information.
To it’s credit, Google Maps is incredibly easy to use. Provided you’ve already GeoCoded your data, it’s as easy as uploading a .csv.
As it turns out, using Mapbox and OpenSteetMap turned out to be just as easy.
Mapbox Implementation
Honestly, there wasn’t much to this. Mapbox provides some gorgeous, tweakable tiles out of box, and provided me with the embed below.
Design-wise, I opted to go slightly modify one of their stock themes, but in the future I might make the time to develop a custom theme.
For now, I’m happy with this rudimentary map.
If you’d like to add to the project, feel free to check out the Github repo.
10 Jun 2014
The Problem and it’s Obvious Solution
It’s been said, that when all you have is a hammer everything looks like a nail. Lately, I’ve been wielding a hammer called “web scraping”, and everywhere I look I see nails in need of hammering.
With that said, I’ve been doing some work with the great Ruby Gem Nokogiri lately and decided to put it to some use, by using it on Craigslist.
In the past, I’ve found work in the Craigslist Computer Gigs [/cpg/] section in various cities. But I’d been irritated by having to open tabs for each new city, and it’s /cpg/ listing.
The solution isn’t graceful, but is obvious: RSS.
RSS isn’t discussed much anymore since Google shuttered it’s Reader service a few months ago. I recall their reasoning being along the lines of, “RSS isn’t really used much anymore …”. The nerds were upset. I was one of them.
Shortly after that, Twitter stopped supporting RSS feeds as well, and RSS entered it’s current phase: that orange icon on many a site, but not fully utilized by visitors more comfortable with Google+ and Facebook “Like” buttons.
To me, the sight of it has the air of an older web. One dedicated to open formats, and open data. Which is probably why Craigslist still supports it.
Craigslist
Craigslist.org features an RSS feed for nearly every page of it’s classified ads. From personals, event listings, and the especially the category we’re intereted in: job postings.
Getting the Data
The main page of Craigslist.org (CL) lists over 400 different sites, which provide the base URL for each city’s particular CL. You could take a few hours out of your day and copy-paste all of those URL’s into a text document or you could fire up an IRB session and tinker with Nokogiri for a few minutes.
I know what choice I made.
To make this process that much easier,I’ve put all the URL’s in a .csv for you, which you can download here [dropbox link].
###Tweaking the URL’s
Here’s a quick peek of the list of URL’s:
URL’s
- http//auburn.craigslist.org
- http//bham.craigslist.org
- http//dothan.craigslist.org
- http//shoals.craigslist.org
- http//gadsden.craigslist.org
The url paths are the same for all of the craiglist clasified pages. Simply adding /cpg/ to the end of any domain will get us the page we need. You can, obviously, add any sub-page — and search parameter — but for for my purposes all I needed was the /cpg added to the 400+ URLs.
This is where SublimeText 2’s multiple cursor feature eats the lunch of all text editors on the block. If you’re not hip to it, watch this video now and forever change the way you see a text editor.
Now, we have a text document with the URL’s appended with exactly what we need. Something like this:
Updated URL’s
- http//auburn.craigslist.org/cpg/index.rss
- http//bham.craigslist.org/cpg/index.rss
- http//dothan.craigslist.org/cpg/index.rss
- http//shoals.craigslist.org/cpg/index.rss
- http//gadsden.craigslist.org/cpg/index.rss
Creating an RSS feed will be the easiest part. There are lots of services online that will do the conversion of a text or .csv of urls into an OPML file.
I used Feedshow, having used them in the past, but googling around can probably yield something comparable or out right better.
After importing my data, and exporting out an OPML file I was then able to take my 500+ RSS feeds on to the RSS reader of my choice.
I first tried Feedly, but discovered they’d had issues updating their Craigslist feeds in the past. To Feedly’s credit, they worked hard and did a great job of fixing the issue, but I still decided to seek other options.
Right now, I’m switching back and forth between Vienna and RSS Owl. I’m a long-time NetNewsWire user, but decided that importing a file of 400+ feeds into a libary of over 600 might not be the best idea.
A Few Notes
I’ve been using this little system for a few days now, and I’m finding it works for me. I’m able to view and respond to ads at a rate, I’m pleased with.
Also, I’m not certain, but I feel that Craigslist might be delaying RSS feeds in order to drive traffic to the site … but that might just be silly thinking.
The system could use a little tweaking. For example, I’m considering creating specialized search URL’s to increase the signal to noise ratio inherent in Craigslist ads.
I’m sure there are any number of refinements I’ll make, but as of now: I’m pleased.
30 May 2014
Background
Over the last few months I’ve gotten more interested in plotting events on a map. This curiousity merges well with my anger over America’s covert wars in Yemen, Afghanistan, and Pakistan.
In particular, the use of “unmanned combat air vehicles” known more commonly as drones to conduct, “drone strikes”. Drones have played a signifiant role in America’s covert wars. Their use on the battlefield has either been directly responsible, or aided air strikes in remote parts of Yemen, Pakistan, and Somalia.
The Map
The map below, created with TileMill, Mapbox, and data provided by dronestream paints an alarming picture.
Each point marks a reported strike. Points in brightest red, draw attention to strikes whose numbers killed exceeded 20 people.
The map is interactive. Clicking a point will provide you with the number of civilian deaths reported, as well as how number of children killed as a result of these attacks.
Data & Methodology
This map was created with free, and publicly available data and tools. Dronestream provides “Real-time and historical data about every reported United States drone strike”. The information is available as a .json API.
I used the Raw JSON Feed dronesteam provided and converted the .json file a .csv using konklone’s JSON to CSV coverter.
After that, it was as easy as using a TileMill’s tutorials to plot the points a map.
Acquiring the tiles, of Yemen, Pakistan, and Somalia wasn’t relatively easy. I used a shapefile of roads provided free by Diva GIS.
If you have questions or comments about this map, please feel free to contact me via email at hey.bernard [at] gmail [dot] com.
16 May 2014
“It’s easy if you try …”
I took a few moments to install and run Jekyll today. I’ve tried all types of static site generators. Even the Jekyll derivitives, like Octopress — which still looks super rad to me.
Here’s what I think is cool thus far.
- markdown support
- supposedly inline images (see the bottom)
- fast builds (in a moment I’ll run
jekyll build and boom. Done. Now that’s awesome.)
“I showed this to my man here… What’d you think?”
I think I’m into it.
