We’ve posted an update to our Iraq Deaths agent at http://twitter.com/iraqdeaths . Here I’m going to journal and document the work involved in making this actually work.
At a high level this agent makes a daily post to twitter to broadcast any deaths reported by the Iraq War Bodycount database. I should mention that without their efforts at keeping and surfacing these records none of this would be possible. This kind of work itself is emotionally challenging and I want to applaud them.
Design-wise this was a project that Paige and I thought up and were excited by. We did it in a morning while we were both supposed to be doing other work – but the fact is it resonated with work we both care about. Speaking for myself at Meedan we’ve been looking for ways to help social networks bridge the barriers of language and I saw this as a way to contribute to a project that helped people keep attention on something that is normally invisible. For me this was a rewarding project because I enjoy removing the technical boundary between design and implementation. Often I like to work with designers to put real muscle underneath their vision.
The technical implementation consists of three stages
- collection
- analysis
- publishing
In the collections stage we talk to the iraq war bodycount database – pulling the entire corpus of deaths and then plucking them out of their csv structured text – like so:
require 'net/http' require 'rubygems' require "fastercsv" url = "http://www.iraqbodycount.org/database/download/ibc-individuals" data = Net::HTTP.get_response(URI.parse(url)).body puts "fetched" @results = FasterCSV.parse(data) @deaths = [] inside_header = true @results.each do |death| if death[0] == "IBC code" inside_header = false elsif inside_header == false @deaths << death end end
After this stage we go and add any new data to our own database. In this way we keep a running track of any changes and can act only on changes rather than on all data I first attempted to use sqlite3 but then ended up using datamapper - like so:
require 'rubygems'
require 'dm-core'
DataMapper.setup(:default, {
:adapter => 'postgres',
:database => "endiraqwar",
:username => 'endiraqwar',
:host => 'localhost'
})
class Death
include DataMapper::Resource
property :id, Integer, :serial => true
property :code, String
property :name, Text
property :age, Text
property :sex, Text
property :marital, Text
property :parental, String
property :earliest, DateTime
property :latest, DateTime
property :location, Text
property :created_at, DateTime
property :posted, DateTime, :default => nil
end
# DataMapper.auto_migrate!
@deaths.each do |death|
if Death.first(:code => death[0] )
puts "We already found this death #{death[1]} #{death[0]} so not saving"
next
end
# take a second to convert the date phrase into a machine date
death[6] = DateTime.parse(death[6])
death[7] = DateTime.parse(death[7])
record = Death.new(
:code => death[0],
:name => death[1],
:age => death[2],
:sex => death[3],
:marital => death[4],
:parental => death[5],
:earliest => death[6],
:latest => death[7],
:location => death[8]
)
puts "recording the passing of #{record.name} at #{record.earliest} and #{record.code}"
record.save
end
The last phase is to report the actual deaths. We rely on the twitter gem to do this - I find I am using this gem more and more and it is quite convenient - like so:
twittercap = 50 # twitter this many posts max
require 'twitter'
twitter = Twitter::Base.new("iraqdeaths",secret_password)
@deaths = Death.all(:order => [:earliest.desc], :limit => twittercap)
@copyofdeaths = []
@deaths.each do |death|
@copyofdeaths << death
end
@copyofdeaths.reverse.each do |death|
# publish deaths that are new
next if death.posted != nil
result = twitter.post("#{death.name}, #{death.age}, #{death.sex}, #{death.marital}, #{death.parental} killed on #{death.earliest.strftime("%d %b %Y")} at L:#{death.location}")
# remember that we already published this death
death.posted = DateTime.now
death.save
puts "posted the death of #{death.name} #{death.code}"
end
I try to be very careful to never update my own understanding of the database record until I am absolutely certain that Twitter has been updated. Even if my agent crashes I want it to crash in a way that doesn't spray garbage all over twitterspace.
Overall, as you can see, the high aspiration of building something that makes a statement is connected to real metal underneath. We're grateful to the http://www.iraqbodycount.org organization for making this possible.