Iraq Deaths TwitterBot

anselm — Thu, 02 Apr 2009 18:03:17 +0000

We’ve posted an update to our Iraq Deaths agent at http://twitter.com/iraqdeaths . Here I’m going to journal and document the work involved in making this actually work.

At a high level this agent makes a daily post to twitter to broadcast any deaths reported by the Iraq War Bodycount database. I should mention that without their efforts at keeping and surfacing these records none of this would be possible. This kind of work itself is emotionally challenging and I want to applaud them.

Design-wise this was a project that Paige and I thought up and were excited by. We did it in a morning while we were both supposed to be doing other work – but the fact is it resonated with work we both care about. Speaking for myself at Meedan we’ve been looking for ways to help social networks bridge the barriers of language and I saw this as a way to contribute to a project that helped people keep attention on something that is normally invisible. For me this was a rewarding project because I enjoy removing the technical boundary between design and implementation. Often I like to work with designers to put real muscle underneath their vision.

The technical implementation consists of three stages

collection
analysis
publishing

In the collections stage we talk to the iraq war bodycount database – pulling the entire corpus of deaths and then plucking them out of their csv structured text – like so:

require 'net/http'
require 'rubygems'
require "fastercsv"
url = "http://www.iraqbodycount.org/database/download/ibc-individuals"
data = Net::HTTP.get_response(URI.parse(url)).body
puts "fetched"
@results = FasterCSV.parse(data)
@deaths = []
inside_header = true
@results.each do |death|
 if death[0] == "IBC code"
   inside_header = false
 elsif inside_header == false
   @deaths << death
 end
end

After this stage we go and add any new data to our own database. In this way we keep a running track of any changes and can act only on changes rather than on all data I first attempted to use sqlite3 but then ended up using datamapper - like so:

require 'rubygems'
require 'dm-core'
DataMapper.setup(:default, {
   :adapter  => 'postgres',
   :database => "endiraqwar",
   :username => 'endiraqwar',
   :host     => 'localhost'
})
class Death
 include DataMapper::Resource
 property :id,         Integer, :serial => true
 property :code,       String
 property :name,       Text
 property :age,        Text
 property :sex,        Text
 property :marital,    Text
 property :parental,   String
 property :earliest,   DateTime
 property :latest,     DateTime
 property :location,   Text
 property :created_at, DateTime
 property :posted,     DateTime, :default => nil
end
# DataMapper.auto_migrate!
@deaths.each do |death|
 if Death.first(:code => death[0] )
   puts "We already found this death #{death[1]} #{death[0]} so not saving"
   next
 end
 # take a second to convert the date phrase into a machine date
 death[6] = DateTime.parse(death[6])
 death[7] = DateTime.parse(death[7])
 record = Death.new(
             :code => death[0],
             :name => death[1],
             :age => death[2],
             :sex => death[3],
             :marital => death[4],
             :parental => death[5],
             :earliest => death[6],
             :latest => death[7],
             :location => death[8]
          )
 puts "recording the passing of #{record.name} at #{record.earliest} and #{record.code}"
 record.save
end

The last phase is to report the actual deaths. We rely on the twitter gem to do this - I find I am using this gem more and more and it is quite convenient - like so:

twittercap = 50 # twitter this many posts max
require 'twitter'
twitter = Twitter::Base.new("iraqdeaths",secret_password)
@deaths = Death.all(:order => [:earliest.desc], :limit => twittercap)
@copyofdeaths = []
@deaths.each do |death|
 @copyofdeaths << death
end
@copyofdeaths.reverse.each do |death|
 # publish deaths that are new
 next if death.posted != nil
 result = twitter.post("#{death.name}, #{death.age}, #{death.sex}, #{death.marital}, #{death.parental} killed on #{death.earliest.strftime("%d %b %Y")} at L:#{death.location}")
 # remember that we already published this death
 death.posted = DateTime.now
 death.save
 puts "posted the death of #{death.name} #{death.code}"
end

I try to be very careful to never update my own understanding of the database record until I am absolutely certain that Twitter has been updated. Even if my agent crashes I want it to crash in a way that doesn't spray garbage all over twitterspace.

Overall, as you can see, the high aspiration of building something that makes a statement is connected to real metal underneath. We're grateful to the http://www.iraqbodycount.org organization for making this possible.

MakerLab Blog » iraq

Iraq Deaths TwitterBot