PyOhio
Building a Twitter Metadata Scraper: Learnings from Working with a New Twitter API
By
Elizabeth Wehner
BuzzFeed identified a need for Twitter to scrape our metadata more frequently, and worked with Twitter to build a new API that is currently in beta testing. In this talk I will go through the process of building a "Twitter Scraper" that accesses this new API. I will highlight various challenges encountered along the way, and show how I ultimately solved these problems.
Saturday 3:45 p.m.–4:15 p.m. in Barbie Tootle
BuzzFeed (especially our News team) creates a lot of time-sensitive content. But when shared on Twitter, the twitter-share data would remain fixed. This could lead to articles that changed from "So-and-so leads in election" to "so and so won the election" still being shared with an old title. BuzzFeed worked with Twitter on the development of a new API that could be used to rescrape the metadata to update share card information on request. I then built a "twitter scraper" using Python 3.5 that would request a rescrape every time we updated a published article. This proved especially tricky as existing twitter API support did not not quite fit our needs, requiring new solutions on our end. In this talk I'll go through the challenges involved in working with the new Twitter API and how I ultimately solved them.
Elizabeth Wehner
I'm a full-stack web developer at BuzzFeed. I currently work on technology supporting quizzes, but have also worked on news products as well as our content management system for making new posts. I live in Minneapolis, but am from Ohio originally. I'm also super into board games.
