twitter4s

twitter4s: A Scala client for the Twitter API

Daniela Sfregola

A few months ago, I started looking into the Twitter API and I have developed twitter4s, an asynchronous non-blocking Twitter client in Scala.

In this article, we will introduce twitter4s providing examples of how to download tweets from a user timeline and how to perform some simple data analysis.

The code shown in this tutorial is available on Github.

Getting Started

The Twitter API can be accessed by means of a registered app. So, first of all, we need to register our app. Login with your twitter account and have a look at the Twitter API terms and conditions.
If you are happy with them, register your app at http://apps.twitter.com.

In order to do so, you will need to provide an app name and an brief description of what it does. After the registration, a consumer key and consumer secret will be provided: save them as we will need them when setting up twitter4s.
Also, generate an access key and access secret and make sure you have the correct permissions: for this tutorial “Read Only” is enough.

Finally, please note that the Twitter API has rate limits — have a look at the Twitter’s development website for more information.
Also, have a look at the rate limits chart where the rate limit for each endpoint is summarized.

Setup

If not already there, add Maven Central as resolver in your SBT configuration:

resolvers += "Maven central" at "http://repo1.maven.org/maven2/"

Also, you need to include the library as your dependency:

libraryDependencies ++= Seq(
  "com.danielasfregola" %% "twitter4s" % "0.2.1"
)

Usage

Add your consumer and access token to your configuration file and initialize your Twitter Client:

import com.danielasfregola.twitter4s.TwitterClient

val client = new TwitterClient()

Alternatively, you can also specify your tokens directly when creating the client:

import com.danielasfregola.twitter4s.TwitterClient
import com.danielasfregola.twitter4s.entities.{AccessToken, ConsumerToken}

val consumerToken = ConsumerToken(key = "my-consumer-key", secret = "my-consumer-secret")
val accessToken = AccessToken(key = "my-access-key", secret = "my-access-secret")
val client = new TwitterClient(consumerToken, accessToken)

Now that our Twitter Client has been initialized, we are now ready to use it! 😀
Have a look at its documentation for a complete list of the supported functionalities.

Top Hashtags in Timeline

As a sample code, let’s collect the tweets in a user timeline and display the top 10 hashtags used. In the tutorial, we will download and analyze tweets by Martin Odersky (the creator of Scala).

First, we need to get the tweets from the user timeline:

client.getUserTimelineForUser(screen_name = "odersky", count = 200)

The method getUserTimelineForUser (see scaladoc) return type is Future[Seq[Tweet]].
Note that a Tweet is a quite rich case class that contains a lot of information (see its scaladoc): it has more than 22 fields!
The need of having huge case classes is the reason why this library doesn’t support Scala versions older than 2.11: previous versions allow up to 22 fields in a case class.

In order to retrieve the hashtags used in a Tweet, we don’t have to parse the text of the tweet, as the Twitter API has already done all the hard work for us: we just need to access the Entities field and count how many times each hashtag is used:

  def getTopHashtags(tweets: Seq[Tweet], n: Int = 10): Seq[(String, Int)] = {
    val hashtags: Seq[Seq[HashTag]] = tweets.map { tweet =>
      tweet.entities.map(_.hashtags).getOrElse(Seq.empty)
    }
    val hashtagTexts: Seq[String] = hashtags.flatten.map(_.text.toLowerCase)
    val hashtagFrequencies: Map[String, Int] = hashtagTexts.groupBy(identity).mapValues(_.size)
    hashtagFrequencies.toSeq.sortBy { case (entity, frequency) => -frequency }.take(n)
  }

Let’s put everything together and add some code to print the results with a nice layout:

import com.danielasfregola.twitter4s.TwitterClient
import com.danielasfregola.twitter4s.entities.{HashTag, Tweet}

import scala.concurrent.ExecutionContext.Implicits.global

object UserTopHashtags extends App {

  def getTopHashtags(tweets: Seq[Tweet], n: Int = 10): Seq[(String, Int)] = {
    val hashtags: Seq[Seq[HashTag]] = tweets.map { tweet =>
      tweet.entities.map(_.hashtags).getOrElse(Seq.empty)
    }
    val hashtagTexts: Seq[String] = hashtags.flatten.map(_.text.toLowerCase)
    val hashtagFrequencies: Map[String, Int] = hashtagTexts.groupBy(identity).mapValues(_.size)
    hashtagFrequencies.toSeq.sortBy { case (entity, frequency) => -frequency }.take(n)
  }

  val client = new TwitterClient()

  val user = "odersky"

  client.getUserTimelineForUser(screen_name = user, count = 200).map { tweets =>
    val topHashtags: Seq[((String, Int), Int)] = getTopHashtags(tweets).zipWithIndex
    val rankings = topHashtags.map { case ((entity, frequency), idx) => s"[${idx + 1}] $entity (found $frequency times)"}
    println(s"${user.toUpperCase}'S TOP HASHTAGS:")
    println(rankings.mkString("\n"))
  }

}

At the time of this writing, running the following code generates the following output:

ODERSKY'S TOP HASHTAGS:
[1] scala (found 25 times)
[2] scaladays (found 5 times)
[3] scalajs (found 4 times)
[4] progfun (found 3 times)
[5] coursera (found 2 times)
[6] scalax (found 1 times)
[7] community (found 1 times)
[8] aws (found 1 times)
[9] iexpectmoreofapple (found 1 times)
[10] scalamatsuri (found 1 times)

Summary

In this article we have introduced a new asynchronous non-blocking Scala Client for Twitter, called twitter4s.
We have described how to register our app, setup the Twitter Client and we have provided a sample code to download tweets from a user timeline and analyze their hashtags.

The code shown in this tutorial can be found here.

Published by Daniela Sfregola

Author of Get Programming with Scala, Open Source Contributor and Maintainer, Software Geek

36 thoughts on “twitter4s: A Scala client for the Twitter API”

  1. Hi, Daniela,

    thank you for the client. To your knowledge, are people using it? Also, if I need to post a text and a picture, will it do it?

    Thank you and further success.

    Like

    Reply

    1. Hi Mark, unfortunately the current version only allows you to post text without a picture. From the next release, you should also be able to post media together with some text.

      Regards,
      Daniela

      Like

      Reply

      1. Hi, Daniela,

        Your library is amazingly complete, hundreds of methods in the implementations. If I can figure it out, I may suggest a patch for uploading an image as well.

        Thank you,
        Mark

        Like

      2. Hi Mark, I have already done something as part of the 0.2-SNAPSHOT version — but I haven’t published yet as I’d like to test it a bit more and add more features. Feel free to have a look and contribute! 🙂

        Like

    2. Hi, Daniela,

      I tried this in my build.sbt

      libraryDependencies ++= Seq(
      “com.danielasfregola” %% “twitter4s” % “0.2-SNAPSHOT”
      )

      and I get this error below. I probably miss some understanding in the package publishing in repos.

      Thank you

      Error:Error while importing SBT project:…

      [info] Resolving org.scala-lang#scala-reflect;2.11.8 ...
[info] Resolving org.scala-lang.modules#scala-parser-combinators_2.11;1.0.4 ...
[info] Resolving org.scala-lang.modules#scala-xml_2.11;1.0.5 ...
[info] Resolving org.scala-lang#scala-compiler;2.11.8 ...
[info] Resolving org.scala-lang.modules#scala-xml_2.11;1.0.4 ...
[info] Resolving jline#jline;2.12.1 ...
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: com.danielasfregola#twitter4s_2.11;0.2-SNAPSHOT: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn] 	Note: Unresolved dependencies path:
[warn] 		com.danielasfregola:twitter4s_2.11:0.2-SNAPSHOT (/home/mark/projects/SiteOps/build.sbt#L9-12)
[warn] 		  +- default:siteops_2.11:1.0
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: com.danielasfregola#twitter4s_2.11;0.2-SNAPSHOT: not found
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: com.danielasfregola#twitter4s_2.11;0.2-SNAPSHOT: not found
[error] Total time: 4 s, completed Jun 20, 2016 11:58:39 PM

      See complete log in /home/mark/.IntelliJIdea15/system/log/sbt.last.log

      Like

      Reply

      4. Hi, Daniela,

        it all works for me (with your help), and here you can observe the results, https://twitter.com/TalmudTweets, daily posts automated instead of me doing it manually. You can also see why image posting would be nice: previously I was posting manually, with images, and it looks more appealing.

        Regards,
        Mark

        Like

      5. Hi Mark, I am glad is working for you. I am finishing the image uploa — it shouldn’t take me more than a week or so — and I am going to publish version 0.2 asap

        Like

    2. Hi, Daniela,

      0.1 used to work, but now neither 0.1 or 0.2 work. Here is a part of my build.sbt

      resolvers += Resolver.sonatypeRepo(“snapshots”)

      libraryDependencies ++= Seq(
      “org.specs2” %% “specs2-core” % “3.8.2” % “test”,
      “org.jsoup” % “jsoup” % “1.9.2”,
      “com.danielasfregola” %% “twitter4s” % “0.1-SNAPSHOT”
      )

      and here is what I get

      [warn] ::::::::::::::::::::::::::::::::::::::::::::::
      [warn] :: UNRESOLVED DEPENDENCIES ::
      [warn] ::::::::::::::::::::::::::::::::::::::::::::::
      [warn] :: com.danielasfregola#twitter4s_2.11;0.1-SNAPSHOT: not found
      [warn] ::::::::::::::::::::::::::::::::::::::::::::::
      [warn]
      [warn] Note: Unresolved dependencies path:
      [warn] com.danielasfregola:twitter4s_2.11:0.1-SNAPSHOT (/Users/mark/projects/SiteOps/build.sbt#L11-16)
      [warn] +- default:siteops_2.11:1.0
      sbt.ResolveException: unresolved dependency: com.danielasfregola#twitter4s_2.11;0.1-SNAPSHOT: not found

      Another question about the keys. You put them into application.conf, and I currently use environmental variables. If I put the keys into application.conf, how do I make sure that I do not accidentally push them to the repository?

      Thank you for your good work.
      Mark

      Like

      Reply

      4. Hi, Daniela, things are working great now. I may be being obtuse, but my process never ends, running in a separate thread. I expected it to terminate, but it does the work, then hangs.

        Thank you

        Like

  5. Hi Daniela. Thank you for such a useful library.

    How do I get to set proxy details for a TwitterClient instance. I am failing to fetch data using the API on a network behind a corporate proxy.

    Like

    Reply

    1. Hi Pepukayi,
      I am afraid this is something that we do not support at the moment. If this is important for you, please create an issue with all the details of what you need or even open a PR about it — contributions are always welcome!

      Cheers,
      Daniela

      Like

      Reply

      1. Hi Daniela, Okay thank you.

        Is there a way I can write the whole contents of the Future[Seq[Tweet]] that is returned by the getUserTimelineForUser() method to a textfile?

        Like

    1. Hi Mark,
      absolutely, this is my highest priority right now! We do want to support Scala 2.12 (see https://github.com/DanielaSfregola/twitter4s/issues/54).

      Unfortunately, it is not as simple as it looks as the project is currently based on Spray, that does not support Scala 2.12. I am migrating the project to Akka Http (see https://github.com/DanielaSfregola/twitter4s/issues/55 for updates on its progress). Once the project has been migrated to Akka Http, supporting Scala 2.12 should not be a problem.

      Like

      Reply

  8. Hi Daniela, this is a great library. Thanks for all your hard work! I do have a quick question about it though. I am using it to write an app that calculates metrics about tweets from the sample stream and I noticed that when I look at the text of the tweets the emojis that were present were replaced with a ‘?’. Is there something else I need to do to ensure that the emojis are properly converted to their unicode characters? Thanks!

    Like

    Reply

      1. Thanks, Daniela. I think I found what was wrong. I was actually developing on Windows and there was some issue where the Windows command line tool wasn’t using the correct encoding. I couldn’t get it to work even when changing my encoding, so I just switched to Ubuntu and the emojis show up just fine in the terminal. Thanks!

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s