Building a Twitter GenServer with ExTwitter part 1

Posted on Thu 08 September 2016 in elixir

Premise

This article tackles my learning experience in building a GenServer process that talks with Twitter. I am learning the Elixir language in my evenings, so bear with me and please comment if you will find inaccuracies in this article.

I will not comment every line of code of my experiment, however feel free to drop a comment if there's something that triggers your interest.

Objective

I wanted to create a self-contained process (a GenServer in erlang/elixir terms) that would speak to Twitter. The interface of the process should be easy enough to be able to integrate with an application (being web or of another type)
My personal goal would be to integrate with Phoenix channels, however this is not the scope of this article ;)

And... I'll present you the GenServer interface

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
defmodule Tweetyodel.Worker do
  use GenServer

  def start_link(name) do
    GenServer.start_link(__MODULE__, [], name: via_tuple(name))
  end

  def init(_)  do
    schedule_cleanup()
    {:ok, %{}}
  end

  # API

  def start_stream(namespace, topic, timer_milliseconds \\ @start_stream_after) do
    GenServer.call(via_tuple(namespace), %{start_stream: topic, timer: timer_milliseconds})
  end

  def entries(namespace)  do
    GenServer.call(via_tuple(namespace), :entries)
  end

  def stop_stream(namespace) do
    GenServer.call(via_tuple(namespace), :stop_stream)
  end

  def search(namespace, topic) do
    GenServer.call(via_tuple(namespace), %{search: topic})
  end

Let's break it down...

Basic GenServer functions

1
2
defmodule Tweetyodel.Worker do
  use GenServer

Every respected GenServer starts with the macro directive use GenServer.

1
2
3
  def start_link(name) do
    GenServer.start_link(__MODULE__, [], name: via_tuple(name))
  end

To have a fully functional GenServer we have to implement some functions of the GenServer "interface", for instance start_link is implemented in this case, which calls the start_link implementation to start the GenServer by passing to it a __MODULE__ parameter which will expand to the current module name (i.e. Tweetyodel.Worker)

I left the second parameter Args empty, because I do not need it for the moment.

The third argument name: via_tuple(name) is a way to register processes by name in a registry. I will leave it for now here, however a registry helps you to lookup your process by name (you could see it as a sort of namespace). Also by locating your process by name, you can obtain its pid (process id) if needed.

1
2
3
4
  def init(_)  do
    schedule_cleanup()
    {:ok, %{}}
  end

To implement a "fully functional" GenServer in Elixir you have also to specify the init function. It's the place where you want to put all the operations to initialize the GenServer or if you want to open a connection to an external service.

init calls schedule_cleanup() which is a function that will purge the tweets periodically. It has to be called in the init phase to schedule a timer that will reduce the number of tweets in the GenServer state. But what is {:ok, %{}}? It's the tuple returned from the init function of the GenServer to initialize the state of the process.

In this case it's just an empty map which will be populated when using the Tweetyodel.Worker.

The Tweetyodel interface

After having our bare-bone GenServer process implementation, we might want to give to the user of our process something to play with ;) Here is the API of my open source pet project

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
  # API

  # Start a twitter stream after timer_milliseconds
  def start_stream(namespace, topic, timer_milliseconds \\ @start_stream_after) do
    GenServer.call(via_tuple(namespace), %{start_stream: topic, timer: timer_milliseconds})
  end

  # Returns the entries stored inside the `GenServer` process (i.e. the state of the process)
  def entries(namespace)  do
    GenServer.call(via_tuple(namespace), :entries)
  end

  # Stops the Twitter streams and does something more ;) We'll see later on
  def stop_stream(namespace) do
    GenServer.call(via_tuple(namespace), :stop_stream)
  end

  # Executes a direct Twitter search
  def search(namespace, topic) do
    GenServer.call(via_tuple(namespace), %{search: topic})
  end

Without going (yet) into the GenServer callbacks implementation, let's see it in action...

How it works

1
2
3
4
5
6
7
8
{:ok, pid} = Tweetyodel.Worker.Supervisor.start_tweet("tweetyodel")

# Apple has always tweets
Tweetyodel.Worker.start_stream("tweetyodel", "apple")

# Fetch only the the first 5 tweets and their text
# NOTE that pulling data from twitter starts after 10 seconds by default (you can change it although)
Enum.map(Tweetyodel.Worker.entries("tweetyodel"), fn tweet -> tweet.text end) |> Enum.take(5)

Quite simple no? I did not described the Tweetyodel.Worker.Supervisor.start_tweet("tweetyodel") incantation, but just think that this command starts and monitors the life of the GenServer, by killing it and re-starting it if needed. For instance, if an error occurs.

Also you might have noticed that start_tweet has an argument which is tweetyodel. This argument is the namespace (or to put it simply the name) that we use to refer/identify a particular process.

The module used to implement the registry is called gproc, you can find in hex

In short, I can talk to one of my processes just by using a name (the namespace) to refer to it. You could create many processes dynamically with different names by just using another string:

1
{:ok, _} = Tweetyodel.Worker.Supervisor.start_tweet("elixir")

End of part 1 (part 2 will come soon...) It's here !

P.S. The full repository of my pet project is available on github

(Things might change inside the life of the repository... shht don't tell anyone)