Notes on building a cron-powered Twitter countdown bot

⌫ Home

This blog post is about @looming_midterm, a bot that counts down the days to the next midterm election. You can find the code for this bot at benlk/looming_midterms.

@looming_midterm was inspired by this tweet from Cordelia Yu, and several experiments with sferik/t. I thought making a simple script that generated a message and piped it to t wouldn't take long. I was wrong. Here's hoping I can save you some time when you make a bot like this.

In the end, it took three months to:

  • get dates to output reliable
  • get my cronjob box to use the correct time zone
  • install t on my cronjob box
  • prevent the script from posting multiple times

Most of that was because I was doing other things, but a lot of it was because I had to wait for cronjobs to run and generate logs.

1. Installing t

You're going to need to make sure that you have a newer version of Ruby. If the computer you're planning to use to run this bot is running Debian 7, you'll want to upgrade to Debian 8 or look into something like rvm or rbenv for managing your Ruby versions.

2. Getting Twitter credentials for t

To get API keys, Twitter requires you to confirm a phone number. Twitter will text you a confirmation code, and may need that phone number for later interactions with the bot's Twitter account. Consider seting up a Google Voice account for your newsroom or organization, so that you aren't depending upon one employee's cellphone to receive confirmation texts.

3. Saving Twitter credentials for t

By default, t creates a .trc configuration file in your home directory. You can run t with the --profile=/path/to/your.trc argument to load credentials from a different file.

Once you've set up t, cut the credentials from the .trc file in your home directory and paste them into a new .trc file in your project directory. Do not version control the .trc - treat it like passwords. Keep it out of version control. Use whatever your newsroom's normal password management scheme is, whether that's a commercial solution like 1Password, a home-brewed cloud storage like NPR's cloud secrets storage rig or just an encrypted file that gets emailed between developers or scp'd from the production environment.

Your .trc will look like this:

---
configuration:
  default_profile:
  - looming_midterm
  - #########################
profiles:
  looming_midterm:
    #########################:
      username: looming_midterm
      consumer_key: #########################
      consumer_secret: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
      token: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
      secret: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It's a yaml file. :shrug:

4. Piping messages into t

Writing your cron job will be easier if you separate the message-generation portion of the program from the part that tweets. In @looming_midterm, I did this by having two bash scripts, message.sh and tweet-countdown.sh. For the purposes of demonstration, we'll use this code as the contents of message.sh:

#!/bin/bash
# message.sh
# generates a tweet
echo "Hello Twitter!"

Once you've saved message.sh, make it executable for your user with chmod u+x message.sh

Now let's create a tweet.sh that pipes the output of message.sh to t, using xargs:

#!/bin/bash
# tweet.sh
# tweet a tweet
./message.sh | xargs -I '{}' t update --profile=/path/to/your.trc '{}'

Bash [pipes] are useful things. They turn the output of one program into the input of another. xargs takes piped input and allows you to use that piped input as an argument on a command.

Since our piped input may contain spaces, punctuation, newlines, or other illegal characters, we're going to use substitution to get the piped input into t. The -I '{}' argument tells xargs to replace (interpolate) {} in t update '{}' with the piped input, inside single quotes so that it's properly escaped.

Once you've saved tweet.sh, make it executable for your user with chmod u+x tweet.sh

5. Making tweet.sh into a proper cronjob

Cronjobs are scripts or programs that are run by the [cron] daemon on your computer at set times. These cronjobs are defined in a file called the crontab. Each user on the computer can have a crontab, so a developer who's logged into the conjob machine may need to switch to the twitter bot's user before editing the crontab. How do you edit the crontab for a user? log in as that user and then run crontab -e, which will open the crontab using that user's default command-line text editor.

If you're stuck in a command-line text editor, try these resources to save yur work and get out of the text editor.

With crontab -e, let's add a line to the crontab to run tweet.sh:

0 9 * * * /path/to/tweet.sh
check this with the latest version of the repo

This runs at 9:00, on the hour, every day, month, and TKTK. Wikipedia [has an excellent description of the crontab syntax] which you should check out, since it explains the possibilities.

Now let's make its output get logged to a file in the directory:

now add the >> /path/to/your.log 2>&1

The 2&>1 portion redirects all messages from 2 to 1. What are 2 and 1? 2 is STDERR, the pstandard error output, and 1 is STDOUT, the place where normal output goes. By redirecting the error messages into the main console output, they'll get caught when we concatenate the output of message.sh into /path/to/your.log using the append operator >>. [Here's an explanation of bash catting with the greater-than arrows].

6. Make sure that the cronjob runs at the correct time.

Let's take another look at the cronjob example from above:

0 9 * * * /path/to/tweet.sh

This runs at 9:00. But 9:00 in what time zone? The time zone of your computer, of course.

If you run date you'll see what time zone your user is using:

Ben@cronmachine $ date
Thu Mar  2 09:14:10 EST 2017

So my user on this machine is in the Eastern Standard Time time zone.

The process for changing the time zone for your computer or your user varies by the operating system you have installed. Here's some guides to common systems:

  • raspbian 8
  • ubuntu
  • Centos/Red Hat
  • Arch
  • OSX

7. Testing stuff.

How to test your cronjob: wait for it to run.

How to test the script your cron job runs: go to your user's home directory and run the command that is in the crontab. Cron runs its scripts from the user's home directory, so the command in the crontab must handle all path assumptions. If @looming_midterm's scripts and resources live in ~/looming_midterm, my crontab will look like this:

0 10 * * * /home/pi/looming_midterm/tweet-countdown.sh >> /home/pi/looming_midterm/tweet-countdown.log 2>&1

Pasting and running that on the command line should work. If it doesn't, then you need to make changes.

It's a lot easier if the script referenced in your cronjob starts with a command to change to the directory of the project. That way, you can use . in your file paths to reference the current directory, which after changing to your project directory is your the project directory.

# . in the cronjob is ~, but when you run this script by yourself, it's .
# t needs the full path to the trc. You can't use ~, although . works.
cd ~/looming_midterm/