Notes on building a cron-powered Twitter countdown bot
- 0. Assumptions
- 1. Installing
t - 2. Getting Twitter credentials for
t - 3. Saving Twitter credentials for
t - 4. Piping messages into
t - 5. Making
tweet.shinto a proper cronjob - 6. Make sure that the cronjob runs at the correct time
- 7. Testing the cronjob
- 8. Continuing development
This blog post is about @looming_midterm, a bot that counts down the days to the next midterm election. You can find the code for this bot at benlk/looming_midterms.
@looming_midterm was inspired by this tweet from B Cordelia Yu, and several experiments with the command-line Twitter client sferik/t. I thought making a simple script that generated a message and piped it to t wouldn't take long. I was wrong. Here's hoping I can save you some time when you make a bot like this.
In the end, it took a lot of poking around spread over three months to:
- get dates to output reliably
- get my cronjob box to use the correct time zone
- install
ton my cronjob box - prevent the script from posting multiple times
Most of that was because I had no idea what I was doing and because I was waiting for cronjobs to run.
463 days until the 2018 midterm elections.
— Midterms Are Coming! (@looming_midterm) July 31, 2017
21 days until the 2017 solar eclipse.
0. Assumptions
I'm assuming that:
- you have some familiarity with the command line, i.e. that you know how to navigate and can run commands
- you have some familiarity with Bash scripting
- you are using a Linux computer (most of this will work on a Mac, though)
1. Installing t
t is a command-line client for Twitter, written by sferik. Its advantage is that it implements most features of Twitter in a way that is easy to use in scripts.
Follow the installation instructions in t's documentation.
You're going to need to make sure that you have Ruby version 2.0 or later. If the computer you're planning to use to run this bot is running Debian 7, you'll want to upgrade to Debian 8 or look into something like rvm or rbenv for managing your Ruby versions.
2. Getting Twitter credentials for t
To get API keys (part of setting up t), Twitter requires you to confirm a phone number. Twitter will text you a confirmation code, and may need that phone number for later interactions with the bot's Twitter account. Consider setting up a Google Voice account for your newsroom or organization, so that you aren't depending upon one employee's cellphone to receive confirmation text messages.
3. Saving Twitter credentials for t
By default, t creates a .trc configuration file in your home directory. You can run t with the --profile=/path/to/your.trc argument to load credentials from a different file. THe advantage of doing this is that you'll isolate your bot from potentially using the bot's credentials with t in other situations on this computer.
Once you've set up t, the credentials for your bot account will have been saved in a .trc file in your home directory. It's a plain text file. Cut the credentials from the .trc file in your home directory and paste them into a new .trc file in your project directory.
Do not version control the .trc - treat it like you treat passwords. Keep it out of version control. Use whatever your newsroom's normal password management scheme is, whether that's a commercial solution like 1Password, a home-brewed cloud storage like NPR's cloud secrets storage rig, an encrypted file that gets emailed between developers, or a file that gets scp'd from the production environment.
Your .trc will look like this:
---
configuration:
default_profile:
- looming_midterm
- #########################
profiles:
looming_midterm:
#########################:
username: looming_midterm
consumer_key: #########################
consumer_secret: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
token: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
secret: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It's a yaml file. :shrug:
4. Piping messages into t
Writing your cron job will be easier if you separate the message-generation portion of the program from the part that tweets. In @looming_midterm, I did this by having two bash scripts, message.sh and tweet-countdown.sh. tweet-countdown.sh runs message.sh and a couple of other things.
For the purposes of demonstration, we'll use this code as the contents of message.sh:
#!/bin/bash
# message.sh
# generates a tweet
echo "Hello Twitter!"
Once you've saved message.sh, make it executable for your user with chmod u+x message.sh
Now let's create a tweet.sh that pipes the output of message.sh to t, using xargs:
#!/bin/bash
# tweet.sh
# tweet a tweet
./message.sh | xargs -I '{}' t update --profile=/path/to/your.trc '{}'
Bash pipes are useful things. They route the output of one program into the input of another. xargs takes piped input and allows you to use that piped input as an argument on a command, instead of just feeding the input directly to the command.
Since our piped output may contain spaces, punctuation, newlines, or other illegal characters, we're going to use substitution to get the piped input into t. The -I '{}' argument tells xargs to replace {} in t update '{}' with the piped input, inside single quotes so that it's properly escaped.
Once you've saved tweet.sh, make it executable for your user with chmod u+x tweet.sh
5. Making tweet.sh into a proper cronjob
Cronjobs are scripts or programs that are run by the cron daemon on your computer at set times. These cronjobs are defined in a file called the crontab. Each user on the computer can have a crontab, so a developer who's logged into the cronjob machine may need to switch to the twitter bot's user before editing the crontab. How do you edit the crontab for a user? Log in as that user and then run crontab -e, which will open the crontab using that user's default command-line text editor.
If you're stuck in a command-line text editor, try these resources to save yur work and get out of the text editor.
With crontab -e, let's add a line to the crontab to run tweet.sh:
0 9 * * * /path/to/tweet.sh
This runs on the hour, at nine a.m., every day, every month, and every day of week. Wikipedia [has an excellent description of the crontab syntax] which you should check out, since it explains the possibilities.
Now let's make its output get logged to a file in the directory:
0 9 * * * /path/to/tweet.sh >> /path/to/your.log 2>&1
>> appends the output of /path/to/tweet.sh to /path/to/your.log. New entries show up at the bottom of the file.
The 2&>1 portion redirects all messages from 2 to 1. What are 2 and 1? 2 is STDERR, the standard error output, and 1 is STDOUT, the place where normal output goes. By redirecting the error messages into the main console output, they'll get caught by >>.
6. Make sure that the cronjob runs at the correct time
Let's take another look at the cronjob example from above:
0 9 * * * /path/to/tweet.sh >> /path/to/your.log 2>&1
This runs at 9:00. But 9:00 in what time zone? The time zone of your computer, of course.
If you run date you'll see what time zone your user is using:
Ben@cronmachine $ date
Thu Mar 2 09:14:10 EST 2017
So my user on this machine is in the Eastern Standard Time time zone.
The process for changing the time zone for your computer or your user varies by the operating system you have installed. Here's some guides to common systems:
7. Testing the cronjob
How to test your cronjob: wait for it to run.
How to test your cronjob now: set the run time to a few minutes from now, then wait for it to run.
How to test the script your cron job runs: go to your user's home directory and run the command that is in the crontab. Cron runs its scripts from the user's home directory, so the command in the crontab must handle all path assumptions. If @looming_midterm's scripts and resources live in the directory looming_midterm/ in the user pi's home directory, my crontab will look like this:
0 10 * * * /home/pi/looming_midterm/tweet-countdown.sh >> /home/pi/looming_midterm/tweet-countdown.log 2>&1
Pasting and running that on the command line should work. If it doesn't, then you need to make changes.
It's a lot easier if the script referenced in your cronjob starts with a command to change to the directory of the project. That way, you can use . in your file paths to reference the current directory, which after changing to your project directory is your the project directory.
# . in the cronjob is ~, but when you run this script by yourself, it's .
# t needs the full path to the trc. You can't use ~, although . works.
cd ~/looming_midterm/
./message.sh | xargs -I '{}' t update --profile=/path/to/your.trc '{}'
8. Continuing development
At the time I wrote this article, this was the state of the art for @looming_midterm. If you're coming to this article later, well, things might have changed. Check out the bot on GitHub for the latest news and features.
I'm planning on adding a whole new feature set to @looming_midterm soon, with the ability to tweet messages at dates about upcoming deadlines in the midterms process. You can help with that development process by contributing to the issues in this milestone on GitHub. Thank you in advance for your advice and support.
And thank you for reading this. If you think you'd do it a different way, let me know. I take tweets and email, and GitHub issues.