In this post, we will cover the design and architectural decisions we made in creating the Billcam platform for Naomi Wolf.
Billcam is a web-based tool that allows the general public to track changes to State and Federal legislation in near real-time and weigh in with their opinions on the bills. It exposes customizable JSON/RSS feeds and embedable widgets that allow bill information to be integrated and displayed on external applications and web sites.
For hosting and continuous integration we chose to use Forge + Envoyer. This allowed us to stay within the Laravel eco-system, and deploy to a 6-server multi-database (read/write split) with very little trouble. The infrastructure includes:
- Load Balancer + 2 web servers + 2 database servers + 1 queue server
- S3 for shared-disk resources
- Lightning-fast deployment with solid test-driven-development principles allowed us to go from an empty folder to a working prototype in about 10 days.
- The code was structured with an eye towards future extensibility and maintainability, which allowed us to rapidly add features that we knew we could come back to and extend in future iterations.
Legiscan is the nation’s first impartial real-time legislative tracking service designed for both public citizens and government affairs professionals across all sectors in organizations large and small. It offers a web service that allows for near real-time monitoring of changes to State and Federal Legislation. One of the goals for Billcam was to integrate with Legiscan’s push service for the most up to date bill information.
The Sync Process
- Since LegiScan pushes out thousands of updates in a very short period of time, synchronized data merges were out of the question. They would consume too many resources and slow the entire system to a crawl.
- To keep the same code base (without needing a separate utility), we stash all of the changes as discreet JSON files on the local disk of whichever web server received the request (we won’t know which one because of the load balancer).
- Then a scheduled job sweeps up all the local JSON files and puts them into the s3 bucket.
- The Queue box continuously checks the s3 bucket for files to process.
Bill Text (HTML converter)
- The vast majority of bills come in with one or two sentences about what the purpose of the bill is, but occasionally we’ll get what appears to be the full text of a bill, and it always comes in a single, full, unbroken block of plaintext.
- Using a bunch of Regex rules, we came up with a converter that can take the unstructured text and turn it into a structured collection of sections, paragraphs, and lists.
- The converter class has an HTML output option, which exports what you would see on the bill viewer screen.
- Because we know the party of a supporter or a bill sponsor, we can calculate partisan support by percentage for a given bill, both in the form of sponsorships and votes.
- Giving people an overview of -how- their representative tends to vote is a good start to showing how well a person’s representatives align with their own beliefs and values.
- JSON Feed 1.0: This was just a cool little add-on; the spec was released right around the time we were getting ready to go live, so we implemented it quickly.
- XML – For the legacy readers
- Widgets – This is an area where we need to improve.
Tags: billcam, laravel
Categorised in: General