Motivation
StockBase aimed to solve several problems that affect committed stock investors as consumers of financial news:
- News sources are disparate
- Monitoring critical company news is a manual process
- Important financial information sources come in legacy formats (e.g. XML SEC filings)
- Stock education isn't continuous and seamless
How it works
Once a user signs up, they add stocks to their portfolio. We curate a set of about 70 hand-picked newsfeeds, identify the companies mentioned in the articles, and add those articles to the user's portfolio of news. Plus, we parse SEC filings, and add those to the per-company news too.
Challenges we ran into
The two of us were learning a bunch of technologies at once: Meteor, MongoDB, Bootstrap, cheerio, various queue management packages and NLP APIs; plus one of us was coaching the other on programming in general and JS and Meteor in particular. It was the first time we were working together, and this was our first large Meteor app. Yet, we managed to put out a product in only three months. The Meteor.js framework was a key part of that success.
What we learned
Our news feed parser was subjected to the vagaries of the publishers, who weren't always respecting the RSS feed XML format. Sometimes, the parser would time out or crash, and take down the front-end with it. To prevent this, we avoided Meteor's monolithic approach, so we split the app into a three-tier architecture:
- a data ingestion pipeline built with Node.js that would parse the newsfeeds from a queue, and store the indexed articles in MongoDB
- MongoDB, storing the articles, company information, SEC filings and other data
- a front-end Meteor app, reading articles from MongoDB
With this approach, the front-end app was always ready to serve requests; the backend could be independently restarted if needed without taking down client connections; and there would always be data in the database, even if a few minutes stale in case the backend choked on a particular news feed.
Log in or sign up for Devpost to join the conversation.