When the final rankings were published, the positions of the vast majority of participants were completely upended. Most top 20 participants dropped hundreds of positions down, and many mid-tier participants surged to the top of the rankings, myself included (I ended up placing 110th out of 2700 teams). My silver medal is mostly undeserved, but there are valuable lessons in what happened, applicable to real-life ML tasks, and that’s what this blog is about. 

Silver Medal placement in the AES 2.0 Kaggle Competition

The competition’s goal was to develop a system to grade English language essays on a scale between 1 and 6. The competition’s sponsors (the Learning Agency Lab at Vanderbilt University, the Bill & Melinda Gates Foundation, Schmidt Futures, and the Chan Zuckerberg Initiative) were looking for an automated grading system that would reduce the expense and time required to hand-grade these essays. It would also allow students to receive regular and timely feedback on their writing. Imagine a system that works like Grammarly and can improve your emails, slack messages, and, yes, this blog, available for free to all. 

Sponsored to the tune of $50,000 in prizes, the competition attracted ~3300 participants organized into ~2700 teams. As one of these participants, I had the chance to observe how the community collaborated to exchange ideas and educate first-timers. As is customary in Kaggle competitions, after their final rankings were announced, the top teams published their analysis of what worked and what did not, and I used these writeups, together with my own observations to come up with 5 main lessons from this dramatic competition. 

But before I share these 5 lessons, let me explain a few important things. 

Public Training data / Private Test Data: In this competition, the organizers publicly shared a training dataset of about 17,000 English-language essays, but kept their test dataset (rumored to be around 8000 essays) private. Furthermore, they divided this private test dataset into:

• 1/3rd used for continuously scoring submissions (every participant was allowed up to 5 submissions per day, and many submitted hundreds of submissions over the course of the competition. I submitted about 70). While the actual test essays were never published, the scores of submissions based on these test essays were published on something called the Public Leaderboard. These scores were used by participants to evaluate whether they were on the right path and making improvements.

• 2/3rds of the test data was held until the very end, to run the final scoring and ranking on the Private Leaderboard. While the smaller test dataset (the first 1/3rd) was indirectly accessible to users via so-called “probing”, the larger, (truly-private) test dataset was a complete mystery until the very end. Btw, this larger test dataset was the one that completely upended the rankings, in a phenomenon known on Kaggle as a “shake-down”.

Submission Scores: Each submission was evaluated on a metric called ​​Quadratic Weighted Kappa (QWK). This metric looks at predicted labels and the true labels, and calculates a level of agreement between the two (how well the predicted labels match the true labels). This metric is similar to a simple percent agreement calculation, but is more robust against the predictions and true values being the same by chance. 

The important thing to remember here is that the maximum score a submission could have achieved would be 1.0. Of course, no submission was as good as a 1.0. The winning submission had a QWK score of 0.84128, the cut-off for Gold Medals was 0.83583 and I had a score of 0.83523 (yes, only 0.0006 apart from Gold, not that I would have deserved it).

Now, let’s get back to our top 5 learnings.

1. The Importance of Good Data and Data Provenance

The best teams realized (through Exploratory Data Analysis) that training data was sourced from two places (dataset A – about 75% of test data, came from Persuade Corpus 2.0, and dataset B – another 25% – appeared to be entirely new and never used before this competition). The #4 participant in this competition, Tascj, theorized that “the scores from the two data sources are collected in different ways, thus not compatible”. 

Another big finding was that the “truly-private” test dataset used for final rankings was more similar to the non-Persuade dataset B. Effectively, our training dataset has some data that looked like test data, and some data that was different from test data. The 1st ranked submission even “fixed” the scores of the Persuade-based dataset A in a technique called “pseudo labeling”. Many top submissions (1st, 2nd, 3rd, 5th) ended up dividing their training into two steps, pre-training on the Persuade dataset A, and fine-tuning on the B dataset. Other submissions used the source of data as a predictive feature (4th).

Insight: If you work in a corporate setting, you will most probably recognize the problem and possible solutions. Knowing where the data came from, having a catalog of data sources with metadata on creators, would save you several months of EDA and guessing. In this competition, this information made the difference between being in the top 5% and top 0.5%.

2. Knowing when to stop prevented overfitting

Data exploration led to the discovery of the two subsets of data. But this was not the only finding. It turned out that the Persuade subset of data consisted of essays on 7 topics, such as “Car-free cities” and “Does the electoral college work?”. Many top performing competition participants spent the final month of the competition trying to determine the topic of the essay based on its text, and use it as a predictive feature. They probed the private test dataset by dozens of submissions trying to determine if the test dataset contained any new topics. 

As it turned out, this was not the case, and the private test data was not from the Persuade dataset, but the damage was done. A lot of participants over-optimized their models for the Persuade data by clustering the essays in groups that represented those 7 essay topics. When their models were applied to the “truly-private” test dataset, they underperformed and teams were “shaken-down” by hundreds of positions on the Private Leaderboard.

Former mid-tier-ranked participants surging to the top 

Former top 50 dropping hundreds of ranks down

3. No Single Model (Compound AI Systems)

All well-performing submissions, with few exceptions, used the ensemble approach, where multiple models were combined to produce a composite prediction. No single LLM placed in the top 1% of this competition. But LLMs were definitely used as part of the model ensembles (the Deberta v3 LLM turned out to be a very popular choice). 

How the outputs of individual models were combined ranged from simple weights to techniques such as stacking. Even the “good enough” model (see Learning #4) that guaranteed a top 5% ranking was an ensemble of 2 Gradient Boosted Trees models. 

Insight: Considering all the data prep and transformations steps in-between, all top submissions were effectively Compound AI Systems. This is a new idea that is enabled by unified data and AI platforms, where one can put together multiple data processing steps and AI training or inference tasks into a graph of interacting components.

Read up on an example of a Compound AI System in my previous blog.

4. Gradient Boosted Decision Trees were Good Enough

The approach that got you into the top 5% of submissions (silver medal territory) was based on gradient boosted decision trees (GBDT), something the Data Science community was crazy about 5-10 years ago. About two weeks into the competition yukiZ, a Kaggle Expert from Tokyo, Japan, shared a notebook implementing a simple ensemble of two GBDT models – a LightGBM and a XGBoost model. That notebook, and derivative notebooks (especially the one from Prem Chotepanit), ended up scoring 0.83518+ – enough to place in the top 5%. 

The “Good Enough” approach calculated tens of thousands of features based on paragraphs, sentences, and words of the essays. It then selected 13,000 most relevant features and used them to make predictions. LLMs were used in this approach, but only to calculate 6 features representing the probability of being the right essay score (1 to 6). These features, btw, ended up being some of the most predictive ones. Other predictive features were sentence and paragraph-level stats like number of errors in words, number of short and long sentences and paragraphs, number of short and long words (representing syntax knowledge and writing conventions). The rest of the 13K features were occurrences of particular words (representing the breadth of the vocabulary of essay writers).

Since this approach was entirely independent of the Persuade/non-Persuade test dataset fiasco, it handled the switch to previously-unseen test data rather well. 

5. Collaboration UX

What made this competition so addictive and fun were 1) people and 2) the Kaggle UX. Chris Deotte, a data scientist at Nvidia, deserves a special call out. He shared countless notebooks with new ideas and answered dozens of questions from folks who were just starting out. Unfortunately for Chris, the “shake-down” in the final scoring of the submissions (see Learning #2) affected him similarly to other top participants, but he promised to continue sharing and helping in future competitions. Fingers crossed.

Since I am working on compute options for Data Science teams in my day-to-day job at Databricks, I paid attention to the Kaggle UX. It was largely conducive to a fun learning and competitive environment, with some exceptions:

• Collaboration in Kaggle is done mostly by cloning Notebooks and sharing Datasets. Users start discussions and share their notebooks that then get cloned and improved by others. 

• There is a lot of bad notebook code out there, and Kaggle does not help with refactoring of it. Notebooks get cloned many many times, code gets modified, and by clone #99 the notebook is full of  code cells from various other notebooks, and training and inference code jammed into one notebook, and controlled by variables. Ease of collaboration is important, but so is readability of code.

• In modern data platforms notebook sharing and collaboration is a common feature. As far as I know, no platform allows smart editing of the same notebook by multiple contributors that actually results in an easily readable artifact. Take note, Databricks, Snowflake, and all you Jupyter clones!

• I saw almost no usage of the Models artifact. Instead, users create file snapshots of their models (sometimes as pickled objects), and make Datasets out of them. These model-datasets are then imported by other users as files and loaded into memory using the joblib or dill libraries. Presumably, if users were encouraged to share their models as Models (via API calls) instead of pickled objects, the code would be a bit easier to read.

Insights: In future collaborative platforms I see teams submitting PRs against notebooks, instead of cloning them. The PRs would be reviewed by the original notebook author and the creator of the PR would get credited by points if the PR were approved. This would improve readability of code, and prevent situations where random participants gain strong positions in the leaderboard just by cloning a good publicly shared notebook. 

What’s next, you ask? 

Have I mentioned that Kaggle is addictive? There is a new top competition on Kaggle – the ARC Prize – and I intend to participate and get some AGI-“achieving” done. Expect a new report in a few months. 

Sample GitHub feature request issue and bot responses

To demonstrate the workings of the bot, let’s look at an actual feature request submitted on dltHub’s dlt repo on GitHub (issue 933). dltHub is an open-source project hosted on GitHub that frequently gets contributions from its community.

Feature description
dlt should support Postgres database replication. This should be efficiently done, using a CDC-based approach leveraging the transaction log (WAL)—not by querying the actual data directly.
Implementation should be as generic as possible, so database replication support for other databases (e.g. SQL Server) can easily be added later.
[EDITED FOR BREVITY]

The response generated by our bot before taking into account the responses by repo maintainers and the FR submitter. 

Implementing PostgreSQL database replication support in dlt using a Change Data Capture (CDC) approach, leveraging the transaction log (WAL), is a great idea! This will not only improve efficiency but also enable future support for other databases like SQL Server.

To achieve this, we can follow these high-level steps:

1. **Establish a connection to the PostgreSQL database**: Use a library like `pg` in Python to connect to the PostgreSQL database and retrieve the transaction log (WAL) files.
2. **Parse the transaction log (WAL) files**: Use a library like `pg_wal` in Python to parse the WAL files and extract the necessary information, such as transaction IDs, timestamps, and affected tables.

[EDITED FOR BREVITY]

And here is the bot response after taking into account all 13 comments that were added to this issue, with the last comment being from the original submitter of the FR:

Last Comment (Link):

@codingcyclist
re. selecting columns: I had a little chat with @rudolfix about this on Slack. Schema contracts are still experimental and likely to change significantly. Hence, not a good idea to rely on this for now. Also, it's more efficient if we can do the filtering further upstream. My idea is to include an argument on the replication resource that let's you provide a column list, which is then used to filter already in the resource.

Bot response:

That makes sense. I didn't know that schema contracts were still experimental. And you're right, it's better to do the filtering upstream rather than relying on the `discard_row` and `discard_value` options.

So, if I understand correctly, you're proposing to add an argument to the replication resource that allows users to specify a list of columns to include in the replication. This would then be used to filter the data in the resource, rather than relying on schema contracts. Is that correct?

Bot’s evaluation of the response being “Safe For Work”: 

{'label': 'SFW', 'score': 0.7747769355773926}

Implementation Chain 

We broke down our implementation into 5 components that match the steps in our “compound AI system” chain: 

• Read from Github using ReadGithubIssue , 

• Convert the Github thread into a chat template that Llama 3 can understand using GithubIssueToChat , 

• Generate a proposed response to chat messages with the newest Llama 3 LLM model with  InferWithLlama3Instruct ,

• Classify the response into Safe or Not Safe for Work using a text classification model from Hugging Face with InferWithHuggingface, and, lastly, 

• Write our output to files using WriteFile. 

In the future, we want, of course, to replace the writes to files with a write to Github itself, by creating a new comment.

We start by importing the little wrapper components that we wrote: the data readers and writers, the inference classes, and the data converters.

from core.readers.github import ReadGithubIssue
from core.transforms.github import GithubIssueToChat
from core.inference.llama3instruct import InferWithLlama3Instruct
from core.inference.huggingface import InferWithHuggingface
from core.writers.dlt import WriteFile
import os


# Get the Huggingface token
HF_TOKEN = os.getenv("HF_TOKEN")

All our data readers and writers are based on dltHub (on Github), an awesome Python library that runs entirely in-process, requires no dedicated compute infrastructure, and solves a bunch of real problems like pipeline restarts, schema detection, mapping and evolution. “Dlt” btw stands for “Data Load Tool”, not the Delta Live Tables offered in Apache Spark.

# Create a Github reader action that will read from Github and store in memory (readitems attributes of the class)
# This reader will read the main issue and all comments for a specific issue number
read_issue = ReadGithubIssue()
read_issue(repo_owner="dlt-hub", repo="dlt", issue_number=933)
issues= read_issue.readitems['issues']
comments = read_issue.readitems['comments']

After we get the issue record and all its comments, we have to convert them into a format that Llama 3 will understand. This is a standard chat messages format popularized by OpenAI, with each message in its own python dictionary, with roles and content fields.

# Start building inputs into an LLM. Issues and comments get converted into a chat-like thread
# Each message in the chat are tagged with one of 3 roles: system, assistant, user
# The first message is by 'system' and instructs our LLM how to behave
messages = []
messages.append({"role": "system", "content": "You are a coding assistant that answers user questions posted to GitHub!"})


# Now convert all comments from Github that we read earlier
convert_gh_issue = GithubIssueToChat()
chat = convert_gh_issue(issues=issues,comments = comments)
messages.extend(chat)

We then run our inference using Meta’s Llama 3 model on a Macbook Pro that has no problems fitting the 8 billion parameter version of the model (it needs about 20 GB of RAM). We use the Apple Silicon GPUs (“mps” device). Loading the model (first line of code) takes about 10 seconds; the time to run inference on chat messages and generate a response depends on the total size of these messages. It takes somewhere between a few seconds to a few dozen seconds (when the content length is about 4K words). 

# Pass this chat message list to the Llama3 model and get a response
infer_with_llama3 = InferWithLlama3Instruct(HF_TOKEN,"mps")
response = infer_with_llama3(messages)

As the second inference call in our chain we run the response through a text classification model “michellejieli/NSFW_text_classifier” that checks if the response is Safe or Not for Work.

# Check that our response is safe to use at work.
# Generate an NSFW score using a model on Huggingface
filter_nsfw = InferWithHuggingface(task="text-classification", model="michellejieli/NSFW_text_classifier", device="mps")
nsfw_score = filter_nsfw(response)[0]

We merge the response and the NSFW classification scores and prepare two output files – one for the response message itself, and another one for the full contents of the chat, including the issue text, all comments, and the proposed bot response. The data will be written in json format.

# Start preparing outputs.
response_message = {"role": "assistant", "content":response}
response_message = {**response_message, **nsfw_score}
messages.append(response_message)


# Write the last response and the full chat to local files
write_last_response = WriteFile(
   "github_bot", bucket_url="file://gh_bot_last_response")
write_last_response([response_message],loader_file_format="jsonl")


write_full_chat = WriteFile(
   "github_bot", bucket_url="file://gh_bot_full_chat")
write_full_chat(messages,loader_file_format="jsonl")

The source code for this implementation is available in this gist.

Now that we know what the bot does and what its design is, let’s dive deeper into the concept of compound AI systems on which it was built.

Background

In a paper published this February, Matei Zaharia (CTO of Databricks) and a bunch of smart folks at Berkeley AI Research (BAIR) and Databricks described a new trend that they were seeing in the market. 

Instead of a single, monolithic AI model being used to achieve state-of-the-art results (think a chat interface to ChatGPT where you type in something, and you immediately get a result from the model), they saw longer chains of components working together to produce the best results. If you are familiar with data processing pipelines, imagine multiple processing steps being chained together, where outputs of one step become input of another. What are these components, you might ask? Think of search engines, data format converters, data readers and writers, and, quite often, different models being used one after another. 

Source: BAIR paper

Examples of Compound AI Systems

Better Inputs

RAG (retrieval-augmented generation) systems are the best-known (and relatively simple) examples of compound AI systems. In a RAG system, there is a retriever and a generator. Questions from users don’t go directly as input into a model (the generator in this system). Instead, there is a search step (the retriever), where a database is searched for content relevant to the user’s input (usually, in a vector database). Documents that are relevant to the user inputs are brought together into what is known as “context” and added to the user question. A simple user question “What is my current sales in EMEA?” becomes a much longer piece of text, with recent Google Docs, PDF files and presentations, all from the corporate file storage system, added to the question. This is important, because the generic model from an AI vendor does not have the latest information and knows very little about your own business. By adding context, one can make a generic AI model answer questions about something very recent and non-public. 

The length of these documents can be quite significant, measured in thousands of words (for comparison, 1 page of English text is ~ 500 words). That’s why you see many Generative AI model builders (Meta, OpenAI, Anthropic, you name it) bragging about the “context length” of their models. The recently released Databricks DBRX model boasts a maximum 32K token context length. That’s about 65 pages of internal, non-public, up-to-date information that you can add to your question about EMEA sales.

Safer Outputs

Another example of components used in Compound AI Systems are so-called Guardrails, in other words, tools that control the outputs of a LLM model (several are mentioned in the February BAIR paper). Whereas in RAG systems we improve the results by feeding better inputs into the model, the Guardrail components make results better by preventing potentially incorrect or embarrassing outputs. 

We use one of such guardrails in our own Github bot implementation – it checks if the response is Safe or Not Safe for Work using a text classification model. 

Frameworks for Building Compound AI Systems

Two recent projects have received significant attention for proposing a new syntax for expressing chains of components and by offering component libraries that implement common actions or tasks: LangChain and, to a lesser degree, Metaflow.

LangChain cleverly uses the Python | (pipe) operator override capability to bring back the syntax that folks who are familiar with Unix pipes, Apache Beam, and PyCascading (by Twitter) find very appealing. Since we count ourselves among these folks, we can’t wait until the next long weekend to hash out the __or__ implementation in Python that would allow us to define our Github bot as:

  ReadGithubIssue()( … params … ) 
| GithubIssueToChat()( … params … )
| InferWithLlama3Instruct()( … params … )
| InferWithHuggingface()( … params … )
| WriteFile()( … params … )

If the above did not quite make sense, review this SO discussion on Python operator overloading.

The idea behind this syntax is that one writes the steps of the pipeline one after another, and that the outputs of the previous step become the inputs of the following step.

Fun fact: the author of this blog onboarded the data pipelines teams from Twitter as they were migrating from the Cascading framework to Apache Beam running on Cloud Dataflow. Luckily for these folks, they could continue using the Python pipe operator in Beam to write marvels like the one above.

Metaflow (open-sourced by Netflix) offers an alternating view on defining chains of actions. By using the @step decorator (and several others) on a function in Python one can define a complete pipeline that combines data processing and ML inference steps. 

With an important part of the industry coalescing around the idea of chains of AI and data processing components, it becomes clear that we don’t have to bet the house to improve the quality of our AI systems. Instead of training an ever larger ChatGPT model, we can combine several smaller ones into something equally good.   

We decided to implement a useful case that would demonstrate the idea of compounded AI systems.

Meta Llama 3

Something else happened very recently that jolted us to action. Meta has finally released their Llama 3 model, which the whole industry was waiting for. The model comes in two variants – the smaller 8B parameter model, and the larger 70B model – and is available for online inference at Databricks Model Serving. Interestingly, the 8B parameter model requires less than 20GB of memory when doing inference, which is totally doable for local inference on a Macbook Pro. 

Within 24 hours of Llama 3 hitting Hugginface and being announced by Meta’s partners (see Databricks blog) the rumors started circulating that it was quite good. 

We knew what we had to do. We would use a small, but state-of-the-art LLM model and combine it with other models and business logic.

Looking for a realistic use case

To make an evaluation more realistic, we thought of some real-life tasks that businesses would want to get solved with AI. We are dealing a lot with application developers, and they live and breathe GitHub and all the cool projects that are out there. A very common workflow in GitHub are developers opening Issues and proposing new features. These feature proposals sometimes result in long comment threads, where project maintainers evaluate the proposals and where the community chimes in. A bot that would automatically evaluate a Feature Request, or propose a response to a user’s question might be quite useful to repo owners and users of Github. As an added bonus, the Github API is open and free when used in reasonable quantities.

We had our use case. We would build a bot for GitHub repo owners.

Getting data in and out

The best AI system is nothing without good data. While most data frameworks offer common connectors to things like databases and file systems, application-specific connectors are harder to find. Here’s where dltHub comes in. Their Data Load Tool (dlt) looks deceptively simple (it packs a big punch once you dive deeper into it). One picks a source and a destination, and the tool generates an initial pipeline that can be customized later. The following commands will install dlt and then initialize source code and config files to read from github and write to files.

pip install dlt
dlt init github filesystem

The thing that differentiates dlt from all the other ETL tools out there is that it’s both a code generation tool and a Python library. This library can run entirely in-process of your main application – you don’t need a separate compute infrastructure to run your data pipelines. This library also takes care of one of the hardest things in software engineering, besides naming and pricing, of course, that of schema detection and evolution. The tool detects schemas at the source (including nested fields) and is able to replicate them at the destination, without you having to specify the mapping rules.

dlt comes with dozens of pre-defined sources and destinations, but it’s quite easy to define new ones as well. In fact, the project positions itself as an Extract-Load-Transform (ELT) tool for the long tail of data sources and destinations. Because if was originally designed to be used in the ELT pattern, it does the E and the L, and delegates the T (transformations) to tools like dbt or Airflow or to the processing capabilities of the destination (e.g. if the destination is Databricks, you could use Databricks SQL to transform your data once you loaded it into the bronze layer of a Delta Lake).

However, for our bot implementation we needed to load data into memory in order to pass it to our model inference components. The results of inference would also be in memory, and they needed to be eventually written to some external destination. We decided to use the dlt framework and build a new dlt destination (and a new source) that would reside entirely in-memory. 

We created a Read class that can take as an input any valid Dlt source and read the contents into two of its attributes: readitems and readtableschema.

import dlt
from dlt.common.typing import TDataItems
from dlt.common.schema import TTableSchema
from dlt.common.destination import Destination
from typing import Any
from core.actions import Action


class Read(Action):


   def __init__(self, actionname: str = None):
       super().__init__(actionname)
       # needs to say from_reference("destination"... to work
       self.dltdestination = Destination.from_reference(
           "destination",
           destination_name=self.actionname+"_destination",
           destination_callable=self.read_destination)
       self.dltpipeline = dlt.pipeline(
           self.actionname+"_destination_pipeline",
           destination = self.dltdestination)


       self.clean_state()


   def read_destination(self, items: TDataItems, table: TTableSchema) -> None:
       tablename = table["name"]
       if tablename not in self.readitems:
           self.readitems[tablename] = []
       self.readitems[tablename].extend(items)
       self.readtableschema[tablename] = table


   def clean_state(self):
       self.readitems={}
       self.readtableschema={}


   def do(self, *args:Any, **kwargs: Any):
       self.clean_state()
       self.dltpipeline.run(*args, **kwargs)

We then defined ReadGithub(Read) and ReadGithubIssue(ReadGithub) classes that implemented an abstraction that allowed us to be very succinct when specifying steps in our compound system chain.  

Here is the definition of the ReadGithubIssue(ReadGithub) class, in case you were wondering.

class ReadGithubIssue(ReadGithub):


   def do(self, repo_owner, repo, issue_number, *args:Any, **kwargs: Any):
       base_github_url = f"https://api.github.com/repos/{repo_owner}/{repo}"
       issues_endpoint = self.build_entityspec(
            entity="issues", issue_number=issue_number)
       comments_endpoint = self.build_entityspec(
            entity="comments", issue_number=issue_number)
       endpoints = [issues_endpoint,comments_endpoint]
       super().do(base_github_url, endpoints, *args, **kwargs)

And here is all the code that one needs to write to pull Issue 933 from the repo “dlt” of repo owner “dlt-hub”. 

# Create a Github reader action that will read from Github and store in memory (readitems attributes of the class)
# This reader will read the main issue and all comments for a specific issue number
read_issue = ReadGithubIssue()
read_issue(repo_owner="dlt-hub", repo="dlt", issue_number=933)
issues= read_issue.readitems['issues']
comments = read_issue.readitems['comments']

Similarly, we also wrote a WriteFile(Action) class that takes an memory object (a list), and writes it to either local or cloud-based files.

The source code for this class can be reviewed in this gist.

This abstraction allows us to write two lines of code to save the contents of an in-memory object in a file. 

write_full_chat = WriteFile("github_bot", bucket_url="file://gh_bot_full_chat")
write_full_chat(messages,loader_file_format="jsonl")

While this, by itself, is no big news, the power of dlt gives us various options

• Format: jsonl, parquet, csv and INSERT VALUES (sql script format)

• Write disposition: append, replace, merge

• Destination: local disk, S3, google storage or azure blob storage

What about the mysterious Action class?

Throughout the code samples in this report we used a base class Action from which many other classes were derived. This is a super simple piece of code, and all it does are two things:

• It defines a constructor __init__ where the initialization of the objects is going to happen.

• And it defines a “do” function that “does all the doing” of the Action class

from typing import Any


class Action:
   def __init__(self, actionname: str = None):
       if actionname is not None:
           self.actionname = actionname
       else:
           self.actionname = "UnnamedAction"


   def do(self,*args:Any, **kwargs: Any):
       pass


   # ThisAction(params) or ThisAction.do(params)
   def __call__(self,*args:Any, **kwargs: Any):
       return self.do(*args, **kwargs)

The additional __call__ function is just syntactic sugar that allows us to write 

write_full_chat(messages,loader_file_format="jsonl")

Instead of

write_full_chat.do(messages,loader_file_format="jsonl")

The purpose of this Action class hierarchy is to enable chaining of Action objects down the road. If every component has the same super-simple interface (the “do” function), it will be easier for us to write concise and readable code. In this regard we are following the footsteps of the Great (= LangChain, Beam, PyCascading). 

LangChain defines an interface – Runnable – from which all other components are derived. LangChain requires all Runnable-derived components to specify three functions that allow chaining: 

• stream: stream back chunks of the response

• invoke: call the chain on an input

• batch: call the chain on a list of inputs

We believe our “do” function can represent both “invoke” and “batch” (if we treat each input as a list) and quite possibly the “stream” function as well. 

Our goal is to be able to write our bot in five lines of code!

  ReadGithubIssue()( … params … ) 
| GithubIssueToChat()( … params … )
| InferWithLlama3Instruct()( … params … )
| InferWithHuggingface()( … params … )
| WriteFile()( … params … )

Summary and Future Plans

It was fun playing with the latest Llama 3 model and evaluating state-of-the-art concepts (like the compound AI systems) and tools (like LangChain and dltHub). As someone who built two startups in the Natural Language Processing space in the 2010s, we are impressed by the progress in language understanding, although most of us probably still can’t comprehend how these Generative models actually produce their output.

The big headline of this report is that Llama 3 is indeed awesome. Its 8B parameter version is a small but powerful LLM that can be used for implementing realistic use cases. However, the text it generates can sometimes appear to just reword the contents of the whole chat, or state facts that don’t add value. Improving the quality of its outputs through techniques such as retrieval augmentation or fine-tuning should make it better for this use case.

The other big headline is that chained or compound systems are instinctively better choices for engineers trying to improve their applications with AI. If we were to make a suggestion to the authors of the BAIR paper, we would use the term “compound systems using AI” instead of “compound AI systems”. And if we could make a suggestion to our friends at LangChain, it would be to rename itself to AppChain. This would reflect what these chains really are – full-blown applications that just happen to be using a lot of AI (but also a lot of data processing and business logic). But naming is hard, and we won’t be sad if our suggestions are not acted on.

There are also a bunch of things that we have an appetite for doing in the near term:

• First, we did not have time to implement a Retriever for our LLM generation step. This retriever could be populated with user documentation from each individual repo. When we build the retriever, we will be ready to move from local execution on our Macbooks to an actual development platform like Databricks and use the Databricks Vector Search feature,

• Second, we are itching to fine-tune the Llama 3 model with the Issues/Comments training data set built from previous conversations in dltHub’s repo. This would give the LLM model the “voice” of the repo maintainers,

• Third, while the 8B Llama 3 model was great, why not use its larger cousin, the 70B parameter one. The larger model won’t fit into a laptop’s memory anymore, so we will use Databricks Model Serving for that,

• Forth, we want make our bot an actual bot and write into Github directly,

• Fifth, we want to improve our Action framework to allow better chaining.

It’s going to be fun to make it all work!

Code

Main gist for Github bot

Actions class

Generic Read class

WriteFile class

GithubIssueToChat class

ReadGithubIssue class

Llama3 wrapper class

Generic class for inference on Hugging Face models

The specs are slightly ridiculous (100 cores and memory larger than some of the biggest boxes on AWS). This calls the question – are we ready for a DuckDB-like movement to application servers running under our desks? Just like it was in the 2000s with the Sun Sparcs and the Unixwares and the baby-AS400s.

But nowadays, of course, with more modern app servers – Flink, Spark, even the good old Apache Beam.

Obviously, you would not run a production process that generates your payroll on a box near where your most dedicated employees take a nap or have a snack.

But what if these high-performance desktops running new, modern app servers were organized in a mesh, exchanged jobs using a standard API, and were put to a better use than generating bitcoin ? Mind blown.

To get to their HQ I had to travel to a small town on the Galician coast of Spain. If you know, you know. The closer I got to the HQ, the more well-dressed folks I saw.

The view from my hotel room was pretty impressive. They say one must try an octopus when visiting.

A nice gesture from event organizers was giving us a tour of the Steven Meisel exhibition, who worked a lot with this fashion retailer.

The campus, located in a small Galician village where the founder of the company is from, looked like as if Apple went on vacation to Sweden and came back wanting to build a summer residence.

But now back to the talk.

A few years back I began thinking about a topic that was just emerging at the time. 

I started seeing more and more companies wanting to do more with their billions of events that they collected and stored. They wanted to react to them in real-time or near real-time. And they wanted to react with almost human intelligence, instead of just counting or aggregating or storing and forgetting.

This led me to naming the three driving forces in the market at the time.

Very large # of events

Global user base creates billions upon billions of interactions with systems

Real-time

Systems must serve intelligence in or near real-time, or opportunities are lost

Near-human intelligence

Systems must be able to apply near-human intelligence to super-human volumes of data

Why are businesses interested in reacting in real-time? This is best visualized by this picture.

Time Value of Data

Imagine this mountain range of business value. The rightmost peak is the Mountain of Wisdom. This is our traditional Data Lakes or Data Warehouses that hold events created over a long period of time. You can run analytical queries on them. You can train your ML models on them. You have years and years of data there. Businesses get a lot of value from that.

But in addition to that mountain of data there is also a lot of value if you can react to an event in the same instant it happens. The traditional example for that is fraud detection and cancellation of a transaction. This is the Summit of Now.

Reacting to an event instantaneously is not always possible, but that’s ok, because there are lots of use cases where you have between several seconds and several minutes to do something smart. This is the Peak of Soon After.

And this brings us to the definition of Real-time AI, and the techniques that implement it.

Near-human Intelligence

Let’s start with the easy part. It is clear that to address the need for near-human intelligence one needs to use Machine Learning. Dealing with billions of events manually, with heuristics, just would not scale.

Real-time

On the time dimension we have two different approaches. 

True Real-time: In cases when we need true Real-time, when intelligent responses need to happen in a second or less, Large-Scale Event Processing is best. Building pipelines is the wrong approach, because pipelines tend to micro-batch (yes, everyone is doing it, they just call it differently) and introduce latency. I happen to live in a country where Amazon Store cards are a bit stingy with credit. My limit is low for American standards and Amazon periodically cancels my orders when I exceed my store card limit, but they do it 1, 3, 5 minutes after they accept my order. Why they can’t check before accepting the order beats me. I sometimes imagine a Kinesis Data Firehose piping all these orders into some file, and a cron job periodically waking up and doing a micro-batch call to their Store Card bank to check the balance of all their recent customers. Not great. For best customer experience you want to be able to do inference on very small batches (if batching them at all) in the Cloud or even on the Edge, if you are in a brick-and-mortar retail or IoT situation.

Near Real-time: However, in a large class of use cases users have between several seconds to several minutes to act. For example, I know at least one Security Alarm vendor that does not react to every signal from a movement and glass-breaking sensor, or to a video camera reporting a movement. They wait multiple seconds, collecting all the signals from various rooms of the house, and then make an inference about an incident being a real case of break-and-entry or a false positive. There is a real cost to their business if they react to every false positive, and driving down the rate of false alarms keeps their costs in line. In cases like this, one can start using techniques customary to the domain of Streaming Analytics (time windows and such). One can bring different event data sources, one can even query historical data (or use it as a predictor). Using streaming pipelines or dumping raw events into Data Lakes and then doing aggregations and inference are all fine techniques here. If you feed these events into an analytical store, you even get an early start for forming your next peak in the data value mountain range – the Mountain of Wisdom!

Snowflake is moving fast to enable users to implement these Real-time and Near Real-time AI use cases. For Streaming Analytics Snowflake now offers in Public Preview Snowpipe Streaming, a streaming ingestion solution that has a high insert rate (GB/sec per table) and low latency (data is queryable in seconds, <5s being typical). We also launched in Public Preview something we call Dynamic Tables – a very simple way for defining streaming pipelines entirely in SQL, and by specifying one of the most important streaming Service Level Objectives (SLO) – the Lag (here is the Private Preview announcement). And for Machine Learning we offer compute capabilities in Snowpark and are working on much more (come to our Summit or listen to announcements online).

Happy to see that the industry is also taking this trend seriously. Perhaps you should too.

I first loaded the Titanic survivors dataset that I downloaded from Kaggle. 900 passengers in the training dataset and a few hundred in the test one. With attributes like age, ticket class, sex, number of relatives on board etc. And the most important variable – whether the passenger survived or not.

Here is a cluster analysis of the dataset, with about 20 clusters identified. Each dot is a passenger, and they are clustered using the built-in Catboost gradient boosted decision trees classifier. The predicted variable is “Survived” (yes or no).

At the start of my evaluation I wanted to see if it were possible to get a feeling for the data without writing any pandas code and using Jupyter, and then set up a predictive model for Survived using UI tools only. The tool largely worked for me, and after ~1.5 hours I felt like I discovered some non-obvious pattern in the dataset. For example, “Male passengers in the 3rd class and no siblings” were unfortunately not very lucky that day. Take a look at Cluster 10 and the side bars.

Or, for example this Cluster 3 “Female passengers in 2nd and 3rd class”, 80% of whom were able to survive. Note how the UI helps telling the story. In the upper left corner we have the “significant variables” telling us that passenger class and sex are significant vars in this cluster. Down left we also have the Error bars and the Confusion Matrix. Then, on the right, we have a comparison of the predicted variable and the actual prediction (upper right). Below we see other variables.

Before doing the predictive model though, I played with variable distributions and correlations.
– Passenger class was pretty important for survival (3rd class was a bit lethal),
– sex had correlation too, and
– age was also important (see that big white circle for the age bucket 20-30?).
Btw, the tool did the bucketing for the continuous age variable automatically.
Other insights:
– if you had 0 siblings/spouses or parents/children (Sibsp and Parch), your survival was questionable

Initially, I started with just exploring the dataset. The “Data” tab is what reminded me of other data wrangling tools. In the middle of the screen you see your data set table, and on the right you see distributions of your columns. You can manually set the types of columns to boolean and categorical, which helps in predictive models. The distribution charts on the right allow you to select a range of values, and this auto-adjusts other columns (this reminded me of Snowflake).

It took me a bit to find how to add derived variables. Turns out one need to select the “Wizard” button.

It then opens a panel with two enabled choices “Social media” and “Others” and under Others you find “Enrichments”.

The available enrichment types include sentiment analysis, entity extraction, fuzzy grouping, geolocation, and some external methods that require an API key. Btw, the origins of Graphext were in text analytics, hence all these text analysis options (me likey).

Still, I could not find an intuitive way to add a new variable based on just some data transformations. Hopefully that can be added soon.
Interestingly, while the tool positions itself as a “no-code tool for data scientists”, it does allow you to review the code for the “recipes” (hello Trifacta) that it creates as you transform columns and define models. As such, I see Graphext as a “tool for non-data scientists who want to do data science (but can’t remember the pandas API)”.

IMO, the largest 3 value adds of Graphext are:
– UI guided creation of prediction models
– graph visualizer for the identified clusters
– built-in reporting tool called “Insights”

The tool is still a little buggy and I could not directly create a model (on the Models tab). The “Create model” button was disabled. I found a way to create a model by going to the “Graph” tab. If you don’t see the Models and Graph tabs, hover over the tab bar and click on an icon that looks like an arrow.

On the Graph tab, click on the “Create Network” button.

A selection tab will open (remember it from the data “enrichment” step?), with many previously disabled options now being available.

I recommend exploring all of them, because I liked what I saw there. There were training and prediction tasks, text analytics tasks (sentiment and entity extraction), commerce tasks (e.g. product basket analysis)…

Btw, this is where the user experience is still a little rough.

I wanted to create a few screenshots of how to create a clustering-based prediction model. But for the life of me I could not remember what option I chose initially on the selection panel. Was it the “Source and target network”? “Train and predict”? I rolled the dice and chose “Source and target network”, and of course ended up wasting time. I could not even roll back the resulting “recipe” and had to reimport data again.

I tried “Train and predict” and finally found what I was looking for.
Interestingly, the tool only offered two types of models: Catboost and Logistic Regression (was it because my predicted variable was Categorical?).

One of the unique (to my knowledge) features of Graphext is the ability to visualize the prediction model as a Graph of Clusters, a Geo Map, or a Treemap. The Graph visualization ended up being very effective in interpreting the results (model explainability).

Training the model took about 5 minutes. Graphext has an interesting architecture. The do the “fast tasks” (render graphs, data transformations etc) in the browser and the “slow tasks” (like model training) in the cloud. The browser takes on quite a bit of tasks, much more than in a traditional BI tool architecture. They even have a data engine in the browser called Durro that is able to render 10 million records as a graph. Durro is based on WebAssembly (aka WASM).

Btw, the resulting model had a 79.9% accuracy on an eval dataset. I liked how the tool highlighted the importance of various features: Sex, Age, Passenger Class. If you know the Titanic story, you will understand.

Graphext does the training on the cloud backend and claims to be able to use GPUs and preemptibles. They mention Apache Arrow as something they use to minimize the overhead on the backend, but are also hinting that their homegrown Durro engine will gradually replace Arrow.

Last but not least, let’s take a look at Graphext’s reporting capabilities. I work a lot with data scientists at Snowflake, and we tag-team on producing reports for our business leaders. We do analyses like root cause, perf and revenue impact of a particular feature launch, drivers of weekly/monthly/quarterly growth or decline. Each activity results in a doc with a narrative and supporting graphs. This is pretty standard for a PM and I’ve done this at AWS, Google and now Snowflake.

The unique thing about Snowflake is that the PMs here have a data scientist buddy who actually does the data analysis for them. They look at statistical significance, correlation-vs-causality, A/B tests and sampling. Fudging favorable conclusions becomes a little harder in this setup. My data scientist buddy likes to work in Jupyter and write code in pandas. A PM like me likes to work in Sheets+Docs. My data analysis tool is Sheets and I write my narratives in Docs (not Slides).

Other functions, based on my own experience, tend to avoid Jupyter. Finance likes Sheets/Excel and can read Docs. Engineering Managers live in Jira, but have to use Docs a lot, because .. reasons. Execs either prefer Docs/Word (if they are from AWS) or decks (if they are from Microsoft), and the Xooglers are still undecided. Sales likes Slack, but let’s set that aside for a moment.
So, this leads to Docs as the least common denominator for communicating insights in a high-tech org.

Graphext recognized that the end goal of every analysis (unless you are just grokking the data) is a report that goes to the product, marketers, execs, finance, etc. They’ve added the ability to annotate interesting findings, save them as “insights” and then product a report that is basically a collection of slides showing each individual insight. Here is, for example, one of these sliderized insights – “Older male passengers from the first class had a pretty decent chance of survival”.

While I like it that they thought about the end goal (the insights report), I wish they had an option for me to produce a Doc instead of a PDF deck that I could then edit. These reports go through a bunch of editing and commenting, and I prefer to use the text editor for doing all of that. Dear Graphext, just give me the initial export of the insights into a doc, and allow me to embed your beautiful interactive charts (super important – no static image stuff, please)!

And this brings us to the end of this review. I liked Graphext and can see myself using it for root cause and business driver analysis. They are different from other BI tools out there, and as a techy I liked that they are experimenting with a new deployment architecture (thick client in WebAssembly in the browser and specialized services on the Cloud backend). The UX seems pretty sleek, albeit a bit buggy. I give the tool extra points for offering text analytics features. Nice job!

I needed to visualize a sentence parse tree of the “constituent” variety (you know, one of those with S, NP, VP and all the goods) and googled “show constituent trees”, expecting to see good quality hits on search result page 1. Turns out, there are no good quality web tools out there. Sure, there are sites that will offer you to download “applications that run on any version of Vista”, but my Vista desktop was temporarily unavailable that day. Google’s Cloud NLP front page offers a nifty tool to parse and display trees, but those are of the “dependency” variety.

I dug a little and here is what I found. Here is a link to the Carnegie Mellon page that will return the constituency tree in text form (update: Stanford NLP has a similar feature at this location)

Take this text, replace round brackets with the square ones, and plug that text into the phpSyntaxTree tool, or any of its clones (e.g. this one).

Voila, here is your tree.

What I found was that the Publishing Domain and the Textual Content of a news article were good predictors of the Subreddit where a news article would end up at, as well as of the person who would share the news post. Furthermore, when used together, the Publishing Domain, Textual Content, and Submitter yielded a model that predicted the Subreddit with a 92% accuracy — something that could be used for anomaly detection, e.g. validating whether a posting to a Subreddit fits already seen behavior or constitutes a new trend or potentially a bot activity.

For user engagement metrics such as the Popularity Score or the Number of Commenters and Comments, the best predictors were the person who posted the article to Reddit (Post Submitter), and the Subreddit where the news was posted in. Models that used the Textual Content of the article did not have the best accuracy for predicting engagement metrics (whereas they were very predictive for Subreddit and Submitter), but I am currently working on improving the text embeddings I used and hope to publish more on this in the near future.

With the notebook now checked in github and shared in Kaggle, you can reproduce the results of the research much easier than before. Here are some tips and tricks for running the notebook in these environments.

Optimizing the notebook execution for the hosting environment

You can run this notebook either in Colab or in Kaggle. To run it in Colab, download the RedditEngagement notebook from github, import it into Colab, and set the current_run_in_colab variable to True in the “Define Constants and Global Variables” code cell, otherwise, set it to False.

You can choose between BigQuery and CSV files for the input dataset

Decide if you want to get the training data from the datancoffee BigQuery dataset or from snapshot CSV files. At present time only Colab allows you accessing the datancoffee BigQuery dataset. To get training data from BigQuery, set the current_read_from_bq variable to True in the “Define Constants and Global Variables” code cell, otherwise, set it to False.

Getting CSV files

Download the reddit-ds.zip snapshot file archive from the Dataflow Opinion Analysis github repository. Instructions for setting up the dataset are available in the notebook itself. In Kaggle, the snapshot dataset is packaged with the kernel. You don’t have to do anything to use it.

Run in Kaggle with GPUs

Running with GPUs really makes the difference in execution time. Training runs with GPUs are ~20s vs. ~400s with regular CPUs.

Improving model accuracy

The number of data points in the full Reddit dataset is large, so the size of available memory is important. The current_sample_frac variable controls the fraction of the input dataset that will be sampled and then divided in training, test and validation subsets. The default settings in the notebook have been selected to run in the publicly hosted versions of Kaggle and Colab. For SubredditClassification goal the setting is current_sample_frac = 0.5 and for the CommentsClassification goal the setting is current_sample_frac = 0.25.

Note that this is about half of what we used in our 3-part blog series and so the accuracy numbers you will get when running the notebook unmodified will be lower than what we have in our blog series.

Kaggle gives 6.5GB of memory when running with GPUs. To run the model with more accuracy, self-host the model in an environment with ~35–40GB of available memory. In this case you can set current_sample_frac = 0.99 for SubredditClassification and current_sample_frac = 0.5 (or higher) for CommentsClassification.

Enjoy and happy deep learning!

You can do that with Stackdriver Monitoring and the metrics that Cloud Dataflow exports to Stackdriver. Here is a handy list of metrics available for Dataflow:

GCP Metrics List | Stackdriver Monitoring | Google Cloud
This page describes the predefined metrics available from Google Cloud Platform (GCP) services. These metrics are…cloud.google.com

My favorites:

job/is_failed — This should be self-explanatory. Very easy to set up alerts on failed jobs.

job/system_lag, job/data_watermark_age — If you have a streaming pipeline, these two metrics will indicate if your pipeline is beginning to lag.

job/elapsed_time — Another easy-to-understand metric. This is how long your pipeline has taken so far. If you wanted to alert on pipelines exceeding predefining duration thresholds, here is your chance.

And if you still prefer a graphical UI for monitoring your metrics, check out this blog post.

Enable billing export into BigQuery.

Go to Billing>Billing export in the Google Cloud Console.

Create a new BigQuery dataset, if needed, to store your billing data.

Give the billing export a little time to populate your dataset.

Then, in BigQuery, run this query:

#standardSQL

SELECT l.value AS JobID,  ROUND(SUM(cost),3) AS JobCost 
FROM `[PROJECT].[DATASET].gcp_billing_export_v1_[account_id]` bill, UNNEST(bill.labels) l
WHERE service.description = 'Cloud Dataflow' and l.key = 'goog-dataflow-job-id'
GROUP BY 1

You can get more than $$s from Cloud Dataflow billing exports. Each consumed SKU unit is labeled and you can calculate how many CPU hours, Memory GB hours, and other units you consumed per job using pretty much the same approach.

And the costs of my Dataflow jobs? They were in line with what I expected, but it was good to get reassured by real usage data.

Here is an example of how an LA Times article about a warehouse fire is tagged in GDELT across 3 fields V2Themes, AllNames, and V2Locations of the GDELT global knowledge graph table gdelt-bq.gdeltv2.gkg.

MANMADE_DISASTER_POWER_OUTAGES Alameda_County__California__United_States WB_1174_WAREHOUSING_AND_STORAGE TAX_FNCACT_FIRE_CHIEF DISASTER_FIRE Oakland__California__United_States CRISISLEX_C07_SAFETY CRISISLEX_T03_DEAD TRIAL Alameda_County TAX_FNCACT_CHIEF EPU_POLICY_AUTHORITIES TAX_FNCACT_VICTIMS WB_793_TRANSPORT_AND_LOGISTICS_SERVICES TAX_FNCACT_CREATIVE_DIRECTOR WB_135_TRANSPORT TAX_FNCACT_ARTISTS TAX_FNCACT_AUTHORITIES MANMADE_DISASTER_IMPLIED Max_Harris TAX_FNCACT_CHILDREN SECURITY_SERVICES TRANSPARENCY CRISISLEX_T02_INJURED TAX_FNCACT_DOORMAN GOV_LOCALGOV TAX_WORLDLANGUAGES_SERRA EPU_CATS_MIGRATION_FEAR_FEAR TAX_FNCACT_INSPECTOR TAX_FNCACT_ATTORNEYS TAX_FNCACT_OFFICIALS WB_2445_NON_STATE_SECURITY_ACTORS Ghost_Ship CRISISLEX_CRISISLEXREC KILL TAX_FNCACT_POLICE WB_2451_REBELS_GUERRILLAS_AND_INSURGENTS Jeffrey_Krasnoff TAX_FNCACT_CORONER WB_2433_CONFLICT_AND_VIOLENCE CRISISLEX_T01_CAUTION_ADVICE SHORTAGE WB_2432_FRAGILITY_CONFLICT_AND_VIOLENCE ARREST TAX_FNCACT_DIRECTOR California__United_States POWER_OUTAGE SOC_GENERALCRIME CRISISLEX_T08_MISSINGFOUNDTRAPPEDPEOPLE TAX_FNCACT_PIRATE

I want to put these tags to good use in Tensorflow models, and need to embed them into multi-dimensional vectors, but their sheer number is causing all kinds of memory, data size, and model accuracy issues. So, to have better control over what I eventually take over into my sequences of tags, I developed a SQL blurb that does the same thing that Keras Tokenizer or Tensorflow VocabularyProcessor do. Having it available right where my source data is allows me to filter and order as I like (review the final query in this repo). I can even create Bag-of-Word or Sequence encodings, depending on my use case.

WITH
   b AS (
    SELECT gkg.DocumentIdentifier, gkg.V2Themes, gkg.AllNames, gkg.V2Locations
    FROM `gdelt-bq.gdeltv2.gkg` gkg 
    LIMIT 1000
  )
  ,c AS ( -- Mentions of Themes
    SELECT b.DocumentIdentifier, SPLIT(theme_mentions,',')[SAFE_OFFSET(0)] AS Entity, SPLIT(theme_mentions,',')[SAFE_OFFSET(1)] AS Offset
    FROM b, UNNEST(SPLIT(b.V2Themes,";")) AS theme_mentions
  )
  ,d AS (
    SELECT b.DocumentIdentifier, 
    REPLACE(SPLIT(name_mentions,',')[SAFE_OFFSET(0)],' ','_') AS Name, 
    SPLIT(name_mentions,',')[SAFE_OFFSET(1)] AS Offset
    FROM b, UNNEST(SPLIT(b.AllNames,";")) AS name_mentions
  )
  ,e AS ( -- Calculate frequency stats for Name mentions
    SELECT Name, COUNT(DISTINCT DocumentIdentifier) FROM d 
    GROUP BY 1 HAVING COUNT(DISTINCT DocumentIdentifier) >= 10
  )
  ,f AS (-- Filter mentions of Names
    SELECT d.DocumentIdentifier, d.Name AS Entity, d.Offset
    FROM d INNER JOIN e ON e.Name = d.Name
  )
  ,g AS ( -- Mentions of Locations
    SELECT b.DocumentIdentifier, SPLIT(loc_mentions,'#') AS LocFieldArray 
    FROM b, UNNEST(SPLIT(b.V2Locations,";")) AS loc_mentions
  )
  ,h AS (
    SELECT 
      g.DocumentIdentifier, 
      REPLACE(REPLACE(LocFieldArray[SAFE_OFFSET(1)],' ','_'),',','_') AS Loc, 
      LocFieldArray[SAFE_OFFSET(8)] AS Offset
    FROM g
  )
  ,i AS ( -- Calculate frequency stats for Location mentions
    SELECT Loc, COUNT(DISTINCT DocumentIdentifier) FROM h 
    GROUP BY 1 HAVING COUNT(DISTINCT DocumentIdentifier) >= 10
  )
  ,j AS ( -- Filter mentions of Locations
    SELECT h.DocumentIdentifier, h.Loc AS Entity, h.Offset
    FROM h INNER JOIN i ON i.Loc = h.Loc
  )
  ,k AS ( -- Join all Themes, Locations, Names
    SELECT DocumentIdentifier, Entity, Offset FROM c
    UNION ALL
    SELECT DocumentIdentifier, Entity, Offset FROM f
    UNION ALL
    SELECT DocumentIdentifier, Entity, Offset FROM j
  ) 
  ,n AS ( -- Create Encoding for Entities
    SELECT Entity, cnt, CAST(RANK() OVER (ORDER BY cnt DESC, Entity ASC) AS STRING) AS EntityIdx 
    FROM (SELECT Entity, COUNT(*) AS cnt FROM k GROUP BY 1) 
  )
SELECT DocumentIdentifier, 
  STRING_AGG(DISTINCT EntityIdx," ") AS BOWEntitiesEncoded, -- For Bag-of-Words encoding order is not important
  COUNT(DISTINCT k.Entity) AS BOWEncodingLength,
  STRING_AGG(DISTINCT k.Entity," ") AS EntitiesBOW, 
  STRING_AGG(EntityIdx," " ORDER BY Offset ASC) AS EntitiesSeqEncoded, -- For CNN and RNN analysis, use Entity Sequence
  COUNT(*) AS EntitiesSeqLength,
  STRING_AGG(k.Entity," " ORDER BY Offset ASC) AS EntitiesSeq
FROM k
  INNER JOIN n ON k.Entity = n.Entity
WHERE k.Entity<>""
GROUP BY 1

The result? Instead of the longish field with tags, I get a sequence of integers that work as good for Embedding purposes as do the tags themselves.

72 153 693 27 68 443 5050 646 42 5333 345 447 15 1350 75 6509 6901 237 49 2073 1935 224 81 38 66 5 10 4 8819 55 14 451 150 1162 27041 942 252 16 409 669 5172 149 28 468 79 128 3478 3980 10288 70

One natural hypothesis to make is that the contents of a news article will influence the subreddit that this news post will end up being posted in. That’s a logical thing to assume, but is there proof, and if so, can we get a list of topics and correlate them with subreddits? In addition to the subreddit, can we deduce the factors that will drive the highest number of participants on the discussion thread, and the highest number of comments?

What’s the use of these insights? Imagine a news publisher or a marketing person who wants to see more engagement with the content that they are creating. Knowing what engages your audience can drive up your page views and ultimately revenue. And if you are a researcher who is tracking how news is being disseminated across various audiences, having this understanding will help you too.

A prediction task such as this one is a prime candidate for Machine Learning, and TensorFlow is the authoritative framework for training ML models and running predictions. In a sense, GDELT’s catalog of news coverage and Reddit discussions are a match made in heaven, because GDELT has the features, the inputs, for training models and Reddit has the labels, or the outputs of predictions. GDELT has articles indexed by the article URL and attributes such as Title, Author, Publication Date, the full content of the article etc. Reddit also has the article URL, but in addition to that brings the submitter of the article to the board, the subreddit (topic area), and all of the comments that the users of Reddit have created based on the submission.

As we went about building the TensorFlow model for this prediction task, we had to first assemble the dataset from which TensorFlow would be able to learn. In this multi-part blog series we will first explain how we built this input dataset, bringing GDELT and Reddit data together, and then we will explore training and predicting of outcomes.

Our first task was to place the GDELT and Reddit data into a database that was easy to query. BigQuery was the natural choice here. There are GDELT and Reddit BigQuery datasets already, but we wanted to do a deeper sentiment analysis on the raw content of news articles and Reddit posts and comments, and for that we used the Dataflow Opinion Analysis project (github repo) we wrote about earlier.

Back in June 2017 Reza Rokni and John LaBarge — two GCP solutions architects — shared their best practices for developing production-quality Dataflow pipelines in a two-part blog series (blog 1, and blog 2). As we were solidifying the design for our data preparation pipeline, we used several of these best practices to future-proof our opinion analysis infrastructure. The rest of the article will explain in more detail how we applied these Dataflow design practices to the data problems we encountered in building our training dataset.

The top 10 design patterns for Dataflow can be roughly divided according to the regular lifecycle of a Dataflow pipeline:

• Orchestrating the execution of a pipeline

• Onboarding external data

• Joining data

• Analyzing data and

• Writing data

We used patterns in almost every of the above categories. For onboarding external data, we used an external service access pattern that allowed us to call to Cloud NLP to enrich our dataset with additional subject information. For joining the Reddit posts and comments, we used the CoGroupByKey pattern. When performing data analysis, we relied on the GroupByKey pattern for key classes with multiple properties. And, lastly, when dealing with invalid or misformed input data, we implemented a Bigtable sink that collected invalid input records according to the Dead Letter queue pattern.

Let’s dive deeper into how these patterns helped us assemble our training dataset.

As we mentioned before, we sourced half of our training set (the features) from GDELT using files in gzipped JSON format. The other half of our training set (the labels) are Reddit posts and their comments, available in a BigQuery dataset in two tables: posts and comments. We wanted to run both the GDELT news articles and the Reddit posts/comments through a very similar processing pipeline, extracting subjects and sentiments from both of them, and storing the results in BigQuery. We therefore wrote a Dataflow pipeline that can change its input source depending on a parameter we pass to the job, but does all the downstream processing in the same way. Ultimately, we ended up with two BigQuery datasets that had the same schema, and that we could join with each other via the URL of the news article post using BiqQuery’s SQL.

Here is the high-level design of the Dataflow pipeline that we used to bring both the GDELT and Reddit data over to BigQuery.

• Read from inputs

• Filter already processed URLs

• Filter exact duplicates

• Index (extract opinions and tags)

• Filter “approximate” duplicates

• Enrich with Cloud NLP

• Write opinions to BigQuery and invalid inputs to Bigtable

As we were implementing the input reader for Reddit data, we had to join the posts and comments table together. This join implementation follows the recommendations of our CoGroupByKey design pattern. For each dataset in the join we created a KV key-value pair and then applied the CoGroupByKey operation to match the keys between the two datasets. After the join we iterated over the results to create a single entity that had both post and comment information.

Pro Tips:

• Side Inputs could be a faster alternative to CoGroupByKey when joining datasets if one of the datasets can fit into the memory of a Dataflow worker VM.

• Use the service-based Dataflow Shuffle for better performance of your CoGroupByKey. Add the following parameter to your pipeline: — experiments=shuffle_mode=service

After reading our inputs — either news articles from GDELT, or posts from Reddit — we proceeded to eliminating duplicates from our data. Duplicate elimination is typically done either by looking up a record ID among a list of already processed IDs, or, if the record wasn’t yet processed, by grouping elements by some key that represents uniqueness.

There are three levels of filtering in our pipeline that use both of these techniques. In the first filter we simply check if the URL of the record (in effect, the ID column in our dataset) was already processed and stored in BigQuery. For this check we use a side input of hashes of the URLs and populate it from the BigQuery dataset we are writing to.

In the second and third filters, which we apply only to news articles in GDELT, we are looking at the contents of the news article. We don’t need to do this filtering to Reddit records because our data source guarantees uniqueness of posts. However, the GDELT data source is highly redundant, reflecting the fact that the world of news is highly repetitive as well.

Original news stories are still being written by staff at major news publications or by bloggers, but a large percentage of news websites, especially regional publications, reposts news from news wires such as Associated Press, Reuters, Agence France Presse, Deutsche Welle etc. Our pipeline makes an effort to identify such reposts and group them together.

A note on terminology we will use in the remainder of this blog:

• Original post: A web page where the original news story was published for the first time

• Repost: A web page which re-posted/republished an original story

• Publisher: the domain where a story (original or reposted) was published

When grouping raw news articles, we first try to match articles that are identical in content, without any variation in text. In our pipeline this is represented by Filter #2, and this filtering is implemented by grouping all incoming text articles by the hash we calculate over the text of the article concatenated with the Publication Date.

After this initial, simplistic grouping, we then try to group articles that have small variations in text, for example, because they prefix the source of the publication, e.g. (“CHICAGO (AP)“), or because they add a copyright statement at the end. This is a more advanced grouping than just taking a hash over the text. Here, we need to group on combinations of attributes of the article.

The grouping pattern recommends defining a class that has all of the attributes by which you want to group, and then tag this class with AvroCoder as the @DefaultCoder of the class. By doing that we get for free the serialization/deserialization of objects of this class, and we can then use the GroupByKey to group by it.

/**

* @param Document indexes

* @return a POJO containing 2 PCollections: Unique docs, and Duplicates

*/

private static ContentDuplicateOrNot filterSoftDuplicates(

  PCollection<ContentIndexSummary> indexes) {

PCollectionTuple dedupeOrNot = indexes

.apply("Extract Text grouping key",

ParDo.of(new GetContentIndexSummaryKeyFn()))

.apply("Group by Text grouping key",

GroupByKey.<ContentSoftDeduplicationKey, ContentIndexSummary>create())

.apply("Eliminate Text dupes",

ParDo.of(new EliminateTextDupes())

.withOutputTags(PipelineTags.indexedContentNotToDedupeTag,

TupleTagList.of(PipelineTags.indexedContentToDedupeTag)));

PCollection<TableRow> dedupedWebresources =

dedupeOrNot.get(PipelineTags.indexedContentToDedupeTag)

.apply(ParDo.of(new CreateWebresourceTableRowFromDupeIndexSummaryFn()));

ContentDuplicateOrNot contentDuplicateOrNot = new ContentDuplicateOrNot(

dedupeOrNot.get(PipelineTags.indexedContentNotToDedupeTag),

dedupedWebresources);

return contentDuplicateOrNot;

}

The document attributes that have shown to be effective in finding similar content include the Title of document, document length rounded to a thousand characters, and the topics that are discussed in the articles. The title and document length is something we get without any additional processing because it is part of the source dataset. To get topics of the document we need to run NLP and sentiment analysis, and this is why this third and last filtering step is slotted behind the “Indexing” step. The content-based grouping is highly accurate and allows us to reduce the number of stories we insert into our analysis dataset by about 45%. For example, in June 2017 there were 6.1M web pages from which the GDELT project extracted English-language news, but only 3.7M unique stories. The additional benefit from this deduplication is that we are able to aggregate the social impact of all the reposts of an original story.

As we processed text documents in the Indexing step, we sometimes encountered cases where the inputs were malformed and could not be processed. For example, in GDELT we sometimes saw text full of CSS formatting, the result of edge cases in the GDELT web crawler. Or, and this is typical for social content on Reddit, we saw posts that consisted entirely of Emoticons or Unicode symbol characters that could not be processed by NLP libraries. This is another opportunity to access our bag of patterns. Normally, a developer would place a potentially “dangerous” block of code into a try/catch block and handle exceptions by logging them. The Dead Letter processing design pattern recommends, in addition to logging, redirecting such bad records into a Side Output and then storing them in a database that supports fast writes, for example Bigtable, Datastore or BigQuery. It then recommends periodically reviewing the contents of the Dead Letter tables and debug these bad records, determining the cause of the exceptions. Usually, a change in business logic, covering the additional edge case, or improving data collection will be needed to address bad records, but at the very least, we are not ignoring these errors and preserve data that would have been otherwise lost. In our pipeline we implemented a side output in the Indexing step and a Bigtable sink for this side output.

We’ve talked about how the Indexing step of the pipeline finds topics of documents by using the Sirocco opinion extraction library. Sirocco is not the only way to get topics of documents. Cloud NLP provides an API for classifying a text document into ~700 categories. Additionally, it provides mechanisms for extracting entities from text, which is very similar to what Sirocco does. We integrated entity extraction by CloudNLP with entity extraction by Sirocco into our pipeline to benefit from a richer classification of text in our pipeline, and to do that we applied another pattern from the pattern blog posts.

After the opinion extraction in the Indexing step, the CloudNLP enrichment, and final filtering steps data is finally ready to be inserted into BigQuery. Note that both the Reddit and GDELT data will end up in distinct datasets, however, the schemas of these datasets are the same, and we can join the news articles in GDELT with the Reddit posts via the URL field of the news post.

This concludes the first part of our blog series, focused on preparing the data for model training. We have extracted features from our GDELT article dataset such as the entities found in the article, the author or the news post, publication date and time, and the sentiment. We also have our training labels extracted from the Reddit posts and comments, including the submitter of the Reddit post, the subreddit, and the sentiment of comments. We have cleansed our training data from duplicates, and in the process gained useful information about reposts which will help us providing a more accurate estimate of the social impact.

Next steps

In the next set of blogs we will introduce our methodology for developing our TensorFlow model. We hope you will be able to apply the Dataflow best practices in your own data processing pipelines. Here are a few useful links if you want to learn more.

Guide to common Cloud Dataflow use-case patterns, Part 1

Guide to common Cloud Dataflow use-case patterns, Part 2

Dataflow Opinion Analysis github repo

INSERT INTO opinions.stattopic (...)
WITH 
p AS (
 SELECT 20170630 AS SnapshotDateId
),
CalcStatSentiments AS (
 SELECT p.SnapshotDateId, t.Tag, ... s.SentimentHash,...
 FROM opinions.document d, p
 INNER JOIN opinions.sentiment s ON s.DocumentHash = d.DocumentHash, UNNEST(s.Tags) AS t
 INNER JOIN opinions.webresource wrOrig ON wrOrig.DocumentHash = d.DocumentHash
 INNER JOIN opinions.webresource wrRepost ON wrRepost.DocumentCollectionId = d.DocumentCollectionId
 AND wrRepost.CollectionItemId = d.CollectionItemId
 WHERE
 d.PublicationDateId = p.SnapshotDateId AND s.SentimentTotalScore > 0
),
CalcStatTopics AS (
…
),
CalcStatCombiTopics AS (
 SELECT 
 css1.SnapshotDateId, CONCAT(css1.Tag,’ & ‘,css2.Tag) AS Topic, [css1.Tag,css2.Tag] AS Tags, true AS GoodAsTopic, 2 AS TagCount,
 ...
 FROM
 CalcStatSentiments css1, CalcStatSentiments css2
 WHERE
 css1.SentimentHash = css2.SentimentHash AND
 css1.Tag < css2.Tag
 GROUP BY css1.SnapshotDateId, css1.Tag, css2.Tag
),
…

Basically, I have a table “opinions.sentiment” with an identity column SentimentHash, a repeated field Tags, and a bunch of other columns. The Tags column contains an array of textual tags that I extract from text using the Opinion Analysis Cloud Dataflow IndexerPipeline. In the pre-BigQuery days I would have used a separate table to store the tags and link it to the main Sentiment table via the identity column SentimentHash. However, in BigQuery this is much more easily done with repeated fields.

When I calculate trends, I build frequency statistics for combinations of tags (e.g. how many news articles were about both “climate change” and “G-20”). To do so, I flatten the Tags field and create a temporary table CalcStatSentiments that contains individual records for each tag, with fields like SentimentHash as well as the actual tag. I then do a self-join of the CalcStatSentiments table to build what I call “topics” (in the CalcStatCombiTopics temp table).

It turns out that self-joins are bad for (your health) the performance of your queries, as illustrated in this blog post. It suggests replacing the self-joins with Windowing. I considered doing that, but I actually needed to end up with permutations of tags instead of aggregate statistics, for which Windowing would work great, so I came up with another technique.

Announcing Sergei’s Self-Join Elimination Technique ©

Instead of flattening my Tags field only once (in the CalcStatSentiments part of my query), I now flatten it the first time, carry the Tags array in the result set, and then flatten it the second time to emulate the CROSS JOIN operation.

Here is how it looks in the new query:

WITH 
p AS (
 SELECT 20170630 AS SnapshotDateId
),
SentimentTags AS (
  SELECT p.SnapshotDateId, s.SentimentHash, t.Tag, t.GoodAsTopic, s.Tags AS Tags
  FROM p, opinions.sentiment s, UNNEST(s.Tags) AS t
  WHERE
    s.DocumentDateId = p.SnapshotDateId AND s.SentimentTotalScore > 0
),
SentimentTagCombos AS (
  SELECT st.SnapshotDateId, st.SentimentHash, st.Tag AS Tag1, stt.Tag AS Tag2 
  FROM SentimentTags st, UNNEST(st.Tags) stt
  WHERE st.Tag < stt.Tag
),
...

The inequality filter `WHERE st.Tag < stt.Tag` ensures that I do not get duplicates in my tag combos. It works the same way as the inequality filter in the original version of my query.

WHERE
 css1.SentimentHash = css2.SentimentHash AND
 css1.Tag < css2.Tag

Once I calculated my Tag1 & Tag2 combinations, I join my result set via the SentimentHash record ID to my main dataset and conclude all the calculations.

The result: instead of a query that takes 380 seconds to complete in billing tier 8, my modified query runs in 12 seconds in billing tier 1.

Conclusion:

Using self-joins = BAD. 
Replacing them with dual UNNEST() = Priceless! (or something like that)

Here are the queries if you want to compare the syntax.

New, performance-optimized query:

INSERT INTO opinions.stattopic (SnapshotDateId, Topic, Tags, TagCount, cntOrigPublishers, cntRepostWRs,
  cntPositives, cntNegatives, cntAmbiguous, cntGeneral, SentimentHashes, OrigWebResourceHashes, RepostWebResourceHashes )
WITH 
p AS (
 SELECT 20170630 AS SnapshotDateId
),
SentimentTags AS (
  SELECT p.SnapshotDateId, s.SentimentHash, t.Tag, t.GoodAsTopic, s.Tags AS Tags
  FROM p, opinions.sentiment s, UNNEST(s.Tags) AS t
  WHERE
    s.DocumentDateId = p.SnapshotDateId AND s.SentimentTotalScore > 0
),
SentimentTagCombos AS (
  SELECT st.SnapshotDateId, st.SentimentHash, st.Tag AS Tag1, stt.Tag AS Tag2 
  FROM SentimentTags st, UNNEST(st.Tags) stt
  WHERE st.Tag < stt.Tag
),
CalcStatSentiments AS (
  SELECT st.SnapshotDateId, st.Tag, st.GoodAsTopic, d.DocumentHash AS DocumentHash, s.SentimentHash,
    wrOrig.WebResourceHash AS OrigWebResourceHash, wrOrig.Domain AS OrigDomain, wrRepost.WebResourceHash AS RepostWebResourceHash,
    s.DominantValence AS Valence, d.PublicationTime AS PublicationTime
  FROM SentimentTags st
    INNER JOIN opinions.sentiment s ON s.SentimentHash = st.SentimentHash AND s.DocumentDateId = st.SnapshotDateId
    INNER JOIN opinions.document d ON d.DocumentHash = s.DocumentHash AND d.PublicationDateId = st.SnapshotDateId
    INNER JOIN opinions.webresource wrOrig ON wrOrig.DocumentHash = d.DocumentHash
    INNER JOIN opinions.webresource wrRepost ON wrRepost.DocumentCollectionId = d.DocumentCollectionId
      AND wrRepost.CollectionItemId = d.CollectionItemId
),
CalcStatTopics AS (
  SELECT
    c.SnapshotDateId, c.Tag AS Topic, [c.Tag] AS Tags, c.GoodAsTopic, 1 AS TagCount,
    COUNT(distinct OrigDomain) as cntOrigPublishers,
    COUNT(distinct RepostWebResourceHash) as cntRepostWRs,
    COUNT(distinct (case when c.Valence=1 then c.SentimentHash else null end)) as cntPositives,
    COUNT(distinct (case when c.Valence=2 then c.SentimentHash else null end)) as cntNegatives,
    COUNT(distinct (case when c.Valence=3 then c.SentimentHash else null end)) as cntAmbiguous,
    COUNT(distinct (case when c.Valence=5 then c.SentimentHash else null end)) as cntGeneral,
    ARRAY_AGG(DISTINCT c.SentimentHash) AS SentimentHashes,
    ARRAY_AGG(DISTINCT c.OrigWebResourceHash) AS OrigWebResourceHashes,
    ARRAY_AGG(DISTINCT c.RepostWebResourceHash) AS RepostWebResourceHashes
  FROM CalcStatSentiments c
  GROUP BY c.SnapshotDateId, c.Tag, c.GoodAsTopic
),
CalcStatCombiTopics AS (
  SELECT 
    stc.SnapshotDateId, CONCAT(stc.Tag1,' & ',stc.Tag2) AS Topic, [stc.Tag1,stc.Tag2] AS Tags, true AS GoodAsTopic, 2 AS TagCount,
    COUNT(distinct wrOrig.Domain) as cntOrigPublishers,
    COUNT(distinct wrRepost.WebResourceHash) as cntRepostWRs,
    COUNT(distinct (case when s.DominantValence=1 then s.SentimentHash else null end)) as cntPositives,
    COUNT(distinct (case when s.DominantValence=2 then s.SentimentHash else null end)) as cntNegatives,
    COUNT(distinct (case when s.DominantValence=3 then s.SentimentHash else null end)) as cntAmbiguous,
    COUNT(distinct (case when s.DominantValence=5 then s.SentimentHash else null end)) as cntGeneral,
    ARRAY_AGG(DISTINCT s.SentimentHash) AS SentimentHashes,
    ARRAY_AGG(DISTINCT wrOrig.WebResourceHash) AS OrigWebResourceHashes,
    ARRAY_AGG(DISTINCT wrRepost.WebResourceHash) AS RepostWebResourceHashes
  FROM SentimentTagCombos stc
    INNER JOIN opinions.sentiment s ON s.SentimentHash = stc.SentimentHash AND s.DocumentDateId = stc.SnapshotDateId
    INNER JOIN opinions.document d ON d.DocumentHash = s.DocumentHash AND d.PublicationDateId = stc.SnapshotDateId
    INNER JOIN opinions.webresource wrOrig ON wrOrig.DocumentHash = d.DocumentHash
    INNER JOIN opinions.webresource wrRepost ON wrRepost.DocumentCollectionId = d.DocumentCollectionId
      AND wrRepost.CollectionItemId = d.CollectionItemId
  GROUP BY stc.SnapshotDateId, stc.Tag1, stc.Tag2
  -- HAVING cntPublisherDomains > 1
),
CalcStatAllTopics AS (
  SELECT * FROM CalcStatTopics
  WHERE GoodAsTopic = true AND cntRepostWRs > 1
  UNION ALL
  SELECT * FROM CalcStatCombiTopics
  WHERE GoodAsTopic = true AND cntRepostWRs > 1
)
SELECT SnapshotDateId, Topic, Tags, TagCount, cntOrigPublishers, cntRepostWRs,
  cntPositives, cntNegatives, cntAmbiguous, cntGeneral, SentimentHashes, OrigWebResourceHashes, RepostWebResourceHashes
FROM CalcStatAllTopics

Original Query:

INSERT INTO opinions.stattopic (SnapshotDateId, Topic, Tags, TagCount, cntOrigPublishers, cntRepostWRs,
  cntPositives, cntNegatives, cntAmbiguous, cntGeneral, SentimentHashes, OrigWebResourceHashes, RepostWebResourceHashes )
WITH 
p AS (
 SELECT 20170630 AS SnapshotDateId
),
CalcStatSentiments AS (
  SELECT p.SnapshotDateId, t.Tag, t.GoodAsTopic, d.DocumentHash AS DocumentHash, s.SentimentHash,
    wrOrig.WebResourceHash AS OrigWebResourceHash, wrOrig.Domain AS OrigDomain, wrRepost.WebResourceHash AS RepostWebResourceHash,
    s.DominantValence AS Valence, d.PublicationTime AS PublicationTime
  FROM opinions.document d, p
    INNER JOIN opinions.sentiment s ON s.DocumentHash = d.DocumentHash, UNNEST(s.Tags) AS t
    -- Need to use Sentiment tags, so that sentiments relate to topics
    INNER JOIN opinions.webresource wrOrig ON wrOrig.DocumentHash = d.DocumentHash
    INNER JOIN opinions.webresource wrRepost ON wrRepost.DocumentCollectionId = d.DocumentCollectionId
      AND wrRepost.CollectionItemId = d.CollectionItemId
  WHERE
    d.PublicationDateId = p.SnapshotDateId AND s.SentimentTotalScore > 0
),
CalcStatTopics AS (
  SELECT
    c.SnapshotDateId, c.Tag AS Topic, [c.Tag] AS Tags, c.GoodAsTopic, 1 AS TagCount,
    COUNT(distinct OrigDomain) as cntOrigPublishers,
    COUNT(distinct RepostWebResourceHash) as cntRepostWRs,
    COUNT(distinct (case when c.Valence=1 then c.SentimentHash else null end)) as cntPositives,
    COUNT(distinct (case when c.Valence=2 then c.SentimentHash else null end)) as cntNegatives,
    COUNT(distinct (case when c.Valence=3 then c.SentimentHash else null end)) as cntAmbiguous,
    COUNT(distinct (case when c.Valence=5 then c.SentimentHash else null end)) as cntGeneral,
    ARRAY_AGG(DISTINCT c.SentimentHash) AS SentimentHashes,
    ARRAY_AGG(DISTINCT c.OrigWebResourceHash) AS OrigWebResourceHashes,
    ARRAY_AGG(DISTINCT c.RepostWebResourceHash) AS RepostWebResourceHashes
  FROM CalcStatSentiments c
  GROUP BY c.SnapshotDateId, c.Tag, c.GoodAsTopic
),
CalcStatCombiTopics AS (
  SELECT 
    css1.SnapshotDateId, CONCAT(css1.Tag,' & ',css2.Tag) AS Topic, [css1.Tag,css2.Tag] AS Tags, true AS GoodAsTopic, 2 AS TagCount,
    COUNT(distinct css1.OrigDomain) as cntOrigPublishers,
    COUNT(distinct css1.RepostWebResourceHash) as cntRepostWRs,
    COUNT(distinct (case when css1.Valence=1 then css1.SentimentHash else null end)) as cntPositives,
    COUNT(distinct (case when css1.Valence=2 then css1.SentimentHash else null end)) as cntNegatives,
    COUNT(distinct (case when css1.Valence=3 then css1.SentimentHash else null end)) as cntAmbiguous,
    COUNT(distinct (case when css1.Valence=5 then css1.SentimentHash else null end)) as cntGeneral,
    ARRAY_AGG(DISTINCT css1.SentimentHash) AS SentimentHashes,
    ARRAY_AGG(DISTINCT css1.OrigWebResourceHash) AS OrigWebResourceHashes,
    ARRAY_AGG(DISTINCT css1.RepostWebResourceHash) AS RepostWebResourceHashes
  FROM
    CalcStatSentiments css1, CalcStatSentiments css2
  WHERE
    css1.SentimentHash = css2.SentimentHash AND
    css1.Tag < css2.Tag
  GROUP BY css1.SnapshotDateId, css1.Tag, css2.Tag
  -- HAVING cntPublisherDomains > 1
),
CalcStatAllTopics AS (
  SELECT * FROM CalcStatTopics
  WHERE GoodAsTopic = true AND cntRepostWRs > 1
  UNION ALL
  SELECT * FROM CalcStatCombiTopics
  WHERE GoodAsTopic = true AND cntRepostWRs > 1
)
SELECT SnapshotDateId, Topic, Tags, TagCount, cntOrigPublishers, cntRepostWRs,
  cntPositives, cntNegatives, cntAmbiguous, cntGeneral, SentimentHashes, OrigWebResourceHashes, RepostWebResourceHashes
FROM CalcStatAllTopics

Here is what I suggest you do:

• Read about Plutchik’s framework for Emotion analysis to understand the theory behind this solution

• Read about the ETL solution

• Go to the github repo and follow the instructions in README. Set up your own processing pipeline and run a test crunching a few news articles that I uploaded to the test folder.

To recap some of my earlier posts, Sirocco parses each news article or user-generated content into subjects and opinions. The subject and the opinion, together with the author of the news article, form a triad that allows us to answer the question: “Who (author) thinks about what (the subject) in what way (the opinion)?”

Finding out who authored a news piece tends to be straightforward. Their names are either on the byline of the article and can be parsed out of the text, or are in the metadata of the record that is part of an imported dataset.

We think of news articles as collections of opinions of various degrees of factuality. The individual opinions can be in a single sentence or in a sequence of sentences, and they are often, but not always, separated by paragraphs. Sirocco analyzes individual sentences and then “chunks” related sentences together, to form an “opinion”.

To determine the subjects of a sentence, we take the sentence parse tree (the constituency-based parse tree, to be precise) and extract Proper Nouns (aka Named Entities) and Noun Phrases. Named Entities are names of persons, organizations, locations, expressions of times, quantities and etc. Typically, they can be found by looking at the first letter of a word (if it is capitalized, and in the middle of a sentence, then it’s a good sign that it’s a Named Entity). There are many more rules for when a combination of words represent a name, but that’s why we have machine learning and training data sets, and the quality of Named Entity extraction has reached pretty high levels. In the terminology of the Penn Treebank Project, Named Entities are Proper Nouns, carrying the NNP part-of-speech tag.

Noun Phrases are 1-to-N-word phrases of nouns, adjectives, etc. For example, “missile test” and “air defense force” are all noun phrases. The NLP algorithms group words together into Noun Phrases based on their role in the sentence, and observed patterns in the training data sets. It’s helpful to process both Named Entities and Noun Phrases as potential subjects, because in many cases a sentence won’t have a name mentioned in it, but they will usually have a Noun Phrase.

Sentiment or opinion extraction is where the bulk of the work is done. Opinions can be roughly divided into emotions (“very interesting”) and expressions of qualities (“nice”, “big”). The Sirocco emotion detection technology is using a framework for human emotions originally developed by Robert Plutchik, a professor at the Albert Einstein College of Medicine.

Plutchik’s Wheel of Emotions identifies 8 basic emotions: Joy, Acceptance, Fear, Surprise, Sadness, Disgust, Anger, and Anticipation.

The 8 basic emotions are divided into 4 pairs, for example Fear is the opposite of Anger, Disgust is the opposite of Trust, Sadness is the negative of Joy, and Surprise is the opposite of Anticipation. Emotional intensity can vary, and taking Sadness as an example, on the extreme end we are looking at Grief, while low intensity Sadness is Pensiveness.

More complex human emotions, according to Plutchik’s research, can be thought of as combinations of two basic ones. For example, Love is Joy + Trust, while Pessimism is Sadness + Anticipation.

Plutchik’s emotion theory is not the only game in town for sentiment analysis; there are other emotion frameworks used by psychologists and researchers. However, Plutchik has something that other frameworks lack, and that is it’s perfect suitability for algorithmic implementation. As we have 4 pairs of emotions, we can use a relatively simple arithmetic for processing negations. And the composability of derived emotions allows us to build search patterns that consist of simple AND operations. Lastly, we can use the emotional intensity gradings, e.g. ranging from Pensiveness to Sadness to Grief on the Sadness emotional dimension, to process degree adverbs, e.g. “very sad” is Grief.

When a computer program attempts to understand the emotions people try to express in text, it faces dozens of signals that indicate the presence of emotions. In addition to adjectives and nouns such as “cranky” or “merry” representing emotions, strong signals are sent by linguistic constructs such as idioms, adjective phrases, adverbs and others, including:

The ability to process abbreviations, interjections, capitalization and emoticons is very handy for dealing with informal text including tweets and blog comments. In fact, when processing informal text so common in Twitter, the goals of sentiment analysis are a bit different than when processing fully formed sentences of an WSJ editorial. Building sentence trees is almost useless for tweets. There, after we tokenized sentences into words, and ran part-of-speech tagging, we stop at the chunking step and do not proceed to parsing at all.

The flat, sequential nature of written text also imposes some very unique challenges for a computer algorithm. For example, should emotion signals in quotes be associated with the author of the news post that is being analyzed? (In our opinion, they should not) How should we deal with emotions in question sentences? (The person asking the question seldom carries that emotion). What about the favorite example in every sentiment analysis textbook — irony? The words say one thing, while both the reader and the writer know they mean something else, based on the context of the previous conversation. To handle questions, quotes, and irony, we introduced a special sentiment dimension and called it “Ambiguous”. “Ambiguous” sentiment is a catch-all sentiment that we have to use when we find phrases that would be normally one of the 8 basic emotions (according to Plutchik’s theory), but are located in quotes or in question sentences or are possibly ironic.

So far we talked about emotions in text, but there are other interesting signals in text that could be considered opinions — word phrases representing qualities, such as “correct” or “elegant”. Expressions of quality cannot typically be mapped to one of the Plutchik’s emotion, but they do usually carry a positive or negative connotation. Sometimes quality words can have different meaning depending on the context, e.g. “massive flood” is negative, while “massive investment” is positive. We use our idiom dictionary to determine the semantic orientation, but when we can’t find a known idiom, we again use the Ambiguous dimension to flag such a phrase occurrence.

Lastly, when processing text sourced from the Internet, especially from non-moderated sites, one comes across profanity that one would not want to republish or repost in a research paper or site. We’ve added the ability to recognize several thousand words and idioms that are considered racist, pornographic, and profane, and flag such occurrences under the “profane” dimension. We also added the “unsafe” sentiment dimension to flag phrases that refer to sexuality, drug use, criminal activity and other potentially delicate opinions.

Let’s also talk about what happens to the entire news article once we processed all constituent sentences of that article. Knowing the subjects of each sentence allows us calculating statistics for the entire article of what the most important subjects are, taking into account the frequency of their occurrence, and their placement in sentences with strong sentiment. Our algorithm produces up to 7 tags to classify the topic of the article (we borrowed the term Tag from the social media context). The tags, together with all the opinions we extracted from the article form what we call a Content Index of the article. Here is an example of a Content Index for an article reporting on a UEFA soccer game between the Besiktas and Napoli soccer clubs.

TEXT STATS Length [3408] Paragraphs [18] Unique Entities [118]

TAGS Begin:

8.5 "Besiktas" [Good Topic]

6.0 "14-time national champion refusal" [Good Topic]

5.9 "Liverpool historic 2005 Istanbul triumph" [Good Topic]

5.5 "Napoli" [Good Topic]

4.9 "quarter-final appearance" [Good Topic]

4.7 "runners-up Bayern Munich" [Good Topic]

4.6 "Turkish" [Good Topic]

TOP SENTIMENTS Begin:

Sentiment {0} Tags: Besiktas, quarter-final appearance

Sentiment {0} Dominant Valence: positive

Sentiment {0} Total Sentiment Score: 34.0

Sentiment {0} Annotated Text: {{Besiktas}} will <<hope>> that their <<best>> previous showing— a 1987 European Cup {{quarter-final appearance}}— will not prove a <<poor>> omen.

Sentiment {0} Serialized Representation: V2|2|Besiktas|0|7|1|equarter-final appearance|73|96|1|e3|14|17|1|t116|119|1|n30|33|1|p

Sentiment {1} Tags: Besiktas, Liverpool historic 2005 Istanbul triumph, Turkish

Sentiment {1} Dominant Valence: positive

Sentiment {1} Total Sentiment Score: 30.0

Sentiment {1} Annotated Text: That {{Turkish}} comeback, rekindling memories of {{Liverpool's historic 2005 Istanbul <<triumph}>, has left the group outcome on a knife-edge with {{Besiktas}} knowing that a <<win>> at <<eliminated>> Dynamo Kiev will send them through.

Sentiment {1} Serialized Representation: V2|3|Besiktas|138|145|1|eLiverpool historic 2005 Istanbul triumph|46|87|1|eTurkish|5|11|1|e3|169|178|1|n81|87|1|j162|164|1|j

Sentiment {2} Tags: Besiktas

Sentiment {2} Dominant Valence: positive

Sentiment {2} Total Sentiment Score: 20.0

Sentiment {2} Annotated Text: The winner of Benfica-Napoli in Lisbon will <<qualify>> as group winners, while a draw would be enough to send the latter through given their head-to-head edge. If {{Besiktas}} lose, both will <<progress>> regardless.

Sentiment {2} Serialized Representation: V2|1|Besiktas|160|167|1|e2|44|50|1|p185|192|1|p

Sentiment {3} Tags: Besiktas

Sentiment {3} Dominant Valence: negative

Sentiment {3} Total Sentiment Score: 20.0

Sentiment {3} Annotated Text: Portugal's Benfica, European champions back in 1961 and 1962 in their <<halcyon>> <<days>> but who have underachieved since, should have secured their passage on matchday five but <<threw>> away a three-goal lead at {{Besiktas}}.

Sentiment {3} Serialized Representation: V2|1|Besiktas|204|211|1|e3|172|176|1|s78|81|1|g70|76|1|g

I hope I was able to provide you with more insight into how Sirocco works. As a parting thought, here is why I think the knowledge of how a person feels towards a subject matters.

When you know that someone is anticipating or is interested in something, you can work with that person to see if you can offer them what they are looking for. Alternatively, when someone is experiencing sadness, disgust, or anger towards a subject, then it’s a good indicator that they are looking for an alternative. When someone is positive, or experiences joy about a subject, you can use that to influence positively other people in their circle. Conversely, when someone is negative about something, you can use that to promote the alternative.

In the old, SharpNLP version of Sirocco, we used WordNet version 2.7 to look up the base forms, aka lemmas, of words. For example, if you have a verb “was”, its lemma, or base form, is “be”. Knowing base forms of words is important for us because we use pattern rules for English idioms that are defined using base forms of words. An example of an English idiom is “rub it in”, defined in Sirocco patterns as “rub/VB it/PRP in/IN”. This pattern allows us catching variations of the phrase, e.g. “rubbed it in”, “rubbing it in” etc.

WordNet is a large dictionary of English words (look up any word in this simple web UI) with a Java and many other libraries to access it programmatically. Its latest version is 3.1 and I was looking for a Java API that would allow me accessing the newest dictionaries and be able to run them in the cloud. The second requirement had a good reason. As I was reading about the different options, I realized that the best way to accomplish efficient execution in the cloud was to have the WordNet dictionary files packaged in a jar, and loaded as resources. I could not use WordNet dictionaries deployed as databases, or as files on the file system, as this would quickly break in the auto-scaled, auto-deployed world of modern clouds.

First, I looked at OpenNLP. OpenNLP version 1.6 supports WordNet via an add-on, which in turn is using the JWNL library , specifically, they have the JWNLLemmatizer class. If you want to use it in your project, add the following dependency to your POM (but read to the end of this blog post, before you do that).

<dependency>
  <groupId>net.sf.jwordnet</groupId>
  <artifactId>jwnl</artifactId>
  <version>1.3.3</version>
  <scope>compile</scope>
</dependency>

However, the JWNLLemmatizer from the 1.6 version of OpenNLP only returned a single base form for a word, which isn’t true for a significant portion of English words. Interestingly, the OpenNLP version 1.5.3 has the JWNLDictionary class which returns a list of base forms. I almost ended up using that class, but didn’t for reasons I will explain below.

After getting JWNLDictionary to work locally, I realized that I just could not deploy it to my preferred cloud provider. The OpenNLP add-on assumed it could access dictionary files directly on disk, which would not work with my data processing setup. After some researching I came across this Stackoverflow post which provided me with more options. I did not try every one, I admit, but I tried many. The package that did the trick was Extended JWNL, or ExtJWNL for short. I used version 1.8, and here is the necessary dependency declaration for your POM:

<dependency>
 <groupId>net.sf.extjwnl</groupId>
 <artifactId>extjwnl</artifactId>
 <version>1.8.0</version>
</dependency>

<dependency>
 <groupId>net.sf.extjwnl</groupId>
 <artifactId>extjwnl-data-wn31</artifactId>
 <version>1.2</version>
</dependency>

The second dependency is for the WordNet data files , btw.

Here is what you need to do to start using it.

import net.sf.extjwnl.dictionary.Dictionary;

... declare your Dictionary variable ...

private Dictionary wnDict = null;

... initializing in constructor ...

String propsFile = ConfigurationManager.getConfiguration().getString(“WordnetPropertiesFile”);
InputStream stream = getClass().getResourceAsStream(propsFile);
wnDict = Dictionary.getInstance(stream);

Then, from your methods that need to read the base forms (or lemmas) of words, you can query the Dictionary.

// calling the WN dictionary

POS fullposobj = POS.getPOSForLabel(fullpos);
List<String> bf = (List<String>) wnDict.getMorphologicalProcessor().lookupAllBaseForms(fullposobj,lowercaseLemma);

Here is how Sirocco is doing it in its BaseFormsDictionary class.

Btw, if you are wondering about the “WordnetPropertiesFile” property in the snippet above, here it is.

<?xml version="1.0" encoding="UTF-8"?>
<jwnl_properties language="en">
    <version publisher="Princeton" number="3.1" language="en"/>
    <dictionary class="net.sf.extjwnl.dictionary.FileBackedDictionary">
        <param name="morphological_processor" value="net.sf.extjwnl.dictionary.morph.DefaultMorphologicalProcessor">
            <param name="operations">
                <param value="net.sf.extjwnl.dictionary.morph.LookupExceptionsOperation"/>
                <param value="net.sf.extjwnl.dictionary.morph.DetachSuffixesOperation">
                    <param name="noun" value="|s=|ses=s|xes=x|zes=z|ches=ch|shes=sh|men=man|ies=y|"/>
                    <param name="verb" value="|s=|ies=y|es=e|es=|ed=e|ed=|ing=e|ing=|"/>
                    <param name="adjective" value="|er=|est=|er=e|est=e|"/>
                    <param name="operations">
                        <param value="net.sf.extjwnl.dictionary.morph.LookupIndexWordOperation"/>
                        <param value="net.sf.extjwnl.dictionary.morph.LookupExceptionsOperation"/>
                    </param>
                </param>
                <param value="net.sf.extjwnl.dictionary.morph.TokenizerOperation">
                    <param name="delimiters">
                        <param value=" "/>
                        <param value="-"/>
                    </param>
                    <param name="token_operations">
                        <param value="net.sf.extjwnl.dictionary.morph.LookupIndexWordOperation"/>
                        <param value="net.sf.extjwnl.dictionary.morph.LookupExceptionsOperation"/>
                        <param value="net.sf.extjwnl.dictionary.morph.DetachSuffixesOperation">
                            <param name="noun" value="|s=|ses=s|xes=x|zes=z|ches=ch|shes=sh|men=man|ies=y|"/>
                            <param name="verb" value="|s=|ies=y|es=e|es=|ed=e|ed=|ing=e|ing=|"/>
                            <param name="adjective" value="|er=|est=|er=e|est=e|"/>
                            <param name="operations">
                                <param value="net.sf.extjwnl.dictionary.morph.LookupIndexWordOperation"/>
                                <param value="net.sf.extjwnl.dictionary.morph.LookupExceptionsOperation"/>
                            </param>
                        </param>
                    </param>
                </param>
            </param>
        </param>
        <param name="dictionary_element_factory"
               value="net.sf.extjwnl.princeton.data.PrincetonWN17FileDictionaryElementFactory"/>
        <param name="file_manager" value="net.sf.extjwnl.dictionary.file_manager.FileManagerImpl">
            <param name="check_path" value="false"/>
            <param name="file_type" value="net.sf.extjwnl.princeton.file.PrincetonResourceDictionaryFile"/>
            <param name="dictionary_path" value="/net/sf/extjwnl/data/wordnet/wn31"/>
        </param>
    </dictionary>
    <resource class="net.sf.extjwnl.princeton.PrincetonResource"/>
</jwnl_properties>

ExtJWNL supports multiple dictionary storage backends (Memory, Database, File Backed) and uses property files to parameterize them. I used, as you can see, the File Backed dictionary, and all the WordNet files are located in the extjwnl-data-wn31–1.2.jar that I deploy to my processing infrastructure when I run Sirocco in the cloud.

In my next blog post I will write a bit more about the parts of OpenNLP I use to do sentiment analysis on text.

When we set out developing Cuesense in late 2000s, my partners and I standardized on the Microsoft stack mostly for skills reasons. Our sentiment analysis piece was built using SharpNLP, the C# clone of OpenNLP. It worked fine for the time, but when I decided to put it back to action recently, I faced a predictable challenge — the SharpNLP project was mothballed and I had my doubts that I would find many people interested in maintaining the C# version. On the other hand, OpenNLP’s Java branch actually graduated to the Top Level Project status at Apache and showed promise of consistent maintenance.

Java was not new to me, in fact, I used to develop pretty complex distributed systems in it before the first Internet bubble (Telco CLEC OSSes, anyone?). Moreover, C# is not that different from Java and I was hoping I could find an automated converter that would help me at least with part of the rewrite exercise.

In addition to converting the C# code base to Java and replacing SharpNLP with the latest Apache OpenNLP version 1.6, I also needed to convert the analytics database that was implemented in MySQL, with a bunch of stored procedures doing trends calculations. I will focus this blog post on the C# conversion and will write more about MySQL conversion in a later post.

I googled “C# to Java conversions” and came across this StackOverflow post. It listed several options for automated converters:

• CS2J

• Grasshopper

• CSharpJavaMerger

• Tangible Software C# to Java Converter

Long story short, the CS2J tool did a pretty awesome job, although it did require installing Mono and running a Windows-native app on my Mac. Here are the steps I took to do the initial conversion of my code base:

1. Install Mono to run CS2J

2. Install GitHub Desktop (because I could never remember the git commands, after polluting my memory with subversion, Visual SourceSafe and a bunch of other source control systems)

3. Get CS2J from GitHub or download from http://www.cs2j.com/

4. Install Homebrew to be able to install ant and maven

/usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

5. Run CS2J

cd cs2j

mono CS2JTranslator/bin/cs2j.exe -net-templates-dir=./NetFramework/ -out-java-dir=<project-dir>/JavaOutput -app-dir=<project-dir>/Common -cs-dir=<project-dir>/Common

mono CS2JTranslator/bin/cs2j.exe -net-templates-dir=./NetFramework/ -out-java-dir=<project-dir>/JavaOutput -app-dir=<project-dir>/Servers -cs-dir=<project-dir>/Servers

The JavaOutput directory was the directory were I wanted my Java code to end up, and the Common and Servers directories were folders were I kept my C# code to be converted.

SUCCESS!

Now that I had the initial output of the converter, the fun was just starting, as there were hundreds of compile errors that I had to resolve. But more on this in a future blog post.