Cognition Studio 3.0 Release Makes It Easier to Detect Risk
The recent release of the Smarsh Cognition Studio 3.0 has enabled customers to use real-world data from their production environment. With this release, customers can rapidly and robustly prototype and evaluate the analytic components they have designed to detect risk behavior. In Cognition Studio, these components are referred to as skill blocks. Skill blocks that are brought together in combination for deployment result in a Scenario.
Before this release, it was hard for customers to tell whether changes being made to a Scenario in Cognition Studio were delivering improved results. Scenario Evaluation gives customers the ability to demonstrate the business value that has been added through Scenario development with excellent levels of explainability and auditability.
In this post, we highlight a couple features of the Cognition Studio release.
Scenario Evaluation
A Scenario Evaluation is the means to measure the performance of a given Scenario (or Scenarios) against a known dataset. The Scenario is fed a collection of communications (a dataset) and processes each one to see whether it would generate an alert or not based upon its component skill blocks.
Given that the user can pre-label the dataset to indicate whether each communication is a positive or negative example of the risk behavior in question, the Evaluation is then able to provide the user with metrics around how successful the Scenario has been at identifying the target risk behavior.
With Scenario Evaluation, customers can:
- Evaluate the performance of up to four Scenarios simultaneously (in a champion vs. challenger style approach)
- Compare evaluations with a sophisticated and user-friendly dashboard
- Drill down into the performance of the individual skill blocks that make up each Scenario
This enables users to quickly spot misconfigured, under or over performing components, which can be amended in Cognition Studio and retested.
Communication Audit Tool
The Scenario Evaluation feature comes with the Communication Audit Tool. This tool allows customers to view the results of the Scenario Evaluation to learn why an individual message alerted or failed to alert.
The Communication Audit Tool:
- Provides a comprehensive view of how every communication in a dataset has been handled by the Scenarios Evaluation
- Finds messages of interest and displays information on how they’re processed in different Scenarios
- Identifies messages of interest, such as false negatives
Evaluations at scale
One of our major goals in designing Cognition Studio was to enable our customers to test and validate their new and augmented scenarios at significant scale in order to have the confidence to promote them into production. We often asked ourselves:
When the customer is in the process of redesigning/re-evaluating a Cognitive Scenario, what scale of data (i.e., number of communications) will they like and expect to use for Scenario Evaluation?
We ultimately decided on a dataset of one million communications.
It should be noted that Cognition Studio offers incredibly high-fidelity results. Each communication is available in the Communication Audit Tool for detailed investigation of how each skill block in the Scenario behaved.
Running a Scenario Evaluation at significant scale on an unlabeled dataset gives a customer an idea of the alert volumes that will be generated if the Scenario was to be pushed into production. This is particularly useful within the context of a Champion vs. Challenger Scenario Evaluation.
For example, if the existing Champion and the new Challenger were evaluated against a large dataset, it’s important that the new Challenger doesn’t produce an enormous spike in alerted communications. If this did occur, it would be a strong indication that the new Challenger has been overly tuned and the precision has suffered.
Streamlining compliance
We are continually building new tools to give you better insights into your communications data. The intent is to make supervision and surveillance more efficient by reducing the number of false positives in your reviews. Stay tuned for more exciting developments with Cognition Studio.
Share this post!
Ed Hallam
Latest posts by Ed Hallam (see all)
Smarsh Blog
Our internal subject matter experts and our network of external industry experts are featured with insights into the technology and industry trends that affect your electronic communications compliance initiatives. Sign up to benefit from their deep understanding, tips and best practices regarding how your company can manage compliance risk while unlocking the business value of your communications data.
Ready to enable compliant productivity?
Join the 6,500+ customers using Smarsh to drive their business forward.
Watch it Work
Contact Us
Chat
Subscribe to the Smarsh Blog Digest
Subscribe to receive a monthly digest of articles exploring regulatory updates, news, trends and best practices in electronic communications capture and archiving.
Smarsh handles information you submit to Smarsh in accordance with its Privacy Policy. By clicking "submit", you consent to Smarsh processing your information and storing it in accordance with the Privacy Policy and agree to receive communications from Smarsh and its third-party partners regarding products and services that may be of interest to you. You may withdraw your consent at any time by emailing privacy@smarsh.com.
FOLLOW US