Orchestra

An open source system to orchestrate teams of experts and machines on complex projects.

Orchestrate teams

Humans where they're best, machines for the rest.

Orchestra orchestrates teams (e.g., reporters and photographers in a newsroom) and projects (e.g., a news story). In Orchestra workflows, you can assign senior experts to review other experts in order to provide feedback and iteratively refine work. Orchestra also brings automation, like classifiers or crawlers written in Python, onto projects to help out. New workflows can be added with some simple Python glue and an html interface.

How it works

Example Workflow

Below we'll walk you through an example of how Orchestra could be used in a newsroom. Take a look at the example implementation in our documentation!

An editor finds a good story and kicks off the reporting workflow, providing information to help a reporter get started.
The reporter picks up the available story and writes up a draft article.
A more experienced reporter then reviews the article and suggests improvements.
In parallel with the reporting step, a photographer picks up the available story as well, capturing relevant photos.
A senior photographer reviews the photos, making edits and selecting the best ones.
The selected photos are resized and recolored for display across different media.
Finally, a copy editor adds headlines and photo captions to complete the story.

Read the docs View repository

Identify a new story

Editor

Photograph a story

Photographer

Senior photographer

Report a story

Reporter

Senior reporter

Automatically crop, resize and color

Machine

Write headline and photo captions

Copy editor

Concepts

We'll now walk you through major Orchestra concepts based on the example above.

Workflows

The entire process above is called a workflow, comprised of five component steps.
Two of these steps require review, where more experienced experts review the original work performed. Custom review policies (e.g., sampled or systematic review) for tasks can be easily created in Orchestra.
The photo resizing step is a machine step, completed by automation rather than by experts.
Each step emits a JSON blob with structured data generated by either humans or machines.
Steps have access to data emitted by previous steps that they depend on. In the example, the copy editor has access to both the story and the resized photos.

Project distribution

Projects are a series of interconnected tasks. A project is an instance of a workflow; a task is an instance of a step.
- Example: An editor with a story about local elections would create an elections project, with tasks for a reporter/photographer/copy editor.
Tasks are carried out by an expert or by a machine.
- Example: Photographers capture the story.
- Example: Machines resize and recolor the photos.

Hierarchical view

Core experts do the initial work on a task.
Reviewers provide feedback to other experts to make their work even better.
The core expert submits the task when their work is complete.
The reviewer can choose to accept the task, which is either selected for further review or marked as complete.
They could also choose to return the task, requesting changes from and giving feedback to the worker they are reviewing.

Worker certification

Certifications allow experts to work on tasks they're great at.
Experts can work toward all sorts of certifications, picking up practice tasks to build experience.
- Example: Joseph is a solid reporter but needs a little more practice as a photographer—let's give him some simple tasks so he can improve!
Experts need additional certification to work in a reviewer role.
- Example: Amy has been reporting for quite some time and would be great at mentoring new reporters.

Life of a Task

Below are two images of the Orchestra dashboard, the launching point for expert workers.
Click to see how tasks move differently across the dashboard for core workers and reviewers.

Core Expert

A core expert performs initial task work which will later be reviewed. The diagram below shows a task's movement through the core worker's dashboard. Click to enlarge.

Reviewer

A reviewer evaluates the core expert's work and provides feedback. The diagram below shows a task's movement through a reviewer's dashboard. Click to enlarge.

Getting involved

Community

Orchestra is a project built around people. We're developers and designers who love to think about how we can take small steps toward improving the world and we love meeting people who feel the same! Reach out to us with requests for help, feature requests, suggestions, complaints, and compliments on our Gitter channel. Alternatively, reach out over our our mailing list!

Chat with us on Gitter

Motivation

Team

B12 has open sourced Orchestra as part of our goal to build a brighter future of work.

At B12 our mission is to help people work smarter. The rise of automation in the workforce is one of the largest challenges society will face in the coming decades. At B12 we look to use technology and automation as a force for good to build a brighter future of work. As a team we strive to create a transparent and collaborative work environment that enables us to build the best possible products for our customers.

Our main office is in Union Square in New York City, but B12ers can be found all over the world in places like Wisconsin, Ottawa, and Bishkek. We hope you will join us!

Research

Orchestra is motivated by years of research into machine-mediated expert teams.

Flash teams by Retelny et al., a study in empowering managers to coordinate interactions between a team of experts contributing to a larger project. The Foundry project is a mature research prototype of these ideas.

Review hierarchies work from Anand Kulkarni et al. and Daniel Haas et al. shows that machines can pair experts with other experts to improve work quality while facilitating mentorship.

Active learning, or human-in-the-loop machine learning, is the study of how machine learning model training can happen in concert with a domain expert completing their work.

In 2019, Bharadwaj et al. published B12's first paper on how we support constrained creativity in Orchestra through Dynamic Checklists, automated quality assurance, and contextual reviewer feedback.