Orchestra
An open-source human-in-the-loop AI system to orchestrate teams of experts and machines
Humans where they're best, machines for the rest
Orchestra automates repetitive, rules-based project tasks so team members can focus on creative and analytical work, learn new skills, and mentor others.
Using a human-in-the-loop AI system like Orchestra will improve your project management processes, resulting in increased accuracy and consistency, less time coordinating projects, and more expert time in the areas they are uniquely positioned to work on.
With Orchestra workflows, you’ll better allocate your team’s expertise, improve communication across teams, and never miss another deadline.
Example workflow
Below we'll walk you through an example of how Orchestra could be used in a newsroom. Take a look at the example implementation in our documentation!
- An editor finds a good story and kicks off the reporting workflow, providing information to help a reporter get started.
- The reporter picks up the available story and writes up a draft article.
- A more experienced reporter then reviews the article and suggests improvements.
- In parallel with the reporting step, a photographer picks up the available story as well, capturing relevant photos.
- A senior photographer reviews the photos, making edits and selecting the best ones.
- The selected photos are resized and recolored for display across different media.
- Finally, a copy editor adds headlines and photo captions to complete the story.
Features
Orchestra’s major features that organize teams of humans + machines
Workflows
- The entire process above is called a workflow, comprised of five component steps.
- Two of these steps require review, where more experienced experts review the original work performed. Custom review policies (e.g., sampled or systematic review) for tasks can be easily created in Orchestra.
- The photo resizing step is a machine step, completed by automation/AI rather than by experts.
- Each step emits a JSON blob with structured data generated by either humans or machines.
- Steps have access to data emitted by previous steps that they depend on. In the example, the copy editor has access to both the story and the resized photos.
Project distribution
Projects are a series of interconnected tasks. A project is an instance of a workflow; a task is an instance of a step.
- Example: An editor with a story about local elections would create an elections project, with tasks for a reporter/photographer/copy editor.
Tasks are carried out by an expert or by a machine.
- Example: Photographers capture the story.
- Example: Machines resize and recolor the photos.
Hierarchical view
- Core experts do the initial work on a task.
- Reviewers provide feedback to other experts to make their work even better.
- The core expert submits the task when their work is complete.
- The reviewer can choose to accept the task, which is either selected for further review or marked as complete.
- They could also choose to return the task, requesting changes from and giving feedback to the worker they are reviewing.
Worker certification
- Certifications allow experts to work on tasks they're great at.
Experts can work toward all sorts of certifications, picking up practice tasks to build experience.
- Example: Joseph is a solid reporter but needs a little more practice as a photographer—let's give him some simple tasks so he can improve!
Experts need additional certification to work in a reviewer role.
- Example: Amy has been reporting for quite some time and would be great at mentoring new reporters.
StaffBot
- StaffBot quickly and easily staffs projects.
- It brings a worker on a project in minutes without relying on human support.
- Staffing and project coordination that used to take hours are now completed in minutes because StaffBot always knows which workers are available.
- StaffBot either automatically finds workers for a task based on custom logic, or reaches out to qualified workers over Slack or email and offers them a task.
- Workers can accept or reject a task after reviewing a summary.
SanityBot
- SanityBot periodically monitors the state of a project and alerts team members if something seems off.
It runs sanity checks every few minutes based on various aspects of project status to determine what’s on track and what’s at risk.
- Example: If you’re close to missing a major deadline, SanityBot will alert team members via Slack that the project conditions aren’t being met.
- SanityBot can’t solve problems, but it makes it easier to identify them so you can scale your attention across more projects.
Nested self-pruning checklists
- Checklists reduce mistakes by reminding workers about important to-do items to complete a task.
In these checklists, related items are grouped and hierarchically nested. For each project, a checklist self-prunes to remove irrelevant to-do items.
- Example: B12’s checklist to complete a customer’s website project includes 153 items and self-prunes depending on the project type.
- When it’s time for a reviewer to provide feedback, the checklist serves as a natural scaffold for that feedback.
- Reviewers can leave notes on individual checklist items to give specific and thorough feedback.
- The checklist reminds workers of specific feedback to keep in mind for future projects.
Time tracking
- Time tracking helps estimate how long a project or task takes.
- This feature provides a timer widget that workers can toggle on to track their time spent working on a task.
- Once the worker stops the timer, a time entry is automatically created for the amount of time worked. The worker can add a description to specify the task they’re working on.
- Orchestra’s timecard page shows all time entries for the current work week, grouped by date. The worker can edit and manually add time entries.
Life of a task
Below are two images of the Orchestra dashboard, the launching point for expert workers. Click to see how tasks move differently across the dashboard for core workers and reviewers.
Core expert
A core expert performs initial task work that will later be reviewed. The diagram below shows a task's movement through the core worker's dashboard. Click to enlarge.
Reviewer
A reviewer evaluates the core expert's work and provides feedback. The diagram below shows a task's movement through a reviewer's dashboard. Click to enlarge.
Community
Orchestra is a project built around people. We're developers and designers who love to think about how we can take small steps toward improving the world and we love meeting people who feel the same!
Reach out to us with requests for help, feature requests, suggestions, complaints, and compliments on our discussion forum!
Chat with us on the forumMotivation
Team
B12 has open-sourced Orchestra as part of our goal to build a brighter future of work.
At B12 our mission is to help people do more meaningful work. The rise of automation in the workforce is one of the largest challenges society will face in the coming decades. B12 looks to use technology and AI as a force for good to build a brighter future of work. As a team we strive to create a transparent and collaborative work environment that enables us to build the best possible products for our customers, including websites and SEO.
We started in New York, but are now a fully distributed company with B12ers in Canada, Kyrgyzstan, Turkey, many US states, and beyond. We hope you will join us!
Research
Orchestra is motivated by years of research into machine-mediated expert teams.
Flash teams by Retelny et al., a study in empowering managers to coordinate interactions between a team of experts contributing to a larger project. Review hierarchies work from Anand Kulkarni et al. and Daniel Haas et al. shows that machines can pair experts with other experts to improve work quality while facilitating mentorship.
Active learning, or human-in-the-loop machine learning, is the study of how machine learning model training can happen in concert with a domain expert completing their work.
In 2019, Bharadwaj et al. published B12's first paper on how we support constrained creativity in Orchestra through Dynamic Checklists, automated quality assurance, and contextual reviewer feedback.
In 2020, Rahman et al. published B12's second paper on how we built a mixed-initiative system to combine multiple sources of semi-structured information into a single structured action plan.