Now, I’d like to introduce our speaker for today. Melissa Valentine is an Assistant Professor at Stanford University in the Management Science and Engineering Department and Co-director of the Center for Work, Technology and Organization. Her award winning research focuses on designing groups and organizations, how the rise of information technology has impacted organizational behavior, and how new technology can improve organizational structures and outcomes. Professor Valentine’s work makes contributions to understanding classic and longstanding challenges in designing groups’ and organizations’ deep knowledge of how the rise- of how the rise of information technology has made it the possible new and different team and organizational forms. Her most recent study examined how the deployment of new algorithms changed the organizational structure of a retail tech company. Thank you very much Melissa, and I will turn it over to you. Good morning everyone. Thank you so much for joining. I look forward to interacting with you throughout the webinar. Um, I wanted to start by putting this research about how data and platforms are changing team design in historical context. I wanna talk about sort of what technological trends, um, this research is a part of and how it’s important for society and for how work is structured. So what I’m showing here is a graph from a book called The Vanishing American Corporation by Jerry Davis, who’s a Professor at University of Michigan. So he, um, motivates this book with this story. In 2009, um, a large retail chain called Circuit City went out of business. Shortly after that, a company called Systemax bought the Circuit City domain name and ran the Circuit City website, um, still continuing to fulfill customer orders for electronic goods. But running this website, running the Circuit City domain name took fewer than 50 employees, in contrast to the 30,000 person workforce who had been running Circuit City previously. So that one story illustrates some of the dynamics of this graph which again, comes from Jerry’s book. He has rigorously studied the number of corporations in the United States over time and he’s shown that the population of public corporations are shrinking. There are more who are exiting than entering this population at this time. Those that come into business typically employ a smaller workforce than the ones that are going out of business. So as illustrated here, in 2015, the combined workforces of Facebook, Yelp, Zynga, LinkedIn Zillow, Tableau, Zulily, and Box were smaller than the number of people who lost their jobs when Circuit City was liquidated in 2009. So there are some macroeconomic trends related to this, including, um, the availability of other kinds of capital. But some of this, and what he talks about in his book and what my research speaks to is the way that technology is underlying some of these trends. So what we’re gonna talk about today is how technologies are changing the basic ways that people organize work and organize businesses. So the historical trends that we’re looking at are, um, and what we should be thinking is about- thinking about is how work was organized in the past and how technologies are changing that. So what I’m illustrating here is some of the oldest organizational charts in the United States. Um, what these are speaking to is a way that public corporations grew up really around railroads and around large-scale manufacturers. So these kinds of corporations required a huge workforce of white-collar professionals who were doing the professional, administrative, and clerical work to coordinate these large-scale operations. So in the past, we had a large workforce doing the coordination and decision-making and now increasingly what we see is that software and different kinds of platforms are doing the kinds of work that the white color administrative, clerical administrative professionals were doing in the past. So the main argument of this book is that the technologies that organized coordination and decision-making have changed. So there are a lot of different ways that this conversation is being had, a lot of different ways that people are making this point. So I’ll just call out a few before we get into the specifics of flash teams and flash organizations, which is what my research is about. So in this paper that I’m illustrating here, which I highly recommend, Jerry Davis talks about the different kinds of robust organizational alternatives that are enabled by new technologies. So for example, um, peer production, so something like Wikipedia. So when you have a template and the Internet, then anyone online can come and edit, um, a page in the encyclopedia, and then you have this large crowd that participates in the creation of something new. So that’s different in a way that, uh, an hierarchical organization is organized and run. So having those kinds of, um, peer production or open innovation platforms is an example of one of the reasons that corporations or organizations are disappearing and shrinking. Um, others have talked about this in different ways. So for example, I was, um, presenting something like this and the venture capitalist named Roy Bahat tweeted me an article about how now software is the new organizational chart. So the example there would be something like holacracy or systems of record. So when you have software that sort of underlies all of the decisions in coordination, um, it really changes the way work is done. So that’s kind of the historical trend that we’re seeing. This is relevant because most of, um, society, right, is organized around these large-scale organizations and as they’re changing, then the structures of society are changing too. So I’m going to, um, use this as the motivating quote for the remainder of the presentation. So, Jerry Davis, having studied the vanishing corporation in the United States, um, has this sort of provocative way of thinking about this. He says that organizational design is now a subfield of computer science. So I’m gonna give you some examples of why he says that, sort of what it means, and then how this idea of using tools and tech for team design fit into this idea. So the first example of organizational design, now being a subfield of computer science, is the web-based enterprise. So let’s start with back at the example of Circuit City. So what was going on there. So the technology that underlies this change is actually not very advanced or revolutionary at this point. Um, as Jerry points out, it’s something simple called the, um, Application Programming Interface, and all that means is the way that the front half of the website talks to the back half of the website. So you’ll have a customer inquiry on the website that can now do a query to the database. So that was something that used to require a white-collar professional who would check, you know, check that customer inquiry and then look at the database. And so that’s now something that is automated. So his point is that with the API doing a lot of the coordination work, um, we’re going to see an increase in what he calls the webpage enterprise. So I’m gonna use his favorite example here. This is kind of a whimsical example. It’s a new kind of consumer cult product called the Instant Pot. So the point with the Instant Pot is, um, this was a new, um, product category on, um, Amazon that now has over 4- almost 40,000 five-star reviews. And what’s interesting about this is the Instant Pot started with a staff of five people and an investment of $350,000. So they were able to weave together this ecosystem of online Internet resources to create, develop, produce, manufacture, ship, sale, uh, sell, and market this new product. So I’m gonna, um, call out this New Yorker article from 2013 by Nathan Heller that sort of explains what’s going on here and why we can expect to see an increase in the webpage enterprise. So Nathan Heller published this in 2013 after interviewing some people who we’re working the Silicon Valley, San Francisco Market. And his informant was explaining this and saying in 2005, the whole thing exploded. Did you need hardware? No. Now you just put it on Amazon or Rackspace. Software? Now it’s all open-source. Distribution? It’s just the App Store, it’s Facebook. Customer service? That’s just Twitter. You just respond to your best customers on Twitter and get satisfaction. Sales and marketing? Now it’s Google Words, AdSense. So the cost to build and launch a product went from five million to one million to 500,000, now it’s 50,000. So the- the revolutionary technology behind all of this is the API. It’s a very simple technology but it makes, um, coordination and decision-making much easier and automated. And we’re seeing this used in a lot of different, um, organizational applications now. So I was giving you an example of the webpage enterprise. And now let’s get into team design which is the motivation of this webinar. So in this whole arc of talking about how technology changes organizations, what we’re seeing now is that you can use APIs basically to interface with the workforce. So we see that in crowdsourcing and we see that in the flash teams research. Let me illustrate a little bit about how that works. So several years ago, um, a PhD student, Daniela Retelny, a Computer Science Professor Michael Bernstein and a, uh, group of other PhD students- master students and I did research, um, to develop a web platform called Foundry. So the thing to think about with Foundry, it’s a- it’s a web platform that- that allows team design. So anybody can come to Foundry which is an open-source platform and can basically author a flash team. And then Foundry interfaces with online labor markets where there are millions of people, millions of workers who are available and registered in Upwork who can come work on the flash team. So I wanna talk about some of the features of this platform and the way that it connects to this overall trend of using more technology to do the coordination and decision making that people and organizational charts were doing in the past. So an important component or feature of Foundry is that it interfaces with large online labor markets. When we did this study several years ago, we configured Foundry to interface with a labor market called Upwork, so Upwork.com. So Upwork has millions of people who are expert in various kinds of professions like web development, mobile app development. Um, we worked with people who are product designers, content creators, editors. We had people who did marketing, business plans. So basically the way that Foundry works is it will issue an open call to the people on Upwork who have registered to work on a flash team. For the first research project that we did, we were trying to demonstrate that this kind of approach could be as fast as previous crowdsourcing approaches. So we can figured Foundry to hire first come first serve, um, which will obviously optimize on speed and the efficiency of hiring. And then I’ll talk later about different ways to configure the composing or assembling of a flash team. So basically Foundry interfaces using this sort of like API approach to a large online labor market. And then, um, the people will click on the link, and then be brought to Foundry, and then they’ll see what their role is in the overall flash team. They’ll see the workflow. They’ll see what their first task is. They’ll see the way that their task is interdependent with other people on the team. They can upload deliverables which will automatically pass them along to other people on the team. And then there are different features for the management of the flash team which will show like what work is on time, what work is, um, at risk, and, uh, the different ways that people can talk to each other. The other feature of Foundry that is interesting for managers who are thinking about team design, is that we configured it so that it could be basically like an organizational chart that anybody who was working on the flash team could edit. So what I’m picturing here on the bottom left, so think of that as a website, a web portal that anyone on the team has access to. So you see sort of the participants of the team down the left side, and then the different tasks that they are part of kind of next to them. So the way that this works is anyone can make a copy of the organizational chart or the workflow, and then they can make a suggestion. So they’ll basically add a task or add a role in their copy, and then when they merge it back in, then everyone else will see what their suggestion is. So what’s a cool property about this is it allows for top-down or bottom-up recommendations to how to change the team design, or how to change the organizational chart. So if somebody on the frontline sees that their team is maybe needing another person, um, or if they see that the hierarchy isn’t working the way that it’s configured, then they can make a suggestion on the org chart and when they merge it back in then somebody will review it. They’ll review what the suggestion is and they can accept the suggestion, change it, reject it, um, and then the suggestion will appear on everybody else’s page for how the work is going to be structured going forward. So again, there’s lots of different ways that you could configure this. You can imagine what would happen if you made this sort of like a democratic vote. Like, let’s say that somebody makes a suggestion that the hierarchy should be switched up for some portion of the task and then everyone on the team could sort of vote if that’s what should happen. So you can have this democratic review of the suggestion. The way that we had it configured during the study was to have the managers were the ones who always, um, approved the different suggestions that were edited in. So again, it’s interesting to think about this as the platform is now something that is dynamically editable by anyone on the team, and that sort of changes the properties of organizational structures, organizational design, and also the way that those decisions get made. So we ran a proof of concept study to show that this kind of approach could be used to crowdsource really complex projects. So what I’m illustrating here are the different organizational structures that emerged in the three different studies that we ran. So each of these started with just the, sort of, um, person who’s our client, the person who had the idea, so illustrated in the top is sort of this black figurehead. So it started like that, and then over time they would add a different, um, role, a different team, a different hierarchy, and these are the organizational structures that emerged. And I’ll show here the workflows that emerged. So each of these tasks was added over time. At the colors of the task boxes illustrate the different teams. So for example, in the top, you’re looking at one of the study- one of the projects that we ran, and you can see that towards the end of it that’s when the QA, which is an orange, the quality assurance team came online and started doing all of their tasks. So the other thing that’s interesting to think about here with flash teams and with this kind of archival record of all the work that was done, is the kind of data science that can be done on the tasks, the roles, the interactions, the hierarchy. So this starts to enable a lot more data science or sort of empirical analysis of team design of how the work is structured, um, of how sort of the work unfolds over time. So I illustrated, uh, the web page enterprise, and then flash teams. So web page enterprise you’re using, um, API to sort of connect different components of a website. With flash teams, you are connecting a client or somebody who has ideas with a whole team of people using an online labor market and a flash team using a flash team platform like Foundry. What I want to talk about now is some of the later research that we did after we ran the flash teams experiments, and where we were continuing to see what was possible when you use kind of software, when you use platforms to do some of this kind of organizing work. So the next example I want to talk about is different ways that you can use software to, to, to sort of think through the flash team members who’s going to be on the team. So the first example that I showed we were optimizing on kind of speed and efficiency because we were in conversation with the crowdsourcing research community. I want to talk now about, um, another model that was developed by Niloufar Salehi who is now a professor at UC Berkeley in the CS department. So she was the PhD student who led this research. And basically her idea, um, was to notice that the work of flash teams and flash organizations was basically sacrificing some of the familiarity of team members in order to get speed. So if you’re just doing a first come first serve to a large labor market, that means you’re just gonna get whose available, you’re going to get fast staffing of the team. What she was wondering is, was there a way that you could staff the team to really focus on the relationships between the team members? In general, this kind of model could be configured in several different ways. You could maybe be optimizing for different kinds of expertise. You could be optimizing for different time zones, different availability, different experience levels. So the variables that we’re optimizing here are the speed that we demonstrated with flash teams in the past, and then the familiarity, with people working together, and sort of accumulating a lot of experience working together over time. So basically, what she wanted to do with this study was to develop a system, an algorithmic system that would assemble or staff, the Flash Teams in a way that optimized the relationships of the past. So team members who had worked together before were weighted more heavily when staffing the next Flash Team that was, um, being developed. So she developed another platform called Huddler, and Huddler would keep track of all of the times that people had worked together in the past, and then use that information to staff the team’s going forward. So we ran a study this is, um, published and available on her website. So we ran a study that basically showed that using the Huddler algorithm that optimized for both familiarity and availability, we were able to quickly staff Flash Teams, where team members had worked together in the past, which is one of the predictors of really effective teams. All right. I want to now give my final example of how organizational design is now a subfield of computer science. So this is a continuation of the Flash Teams’ research stream. So if you think about this kind of over all or idea you have a platform that interfaces with labor markets or even large internal populations of employees, and then convenes teams to work together on a project. So there’s different features that we’re talking about here. One is the Assembly Feature, which is figuring out who should be on the team and sort of assembling them together. Um, the second piece is how the platform can help people work together more effectively. So this last example that I’m giving is, um, was led by a PhD student named Sharon Jew. And she was looking at how the platform could interact with team members to help them work together more effectively. So this is the paper that she led. She talked about it as, um, searching for the Dream Team. So I like this visual because what this is showing basically is the idea of how the platform helps the team members work together effectively. So basically, what you’re seeing here is three different teams. You have A, B, C. And then on the left, you have different kinds of team structures that the teams could use to coordinate their work. So you could have a centralized hierarchy or a decentralized hierarchy. You could have different ways that the team members are interacting with each other. They could have like, an emergent kind of interaction pattern. They could have some sort of like round robin or equally distributed interaction pattern. So what you see illustrated here is at the 10 rounds that the teams worked together, they adopted different team structures based on the algorithm that I will talk about in just a minute, that was recommending what structures they should be using. So what’s interesting about this research I think in particular is it is a really nice illustration of the kind of technical innovation that’s needed for this kind of work. And then the kind of basic social science or like managerial intuition that’s needed for this kind of work. So what Sharon was sort of speaking to when she was leading this project is a sort of known body of research in organizational design or management about how to structure teams. And what all of this research has shown is that there’s basically not a universally ideal way to structure a team. It’s gonna depend on who the team members are, what kind of task they’re working on, is it like an execution task or an innovation task? So knowing this and sort of being familiar with this research, she developed this algorithmic approach that was going to explore all the different spaces that teams could use to structure, and then recommend different ones to, um, the team leads, which are the Managers. She ran an experiment that showed that the teams that were sort of running with this Dream Team system, um, outperformed the Manager lead teams or the self-managed teams by about 40 percent in this one experiment. So the different structures that could be recommended to the teams are sort of, um, listed here. So there are a bunch of different ones that they could look at. Um, I talked about hierarchy and interaction patterns. I’ll notice at the bottom there’s one feedback norms, so people could give- could be encouraged to recommend it to get really encouraging feedback or really critical feedback. So this is sort of the space of all the different structures that the teams could use to facilitate or coordinate their work. This- this is sort of a decision space that the managers and team members were exploring to figure out, how they should be working together, how they shou- should structure their work. So the way this worked is, um, she configured basically a Slack Bot. And the Slack Bot was- so the team members would come into Slack, and they would be interacting with each other and they would play a game for one round. At the end of the round, um, the Slack Bot would collect feedback from all of the team members about how things had been going, and then using that data the Slack Bot would then recommend that they stay with the same structure that they had before or try and experiment with a new structure. So you can see an example here. Um, this is one of the actual teams that was running, so you can see that team member is kinda joking around and he’s saying, “I’m too scared to type anything.” And then the Slack Bot recommends that they adopt a very, uh, cheerful encouraging way of giving feedback to each other. And then after that the team members, again, this is kind of joking and interaction with the Bot, the team members are saying like, “Go team. We’re killing it.” So there’s kind of like feedback, um, evaluation, recommendation feedback loop is the kind of thing that we’re thinking about here. And what the algorithm was doing to sort of, um, run the Bot behind the scenes was figure out if the Bot should recommend staying with the same structure or trying a new structure. And what Sharon found in a study is basically when- so the Dream Team teams outperformed the other teams. So when she dug into so- some of the dynamics of what was happening there, she saw that the managers- when the managers were running the team, they weren’t trying enough variations. They sort of found a structure that worked and just stayed there, stayed with that structure. So the Dream Teams outperformed the manager-lead teams because they explored more- they explored more structures. And then these, uh, self-managed teams are the teams that kept getting recommendations about how to change, adopted the changed too often, which gave the- the team members started to ignore the recommendations because it was too many recommendations and too many changes. So the Dream Team algorithm had this way of sort of figuring out how much to explore versus how much to- to sort of stay with the same structure that had been working in the past. So in general, the Dream Team, uh, Bot interface is just this proof of concept of the idea that now that teams are increasingly working on platforms and everyone’s sort of interacting in this platform, the platform can be configured to have these kinds of like latent measurements of feedback or how things are going, sort of a latent collection of data, analysis of data, and then some sort of recommendation to try to help the teams perform better. Now, obviously across all of this there’s going to be um, it’s going to be really important that whatever platform, whatever algorithm is being designed is done in collaboration with the people who it’s acting upon, otherwise there will be a lot of kind of gaming or not using the system. So it has to be something that is working for the people that it’s trying to help instead of some sort of like top-down tech design, which tends to get ignored subverted or gained. So those are the examples of how team design is now happening increasingly on platforms using data for assembling teams, for helping them coordinate, um, and how this sort of allows for different properties of the teams, they’re happening, you know, increasingly outside of organizations or in these sort of optimized ways within organizations. So I wanted to conclude on this slide, which is sort of what do you take away from all of these kinds of studies and this is the kind of future of work conversation. And across the number of different studies that we’ve done or the different kinds of conversations that we’ve been having with different leaders of organizations, here’s sort of the takeaways. This is kind of where I see things- this is where I see the conversations sort of happening and getting a little bit stuck in organizations, this is opportunities basically. So the first is, um, all of this data can be used for making decisions in kind of smarter more optimized ways, data can be helpful. What I see is sort of the limit in a lot of organizations right now is actually access to data or to data infrastructure. So a lot of the managers that I talk to will have- will sort of be bought into this vision of how software and how data can be used for organizing and for coordination, but basically it’s hard to get it done in their company or in their organization just because the data infrastructure is not there, like people don’t have access to data. Um, they don’t have- yeah, the- the tools with the access to the data in non-siloed ways to sort of do this kind of optimized decision-making. So investing in access to data and sort of like a really broad open data infrastructure seems to be, um, a capability that a lot of people are working on right now. Um, and then part of that is now that managers and different occupations are using data to do things like team design, team assembly to support team coordination, or just other kinds of decision-making like market research or things like that. I’m seeing a lot of conversation about companies and occupations needing to develop a new kind of data fluency. So people who were not necessarily having to use data for the kinds of decisions they were making in the past are now needing like data curriculum. What is- what is like an A/B test, what is, you know, data science 101? Um, the second takeaway I think for managers, um, that I’m seeing as a result of all of these technological changes is that people are needing new processes for thinking about how to manage the boundary of the firm. What I mean by that is like Upwork is this large online labor market of really expert people, and then there are companies that are interfacing with online labor markets or sort of like, I mean it’s- it’s a- it’s another way of thinking about outsourcing. So I’m seeing a lot of conversation around how organizations can interface with the crowd, can interface with online labor markets, can interface with freelancers or people who are sort of working outside of the organization. There are some really important social implications and policy implications that are part of a larger conversation. Um, for the purposes of- of this webinar, um, what I’m seeing is that people are talking a lot about how to manage the boundary of the firm, and how to sort of coordinate with the crowd. The last piece of it is a lot of this is new, a lot of it is emerging, a lot of it is changing, and there’s a lot of cool opportunities. So there’s not really a “thou shalt” or like a right way to do this at this point. What I’m seeing be the most successful is, um, companies that are investing in kind of human-centered design around this and looking for really kind of tight feedback loops in trying something, er, doing some sort of like co-design with workers, with the employees and figuring out how to configure these things in a way that supports work, and that supports workflows, and helps people kind of do the kind of work in the way that they want to in a really what we’re- kind of virtuous feedback loop. So these are the kinds of problems that I’m seeing in a lot of the, um, examples of how- of the conversations of how to sort of start to think about using software for organizational design and team design in organizations. And I look forward to, uh, continuing conversation. All right. Thank you so much Melissa, uh, for this very informative presentation, it’s such a timely and important topic. Um, I think it’ll really help everyone. The first one is, “your paper on flash teams stated that this method can be used for engineering and design work. Um, but is there a limitation on the type of work that these kinds of methods can be used on, and do you plan any future research on different types of work?” Yeah, thank you for that question. Um, that reminds me of something that I had wanted to mention actually. Um, we’re- so we’re seeing a lot of different examples of now industry use cases of something like flash teams. So for example, there’s a San Francisco startup called Gigster that does software and engineering, um, so that’s similar to what we were talking about. But there’s actually a lot of other models that I’m seeing, um, some cool examples include there’s something called Artella. Artella is artists, so they’re digital artists who do, um, they sort of have like the way that- the way that the founder explained it to me is they have like the Pixar workflow but then they have like a community of people who do digital animation who can come participate in the different projects that each other pitch. Um, there’s something called Cataline which does this kind of model for, um, business plans. So it’s basically a network of like MBA’s. So if you need to do a flash business plan [LAUGHTER] you can interface with MBA’s who do this. So, um, I’ve seen- I’ve seen it. And I- I’ve, as I’ve given this presentation I’ve heard people have ideas for like flash teams of accountants, for like audits. So I think for proj- I think for project-based work we’re gonna see an increase in this kind of idea of like platform-mediated projects. So I think it works well for project work. I think the open question, and this is something that we’ve thought a lot about as well, the open question is, what does the sort of maintenance of whatever the deliverable was, so if you have this like body of code, if you have like the- the sort of like you know that code that- of the mobile app, like who maintains that over time when you have this kind of flash model? So the idea of the like core pr- the core team that stays on and maintains versus like the peripheral team that in a flash comes and helps participate in building something out. So the- the- I think that ends up being the- the, um, where we’re thinking about, where innovation would have to happen or like new thinking would have to happen is in, uh, just like the maintenance of longer projects, because this works best for the kinds of like short flash projects that I’ve been talking about. Thanks. So here’s a more general question. So teams have always been used to get work done. So why aren’t we better at it, and why does technology make a difference? Yeah. That’s- so that’s a great question. Um, I think that’s an important one. And, um, it’s- what- what I’m thinking about with that question is when I assign, uh, team projects in my classes then it’s- it’s to groans. It’s met to groans, nobody likes working on a team because you end up with all the social problems of, you know, maybe some people are working harder than other people, um, you know maybe you have some like funky hierarchy where the person who’s most expert is not the one who’s in- who’s- who has the authority. Um, so all of the like social psychological problems and challenges of groups remain. Technology does not solve that at all. [LAUGHTER] Sometimes technology can make it worse. We’ve seen like virtual teams and online teams that research has shown tend to have more conflict than others. Um, so I will say, you know sometimes there are- sometimes teams can be better than working alone when you have like a synergistic team where you’re like bouncing ideas off each other and- and you really like working together in this innovative way, that’s the dream. And I don’t think that technology in any deterministic way gets you from like the bad team to the good team. So the- the point is- the point is- is good. [LAUGHTER] The- the point stands, teams are hard. [LAUGHTER] So has your research found any way to control for the quality of the final deliverable? Um. So- so thank you. Again, that’s a really good question. Um, so, uh, when we did the flash teams, field experiment, um, one of the challenges that the clients- so they were the ones who had the idea of what they would build using the flash team. So they were very concerned about the quality of the product that was being built. And, um, what the feedback that we got from them is that that is definitely a place where, um, the interface between an existing organization, where you oftentimes have a lot of, kind of like quality checks and quality assurance built in place. That’s where that interface can be pretty problematic or place to really watch for. So again it’s a really important question and I think it’s the right kind of thing to be thinking about. Um, in our study the- the solve for it was- was simply around hierarchy. It was simply around having like managers being the ones who are accountable for whatever it was turned in and for kind of checking it. Um, so I have seen kind of studies or I have seen different proofs of concept where people are looking at like technical solution for trying to look at, um, how to, you know, assess quality and more like automated ways or whatever. Um, so in- in our instance we just used, uh, we just used managers. So the managers were accountable for the quality of the product, with quality assurance teams, um, so that’s- that’s the, uh, the kind of like old school way. Managers armed with technology sort of was the- was the fix. And, um, I think we’ll see more of kind of studies of this in the future, because it is one of the main problems with crowdsourcing. Great. So, um, you said something, uh, about accountability and setting it up and it reminded me of this question that was asked. So essentially, they’re asking if leadership or managers ultimately make the decision to use a digital team, should they also design the process, will the project teams do a better job, how do you integrate knowledge and decisions? Um, yeah, good question. Um, so the managers should not design the team. I agree with you. I think- I think we are in- in radical agreement here. Um, so I think one thing- so we had, uh, a flash team study, um, several- like five years ago, I then the one I was presenting this, it’s a little newer. Um, so the- the one we had done initially, we used a- a workflow where everything was designed up front. So that would be something more like a manager making all the decisions for like how things should be structured. For a very, very simple execution, it was kind of fine, but what we saw is that like, you know, like one task into those flash teams the workflow was obsolete. This sort of like top-down design of how things should go was already like out-of-date, obsolete, and then the people who were actually doing the work had to improvise and they went very much like off platform and just improvised a better way to do it that fit their style, that fit their expertise, that fit the deliverable better. So that’s actually what led us to work on the new functionality and Foundry that allows for this kind of editing over time. So now a manager like the client who has the idea, basically just starts with like a blank slate and then hires people, and then they themselves edit, add teams, add tasks, kind of decide what should happen over time. So it is much more emergent and much more kind of team member designed together over time. There are- there were some challenges with that, like we got feedback from the- from the team members that sometimes they wanted more direction from the client manager and sometimes they wanted less. So it’s still is a little bit of a balance. But the functionality that I would sort of look at there is the idea of kind of like co-designing the workflow, co-creating the structures together over time with some sort of dynamic interface. So we have, um, I was just re-reviewing. So we have several questions on concerns about IP and data. Um, so how do you, uh, how do you address those kinds of concerns when working on a flash team, or- or using a flash team for work? Yeah. So that’s a great question. So intellectual property is another really kind of key issue around this flash team model. When we did the study, we just followed the Upwork policy for intellectual property. So Upwork has protections in place- protections and policies in place, that basically says that the client owns the IP. And there’s some, um, there are some ways that they, through their sort of user agreement, uh, deal with the IP issues of having freelancers work on, um, intellectual property of a company or a client. Um, so it’s- it’s a little bit of a punting the question and just saying about the Up- the Upwork labor market had their protections in place and that’s what we were relying on when we did the flash teams. Um, I think as organizations become increasingly disintermediated, we see more of these kind of platform models and I suspect there will be changes to policy, um, I’m not an expert in that but maybe changes to the way that IP is protected. All right. This is- it’s the right conversation and, um, I think the future might look different than what we have now. Okay. So I’m going to, uh, give you two more, uh, two more questions that are very, uh, common themes through here. So the first one is how- how should people go about trying to get this kind of work started within a more mature organization, how did they get people to be receptive to this, not feel replaced, still feel like their work is important and while also having these- these flash teams? Um, so there’s- there’s sort of two different concerns in that question and I think both are really valid. One is sort of, um, like how to have conversations in large organizations that have an existing way of doing things and one is concerns about, uh, worker replacement and kind of like worker rights, and I think both of those are really- really important. Um, so I’m actually start with the first- the second one first. Um, I think that there is a really broad conversation going on right now about how, um, kind of like this social fabric of our society and not to be dramatic, but kind of like the social fabric of our society. So for years and years, the social safety net has existed around organizations, people have had their entire careers in org- in like a single organization, you know, health insurance, retirement, pension, career ladders, like post-industrial revolution, society just evolved around organizations, and as that changes, then that means that there’s a lot of- a lot more precarity. And like a lot more precarity for workers, and again that’s the right conversation, that should be like the foreground of what we’re talking about with any of these like platform models for organizing. Um, so it’s just such a larger conversation, thank you for bringing it up. Um, and I’ll- I’ll note that there are just- there are so many places that conversations are happening and I- I hope we all continue to participate in that. Um, in terms of inside organizations, how to start this conversation, what I’ve seen is, um, that there are- are people who will hear something like this and they’ll- they’ll sort of recognize it, there’s like some sort of data source and their organization that could be used for this kind of organizing. And- and then like a small pilot, like it’s just a small proof of concept that shows how to use those data to assemble a team, or like a small pilot that shows how, um, I don’t know like a- Slack. I mean Slack doesn’t solve everything. I’m not- this is not me being tech deterministic. Um, but just I guess what I’m saying is like small pilots to demonstrate proof of concept, can start a really interesting conversation and also if you’re interested in doing research, you can always contact, um, academics, see if you can do a study together because it sometimes that will be a low risk kind of outside of the workflow that the operational workflow way that kind of explore these kind of ideas. That’s like another idea. Okay. Last- last one. Um, so how do- how does ideation, innovation and execution, integrate, um, to assign value for different contribution? So would you pay, you know, if you had a base rate but then, certain people within the flash team were performing at a much higher rate or perhaps were not performing that well, would you then adjust the compensation? Um, so it’s an interesting question to think about kind of how people are, how people’s contributions are recognized in different ways. Um, I’ll tell you things that I’ve heard that I’m not satisfied by, but I think, um, and then I’ll [LAUGHTER] and then I’ll say what I think. Um, so there is- well I shouldn’t say it like that. So in terms of like- because what I would wanna avoid with this conversation is basically anything that is like again, like, too like tech deterministic, like people are always interesting in how they will enact these different social systems. Um, so what I’ve heard people talk about is like, is different kinds of like micro equity or sort of like- some sort of like, you know, how do you do like equity in a flash team. So that’s an interesting idea, right? If somebody- if the client has this idea and then people are contributing to the idea in different ways, is there a way to sort of like measure and use data to do equity and sort of like- like sort of reimburse people like [LAUGHTER] or reward people for different- thank you, rewarded, people for different ideas. So yeah, maybe- maybe, that would be cool. Um, I think, you know, oftentimes this is playing out just like in a market system and capital, we’ll have capital and labor is recompense for labor. So it’s part of I think the overall ecos- ecosystem will develop in a way that, um, that sort of changes that conversation in different ways. Um, but I have heard- I have heard like different, like tech ideas about it. So I would just encourage you to like Google, like equity for flash teams or something like that if that’s of interest, but it plays out in the larger ecosystem of startups and like large organizations and things like that. Thank you. So, um, people have been chiming in that that this was a really wonderful webinar. So I just wanted to thank you for- for taking the time. Um, and just for everyone who’s online, I wanted to let you know that Melissa has a- a course coming out, um, in November on this very topic. So, um, we will provide information with a, uh, with a follow-up email that you- that you will be receiving, which will also include a recording of today’s webinar. Um, we wanna thank you so much for your time today, and have a great week.