Season 8 Episode 7: Rethinking Consent and AI
Season 8 Episode 7 [download this episode (MP3, 49.0 MB)]
In this episode Elizabeth chats with Jane Bailey, Professor of Law at the University of Ottawa and co-leader of the Rethinking Consent Project, to explore why the individual informed consent model is struggling to keep up with how data is used in AI and digital environments. They discuss how data is used in AI and digital environments. Jane explains how algorithmically assembled groups can hide discrimination and fall hardest on marginalized communities with implications for credit, employment, and democratic participation. They also look at why Canada’s rush to regulate risks centering industry voices may lead to getting something wrong. Jane makes the case for keeping consent where it is meaningful, treating collective consequences as structural policy issues. Setting limits on what systems can be released at all, and slowing things down so democratic deliberation can happen.
Additional Resources:
Learn more about Jane’s Rethinking Consent Project where she explores alternatives in the age of AI and big data.
Read The Great Scrape: The Clash Between Scraping and Privacy, for a more scholarly analysis on how scraping AI for conflicts with nearly all core privacy principles and that data being publicly available does not give scrapers a free pass.
To learn more about Canada's privacy case involving Facebook's role in the 2010s Cambridge Analytica scandal, read Supreme Court of Canada reserves decision on landmark Facebook data-harvesting case.
Racialized impacts in the age of AI discusses how marginalized groups end up being disproportionately impacted by these kinds of tools.
Check out She Didn’t Tell Anyone She Was Pregnant. But Target Knew, which describes how purchasing patterns can reveal something a person has not shared.
Safiya Noble's work in the Algorithms of Oppression highlights how Google search engines participate in data discrimination, making it a real social problem.
Contributors:
Host: Elizabeth Dubois
Research Lead: Julianne Bernier Robert
Transcript: Areej El-Sharif
Audio Editing: Sarah Pledge-Dickson
Translation: Julianne Bernier Robert
Publishing and Promotion: Aaron Brisson
Producer: Sarah Pledge-Dickson
Episode Transcript: Season 8 Episode 7: Rethinking consent and AI
Read the transcript below or download a copy in the language of your choice:
Elizabeth Dubois: [00:00:05] Welcome to Wonks and War Rooms, where political communication theory meets on the ground strategy. I'm your host, Elizabeth Dubois. I'm an associate professor and University Research Chair in Politics, Communication and Technology at the University of Ottawa. My pronouns are she/her. This season, we're talking about social and policy issues surrounding uses and governance of artificial intelligence. And today we're talking with Doctor Jane Bailey about consent and AI. Jane, can you introduce yourself, please?
Jane Bailey: [00:00:29] Yes, I'm Jane Bailey, I'm a law professor at the University of Ottawa, and I'm the co-leader of the Rethinking Consent Project, which is a shirk funded project where we're considering the role and continuing validity of the individual informed consent model as a justification for a lot of the way that our data is being handled in our current AI and digital environment.
Elizabeth Dubois: [00:00:56] Amazing. Thank you so much for being here today. I'm really excited to talk about this more. Obviously, the use of our data to train AI models is a hot topic. It comes up a lot. We also have heard a lot over the years about, you know, is it truly consent if you're clicking on this crazy long list of texts that is anybody actually reading? So I think today's conversation is going to be really great. I thought we could maybe start by laying a little bit of foundational groundwork. Can you talk to me about what consent actually looks like right now?
Jane Bailey: [00:01:29] Yeah. So in the current context, of course, it's important to respect people's agency. And one of the ways that we do that is by asking them, what would they would like to have done with their data? Are they okay with certain things being done with their data? And so it is important to respect that. So I'm not a person who says we abandon consent altogether, but the reality of it is that in the digitally networked environment. All kinds of things are happening. People and organizations are using our data without even asking us. So for example, scraping data to train AI models. There's a lot of us who feel like we really have little choice but to agree when we're asked, do you agree? There are all sorts of situations, like in education, for example, we may have to agree to using certain educational software if we want to be able to complete a course. The complex things that are being done with our data make it really difficult, if not impossible, for us to actually truly be meaningfully informed about what is happening with our data and what might happen to it. And as a result of that, we've got another really thorny problem that we've been thinking a lot about in the project, which is that when I make decisions or when groups of us make decisions individually about our data that has consequences for entire groups of people and members of groups of people who themselves might never have agreed to doing that. So it's a really complex environment that we're working in.
Elizabeth Dubois: [00:03:13] Yeah. There's a lot to unpack here. I want to go through kind of what are the main challenges or barriers to consent? Maybe we could start with something. You mentioned this idea of like, there's a lot of complex things that are being done with our data. What does that look like? How is that different with AI from what it maybe was before? How does that change our ability to, as you say, even consent, regardless of whether or not we're pressing the button?
Jane Bailey: [00:03:39] Yeah. I go back and forth in thinking about whether AI is a change in kind or whether it's a magnification of really [...] ongoing problems with consent. Individual informed consent has been problematic for a very long time because we know we're not all equally situated to make choices. There are all sorts of social constraints on that, but in the AI environment or the digital environment more generally, really, there are a few things that we've sort of identified that impinge on what an individual, whether an individual, can really meaningfully be said to be consenting. So one of them is something that we've been labeling now for later. So I might have consented years ago to whatever Facebook's terms of service were in terms of what it could do with a photo of me. But at that time, there was no, for example, facial recognition technology. And I certainly couldn't have foreseen that anything like that would happen. And that has a material impact on the significance of my photo and what can be done with it and its implications for me and for my privacy. We look at that and it's important to question, you know, is my consent at a time when a technology with that much significance for my privacy didn't exist, can my consent in the past really be a justification for the use of [...] my data [...] for those purposes now? We can also think about something we talk about as this for that which sort of builds on that prior example.
Jane Bailey: [00:05:20] We may now know that my photo, combined with those of millions of other people, can be used to train AI systems to make inferences about aspects of me that I might not have chosen to reveal and didn't understand, where a possible inference that could flow from a photograph, for example. This includes inferences that may or may not be true, that can be made about my sexual preferences. For example, based on an analysis of images of my face. So my consent to any particular use or terms of service with respect to my photo were made in a context where it never could have dawned on me that my photo, in combination with others, could lead to inferences about something that I may or may not have chosen to reveal myself. So those are a couple of examples.
Elizabeth Dubois: [00:06:15] Those are really helpful and tangible examples. And it makes me think, when I was first learning about the potential of data to be collected and analyzed, one of the risks that was flagged [was] a clear example [that] imagine[d] you [as], you know, an expecting mom, but you haven't told anyone yet. But your purchasing patterns online make it very clear to some algorithm that you're pregnant and they start sending you coupons and samples and different things. That seems like a great service. But also, if you live with your parents who have not been told yet that you're expecting it, like outs you as a pregnant person right there and then and when we think about what adding that kind of detail to some of these AI systems where there's just massive amounts of data collected about so many different people, and then like the inferences that can be made not just about you, but also across groups like that gets really huge and kind of way beyond, I think, what most people imagine their data gets used for. And so that actually connects back to one of the things you mentioned, the difference between individual and group consent. Can you talk a little more about the individual component?
Jane Bailey: [00:07:32] I guess things to break down the sort of two examples we've been talking about, I think a lot of people are able to relate to because they understand it's where their decision connects to a consequence for them. It tends to be a little more complicated for people to think about where their individual decision has a consequence for a group of people. And this can manifest in sort of a variety of ways, right? One is a phenomenon that we call me for you, which is suppose in this, you know, world of AI glasses that right now my glasses are actually AI glasses and I'm filming what's happening here, and I'm uploading it directly to my social media feed as we speak. The video I'm posting obviously isn't just information about me, it's information about you. And if there were other people here, or if there were other people that I can hear or can be seen, it's information about them. And though my social media service may accept my consent or my agreement to the terms of service, to uploading this content as either consent for everyone, or they might just ignore the impacts of everyone else, which I think is really more the case. If I haven't consulted you prior to posting. You didn't have any say in the matter, so why should my consent justify posting and using images of you? That's a little more, again, a little more tangible one. But for groups, there's kind of two sorts of situations to think about. One of them is when I make a decision that has an impact on a group that I know I belong to, right? So suppose I realized that my photo and other information I post about myself on social media can and will be used to train an AI system to make predictions about groups to which I'm kind of aware I belong to, even though they're somewhat amorphous.
Jane Bailey: [00:09:38] Like, for example, what sort of products do middle aged, white North American women want. Or better yet, do we think they need to be encouraged to buy, including things like hair color and age reversing beauty products? So even if let's assume, and I think this is a big assumption to make, that I can understand the consequences of that for me and for this demographic group. Still, we have to ask the question, how can that justify creation of profiles of that collective of people for corporate and others uses without consultation? And then also, so the group itself, but also other middle aged, white North American women in my calculated demographic suffering the consequences of my choice. Now, this is a quite benign example, but you don't have to go far to think about how significant this process can be for other marginalized communities, and the kinds of decisions that make about whether they're credit worthy, whether they're deserving of having a job, whether they're likely to be predicted as having criminal tendencies. There are all sorts of implications beyond this benign example that I've given. And then things amp up one more level, right? We've already talked about this sort of idea of algorithms and algorithmically assembled groups. So when I make a decision and thousands of other people make similar decisions to release their data and to agree to certain kinds of terms of service about what can be done with their data, what the algorithmic assembly, the aggregation of that data is done on a dynamic basis.
Jane Bailey: [00:11:31] So it's creating categories of people that we don't even realize we're part of, right? We don't identify with them in the kind of way that we would, with respect to other sorts of more identity factors that were a little more familiar with or categories of discrimination that we're more familiar with. It's dynamic. So it's an incredibly ephemeral group in the sense that that's just sort of happening in an ongoing way to predict how much might this person be willing to pay for this product so that we can do discriminatory pricing? Right? What sort of ads are we going to tell them? What sort of job advertisements are we going to allow them to see? Are these credit worthy people? And this is likely to have more detrimental impacts on marginalized communities. So communities of people where we already recognize the factors that are used for discrimination. But in the instant it might be based on, you know, people who wear red coats and walk their dogs at 6 a.m. Our worst credit risks because humans haven't been able to crunch the data that lets us identify those patterns and relationships. That's not even something that we're thinking of. So we're not even aware that we're being categorized in this way, which creates a really serious challenge for thinking about, well, how do you respond to or complain about discrimination?
Elizabeth Dubois: [00:13:05] Yeah. If the discrimination is completely hidden from us, we can't go complain about it. We can't go and say, this is unjust, or I've been miscategorized or you've labeled this whole category of people as being not creditworthy, and that's completely unfair. Or there's a data problem, right? Like we also know that data is not perfect. And when it's in this massive black box of, oh, it's been algorithmically generated understandings like we don't have good capacity to go and do even quality control checking. And so that can create so many more problems. I also just want to take a quick minute to shout out to a previous episode of the podcast we have with David Coletto, where we were talking about the idea of publics and public opinion. And we actually get in a little bit to this idea of like known and knowing publics, calculated publics, algorithmic publics, and the implications that has for like democratic participation. Because when you use these assessments of what people care about or don't care about to decide who you're going to bother trying to engage in the democratic process, like now you get really close to potential problems of alienating people from the democratic process, or worse, intentionally pushing certain groups away. And that then undermines the whole integrity of a democracy.
Jane Bailey: [00:14:36] Yeah, absolutely. And so there are such a range of consequences of the reliance on this model. And the one that you've described is, is such an important one to think about, because it goes to sort of the very essence of the democratic system and can have very serious impacts. And I think we've seen that historically. On the exclusion of members of marginalized communities from participation in even, you know, voting processes. So very serious consequences that, again, I think it's important to remember are not necessarily equally borne by everybody, right? The negative repercussions tend to be disproportionately borne by members of communities that we already know are marginalized by socioeconomic status, by race, by gender, by gender identity, all sorts of these kinds of categories.
Elizabeth Dubois: [00:15:34] So we know that there are certain marginalized groups that end up being disproportionately impacted by these kinds of tools. This makes me think of an example. You know, in the US where police forces started using facial recognition tools and other kinds of AI supported tools in police work. But the problem was it was all trained on data from existing arrest records and existing data that had been collected. And we know that in the US system, there's an overrepresentation of black people in prisons. And so the systems were getting really, really good at identifying racialized groups and not so good at others, which means that there was kind of this self-fulfilling prophecy that started happening. Are there other examples that you can think of, or other kinds of ways that the data is playing in and exacerbating this marginalization?
Jane Bailey: [00:16:30] Yeah. One of the concerns I have around the digital environment. And AI included in that is that it does create this sort of mathematical sheen on things. So it people think, well, if statistically this group is more likely to be criminalized than this other group of people. And that's why it's really important that we sort of go back and ask ourselves about what's being fed into this system that's making these sorts of decisions. I mean, if you go right back to the work of Safiya Noble in Algorithms of Oppression and looking at the way that a Google search engine at that time, the kind of responses that it returned to a query about black girls, which were highly stereotypical, offensively oversexualized and demeaning. And then think about, so what? It's a search engine result. But the fact of the matter is it both reflects ongoing discriminatory and stereotypical thinking in the sorts of things that people search for that are wind up getting baked in. But we also have to think about the impact again on the public. What is the impact of that in terms of not only reinforcing stereotypes amongst members of non racialized communities about racialized communities, but also what is the message that that's sending in this case to black girls themselves? And so what is the impact of this kind of repetition mathematically generated repetition of harmful negative stereotyping.
Jane Bailey: [00:18:21] So I think I always think of that as another really powerful example. And the thing I love about that book, I, it's been around for a while. It's a fascinating book. At the end, Safiya Noble talks about the fact that right at about that time, people were starting to recognize the impact of these systems on democratic processes, right? On targeting for exclusion or inclusion of people in participating in voting and in Democratic messaging and so forth or political messaging and so forth. And she just noted, well, you know, isn't it interesting that now people care about it? But when it was about black girls, members of other racialized communities, it wasn't a central sort of issue that everybody was hearing about, which I think is a really important point, and one I know we're going to come back to policy making. It's one, a really important one that people need to be – policy makers, need to be keeping in mind.
Elizabeth Dubois: [00:19:24] Yeah, yeah, I agree. And let's jump into the policy conversation. So Canada's been trying to move pretty quickly on AI governance. There's a lot of push, it feels like to me as an outsider, there's a lot of just like sprinting towards we need something which, you know, arguably we do, we do need something. And policy making and law making can be very slow, but it also feels like going fast we might risk getting things wrong. Or we might risk using old models of consent that it sounds like aren't necessarily a great fit for the actual context we're in now. What do you think on that big open ended question? I'm sure you have a super easy answer to it.
Jane Bailey: [00:20:07] Yeah, no, I don't have an easy answer to it. I wish I did, but there's a sort of few things that I would like to unpack there. One of the things is that we're working in an environment where AI is presented as inevitable, synonymous with progress, and key to securing our socio economic future. And that therefore, we need to move quickly to prioritize this so-called innovation agenda. And typically, and I respect the fact that, in fact, policy makers in Canada are now thinking about developing policy. That's somewhat different from the policy that's gone before, which has sort of paved the way for, you know, a handful of very powerful technocrats to really be making decisions that are having fundamental impact, the kinds of fundamental impacts we've been talking about on our society. So it's not only that we didn't have policy that regulated, but also that we had policy that made it possible for these kind of decisions to be made and encouraged them and incubated those. And again, not out of, you know, ill will, but out of the idea that somehow, you know, way back when the internet, the information superhighway, as it was called, was the goose that laid the golden egg. Right. That this is where the economic future goes. So I think even as we're talking about regulation, I think there's a there's a certain way in which the whole point of the regulation is really to make Canadians feel like they can trust these systems, even when there's all sorts of reasons for them to believe that they shouldn't and to ask the questions. Right. And this sort of particular approach that we're taking, you know, a 30 day sprint last October, which is really just an insult in terms of the ability of marginalized communities who are dealing with so many other things in their lives, should now have 30 days to say what they think about AI.
Jane Bailey: [00:22:18] So that in and of itself was an exclusionary approach. And because, as we've talked about, marginalized communities have the most to lose. In fact, our policy needs to be centered on their experiences. These are the things that we need to be thinking about when we're thinking about governance and regulation. Right now, the loudest voices have tended to be powerful members of industry. You know, again, it's not a conspiracy theory. We know what they're interested in and it has to do with making money. But the idea that they dominate the discourse and are able to participate in quick, fast processes is a reflection of the fact that they've been steeped in all of this from the beginning. And the only other thing I want to add about that is that I'm, I'm very concerned that we have for so long treated ordinary citizens, everyday people as not really having anything to contribute because we don't really understand the technicalities. And I think if, unless and until we move away from that model and recognize that people's life experiences and how they're affected by this technology, that's what it should be all about. What are the outcomes that we're looking for? What's the digital future we want to see? Not you can't understand the technicalities of how an algorithm works, for example. That's going to take time. And we're going to have to empower people to understand that their experiences really are relevant.
Elizabeth Dubois: [00:24:06] Yeah, I agree, and I think it's really important to be able to separate, you know, the technical standards from the goals of those technical standards. Like, of course, we don't want somebody who doesn't know the intricacies of the technology to be setting out very specific technical standards. We also don't want general policy to be doing that, because usually the technical standards shift and evolve over time. What we want is when we're setting out, you know, Canada's agenda for AI development and governance. We want it to be really rooted in the core values and goals we have for how these technologies might impact our society and the Canadian public good. And so I really like what you're saying of, you know, we need to be centering people's actual lived experiences and in particular, the lived experiences of those who are most at risk when we're thinking about these technologies being added into our daily lives even more than they already are. So that then brings me back to the question of how do we get this right? Like, what does it look like to do this? Well?
Jane Bailey: [00:25:14] Yeah. To bring it back to the idea of Individual consent and sort of think about, because that's mainly what I've been thinking about in this context. I think the first thing that we have to be thinking about is really, what are the situations where individual consent can really be legitimately understood as a justification for particular kinds of outcomes? We shouldn't be leaving the kind of collective outcomes. We shouldn't be hinging their legitimacy on the fact, well, thousands of individual people consented to those terms because we know already about all the problems associated with whether that consent is is meaningful in the first place, but also because an individual making that decision is it's a mismatch between my decision and the group consequence. How do I get to speak for a group? Or how do thousands of us or millions of us individually get to speak for a group when often. I think the reality is we're not even thinking of the group related consequences of what we're doing. That's one of the first things we have to do is to sort of ask ourselves, it's not abandon the consent model, but where is that model meaningful in terms of outcomes? Right? Where do we feel like knowing that someone consented to that makes us look at the outcome and say, well, that's a legitimate outcome. That individual was informed that individual was given the information that they needed. How constrained were their choices? That's another really important one. And in those situations where we think that's the case, then we need to think about how do we make the consent process robust, right? How do we make sure that the consents coming from the person affected, that it's informed.
Jane Bailey: [00:27:15] In what situation should it be revocable? In what situations should there be opportunities to revisit when things change like. So we can think about making individual informed consent where it's appropriately connected to an outcome more robust. So that's sort of one part of it. But I think that for the in the collective context, for the thinking about more broadly about collective consequences, I really think that's a structural policy issue where we want to talk about certain kinds of outcomes or in that we don't want or in the reverse, we might actually want to engage in a process where we imagine a future we do want, and then work backward from that to say, okay, what kind of duties do we have to impose on the organizations that are involved to make sure that our collective vision of what we want is achieved, and that they are understood to be stewards, acting in kind of a trust capacity, obviously, and clearly overseeing through regulatory authority. And so I think we're going to have to look at certain kinds of outcomes. And I know there are sorts of problems with the EU AI act, and I recognize all the limitations. But just the fundamental idea that there are certain kinds of systems that are just not going to be releasable to the public. They're just off the table.
Jane Bailey: [00:28:58] Those are the kinds of decisions I think we're going to have to make. We've made them before. We've regardless of individual consent in some circumstances, in some provinces, for example, I can agree that I'm giving up my right to a class to participate in a class proceeding against a company that I'm involved in a contract with. But as a matter of law, that choice has no force. Right. It's not because no one respects my consent or my agency. But we say as a society, there are certain kinds of outcomes that are not in the public interest. And so we're not going to hinge the legality of those on whether we asked an individual to agree or not agree to those. And, you know, we made that the decision in terms of human cloning. And I don't and again, these are not fast processes. This is not fast because it is about the vision of the society that you want to have. But it's a fruitful conversation. And we really need to be thinking about how we can strengthen our policy processes so that we empower involvement and in collaboration with broader groups of people. And as we've said in particular, the marginalized communities that have the most at stake, it's it's simply unethical and untenable to me, making decisions that, you know, have an uneven impact and not even be including centering the people who stand to bear the brunt of that in the process itself.
Elizabeth Dubois: [00:30:43] Yeah, absolutely. I mean, there's so many components here. It does give a sense of like, well, how are we ever, ever going to get anything resembling good enough out? Because at the same time, the technologies are moving fast and decisions do need to be made quickly. If we're going to stop particular kinds of things from being created or released, for example. And so there's just a lot of pieces. I think we could go on and have this conversation for a lot longer, but we're actually coming to time. So what I want to do is just finish off with a final question that I'm using as the final question for everyone this season. And that is, if you had to pick just one thing that you think policy makers are missing or just need to be reminded of, like, what's the one thing?
Jane Bailey: [00:31:31] Yeah. Oh, it's so hard to pick one thing and I'm sure everybody else is doing the same thing I'm going to do, which is to make one thing with parts, but subsections. I want to go back to the thing you talked about, about the speed of technological development. And one of the things that I think is important for us to be thinking about, and we have examples that are happening all around us with respect to data centers, for example, sometimes the policy process might need to slow things down. Right. Like even tech corporations in the United States are asking. They're begging with respect to agentic AI. Give us a process to slow things down so that things don't get developed and thrown on the market that are going to lead to catastrophic consequences. So I think that policy makers, I know there's a sense of urgency in this environment. And so one thing they might think about doing is figuring out how do we slow this down? The reason that law and policy are quote and quote slower. I, and I hate that analogy. I hate that idea, the whole sort of concept, then the development of technology is because policy done in the right way is about a democratic process. And deliberation takes time and it's worth it at the end of the day. We can't. Instead of having technology run our policy, we should be thinking about using policy to run, not to run technology in an expert sense, but to slow things down so we can get a grip on this and make good decisions.
Elizabeth Dubois: [00:33:30] Yeah. And that's also how you get decisions that there's buy in for, right? Like if you just make decisions quickly without having that open kind of deliberation, without getting the perspectives of lots of different people, you end up with a bunch of people who are all angry about the decision that got made. And that's not great for democracy either. So I hear you, and that makes a lot of sense to me. Thank you so much for your time today. This has been a really, really fascinating conversation. I'm really glad you were able to make it work.
Jane Bailey: [00:34:01] Thank you so much. I really enjoyed talking to you.
Elizabeth Dubois: [00:34:04] All right. That was our episode on consent and AI. Thank you so much for listening. As always, you can find additional resources in the show notes, and you can head over to polcommtech.com to find annotated transcripts available in English and French. In case you'd like to do some further reading, this season of Wonks and War Rooms is supported in part by the University of Ottawa's Office of Vice President, Research and Innovation, my University Research Chair in Politics, Communication and Technology. The Faculty of Arts and the Center for Law, Technology and Society. I also want to acknowledge that I'm recording from the traditional and unceded territory of the Algonquin people, and I want to pay respect to the Algonquin people acknowledging their long standing relationship with this unceded territory.
Comments