The Paradox of Tolerance for Tech Companies


I’m Valerie Aurora I’m here to
talk about the paradox of tolerance and how it applies to tech
companies today. In particular I want to address the issue of how to support free
speech without also supporting white supremacy – tough question a lot of
people are struggling with. Just a quick disclaimer before we get into this. I’m
representing only myself and Frame Shift Consulting, no other
organization or person. Many of the companies I talk about in this talk are
my clients but any information in this talk about them is totally from public
sources. I’m not using any private information for that. Finally I’m not a
lawyer and this is not legal advice so talk to your lawyer
if you have questions about legal problems. Just to point out to some
online resources I’ve put up the slides and some various related research
resources at this bitly link bitly slash intolerable speech and you can see some
more stuff on Twitter at frameshift LLC If you’re live tweeting this talk thank
you very much please tag frameshift LLC instead of my personal account thanks.
All right so I just I’m going to set the scene with reviewing the common u.s.
approach to free speech and I’m talking about the the general common knowledge
or how people think about it and it can be summarized in this way with these
words: I disapprove of what you say but I will defend to the death your right to
say it and that’s author Beatrice Hall describing Voltaire’s beliefs. Right so
there’s an interesting problem here: what if the people whose right to speech you
are defending actually want you to be dead. That this is their goal they want
to kill you and that makes them happy that you’re dying defending their speech.
Well back in 2003 the onion took on this kind of problem with this approach to
free speech at a joke satirical newspaper article entitled ACLU American
Civil Liberties Union defends Nazis rights to burn down ACLU headquarters. Right the Nazis would like for the ACLU to not exist, all right excellent.
Somehow appropriate now we fast forward to 2017 and it’s looking more and more
like actual reality the ACLU did defend the rights of people with actual Nazi
flags to march in Charlottesville and they carried torches, they threatened to
light people on fire, they beat people and they injured dozens and they
actually murdered a person as we all know so this is an actual real problem.
Something that was a joke in 2003 we are concerned about being in reality even worse
than that our president responded by saying that there were fine people on
both sides of this violence and various other comments which seem
to be supporting white supremacist and as a result about 37% of the people
polled thought that his response was either somewhat appropriate or very
appropriate. So this is a new world we are finding ourselves in it’s so new and
different that the ACLU did actually change its policy after a
Charlottesville. This is what Anthony Romero the executive director of the
ACLU has to say: if a protest group insists no we want to be able to carry
loaded firearms well we don’t have to represent them they can find someone
else. Keep in mind the ACLU has been defending Nazis since 1934. This is an
enormous change to make for this organization (audience question) I’m sorry I can’t
explain why the ACLU is supporting Nazis they but there’s lots of articles about
why they do it. They have a theory. So what I want to talk about is what this
means for tech companies and tech workers in particular, everyone here. The
outline of this talk is I’ll first talk about what the paradox of Tolerance is
then I’m going to go over my proposed Intolerable speech rule. It’s proposed
because I’m hoping you’ll improve it. I’m going to talk about examples of tech
companies taking action based on the paradox of tolerance just to show nobody is alone in this. A lot of people are doing it now, and I’ll
talk about how to implement the paradox of tolerance at your company.
So David told you a little bit about this but I’ll repeat for the people who
won’t get the introduction. I was a software engineer specializing in the
Linux kernel and file systems for over ten years
at which point I got tired of being treated in a sexist way and I co-founded
and was executive director of the Ada initiative which supported women in open
source software, Wikipedia, and things like that. I founded Frame Shift
Consulting which is a diversity and inclusion and technology consulting
company here in San Francisco. As part of that I teach the Ally Skills Workshop
which teaches people with more privilege and power how to support people with
less privilege and power. I’ve taught it to over 1,900 people in eight countries
at this point. More relevant to this topic –
it was interesting digging up things I was proud of – Breitbart news wrote about
me in 2015 back when most people had never heard of it. They said that I was
trying to frame Linus Torvalds for sexual assault. No I wasn’t. So I’m just
proud they wrote a scurrilous hateful wrong article about me back then.
Starting around 2016 I started doing a number of other things around
politics. One of the things I started doing is tweeting using my company
account about politics in mid-2016. After the election I co-organized the never again tech pledge where over 2,800 tech workers
took a pledge not to support human rights abuses with their work. I wrote a
post explaining how to change people’s hearts and minds and your families and
friends who might have bigoted or intolerant opinions. I went on marches
and made protest signs. I called my representatives. I donated thousands of
dollars. I did all of the thing. One of the things I did is I studied fascism. I
read these books and many others and many other sources of information on
both the rise of fascism and the situations in which fascism was thwarted
before it could come to full flower and that’s where I learned about the paradox
of tolerance. So the paradox of tolerance is actually just a rule with one
exception but paradox makes it sound cooler. So the paradox is: a tolerant
society should be tolerant by default with one exception it should not
tolerate intolerance itself. So there’s a great Wikipedia page you can send to
people or look for more details on it but I’ll talk a little bit more about
it. This is a graph of deaths by country in World War two both in
absolute numbers of millions and the percentage of the total population. The
paradox of tolerance was formulated in 1945 at the end of World War two by Karl
Popper. This is absolutely no coincidence because we just went through a situation
in which being too tolerant put us in a situation of having a gigantic World War. When we tolerated Mussolini in invading Ethiopia and said hey you know
what they’re intolerant, they’re terrible, but if we just give them this they’ll
stop and be satisfied, we made a big mistake. Same thing happened with Hitler
annexing various places around Germany. We said ok he’ll be happy once he has
this and he’ll stop. This was known as appeasement. You can also call it being
tolerant of intolerance. Another way to look at it is that a tolerant society has a right to self-defense. If the tolerant societies had not fought
back against intolerance they would have been wiped from the face
of the earth. There would be no tolerant society. So if you want there to be tolerance it must defend itself and sometimes that means being
intolerant. All right so hopefully that’s enough on what can be a tricky issue
but that’s the paradox of tolerance. So let’s look at what it looks
like in practice and I’ll start here with the European Convention on Human
Rights which says everyone has the right to freedom of expression. The exercise of
these freedoms may be subject to such formalities conditions restrictions or
penalties as are prescribed by law and are necessary in a Democratic Society
for the protection of the reputation or rights of others and in particular the
protection of the rights of others is important here. So an example of this is
laws against denying the Holocaust. The idea here is that the Holocaust was such
a terrible event that we can’t risk debating the
facts of how it happened because if – or whether it was important – because if
we lose that information, if we lose that knowledge, we could repeat it. There are laws against denying the Holocaust in around seventeen countries.
Most of them are located in Europe or were heavily affected by World War two –
that would be Russia – with the exception of the country that was created in part
by the Holocaust which is Israel. There’s some sort of experience with
this with the Holocaust that led people to say we need to restrict speech in
this particular way. So here’s a more down-to-earth everyday example and also it’s not a Valerie Aurora talk unless I have a tweet from Jenn
Schiffer in it. So Jenn Schiffer is a software engineer artist and humorist
and she tweets: a very easy and fun activity is not letting people use your software for racist garbage. This is a very important
thing to bring up when talking to future current employers: what’s your stance on
Nazis using your product. Right so just as an individual software developer you
can say, hey you know I’m not going to support people who are white
supremacists. That’s just my choice. So let’s talk about tech companies and
freedom of speech and sort of this – what was the norm maybe a few years ago
and which is becoming less and less the norm. So historically a lot of people
would approach the the topic of what content to host as just: support all
speech as long as it’s not actually literally illegal, if it’s not spam, or if
it’s not directly harming the service. A bunch of companies will say that they
have rules against harassment and abuse on their platforms but that’s pretty
indifferently implemented so I won’t put that – I won’t include it as totally
common. One of the issues we suffer from when we talk about free speech and tech
companies and how that affects our decision what to host is something
called legal talisman syndrome. This is coined by the lawyer Kendra Albert and a legal talisman is a legal term of art that’s out of place. It’s
invoked to make or justify substantive decisions that don’t involve formal
legal process. So you’re taking – often times when you use
the phrase free speech you’re bringing all of these connotations of a
vast powerful government which could greatly harm things if it censors
and this huge body of First Amendment law and very serious people arguing in
suits in front of courts and you’re applying it to a private company
making decisions about which clients to support. These are two totally
different situations but through the magic of legal talisman you can then
introduce a great deal of uncertainty and doubt. It’s so common that there is a
meme that’s been created with the concept that free speech is so
meaningless if you say it over and over again free speech free speech you end up
saying freeze peach which is approximately as meaningful as free
speech in the context it’s being used. so this is a great comic from Naoise Dolan.
Let’s see: Now 100 percent nuance free right like okay I can’t get complicated, just
free speech the end. So I recommend looking that up it’s great. So here’s a
freeze peach example. This is Twitter in 2012. Tony Wang says: generally we remain
neutral as to the content because our general counsel and CEO like to say that
we are the free speech wing of the free speech party. To which I say what you’re
not only the government, Twitter is the government like, we are like the
government but you also have parties and you’re the good party and what is he –
what is even happening here. Like free speech doesn’t have meaning in this
context.It’s a great example of free speech or legal talismans. So there are
totally legitimate concerns about free speech that do apply to individual
private companies making decisions about what which content to host. Governments
do pressure companies to censor their their work in ways that harm society.
It’s a thing to worry about a lot of people express a concern like, how do I
know I’m banning something because it’s bad or if I’m banning it because I just
personally am prejudiced against them because of the way I was I was raised.
Are you going to have to reverse your decision and look foolish?
Will users leave the service because we made a bad decision about what to host?
Are we are we actually harming society is a big question that people ask. So
there’s plenty of legitimate concerns there but taking away this sort
of blanket freeze peach approach is really helpful. So here’s what Karl
Popper has to say and Karl Popper again is the the philosopher who
formulated the paradox of tolerance. He says I do not imply for instance that we
should always suppress the utterance of intolerant philosophies. As long as we
can counter them by rational argument and keep them in check by public opinion
suppression would certainly be unwise. So he’s saying there’s a lot of cases in
which censorship or banning or suppressing speech is not the right
answer but here’s what he goes on to say. But we should claim the right to
suppress them if necessary even by force for it may easily turn out that they are
not prepared to meet us on the level of rational argument but begin by
denouncing all argument; they may forbid their followers to listen to rational
argument because it is deceptive and teach them to answer arguments by the
use of their fists or pistols. This is precisely what’s happening today. What do
you think fake news is, right? This is a form of intolerance and hate which is impervious to argument, to speech, to the usual methods
of fighting bad ideas, and instead they’re coming out and saying you know
what we’ll just load up our guns and we’ll light our torches and we’ll
get our way by violence. So this is the situation that we’re talking about here,
when we’re talking about the paradox of tolerance. So even the US Bill of Rights
has something to say and I love this quote from Waldo Jaquith
who is the former ACLU Virginia Board member who resigned as a result of the ACLU’s decision to defend the white supremacists marching in
Charlottesville. He says the First Amendment guarantees the right of the
people peaceably to assemble. Peaceably. When it became clear that the organizers’
goal was violence that left me with no qualms about their lack of First
Amendment rights in this matter. One more point I want to make here is
that the level of tolerance changes with context. So there’s a great
thing from Susan Benesch the director of the dangerous speech
project she says even in the US where we have the most speech protective law some
acts of speech are illegal. Nobody has suggested that to fulfill
freedom of expression every act of speech has to be allowed. It doesn’t mean
you can post absolutely anything. Everyone is figuring out how to draw the
lines. And I like this this idea that like we are – it is an ongoing process,
there’s no hard and fast rules, and it changes depending on what’s going on. So
I would like to talk about tech companies that are actually using
the paradox of tolerance. I made a spreadsheet because that’s what
you do. If you go to bitly slash intolerable speech you’ll find the link
for that. I’ve currently got 37 entries for 34 different companies including
heavyweights like Apple Google Twitter Uber even and of course Airbnb, you all.
So let’s go into a few of these. So I’m gonna start with people who took action
before Charlottesville. I think that’s only fair.
October 27th 2016 Airbnb created the community commitment for all hosts and
guests so it goes like this: I agree to treat everyone in the Airbnb community
regardless of their race religion national origin ethnicity disability sex
gender identity sexual orientation or age with respect and without judgment or
bias. Which came in handy on August 8th – still before Charlottesville – when Airbnb
revoked the bookings and accounts for dozens of people attending the unite the
right rally in Charlottesville. Here’s the company’s statement: when through our
background check processes or from input of our community we identify
and determine that there are those who would be pursuing behavior on the
platform that would be antithetical to the Airbnb community commitment we seek
to take appropriate action including as in this case removing them from the
platform. So this is a great you know set up and then spike. Spotify took action
before Charlottesville as well they have for many many years used a database of
objectionable material maintained by the German government to
proactively ban white supremacist material and bands and things like that.
They do other bands on a case-by-case basis and of as of November 2014 they’d
already banned 22 of the 54 white supremacist bands listed by the Southern
Poverty Law Center and they were deleting more
after Charlottesville. They responded to additional requests to delete more
obscure comp white-supremacist content and they said: we are glad to have been
alerted to this content and have already removed many of the bands identified
today whilst urgently reviewing the remainder. Facebook took action
before and after Charlottesville. On August 11th they removed the organizing
page for the unite the right rally which was the day before it and on the
fourteenth afterwards they deleted many more white nationalist groups’ pages and
profiles things like that. But I can’t give them a ton of credit because they
had several warnings before that. In 2016 the Southern Poverty Law Center let them
know about over 200 items, Facebook deleted fewer than 10. May 10th Fast
Company told them about 17 items Facebook deleted two. July 31st The
Guardian told them about 175 items they deleted nine. Right it took
Charlottesville it took frankly a white woman being murdered before they were
willing to take this action. I just want to point out the moral of the story here: if
the Southern Poverty Law Center reaches out to you and says there’s a problem
with the content on your site you should listen to them because they’re the
experts. An example from a company that’s proactively doing this: Bumble is in
online dating websites or dating service similar to Tinder but with a pro-woman
slant. It was co-founded by Whitney Wolfe who experienced being
sexually harassed and discriminated against as a co-founder of Tinder. So as an
explicitly pro-woman service they experienced a lot of attacks motivated
by misogyny, so they were aware bad people are out there and they need to do
something about it and they extended that into hate symbols – being very
proactive about finding and removing profiles with hate symbols, especially
white supremacist hate symbols. So after Charlottesville they formally
partnered with the Anti-Defamation League which maintains an up-to-date
database of the latest symbols so they could stay ahead of the
curve. After Charlottesville, here’s – WordPress which is run by the company
Automattic. Before Charlottesville they were asked by Fast Company why are you
hosting a lot of these white supremacist sites and their answer is we censor
nothing, if it’s legal we put it up. August 15th
they deleted American Vanguard without further explanation. American Vanguard is
the group that James Fields claimed membership, in the alleged murderer in
Charlottesville. So not exactly covering themselves in glory but they did it. I
just want to say something about Terms of Service versus enforcement. So as I
went through and collected this database I noticed that most companies didn’t
change their terms of service, they just changed the way that they enforce them
and this is totally fine it’s normal to have a public-facing document and then a
much more detailed much more quickly changing set of internal guidelines.
It’s important is to keep those up to date and actually put in the effort to
enforce them. So you can check out the various forms that terms of services
used to ban this kind of hate speech and hateful content and I just want to note
that while Amazon has an extremely explicit ban that covers white
supremacist content, they are taking no action before or after Charlottesville
to remove white supremacist content. So I want to talk about a few examples that
weren’t based on the paradox of tolerance. CloudFlare on August 16th. It’s
a company that provides protection against denial of service attacks. The
daily stormer which was a white supremacist website had an account with
them. They terminated it on August 16th and the leaked internal
email said literally, I woke up in a bad mood and decided someone shouldn’t be
allowed on the internet. No one should have that power. And I do
sympathize with this and agree you shouldn’t be making decisions about who
is allowed to be on the Internet based on being in a bad mood and I have a
proposal for making that decision based on other things.
Here’s another non paradox of tolerance example Dreamhost. Dreamhost’s official
policy is we will host any website as long as its content is legal in the
United States of America. On August 24th after the Daily Stormer had lost its
denial of service protection, Dreamhost terminated Daily Stormer’s account
for basically being the target of a denial of service attack. I don’t think
that’s a great way to make this kind of decision is: it’s unpopular and people
are doing denial of services because plenty of positive things get denial
of service attacks. All right so I
just wanted to give you another freeze peach reminder. I actually was given a
little freeze peach model and I’ll pass it around the audience. My friend
Gretchen Koch made it and if you look in the slides you can find out more
about it. Freeze Peach, think of it next time
somebody just says that with no context. All right so here’s my proposed rule and
I’m calling it the intolerable speech rule. It’s probably
going to undergo some more revisions but here’s the basics and I’ll go over each
element. I suggest that you ban people from using your products if they are
advocating for the removal of human rights from people based on an aspect of
their identity in the context of systemic oppression primarily harming
that group in a way that overall increases the danger to that group. So
we’ll go over each one and then we’ll do a few examples.
so the advocating for the removal of human rights this covers a lot of things
it includes like the right to not just live but vote travel get medical care. This includes speech. There’s a bunch of things in this in this realm. I
count dehumanizing or treating a group as inferior categorically as meeting
this criteria because humans don’t need human rights right you have to be human first
many tech companies Terms of Service already forbid – and they usually use the
words promoting advocating or encouraging violence or hate of this
form so that’s relatively uncontroversial – from
people based on an aspect of their identity so it has to be based or rooted
in a judgment on a person based on a part of their identity that’s something
that’s difficult to change so it’s something like race or gender or
sexuality or disability religion is interesting because parts of it you do
not need to respect the intolerant parts this is part of the paradox of tolerance
just to be super clear it’s not part of your identity – your
intolerant or bigoted opinions are not part of your identity and it’s okay to
treat people differently based on that in general it’s it’s according to this
rule it’s fine to advocate removing rights from people based on things they
actually did or they freely chose so we feel as a society it’s reasonable that
someone convicted of murder may lose their ability to travel as they are in
prison that sort of thing there’s limits there’s details I don’t take that too
far but that’s just explaining that so three in the context of systemic
oppression primarily harming that group this is really like not
somebody says so on the internet but there – people have done studies and
they’ve run tests and they have concluded there is a systemic oppression
so as an example systemic sexism hurts everyone it’s true but it hurts women
and non-binary people more than it hurts men so you can’t get away with
like patriarchy hurts men too so don’t say anything mean about men and I’ll
give some more examples later on there’s plenty of reliable research out there
people are often saying like well who is systemically oppressed I’m like it’s not
actually rocket science the difficulty comes if you believe that you
have to convince the person you’re banning that they aren’t part of an
oppressed group you don’t you only have to convince yourself so do that all
right that overall increases the danger to that group so as Karl Popper said a
lot of speech is better off if you let it go out there and then you just treat
it the normal way with like other people responding to that speech right
criticism and all that and there’s some some hate speech that makes the target
group safer and here’s some examples quoting hate speech in order to denounce
it providing material on hate speech for those studying hate groups
publishing private communications of hate groups in order to embarrass them
or ruin their or their organization right so there’s there’s a way to say
like hey we can allow this content because it’s making the world better so
I’m just gonna give you a few examples so say a famous black person tweeted
maybe white people shouldn’t vote for 10 years just take a break I argue
according to this rule that you should allow that kind of speech first it
doesn’t meet number three there’s no context there’s – white people’s votes are
safe that’s not going to change there’s no danger of that and in reality this is
a joke meant to raise the awareness of how normalized it is that there are
people sitting in rooms right now trying to figure out how to suppress the vote
of people of color powerful people who are succeeding and doing it right
sothis is the kind of speech you would allow on your platform here’s
another example that doesn’t meet this rule Wall Street Journal op-ed
suggesting that Democrats should not be allowed to vote so being a Democrat is
not part of your identity so it doesn’t meet number two you because you can
choose your political opinions but you would might want to suppress the speech
for other reasons like believing that universal suffrage is really important
or wanting to protect your newspapers’ reputation right so this isn’t the only
rule it’s just one of many rules here’s another case that doesn’t meet the test
the Fast Company reporter in an interview with the CloudFlare CEO said if I
were to write on my blog I think CloudFlare is homophobic and that’s good
because I hate gays that puts you in the same position that you were with the
Daily Stormer right so equating like this little tiny random blog with a
powerful hate site for white supremacy I would say it doesn’t meet
number four this is the kind of thing that you can have a normal
response of other people criticizing your little tiny blog maybe maybe not
but I would say not all right and here’s one more example people posting the
Daily Stormer article attacking Heather Heyer on facebook with comments
denouncing it I would say number four it doesn’t increase the overall danger to
the group and actually it helps because a whole bunch of people were expressing
disgust Facebook also made this exception so this is not the only test
it’s one of many you should keep using all your other tests but it covers a
really bad loophole that a lot of companies have alright just another free
speech this one by my friend Stephanie Szvan I have on the little subtitle saying not
sure what it is but it’s mine you know this vague idea like vague I have
something that makes whatever I’m doing okay so I like that all right so we’re
close to the end here implementing the paradox of tolerance here’s some here’s
the basics of it update your Terms of Service if necessary you’d be surprised
how often that isn’t necessary there’s a lot of language out there already
continuously update your enforcement guidelines including proactively
seeking out evolving threats so not just waiting until you notice or somebody
else points it out to you a really important part of this is having a
diverse and empowered team that’s not only implementing the terms of service
but also creating the guidelines the enforcement guidelines and the Terms of
Service so there needs to be power and feedback going on there and you should
listen to and partner with experts there are people who founded whole nonprofits
to study these kinds of things they’re experts they want to help you
they’re often willing to do it for free so do that so I just want to go into a
few details around this legal issues people sometimes are worried about as
always ask your lawyer if you have a real legal concern I just want to point
out that in many cases the law is actually supportive of implementing the
paradox of Tolerance. Examples there’s anti-discrimination laws for how
you provide lodging in the United States sure helps if you don’t have a
bunch of white supremacist hate speech in your users’ profiles right I’m sure
that will help that anti harassment or anti-stalking laws if you’re not helping
people do those things and non-US laws tend far more towards restricting speech
in fact Twitter has like a special Holocaust denial mode that in countries
where that law is active it just won’t show certain Twitter accounts why can’t
you do that for all the countries oh I don’t know okay so structural issues
there’s a good question here this is a question the CloudFlare CEO is
asking why are tech companies making these decisions? it’s a good question
it’s not a great situation that we’re in that tech companies have this kind of
power he was saying and many others would rather that there were clear legal
limits and they could just implement those and they would be fine many
countries do have better laws including most of the EU but the laws will always
lag or be out of step and we’ll have to be willing to take action ourselves so
we need to stay ahead of that and still do what we can what where we can all
right so advice on advocating for change at your company so Liz Fong-Jones has a
really great summary of how she led several campaigns to change company
policy at Google the gigantic very important company you can find that on
my blog you can also hire me or I would suggest why Y-Vonne Hutchinson at ReadySet to help you with implementing these sort of things
here’s some arguments you can use depending on what works for the person
you’re talking to I don’t necessarily agree with all of these or think they
are good arguments they’re just arguments that sometimes work so hosting
hate groups is bad for the brand that was one of the things that the
CloudFlare CEO talked about. He was like we don’t want to be the denial of service
protection company for white supremacists like that’s not it no that won’t help us
hate groups are just bad clients. They are unethical they are cheap they are mean
they’re just jerks nobody likes working with them it’s just
miserable fascism overall is bad for business as it mobilizes the entire
economic output of a nation for the purpose of going to war so I suggest
that you help prevent it it’s harder to recruit and retain employees if you host
a bunch of hateful groups on your service people just don’t want to do
that they don’t want to work for a company that does it and that’s not how
they want to spend their days it reduces your legal exposure because frankly hate
groups do illegal things they want to kill people and do many other bad things
so it’s helpful personal arguments you can stop feeling bad about your job and
sleep better at night you like living in a free society you
care about fairness and justice your friends might start avoiding you if you
work for a company that supports hate groups and this is totally a real
concern yeah random arguments I don’t these are terrible but if they work use
it everyone else is doing it and now you can give them a spreadsheet at least 34
other tech companies with big names Karl Popper and a bunch of other old white
male philosophers say that this is the right thing to do if you think they think
that’s a good argument go for it and World War II is what happens when people
are tolerant of intolerance we could skip that this time that would be great
all right so here’s the call to action please tell other people about the
paradox of tolerance the wiki page is great there’s other options I keep
thinking about making business cards with the paradox of tolerance just printed
it on them organize other tech workers at your company to adopt the Intolerable
speech rule at your company ask your legislators to pass laws that take the
paradox of tolerance into account let me tell you though the US
anti-discrimination laws are on terrible footing it would be really nice to
improve this and finally donate to the Southern Poverty Law Center that’s
something you can do right now they’re doing great work they’re amazing
and you should support it all right that’s it thanks I appreciate it

, , , ,

Post navigation

Leave a Reply

Your email address will not be published. Required fields are marked *