What does “Challenge” term stand for?

ControllerBase class has Challenge method, that returns an object of the ChallengeResult class. CookieAuthenticationOptions class has AutomaticChallenge property.

I believe ChallengeResult has something to do with external logins. But how does it actually work? Where does the term "Challenge" come from? What does lay inside this.

A ChallengeResult is an ActionResult that when executed, challenges the given authentication schemes' handler. Or if none is specified, the default challenge scheme's handler. Source code for ChallengeResult

So for example, you can do:

return Challenge(JwtBearerDefaults.AuthenticationScheme); //Can specify multiple schemes + parameters

This will challenge the JWT Bearer authentication handler. In this handler's case, it sets the response status code to 401 to tell the caller they need authentication to do that action.

AutomaticChallenge (in ASP.NET Core 1.x) is the setting that says this is the default challenge handler. It means it will be called if no authentication scheme is specifically named.

In 2.x, this was changed such that you now specify the default challenge scheme or the higher-level default scheme.

services.AddAuthentication(o =>
{
    o.DefaultScheme = JwtBearerDefaults.AuthenticationScheme; //Default for everything
    // o.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme; //Default specifically for challenges
})

A challenge is basically a way of saying "I don't know who this user is, please verify their identity". So if the authentication handler triggered is e.g. the Facebook authentication handler, it will react to the challenge by issuing a redirect to the Facebook authentication page. A local account authentication handler might issue a redirect to the local sign-in page.

In the case of JWT Bearer authentication, the handler cannot do anything other than respond with a 401 status code and leave it up to the caller to authenticate themselves properly.

You can see this in action in OAuthHandler (HandleChallengeAsync), which Facebook auth uses (and Microsoft and Google authentication).

You typically return a Challenge when you don't know who the user is, and a Forbid if you know who they are, but they are not allowed to do the action they tried to do.