Capture values submitted by Adaptive Card in water

2020-05-09 23:43发布

问题:

I have followed the advice of this question, the comment on this issue and also this answer.

Inside my Waterfall Dialog:

  • Displaying the adaptive card
  • Sending a text prompt right after displaying an adaptive card

Inside my main bot class:

  • Setting the Text property of the Activity to that of the extracted value from the Value property of the activity if the activity is a message that contains postback data.

I have tried using a AdaptiveTextInput or AdaptiveDateInput as the control to submit the value but it doesn't make a difference. I feel like it's going to be something really silly..

I am currently using a mix of Hero and Adaptive cards in my waterfall, the Hero cards are working as they should.


Edit

I have added // !Relevant- comments to my code to the important parts, the rest is left for context.

So my question is: what is preventing my submit on the adaptive card from flowing through correctly - is it a problem in how I am displaying in the waterfall, a problem with how the action is constructed in the card, or how I am handling the action in the main bot class?


Building my cards in AdaptiveCardService:

public List<Activity> BuildCardActivitiesFromDecisionFlow(BotDecisionFlow botDecisionFlow)
{
    List<Activity> cardActivities = new List<Activity>();

    foreach (Step step in botDecisionFlow.FormSchema.Steps)
    {
        Control control = step.Details.Control;

        cardActivities.Add(CreateCardActivity(step, control));
    }

    return cardActivities;
}

private Activity CreateCardActivity(Step step, Control control)
{
    Activity cardActivity = (Activity)Activity.CreateMessageActivity();

    if (control.Type == ControlTypeEnum.RadioButton)
    {
        HeroCard heroCard = BuildHeroCard(step, control.DataType);
        Attachment attachment = heroCard.ToAttachment();

        cardActivity.Attachments.Add(attachment);
    }
    else if (control.Type == ControlTypeEnum.DatePicker)
    {
        AdaptiveCard adaptiveCard = BuildAdaptiveCard(step, control.DataType);

        Attachment attachment = new Attachment
        {
            ContentType = AdaptiveCard.ContentType,
            // Trick to get Adapative Cards to work with prompts as per https://github.com/Microsoft/botbuilder-dotnet/issues/614#issuecomment-443549810
            Content = JsonConvert.DeserializeObject(JsonConvert.SerializeObject(adaptiveCard))
        };

        cardActivity.Attachments.Add(attachment);
    }
    else
    {
        throw new NotImplementedException($"The {nameof(control)} with value {control} is not yet supported.");
    }

    return cardActivity;
}

private HeroCard BuildHeroCard(Step step, DataTypeEnum dataType)
{
    string question = step.Details.Question;

    HeroCard heroCard = new HeroCard
    {
        Text = question,
        // PostBack is required to get buttons to work with prompts, also the value needs to be a string for the
        // event to fire properly, as per https://stackoverflow.com/a/56297792/5209435
        Buttons = step.Details.EnumValueToDisplayTextMappings.Select(e => new CardAction(ActionTypes.PostBack, e.Value, null, e.Value, e.Value, JsonConvert.SerializeObject(new DialogValueDto(step.Name, e.Key, dataType)), null)).ToList()
    };

    return heroCard;
}

private AdaptiveCard BuildAdaptiveCard(Step step, DataTypeEnum dataType)
{
    const string ISO8601Format = "yyyy-MM-dd";
    string question = step.Details.Question;

    DateTime today = DateTime.Today;
    string todayAsIso = today.ToString(ISO8601Format);

    AdaptiveCard adaptiveCard = new AdaptiveCard("1.0")
    {
        Body =
        {
            new AdaptiveContainer
            {
                Items =
                {
                    new AdaptiveTextBlock
                    {
                        Text = question,
                        Wrap = true
                    },
                    new AdaptiveDateInput
                    {
                        Id = "UserInput",
                        Value = todayAsIso,
                        Min = today.AddDays(-7).ToString(ISO8601Format),
                        Max = todayAsIso,
                        Placeholder = todayAsIso
                    }
                }
            }
        },
        Actions = new List<AdaptiveAction>
        {
            // !Relevant-Start
            new AdaptiveSubmitAction
            {
                Data = new DialogValueDto(step.Name, dataType),
                Title = "Confirm",
                Type = "Action.Submit"
            }
            // !Relevant-End
        }
    };

    return adaptiveCard;
}

Inside my waterfall class:

private readonly IUmbracoApiWrapper _umbracoApiWrapper;
    private readonly IUmbracoResponseConverterService _umbracoResponseConverterService;
    private readonly IAdaptiveCardService _adaptiveCardService;

    private IStatePropertyAccessor<DynamicWaterfallState> _accessor;
    private DynamicWaterfallState _state;

    public DynamicWaterfallDialog(
        IUmbracoApiWrapper umbracoApiWrapper,
        IUmbracoResponseConverterService umbracoResponseConverterService,
        IAdaptiveCardService adaptiveCardService,
        UserState userState)
        : base(nameof(DynamicWaterfallDialog))
    {
        _accessor = userState.CreateProperty<DynamicWaterfallState>(nameof(DynamicWaterfallState));
        _umbracoApiWrapper = umbracoApiWrapper;
        _umbracoResponseConverterService = umbracoResponseConverterService;
        _adaptiveCardService = adaptiveCardService;

        InitialDialogId = nameof(WaterfallDialog);

        // !Relevant-Start
        var waterfallSteps = new WaterfallStep[]
        {
            // TODO: Rename this DisplayCardAsync
            UserInputStepAsync,
            // TODO: Rename this ProcessCardAsync
            LoopStepAsync,
        };

        AddDialog(new TextPrompt(nameof(TextPrompt)));
        AddDialog(new WaterfallDialog(InitialDialogId, waterfallSteps));
        // !Relevant-End
    }

    // TODO: Does it make more sense for the collection of dialogs to be passed in? It depends on how this dialog is going to be called, 
    // maybe just passing in the ID is fine rather than having code sprinkled around to fetch the dialog collections.
    public async Task<DialogTurnResult> UserInputStepAsync(WaterfallStepContext sc, CancellationToken cancellationToken)
    {
        // Get passed in options, need to serialise the object before we deserialise because calling .ToString on the object is unreliable
        string tempData = JsonConvert.SerializeObject(sc.Options);
        DynamicWaterfallDialogDto dynamicWaterfallDialogDto = JsonConvert.DeserializeObject<DynamicWaterfallDialogDto>(tempData);

        // Read out data from the state
        _state = await _accessor.GetAsync(sc.Context, () => new DynamicWaterfallState());

        List<Activity> activityCards = _state.ActivityDialogs ?? new List<Activity>();
        int dialogPosition = _state.DialogPosition;
        bool flowFinished = _state.FlowFinished;
        bool apiDataFetched = _state.ApiDataFetched;

        if (DynamicWaterfallDialogDtoExtensions.IsDynamicWaterfallDialogDtoValid(dynamicWaterfallDialogDto) && !apiDataFetched)
        {
            // Fetch from API
            JObject decision = await _umbracoApiWrapper.GetDecisionById(18350);

            UmbracoDecisionResponseDto umbracoResponseDto = JsonConvert.DeserializeObject<UmbracoDecisionResponseDto>(decision.ToString());

            BotDecisionFlow botDecisionFlow = new BotDecisionFlow(_umbracoResponseConverterService, umbracoResponseDto);

            activityCards = _adaptiveCardService.BuildCardActivitiesFromDecisionFlow(botDecisionFlow);

            _state.ApiDataFetched = true;
            _state.ActivityDialogs = activityCards;

            await _accessor.SetAsync(sc.Context, _state, cancellationToken);
        }

        var cardToShow = activityCards.ElementAt(dialogPosition);

        _state.FlowFinished = _state.DialogPosition == activityCards.Count - 1;
        _state.DialogPosition++;

        await _accessor.SetAsync(sc.Context, _state, cancellationToken);

        // TODO we need to determine the control type to figure out the prompt type?

        // !Relevant-Start
        await sc.Context.SendActivityAsync(cardToShow);
        return await sc.PromptAsync(nameof(TextPrompt), new PromptOptions() { Prompt = MessageFactory.Text("") });
        //return await sc.PromptAsync(nameof(TextPrompt), new PromptOptions { Prompt = new Activity { Type = ActivityTypes.Message } });
        // !Relevant-End
    }

    public async Task<DialogTurnResult> LoopStepAsync(WaterfallStepContext sc, CancellationToken cancellationToken)
    {
        object result = sc.Result;
        DialogValueDto userInput = JsonConvert.DeserializeObject<DialogValueDto>(sc.Result.ToString());

        await sc.Context.SendActivityAsync($"You selected: {userInput.UserInput}");

        _state = await _accessor.GetAsync(sc.Context, () => new DynamicWaterfallState());

        bool flowFinished = _state.FlowFinished;

        // TODO: Do we want to do state manipulation in here?

        if (!flowFinished)
        {
            // TODO: Do we want to pass in custom options here?
            return await sc.ReplaceDialogAsync(nameof(DynamicWaterfallDialog), sc.Options, cancellationToken);
        }
        else
        {
            // TODO: We probably want to pass the state in here instead of null if we want to show outcomes etc
            return await sc.EndDialogAsync(null, cancellationToken);
        }
    }
}

Inside my main bot class:

public override async Task OnTurnAsync(ITurnContext turnContext, CancellationToken cancellationToken)
{
    // Client notifying this bot took to long to respond (timed out)
    if (turnContext.Activity.Code == EndOfConversationCodes.BotTimedOut)
    {
        _telemetryClient.TrackTrace($"Timeout in {turnContext.Activity.ChannelId} channel: Bot took too long to respond.", Severity.Information, null);
        return;
    }

    var dc = await _dialogs.CreateContextAsync(turnContext);

    // !Relevant-Start
    if (turnContext.Activity.Type == ActivityTypes.Message)
    {
        // Ensure that message is a postBack (like a submission from Adaptive Cards)
        if (dc.Context.Activity.GetType().GetProperty("ChannelData") != null)
        {
            var channelData = JObject.Parse(dc.Context.Activity.ChannelData.ToString());

            // TODO: Add check for type, we should only handle adaptive cards here
            if (channelData.ContainsKey("postBack"))
            {
                var postbackActivity = dc.Context.Activity;

                string text = JsonConvert.DeserializeObject<DialogValueDto>(postbackActivity.Value.ToString())?.UserInput;


                // Convert the user's Adaptive Card input into the input of a Text Prompt
                // Must be sent as a string
                postbackActivity.Text = text;
                await dc.Context.SendActivityAsync(postbackActivity);
            }
        }
    }
    // !Relevant-End

    if (dc.ActiveDialog != null)
    {
        var result = await dc.ContinueDialogAsync();
    }
    else
    {
        await dc.BeginDialogAsync(typeof(T).Name);
    }
}

My DialogValueDto incase you need it:

public string StepName { get; set; }
public string UserInput { get; set; }
public DataTypeEnum DataType { get; set; }

/// <summary>
/// For JSON deserialization
/// </summary>
public DialogValueDto()
{
}

/// <summary>
/// For use with DateTime deserialization.
/// The control id is set to "UserInput"
/// so this property will be set automatically
/// </summary>
public DialogValueDto(string stepName, DataTypeEnum dataType)
{
    StepName = stepName;
    DataType = dataType;
}

/// <summary>
/// This is the constructor that should be used most
/// of the time
/// </summary>
public DialogValueDto(string stepName, string userInput, DataTypeEnum dataType)
{
    StepName = stepName;
    UserInput = userInput;
    DataType = dataType;
}

Interestingly enough my OnEventAsync function of my MainDialog (the one which is wired up in Startup.cs via services.AddTransient<IBot, DialogBot<MainDialog>>();) gets fired when I set the text property of the activity.

回答1:

My issue turned out to be two-fold


1) Inside my OnTurnAsync method in my DialogBot file I had:

var postbackActivity = dc.Context.Activity;
string text = JsonConvert.DeserializeObject<DialogValueDto>(postbackActivity.Value.ToString())?.UserInput;

postbackActivity.Text = text;
await dc.Context.SendActivityAsync(postbackActivity);

I was setting the Text property of postBackActivity variable instead of directly setting the Text property directly on dc.Context.Activity. Because I was sending the the variable through SendActivityAsync it was covering up this mistake because I was getting the value I wanted passed through to the OnEventAsync method in my MainDialog class.

The correct way was to set this directly on the context, not on a copy of it (DOH!)

dc.Context.Activity.Text = text

2) Inside the OnEventAsync method in my MainDialog class I had an empty block which caught the response but did nothing with it (it needed to call await dc.ContinueDialogAsync()). However, this was already handled by an existing block of code in the Virtual Assistant template which my empty block was preventing from being hit.

object value = dc.Context.Activity.Value;

if (condition)
{
    // do nothing
}
else if (value.GetType() == typeof(JObject))
{
    // code from the Virtual Assistant template to check the values passed through
    var submit = JObject.Parse(value.ToString());

    // more template code

    // Template code
    if (forward)
    {
        var result = await dc.ContinueDialogAsync();

        if (result.Status == DialogTurnStatus.Complete)
        {
            await CompleteAsync(dc);
        }
    }
}

Once I removed my empty if block then it fell through to the code it needed (the forward part).


Change list:

DynamicWaterfallDialog:

public DynamicWaterfallDialog(
    ...
    )
    : base(nameof(DynamicWaterfallDialog))
{
    ...

    InitialDialogId = nameof(WaterfallDialog);

    var waterfallSteps = new WaterfallStep[]
    {
        UserInputStepAsync,
        LoopStepAsync,
    };

    AddDialog(new TextPrompt(nameof(TextPrompt)));
    AddDialog(new WaterfallDialog(InitialDialogId, waterfallSteps));
}

DialogBot:

public override async Task OnTurnAsync(ITurnContext turnContext, CancellationToken cancellationToken)
{
    ...

    var dc = await _dialogs.CreateContextAsync(turnContext);

    if (dc.Context.Activity.Type == ActivityTypes.Message)
    {
        // Ensure that message is a postBack (like a submission from Adaptive Cards)
        if (dc.Context.Activity.GetType().GetProperty("ChannelData") != null)
        {
            JObject channelData = JObject.Parse(dc.Context.Activity.ChannelData.ToString());
            Activity postbackActivity = dc.Context.Activity;

            if (channelData.ContainsKey("postBack") && postbackActivity.Value != null)
            {
                DialogValueDto dialogValueDto = JsonConvert.DeserializeObject<DialogValueDto>(postbackActivity.Value.ToString());

                // Only set the text property for adaptive cards because the value we want, and the value that the user submits comes through the
                // on the Value property for adaptive cards, instead of the text property like everything else
                if (DialogValueDtoExtensions.IsValidDialogValueDto(dialogValueDto) && dialogValueDto.CardType == CardTypeEnum.Adaptive)
                {
                    // Convert the user's Adaptive Card input into the input of a Text Prompt, must be sent as a string
                    dc.Context.Activity.Text = JsonConvert.SerializeObject(dialogValueDto);

                    // We don't need to post the text as per https://stackoverflow.com/a/56010355/5209435 because this is automatically handled inside the
                    // OnEventAsync method of MainDialog.cs
                }
            }
        }
    }

    if (dc.ActiveDialog != null)
    {
        var result = await dc.ContinueDialogAsync();
    }
    else
    {
        await dc.BeginDialogAsync(typeof(T).Name);
    }
}

MainDialog:

protected override async Task OnEventAsync(DialogContext dc, CancellationToken cancellationToken = default(CancellationToken))
{
    object value = dc.Context.Activity.Value;

    if (value.GetType() == typeof(JObject))
    {
        var submit = JObject.Parse(value.ToString());
        if (value != null)
        {
            // Null propagation here is to handle things like dynamic adaptive cards that submit objects
            string action = submit["action"]?.ToString();

            ...
        }

        var forward = true;
        var ev = dc.Context.Activity.AsEventActivity();

        // Null propagation here is to handle things like dynamic adaptive cards that may not convert into an EventActivity
        if (!string.IsNullOrWhiteSpace(ev?.Name))
        {
            ...
        }

        if (forward)
        {
            var result = await dc.ContinueDialogAsync();

            if (result.Status == DialogTurnStatus.Complete)
            {
                await CompleteAsync(dc);
            }
        }
    }
}

I guess I was expecting having the Text property set on the context to automatically fire through to my LoopStepAsync (DynamicWaterfallDialog) handler rather than falling into OnEventAsync (MainDialog). I knew I needed to call ContinueDialogAsync somewhere and should have been more suspicious of the final paragraph of my question:

Interestingly enough my OnEventAsync function of my MainDialog (the one which is wired up in Startup.cs via services.AddTransient>();) gets fired when I set the text property of the activity.

So close, yet so far. Hopefully this helps someone else out in the future.

Link that I found helpful were:

  • ComplexDialogBot.cs.
  • Question about adaptive cards and waterfalls.
  • GitHub issue about Adaptive Cards and prompts.