Scrolling RecyclerView freezes on long TextViews

I am trying to implement a comment section-style layout using a RecyclerView. I have a list of CharSequence objects (each the result of Html.fromHtml(String, null, null)) which I use to populate this RecyclerView.

Here is the RecyclerView layout:

<android.support.v7.widget.RecyclerView
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/list"
    android:layout_width="match_parent"
    android:layout_height="match_parent"/>

Here is the TextView layout:

<TextView
    android:id="@+id/text"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_marginTop="16dp"
    android:lineSpacingMultiplier="1.4"
    android:textAppearance="@style/TextAppearance.AppCompat.Body1"
    android:textIsSelectable="true"
    android:textSize="15sp"/>

Here is the ViewHolder:

public final class ItemHolder extends RecyclerView.ViewHolder {
    @Bind(R.id.text) TextView text; // ButterKnife

    public ItemHolder(View itemView) {
        super(itemView);
        ButterKnife.bind(this, itemView);
        text.setMovementMethod(new LinkMovementMethod());
    }

    public void bind(CharSequence cs) {
        text.setText(cs);
    }

The rest is pretty standard. The RecyclerView is given a LinearLayoutManager and an Adapter that reads from the list and creates ViewHolder instances. I did not call setHasFixedSize(true) because I want to be able to swap the contents of the adapter and the number of items might not always be the same.

The setup works for most input, but the problem is that some of the CharSequences are over 100 lines long, and when scrolling, the RecyclerView pauses for about half a second just before it reaches them and then continues scrolling. It doesn't just happen once either. When I scroll back up to the item, it freezes again, and then everytime I scroll the item out of view and scroll back down to it, the view freezes again. How do I fix this?