How to Cache Real-time Data?

Question

• I'm working on a windows forms application (.NET 4.0).
• My form contains a 'Fast Line' chart using the Microsoft chart control included in VS2010.
• The chart gets filled with about 20,000 datapoints.
• My application then starts receiving market data from a server via DDE (Dynamic Data Exchange) in real-time and adds it the chart.

Note: I have no control over the server and so I have to deal with DDE only even though it's an outdated technology. VS doesn't support DDE anymore and so I use the Ndde library which works like a charm.

First we connect to the server, create an advise loop, and then subscribe to the OnAdvise event to receive notifications of new data:

Dim client As DdeClient = New DdeClient("ServerApplication", "Bid")

Private Sub StartDDE()
    client.Connect()
    client.StartAdvise("EURUSD", 1, True, 60000)
    AddHandler client.Advise, AddressOf OnAdvise
End Sub

Now we can put the commands to update the chart inside the event:

Private Sub OnAdvise(ByVal sender As Object, ByVal args As DdeAdviseEventArgs)
    Dim myPrice As Double = args.Text
    Chart1.Series("Bid").Points.AddY(myPrice)
End Sub

You get the idea.

THE PROBLEM:

This works fine for a few seconds until the chart crashes throwing the exception: "Collection was modified; enumeration operation may not execute."

I spent a lot of time researching what may be the cause of this in my particular case, and I've come to the conclusion that it's because the chart is receiving data quicker than it can handle. It's already loaded with a lot of data and needs a certain time (less than a second) to add the received data in a new DataPoint and invalidate (refresh) itself. Whereas the server often sends data values very quickly (for example 5ms in between). So I've tried the following:

System.Threading.Thread.Sleep(800)
Chart1.Series("Bid").Points.AddY(myPrice)

thus pausing the application to give time to the chart to finish its work before adding a new point, and guess what? The application now works for minutes before throwing the exception. (altering the value in Sleep() doesn't help any further however)

The only help I could find online is an old post of someone mentioning that you should put incoming data on a cache queue, with one new data value released from the cash at a time (every time the chart finishes working).

My question is how would you do this?

Other suggestions are welcome!

Answer 1

This is most likely an issue caused by attempting to modify a UI element from a thread other than the UI thread.

The way you have it coded now the DdeClient.Advise event handler is being executed on a worker thread managed by the library. See, DDE sucks and because it sucks it has these requirements that it has to run on a thread with a message pump.¹ To make the library compatible with other types of applications besides windows forms I coded it in such a manner that it will create a dedicated thread with a message loop and marshal all of the operations onto that thread by default.

But, you can override this behavior by specifying an ISynchronizeInvoke instance manually in the DdeClient constructor. The library will then use whatever thread is hosting the ISynchronizeInvoke instance for all of its DDE operations. All Form and Control instances implement ISynchronizeInvoke so it is easy enough to tell the library to use the main UI thread.

Dim client As DdeClient = New DdeClient("ServerApplication", "Bid", yourForm)

If you tell the library to use your Form instance then the Advise event handlers will be executed on the same thread hosting that Form; the UI thread.

By the way, I realize that you have no control over the server, but I would at least begin talking with the vendor of the software to use more modern (not 20 years old) mechanisms for doing interprocess communications.

---

¹It also has the unfortunate requirement of thread affinity which made dealing with the garbage collector a real pain.

Answer 2

Get real ;) DDE is slow, graphics is slow. Do not do them in the same thread.

Try that:

• Create a second thread that handles DDE, queues the items.
• The chart thread then pulls the updates and draws them.

Now, here comes the point:

• ONLY the ui thread is allowed to modify the chart control. Yes, sucks. No, not negotiable. - old UI rule since the dawn of time.
• Threads needs locking ;)