How to allocate a memory to send a large pcap file

Question

I have used the code from winpcap example to send pcap file(original code from winpcap documenation found at this link)

It works fine to send a small pcap files but if I tried to send a large pcap file (larger than available memory size say 2 Gb) it will fail for sure. This code is used to allocate size of the file in memory in order to send it later

caplen= ftell(capfile)- sizeof(struct pcap_file_header);
...
/* Allocate a send queue */
squeue = pcap_sendqueue_alloc(caplen);

The question is how to make this work for large files(in Gb or larger than maximum memory space available for allocation)? Should I allocate only 100 Mb for example and send the queue and then take the next 100Mb? If yes what is the proper buffer size? And how to do this?

an important issue here is the performance of sending this file which needed to be done as fast as can (i am sending a video packets).

In short how to manage the memory here to achieve this goal?

Would anyone have a proper solution or suggestion?

---

Snippet from example code (unnecessary code for this question replaced by ... )

...
#include <pcap.h>
#include <remote-ext.h>

...

void main(int argc, char **argv)
{
pcap_t *indesc,*outdesc;
...
FILE *capfile;
int caplen, sync;
...
pcap_send_queue *squeue;
struct pcap_pkthdr *pktheader;
u_char *pktdata;
...

/* Retrieve the length of the capture file */
capfile=fopen(argv[1],"rb");
if(!capfile){
    printf("Capture file not found!\n");
    return;
}

fseek(capfile , 0, SEEK_END);
caplen= ftell(capfile)- sizeof(struct pcap_file_header);
fclose(capfile);

...

...

/* Open the capture file */
if ( (indesc= pcap_open(source, 65536, PCAP_OPENFLAG_PROMISCUOUS, 1000, NULL, errbuf) ) == NULL)
{
    fprintf(stderr,"\nUnable to open the file %s.\n", source);
    return;
}

/* Open the output adapter */
if ( (outdesc= pcap_open(argv[2], 100, PCAP_OPENFLAG_PROMISCUOUS, 1000, NULL, errbuf) ) == NULL)
{
    fprintf(stderr,"\nUnable to open adapter %s.\n", source);
    return;
}

...

/* Allocate a send queue */
squeue = pcap_sendqueue_alloc(caplen);

/* Fill the queue with the packets from the file */
while ((res = pcap_next_ex( indesc, &pktheader, &pktdata)) == 1)
{
    if (pcap_sendqueue_queue(squeue, pktheader, pktdata) == -1)
    {
        printf("Warning: packet buffer too small, not all the packets will be sent.\n");
        break;
    }

    npacks++;
}

if (res == -1)
{
    printf("Corrupted input file.\n");
    pcap_sendqueue_destroy(squeue);
    return;
}

/* Transmit the queue */



if ((res = pcap_sendqueue_transmit(outdesc, squeue, sync)) < squeue->len)
{
    printf("An error occurred sending the packets: %s. Only %d bytes were sent\n", pcap_geterr(outdesc), res);
}


/* free the send queue */
pcap_sendqueue_destroy(squeue);

/* Close the input file */
pcap_close(indesc);

/* 
 * lose the output adapter 
 * IMPORTANT: remember to close the adapter, otherwise there will be no guarantee that all the 
 * packets will be sent!
 */
pcap_close(outdesc);


return;
}

Answer 1

Just cap it at say 50 MB (this is a somewhat arbitrary but I think reasonable starting point), and enhance the bit of code that warns when not all the packets fit in the queue so that it just sends what it has so far and starts filling the queue from the beginning with the remaining packets.