Sending large chunks of data over Boost TCP?

I don't see why 2MB would be an issue to transfer. I'm assuming we're talking about a LAN running at 100mbps or 1gbps, a computer with plenty of RAM, and don't have to have > 20ms response times? If your goal is to just get all 2MB from one computer to another, just send it, TCP will handle chunking it up for you.

I don't see why 2MB would be an issue to transfer. I'm assuming we're talking about a LAN running at 100mbps or 1gbps, a computer with plenty of RAM, and don't have to have > 20ms response times? If your goal is to just get all 2MB from one computer to another, just send it, TCP will handle chunking it up for you.

I have a TCP latency checking tool that I wrote with Boost, that tries to send buffers of various sizes, I routinely check up to 20MB and those seem to get through without problems. I guess what I'm trying to say is don't spend your time developing a solution unless you know you have a problem :-) --------- Solution Implementation -------- Now that I've had a few minutes on my hands, I went through and made a quick implementation of what you were talking about: boost.teeks99.com/temp/DataChunker.zip There are three big parts: The serializer/deserializer, boost has its own, but its not much better than rolling your own, so I did. Sender - Connects to the receiver over TCP and sends the data Receiver - Waits for connections from the sender and unpacks the data it receives.

I've included the . Exe(s) in the zip, run Sender. Exe/Receiver.

Exe --help to see the options, or just look at main. More detailed explanation: Open two command prompts, and go to DataChunker\Debug in both of them. Run Receiver.

Exe in one of the Run Sender. Exe in the other one (possible on a different computer, in which case add --remote-host=IP.ADD.RE.SS after the executable name, if you want to try sending more than once and --num-sends=10 to send ten times). Looking at the code, you can see what's going on, creating the receiver and sender ends of the TCP socket in the respecitve main() functions.

The sender creates a new PrimitiveCollection and fills it in with some example data, then serializes and sends it...the receiver deserializes the data into a new PrimitiveCollection, at which point the primitive collection could be used by someone else, but I just wrote to the console that it was done.

Yeah I guess I'm trying to use the Boost TCP stuff as efficiently as possible... there don't seem to be many examples using it effectively. – Polaris878 Oct 22 '09 at 21:51 Now that I've had a few minutes on my hands, I went through and made a quick implementation of what you were talking about: boost. Teeks99.Com/temp/DataChunker.

Zip There are three big parts: The serializer/deserializer, boost has its own, but its not much better than rolling your own, so I did. Sender - Connects to the receiver over TCP and sends the data Receiver - Waits for connections from the sender and unpacks the data it receives. I've included the .

Exe(s) in the zip, run Sender. Exe/Receiver. Exe --help to see the options, or just look at main.

– teeks99 Oct 24 '09 at 16:52.

Without anything fancy, from what I remember in my network class: Send a message to the receiver asking what size data chunks it can handle Take a minimum of that and your own sending capabilities, then reply saying: What size you'll be sending, how many you'll be sending After you get that, just send each chunk. You'll want to wait for an "Ok" reply, so you know you're not wasting time sending to a client that's not there. This is also a good time for the client to send a "I'm canceling" message instead of "Ok".

Send until all packets have been replied with an "Ok" The data is transfered. This works because TCP guarantees in-order delivery. UDP would require packet numbers (for ordering).

Compression is the same, except you're sending compressed data. (Data is data, it all depends on how you interpret it). Just make sure you communicate how the data is compressed :) As for examples, all I could dig up was this page and this old question.

I think what you're doing would work well in tandem with Boost.Serialization.

I would like to add one more point to consider - setting TCP socket buffer size in order to increase socket performance to some extent. There is an utility Iperf that let test speed of exchange over the TCP socket. I ran on Windows a few tests in a 100 Mbs LAN.

With the 8Kb default TCP window size the speed is 89 Mbits/sec and with 64Kb TCP window size the speed is 94 Mbits/sec.

In addition to how to chunk and deliver the data, another issue you should consider is platform differences. If the two computers are the same architecture, and the code running on both sides is the same version of the same compiler, then you should, probably, be able to just dump the raw memory structure across the network and have it work on the other side. If everything isn't the same, though, you can run into problems with endianness, structure padding, field alignment, etc.In general, it's good to define a network format for the data separately from your in-memory representation.

That format can be binary, in which case numeric values should be converted to standard forms (mainly, changing endianness to "network order", which is big-endian), or it can be textual. Many network protocols opt for text because it eliminates a lot of formatting issues and because it makes debugging easier. Personally, I really like JSON.

It's not too verbose, there are good libraries available for every programming language, and it's really easy for humans to read and understand. One of the key issues to consider when defining your network protocol is how the receiver knows when it has received all of the data. There are two basic approaches.

First, you can send an explicit size at the beginning of the message, then the receiver knows to keep reading until it's gotten that many bytes. The other is to use some sort of an end-of-message delimiter. The latter has the advantage that you don't have to know in advance how many bytes you're sending, but the disadvantage that you have to figure out how to make sure the the end-of-message delimiter can't appear in the message.

Once you decide how the data should be structured as it's flowing across the network, then you should figure out a way to convert the internal representation to that format, ideally in a "streaming" way, so you can loop through your data structure, converting each piece of it to network format and writing it to the network socket. On the receiving side, you just reverse the process, decoding the network format to the appropriate in-memory format.My recommendation for your case is to use JSON. 2 MB is not a lot of data, so the overhead of generating and parsing won't be large, and you can easily represent your data structure directly in JSON.

The resulting text will be self-delimiting, human-readable, easy to stream, and easy to parse back into memory on the destination side.

Wow, lot of information in here. I was going to upvote you for the comment to separate the protocol serialization from the in-memory representation, but... JSON, for what amounts to a big blob of scientific data? No way, man.

– Tom Oct 24 '09 at 6:02 For 2 MB, there's no point in doing anything else. So it expands to 5 or 6 MB, so what? You're still talking a tiny fraction of a second to transfer it.

JSON is easier and safer than mucking around with binary formats. In many cases, piping it through a compressor/decompressor will result in a smaller transfer size than a binary format anyway. If we were talking about large volumes of data, I'd give a very different answer, but for this, make it easy.

– divegeek Oct 25 '09 at 1:12.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.