Unfortunately I end up quite often with some kind of device that has an RS-232 or RS-485 port, but that I don't have any idea what baud rate it uses, comms protocol, etc.
I will not say that it is easy but, if you're lucky, you can get it. The direct way (and also the hard one) would be using a serial converter such as the MOXA UPort 1250 and a comm analyzer software such as Docklight (much better than the old but useful Hyperterminal). Just in case you fail using the usual standard configurations such as 9600/8/n/1 and so on, you could also be trying for some hours the other hundred of combinations for baud rate, parity bit, stop bits, etc...but as sometimes you're not lucky, I will explain here how you could reverse serial communications:
1. The first thing is to discover whether it uses RS-232 or RS-485 communications. It seems obvious, but sometimes you can not trust what they tell you. Be sure!.
2. You should also know the Tx pin (in case of RS-232 comm) or Line A/B pins if it is a RS-485 communication. Simply hook up an oscilloscope and set the scope to trigger on a pulse.
Fig.1: Select a pulse trigger and try to catch a complete frame
3. Once you can see a frame with your oscilloscope, you must measure the time of the shortest pulse and take the reciprocal to figure out the baud rate.
So change the trigger mode from the usual Auto to Single and try to get a single shot of the frame.
Fig. 2: Select Trigger Single mode and capture a single shot of the frame
4. Now select a smaller time/div to zoom in into the frame bits.
Fig 3: Move the Horizontal position knob to watch the end of the frame. You should see the final stop bit
Fig 4: Continue zoom in until you can see clearly the bits width
Fig 5: Get the best time/div that allows you measuring the bit width
5. You can now use the oscilloscope vertical cursors to measure the time between the falling and the rising edge of a bit. Of course you are not going to get an exact number of baud rate, but you will realize easily about the closest value.
Fig. 6: Single bit width is 52 us aprox. so baud rate is 1/52us = 19200 baud
6. So far this was the easy thing. Now comes the hard one: discover how many data bits are in the frame, whether it uses parity or not, if it uses one stop bit, flow control, bla bla bla...
I usually try to enumerate the latest bytes, as shown in the figure below.
Fig. 7: Try to enumerate as many bits as you can
7. Try to identify the start and the stop bits. Let's consider most of the communications use 8 data bits and at least one stop bit, so starting from the ending stop bit backwards, you could try to identify where the start bit could be.
Once you had identified both start and stop bits, it is easy to discover if it uses any parity bit. In this example if uses 1 start bit, 8 data bits, 1 even parity bit and 1 stop bit.
Fig. 8: Try to identify several transmission bytes
8. Bear in mind that in most common serial communications LSB (Least Significant Bit) is typically transmitted first.
So,considering this, in the figure above the latest 3 bytes are:
0b11111101 = 0xFD (even parity bit = 1)
0b01100101 = 0x65 (even parity bit = 0)
0b01101110 = 0x6E (even parity bit = 1)
9. I know sometimes it is not so simple but I never said that. In fact, the protocol above took me some days to get it... ;o)
I hope this helps! Good luck!