Question about debugging virtual machines over the com ports. dos, linux, reactos, windows

[coffeeprogrammer]

Ok, I am sorry if this is not the place to ask so many questions, but I really think Steve and maybe/hopefully someone else will know the answer to this question. I might end up asking on StackOverflow and reddit too, but again I feel like this is the kind of thing Steve would know and understand. I am trying to understand how to debug the contents of one virtual machine with another virtual machine or as least the host. I am including a link to an article for an old version of vmware workstation, but I think is should work for both vmware workstation and virtual box. I remember reading in a masm dos book a long time ago, that you could debug assembly on one computer with another computer via the com ports. As virtual machines have com ports, I can’t think of a reason why this could not be done with two virtual machine no matter if they are FreeDOS, ReactOS, Windows, or Linux. I been reading about OS internal and then I remembered ReactOS and I wondered if there would be a way to use a debugger in one ReactOS instance with another ReactOS/Windows/Linux instance, both in virtual machines. Then I realized if it is possible for React OS, it is likely to be possible with FreeDOS, Linux and maybe Windows. I would want to have the source code in the debugger vm and not just the disassembly. I am sure people out on the internet know and understand the answer to this question, but Steve is the exact first person that comes to mind for me. Thanks.



https://www.vmware.com/support/ws3/doc/ws32_devices3.html

 

[PHolder]

This might be interesting to you:

 

[coffeeprogrammer]

Well actually I did make a little bit of progress, but it is not using FreeDOS or ReactOS, just XP and I don’t know any commands. Ya that Dave Plummer guy is cool, but he jump around too much. He should just do x86 assembly for Windows and DOS. No one that I have found does it that well on YouTube. There are a few people that made their own OS on YouTube however, try searching for that. Some of them used assembly too.

 

[Steve]

@coffeeprogrammer : A useful tool that you might explore is "com0com". Its read.txt file explains:

The Null-modem emulator is an open source kernel-mode virtual serial port driver for Windows, available freely under GPL license. You can create an unlimited number of virtual COM port pairs and use any pair to connect one application to another. Each COM port pair provides two COM ports with default names starting at CNCA0 and CNCB0. The output to one port is the input from other port and vice versa.

And... it works. I've used it successfully to create virtual com port pairs to link various things hosted on Windows in otherwise isolated VMs. They all saw real-appearing com ports and were able to “talk” to each other for debugging purposes.

http://com0com.sourceforge.net/

 

[coffeeprogrammer]

Does is work with the Visual Studio debugger and Debugger for DOS? Or just any debugger is the biggest question? I know it seems to work with WinDBG, but that had menu options for COM ports and remote debugging. I have Visual Studio 2015 in a W10 VM and I just now created ReactOS sln file, I will be compiling in a few minutes. I want to compile it and then deploy to a VMware VM and then debug in VS2015 in the first VM.

Update:
I found this article, I think it might work

Install WinDbg - Windows drivers

Start here for an overview on the Windows debugger and installing WinDbg.

docs.microsoft.com

at the moment I don't have WDK, but I am still wonder if this is something to debugger has to explicitly support. It's been seven minutes since I started compiling ReactOS and it is still going. Kinda cool.

Also, thank for the com0com, I downloaded a copy

 

[Steve]

I would be all over ReactOS for SpinRite if its authors had the foresight to consider it and UEFI. Any OS project that's locked to the original PC BIOS is already a zombie OS. It's sad, since so much very good work was done there.

 

[Dave New]

Many years ago, I had a setup as mentioned in the article you linked to. It required an 'instrumented' Windows NT kernel, which was supplied in the WDK, so that it would spill its guts over a hardware serial port while running various kernel calls and during boot up. Even with an instrumented kernel, developers still didn't get any Windows kernel source code, so that you could trace a call from the user mode up until it made the call into the kernel, and then it was opaque until the call returned to the user mode. The instrumented kernel allowed seeing certain things going on under the covers.

I was developing kernel mode device drivers for a machine vision SIMD array board on PCI bus, so this was about the only way I could see what was happening when my NT device driver made calls into kernel mode. The debugging machine was another Windows NT computer running a normal kernel, but hosting the debugging tools that were hooked to the target machine via the com ports of the two machines, using a null modem cross-over wired cable.

It was all pretty clunky, but it was the only game in town, so I made it work.

 

[coffeeprogrammer]

Yes, it is strangely satisfying reading about and figuring out operating systems internals. I want to start with Windows for some reason. I guess it is just because I have been using Windows the longest. To @Steve comments about the UEFI and ReactOS, I would think the developers would understand that and would add support. I personally am not sure how to go about adding that, but I would think as far as booting OS goes, both and BIOS and UEFI would just look at the beginning of the storage for executable code to boot strap. That should not be so hard. Not many people seem to develop for ReactOS and it is slow moving. I am not sure who is in charge over there.

I figured out a few commands with WinDBG and my XP Vms, but it seems I need the symbols just to play, I hope MS still hosts them. My VS2015 had a problem compiling React OS the other day, ml.exe was trying to assemble a file in the crt folder and “exited with code 1”. I don’t remember the exact error, but hopefully I can figure it out. It was also hard on the memory, VS used almost 2GB but the MSBuild child process was only like 400MB I think.

I created a FreeDOS virtual box VM last night and I am going to put masm in it at some point and hopefully I can figure out remote debugging with another VM. I shouldn’t get board.

Do either of you know anything about the “Windows Research Kernel”. Apparently MS released some of the code to the XP and 2003 kernel for students to hack around with.

Update:
I was think about what @Steve said about ReactOS and UEFI and did some googling. I found this:

An Open-Source Bootloader For Windows Lets You Run Off Btrfs, Other Possibilities - Phoronix

www.phoronix.com

It is open source boot loader that works with ReactOS and UEFI. Maybe he knows about it and wants native support. IDK, but honestly, I wish ReactOS would move faster.