I think OBS just captures the pixels on your display, not the logical windows themselves. So OBS doesn't know anything about your windows aside from maybe the first one to get the dimensions/position of the rectangle to capture.

The recursion you're seeing is just old pixels from a previous capture that were copied into a new capture. It's the same thing that happens if you connect a video camera so its output is displayed on a TV/computer, then point the video camera at that output. To the video/camera computer system, it's no different than if there were no recursion "loop."

A special case is a microphone + speaker. If you feed the speaker output from the microphone back to the microphone, there's a feedback loop that amplifies the signal making that really loud humming sound.

When you take a screenshot what happens? The screen is copied into a buffer (and saved to disk). If you open that image and then take another screenshot what happens? The screen is copied into a buffer (and saved to disk). There is no special handling of the open screenshot because it is just an image like any other. It does imply 2 and you can see that when you move the preview or change something next to it you can see it cascading through the old previews being captured.

There is no infinite recursion in code? All it's doing is copying the final pixels of the window from the OS and putting it into it's framebuffer. I doubt it's looping through the components/widgets/buttons/whatever of the selected program, telling it to render to the OBS framebuffer a second time.

I take your meaning, since I've seen people do "Share my screen" with WebEx, but... What's "OBS"?

Try taking a screenshot of Paint, then open the screenshot in Paint. Repeat.

This is what OBS does -- no recursion involved.

It doesn’t. Infinite recursion is just an infinite loop, which a program that records a screen is in.