#1: Already answered above, but it’s a huge thread and easy to miss: Yes, the HD versions play fine on all HD-capable Kaleidescape players.
#2: The way updates work is that the new version is downloaded to your system, and once downloaded, the old version is deleted. This keeps the movie available during the update and also allows us to restore things like collection membership. For this reason, especially if your systems are jam-packed full, they are not guaranteed to download to the same component.
#3: They may be different, although not necessarily substantially so.
There are no APIs exposed that would permit this type of third-party app.
A manual storage manager like you describe in #2 is really kind of contrary to the intent of the Kaleidescape system, which is to be as hands-off and automatic as possible. What would be more likely to come would be improvements to the automatic management. It sounds like in your case, an option (or even a default behavior) to save 4K-capable storage for 4K content until other storage is exhausted would probably accomplish what you wanted, correct?
One improvement that is part of the updated HD format is that in most cases, the extras are now bundled into a separate object, so I expect there to be more options for extras in the future.