Do not dismiss the window via its close button. Instead, choose Hide Character Viewer or Hide Keyboard Viewer from the Input menu (looks like a flag).
Note: The Input menu may display Show Character Viewer or Show Keyboard Viewer choices (instead of Hide) even though the viewer is already visible, in which case you should select that choice from the menu.
Note: If the Input menu is not in your menu bar, see this article to learn how to enable it. Once it's enabled, follow the instructions above to hide the window.