Let's not play pretend: most of us have done this at at some point.
The boat which takes you from Battery Park to Liberty Island also stops at Ellis Island as one amongst its destinations, so it's on means back.