If you choose "Always route to call flow" then all schedules are ignored, and you get the described behavior.
The Schedule Group uses basically an "OR/AND NOT" structure, where it looks at the Open Schedules first to see if there is a match for the current date/time. If it finds a match on any of the open schedules it will consider the queue open unless it also finds a match for any of the Closed Schedules, in which case it then considers the queue closed.
So, all you need to do is create the Schedule for when your queue is Open. If the current date/time matches that schedule (*:30 - 5:30 in the designated Time Zone) then the IVR will use the one designated under Open Hours Call Flow. If the current date/time does not match that schedule it sees the queue as closed and uses the flow designated under Closed Hours Call Flow.